Loading demonstrations: 0it [00:00, ?it/s]
WARNING:root:NO DEMONSTRATIONS AVAILABLE

Loading GloVe embeddings:   0%|          | 2/100000 [00:00<?, ?it/s]
Loading GloVe embeddings:   3%|2         | 2720/100000 [00:00<00:03, 27176.72it/s]
Loading GloVe embeddings:   5%|5         | 5477/100000 [00:00<00:03, 27291.33it/s]
Loading GloVe embeddings:   8%|8         | 8417/100000 [00:00<00:03, 27889.17it/s]
Loading GloVe embeddings:  11%|#1        | 11488/100000 [00:00<00:03, 28676.16it/s]
Loading GloVe embeddings:  14%|#4        | 14442/100000 [00:00<00:02, 28927.41it/s]
Loading GloVe embeddings:  18%|#7        | 17511/100000 [00:00<00:02, 29432.79it/s]
Loading GloVe embeddings:  21%|##        | 20617/100000 [00:00<00:02, 29902.16it/s]
Loading GloVe embeddings:  24%|##3       | 23642/100000 [00:00<00:02, 30005.21it/s]
Loading GloVe embeddings:  27%|##6       | 26761/100000 [00:00<00:02, 30350.99it/s]
Loading GloVe embeddings:  30%|##9       | 29876/100000 [00:01<00:02, 30580.13it/s]
Loading GloVe embeddings:  33%|###2      | 32964/100000 [00:01<00:02, 30666.37it/s]
Loading GloVe embeddings:  36%|###6      | 36155/100000 [00:01<00:02, 31026.99it/s]
Loading GloVe embeddings:  39%|###9      | 39223/100000 [00:01<00:02, 30149.82it/s]
Loading GloVe embeddings:  42%|####2     | 42219/100000 [00:01<00:02, 28517.25it/s]
Loading GloVe embeddings:  45%|####5     | 45076/100000 [00:01<00:01, 28425.99it/s]
Loading GloVe embeddings:  48%|####7     | 47940/100000 [00:01<00:01, 28488.32it/s]
Loading GloVe embeddings:  51%|#####     | 50940/100000 [00:01<00:01, 28924.66it/s]
Loading GloVe embeddings:  54%|#####3    | 53875/100000 [00:01<00:01, 29048.89it/s]
Loading GloVe embeddings:  57%|#####6    | 56783/100000 [00:01<00:01, 28176.88it/s]
Loading GloVe embeddings:  60%|#####9    | 59609/100000 [00:02<00:01, 26903.77it/s]
Loading GloVe embeddings:  62%|######2   | 62317/100000 [00:02<00:01, 26301.27it/s]
Loading GloVe embeddings:  65%|######4   | 64963/100000 [00:02<00:01, 25029.98it/s]
Loading GloVe embeddings:  67%|######7   | 67490/100000 [00:02<00:01, 24671.34it/s]
Loading GloVe embeddings:  70%|#######   | 70223/100000 [00:02<00:01, 25410.67it/s]
Loading GloVe embeddings:  73%|#######2  | 72849/100000 [00:02<00:01, 25657.25it/s]
Loading GloVe embeddings:  75%|#######5  | 75428/100000 [00:02<00:01, 22306.73it/s]
Loading GloVe embeddings:  78%|#######7  | 77747/100000 [00:03<00:01, 14040.83it/s]
Loading GloVe embeddings:  80%|#######9  | 79596/100000 [00:03<00:01, 12411.87it/s]
Loading GloVe embeddings:  81%|########1 | 81468/100000 [00:03<00:01, 13757.82it/s]
Loading GloVe embeddings:  83%|########3 | 83342/100000 [00:03<00:01, 14949.07it/s]
Loading GloVe embeddings:  86%|########5 | 85522/100000 [00:03<00:00, 16504.56it/s]
Loading GloVe embeddings:  88%|########8 | 88462/100000 [00:03<00:00, 19004.96it/s]
Loading GloVe embeddings:  91%|#########1| 91496/100000 [00:03<00:00, 21403.55it/s]
Loading GloVe embeddings:  95%|#########4| 94565/100000 [00:03<00:00, 23539.45it/s]
Loading GloVe embeddings:  98%|#########7| 97521/100000 [00:03<00:00, 25070.52it/s]

Outer training loop:   0%|          | 0/50000 [00:00<?, ?it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.21s/it]

Evaluating policy:  67%|######6   | 2/3 [00:01<00:01,  1.02s/it]

Evaluating policy: 100%|##########| 3/3 [00:02<00:00,  1.19it/s]

Outer training loop:   0%|          | 1/50000 [00:02<30:55:07,  2.23s/it]
Outer training loop:   0%|          | 95/50000 [00:02<21:36:24,  1.56s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:01,  1.84it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  1.96it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  1.86it/s]

Outer training loop:   0%|          | 123/50000 [00:03<15:21:20,  1.11s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:00,  2.57it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  2.40it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.11it/s]

Outer training loop:   0%|          | 201/50000 [00:05<10:48:55,  1.28it/s]
Outer training loop:   1%|          | 291/50000 [00:05<7:33:42,  1.83it/s] 

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:00,  2.57it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  2.65it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.61it/s]

Outer training loop:   1%|          | 329/50000 [00:06<5:25:05,  2.55it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:00,  2.15it/s]

Evaluating policy:  67%|######6   | 2/3 [00:01<00:00,  1.74it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  1.90it/s]

Outer training loop:   1%|          | 401/50000 [00:08<3:53:24,  3.54it/s]
Outer training loop:   1%|          | 487/50000 [00:08<2:43:23,  5.05it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:00,  2.81it/s]

Evaluating policy:  67%|######6   | 2/3 [00:01<00:00,  2.23it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.23it/s]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:00<00:46,  1.76it/s]

Evaluating policy:   2%|2         | 2/83 [00:00<00:39,  2.04it/s]

Evaluating policy:   4%|3         | 3/83 [00:01<00:35,  2.24it/s]

Evaluating policy:   5%|4         | 4/83 [00:01<00:35,  2.24it/s]

Evaluating policy:   6%|6         | 5/83 [00:01<00:32,  2.43it/s]

Evaluating policy:   7%|7         | 6/83 [00:02<00:33,  2.28it/s]

Evaluating policy:   8%|8         | 7/83 [00:02<00:34,  2.19it/s]

Evaluating policy:  10%|9         | 8/83 [00:03<00:33,  2.24it/s]

Evaluating policy:  11%|#         | 9/83 [00:04<00:44,  1.68it/s]

Evaluating policy:  12%|#2        | 10/83 [00:04<00:38,  1.88it/s]

Evaluating policy:  13%|#3        | 11/83 [00:05<00:37,  1.92it/s]

Evaluating policy:  14%|#4        | 12/83 [00:05<00:32,  2.17it/s]

Evaluating policy:  16%|#5        | 13/83 [00:05<00:30,  2.28it/s]

Evaluating policy:  17%|#6        | 14/83 [00:06<00:36,  1.87it/s]

Evaluating policy:  18%|#8        | 15/83 [00:07<00:34,  2.00it/s]

Evaluating policy:  19%|#9        | 16/83 [00:07<00:28,  2.36it/s]

Evaluating policy:  20%|##        | 17/83 [00:07<00:24,  2.73it/s]

Evaluating policy:  22%|##1       | 18/83 [00:08<00:28,  2.27it/s]

Evaluating policy:  23%|##2       | 19/83 [00:08<00:31,  2.02it/s]

Evaluating policy:  24%|##4       | 20/83 [00:09<00:26,  2.34it/s]

Evaluating policy:  25%|##5       | 21/83 [00:09<00:27,  2.26it/s]

Evaluating policy:  27%|##6       | 22/83 [00:10<00:30,  2.01it/s]

Evaluating policy:  28%|##7       | 23/83 [00:10<00:25,  2.35it/s]

Evaluating policy:  29%|##8       | 24/83 [00:10<00:26,  2.21it/s]

Evaluating policy:  30%|###       | 25/83 [00:11<00:23,  2.44it/s]

Evaluating policy:  31%|###1      | 26/83 [00:11<00:22,  2.50it/s]

Evaluating policy:  33%|###2      | 27/83 [00:11<00:20,  2.71it/s]

Evaluating policy:  34%|###3      | 28/83 [00:12<00:24,  2.24it/s]

Evaluating policy:  35%|###4      | 29/83 [00:12<00:21,  2.50it/s]
          
Outer training loop:   1%|          | 487/50000 [00:23<39:30, 20.89it/s]

Evaluating policy:  36%|###6      | 30/83 [00:13<00:21,  2.46it/s]

Evaluating policy:  37%|###7      | 31/83 [00:14<00:26,  1.98it/s]

Evaluating policy:  39%|###8      | 32/83 [00:14<00:29,  1.73it/s]

Evaluating policy:  40%|###9      | 33/83 [00:15<00:26,  1.89it/s]

Evaluating policy:  41%|####      | 34/83 [00:15<00:23,  2.10it/s]

Evaluating policy:  42%|####2     | 35/83 [00:15<00:21,  2.22it/s]

Evaluating policy:  43%|####3     | 36/83 [00:16<00:25,  1.87it/s]

Evaluating policy:  45%|####4     | 37/83 [00:17<00:22,  2.05it/s]

Evaluating policy:  46%|####5     | 38/83 [00:17<00:19,  2.29it/s]

Evaluating policy:  47%|####6     | 39/83 [00:17<00:20,  2.16it/s]

Evaluating policy:  48%|####8     | 40/83 [00:18<00:20,  2.11it/s]

Evaluating policy:  49%|####9     | 41/83 [00:18<00:20,  2.01it/s]

Evaluating policy:  51%|#####     | 42/83 [00:19<00:19,  2.13it/s]

Evaluating policy:  52%|#####1    | 43/83 [00:19<00:16,  2.40it/s]

Evaluating policy:  53%|#####3    | 44/83 [00:20<00:16,  2.31it/s]

Evaluating policy:  54%|#####4    | 45/83 [00:20<00:16,  2.30it/s]

Evaluating policy:  55%|#####5    | 46/83 [00:20<00:15,  2.41it/s]

Evaluating policy:  57%|#####6    | 47/83 [00:21<00:16,  2.17it/s]

Evaluating policy:  58%|#####7    | 48/83 [00:21<00:15,  2.24it/s]

Evaluating policy:  59%|#####9    | 49/83 [00:22<00:15,  2.22it/s]

Evaluating policy:  60%|######    | 50/83 [00:22<00:15,  2.09it/s]

Evaluating policy:  61%|######1   | 51/83 [00:23<00:14,  2.20it/s]

Evaluating policy:  63%|######2   | 52/83 [00:23<00:14,  2.09it/s]

Evaluating policy:  64%|######3   | 53/83 [00:24<00:17,  1.74it/s]

Evaluating policy:  65%|######5   | 54/83 [00:25<00:16,  1.73it/s]

Evaluating policy:  66%|######6   | 55/83 [00:25<00:16,  1.71it/s]

Evaluating policy:  67%|######7   | 56/83 [00:26<00:15,  1.77it/s]

Evaluating policy:  69%|######8   | 57/83 [00:26<00:14,  1.81it/s]

Evaluating policy:  70%|######9   | 58/83 [00:27<00:12,  1.96it/s]

Evaluating policy:  71%|#######1  | 59/83 [00:27<00:11,  2.15it/s]

Evaluating policy:  72%|#######2  | 60/83 [00:28<00:10,  2.14it/s]

Evaluating policy:  73%|#######3  | 61/83 [00:28<00:10,  2.06it/s]

Evaluating policy:  75%|#######4  | 62/83 [00:29<00:11,  1.90it/s]

Evaluating policy:  76%|#######5  | 63/83 [00:29<00:10,  1.83it/s]

Evaluating policy:  77%|#######7  | 64/83 [00:30<00:11,  1.69it/s]

Evaluating policy:  78%|#######8  | 65/83 [00:30<00:09,  1.96it/s]

Evaluating policy:  80%|#######9  | 66/83 [00:31<00:08,  2.12it/s]

Evaluating policy:  81%|########  | 67/83 [00:31<00:07,  2.16it/s]

Evaluating policy:  82%|########1 | 68/83 [00:32<00:06,  2.20it/s]

Evaluating policy:  83%|########3 | 69/83 [00:32<00:06,  2.11it/s]

Evaluating policy:  84%|########4 | 70/83 [00:33<00:06,  2.00it/s]

Evaluating policy:  86%|########5 | 71/83 [00:33<00:05,  2.14it/s]

Evaluating policy:  87%|########6 | 72/83 [00:34<00:04,  2.22it/s]

Evaluating policy:  88%|########7 | 73/83 [00:34<00:05,  1.91it/s]

Evaluating policy:  89%|########9 | 74/83 [00:35<00:04,  1.91it/s]

Evaluating policy:  90%|######### | 75/83 [00:35<00:04,  1.92it/s]

Evaluating policy:  92%|#########1| 76/83 [00:36<00:03,  2.11it/s]

Evaluating policy:  93%|#########2| 77/83 [00:36<00:02,  2.16it/s]

Evaluating policy:  94%|#########3| 78/83 [00:37<00:02,  2.04it/s]

Evaluating policy:  95%|#########5| 79/83 [00:37<00:01,  2.39it/s]

Evaluating policy:  96%|#########6| 80/83 [00:37<00:01,  2.58it/s]

Evaluating policy:  98%|#########7| 81/83 [00:38<00:00,  2.42it/s]

Evaluating policy:  99%|#########8| 82/83 [00:38<00:00,  2.43it/s]

Evaluating policy: 100%|##########| 83/83 [00:39<00:00,  2.34it/s]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:00<00:35,  2.30it/s]

Evaluating policy:   2%|2         | 2/83 [00:00<00:36,  2.24it/s]

Evaluating policy:   4%|3         | 3/83 [00:01<00:33,  2.35it/s]

Evaluating policy:   5%|4         | 4/83 [00:01<00:37,  2.09it/s]

Evaluating policy:   6%|6         | 5/83 [00:02<00:31,  2.46it/s]

Evaluating policy:   7%|7         | 6/83 [00:02<00:27,  2.80it/s]

Evaluating policy:   8%|8         | 7/83 [00:03<00:33,  2.27it/s]

Evaluating policy:  10%|9         | 8/83 [00:03<00:34,  2.21it/s]

Evaluating policy:  11%|#         | 9/83 [00:03<00:33,  2.23it/s]

Evaluating policy:  12%|#2        | 10/83 [00:04<00:28,  2.58it/s]

Evaluating policy:  13%|#3        | 11/83 [00:04<00:29,  2.46it/s]

Evaluating policy:  14%|#4        | 12/83 [00:05<00:31,  2.25it/s]

Evaluating policy:  16%|#5        | 13/83 [00:05<00:33,  2.09it/s]

Evaluating policy:  17%|#6        | 14/83 [00:06<00:33,  2.08it/s]

Evaluating policy:  18%|#8        | 15/83 [00:06<00:31,  2.18it/s]

Evaluating policy:  19%|#9        | 16/83 [00:06<00:28,  2.35it/s]

Evaluating policy:  20%|##        | 17/83 [00:07<00:26,  2.52it/s]

Evaluating policy:  22%|##1       | 18/83 [00:07<00:24,  2.64it/s]

Evaluating policy:  23%|##2       | 19/83 [00:07<00:23,  2.67it/s]

Evaluating policy:  24%|##4       | 20/83 [00:08<00:25,  2.48it/s]

Evaluating policy:  25%|##5       | 21/83 [00:08<00:24,  2.49it/s]

Evaluating policy:  27%|##6       | 22/83 [00:09<00:25,  2.41it/s]

Evaluating policy:  28%|##7       | 23/83 [00:09<00:28,  2.13it/s]

Evaluating policy:  29%|##8       | 24/83 [00:10<00:31,  1.87it/s]

Evaluating policy:  30%|###       | 25/83 [00:11<00:35,  1.65it/s]

Evaluating policy:  31%|###1      | 26/83 [00:11<00:30,  1.88it/s]

Evaluating policy:  33%|###2      | 27/83 [00:12<00:26,  2.12it/s]

Evaluating policy:  34%|###3      | 28/83 [00:12<00:25,  2.18it/s]

Evaluating policy:  35%|###4      | 29/83 [00:12<00:25,  2.16it/s]

Evaluating policy:  36%|###6      | 30/83 [00:13<00:25,  2.08it/s]

Evaluating policy:  37%|###7      | 31/83 [00:13<00:25,  2.06it/s]

Evaluating policy:  39%|###8      | 32/83 [00:14<00:22,  2.28it/s]

Evaluating policy:  40%|###9      | 33/83 [00:14<00:23,  2.10it/s]

Evaluating policy:  41%|####      | 34/83 [00:15<00:20,  2.33it/s]

Evaluating policy:  42%|####2     | 35/83 [00:15<00:24,  2.00it/s]

Evaluating policy:  43%|####3     | 36/83 [00:16<00:23,  1.96it/s]

Evaluating policy:  45%|####4     | 37/83 [00:16<00:23,  1.97it/s]

Evaluating policy:  46%|####5     | 38/83 [00:17<00:23,  1.88it/s]

Evaluating policy:  47%|####6     | 39/83 [00:17<00:21,  2.07it/s]

Evaluating policy:  48%|####8     | 40/83 [00:18<00:18,  2.28it/s]

Evaluating policy:  49%|####9     | 41/83 [00:18<00:17,  2.36it/s]

Evaluating policy:  51%|#####     | 42/83 [00:19<00:18,  2.19it/s]

Evaluating policy:  52%|#####1    | 43/83 [00:19<00:17,  2.26it/s]

Evaluating policy:  53%|#####3    | 44/83 [00:19<00:17,  2.26it/s]

Evaluating policy:  54%|#####4    | 45/83 [00:20<00:16,  2.36it/s]

Evaluating policy:  55%|#####5    | 46/83 [00:21<00:19,  1.93it/s]

Evaluating policy:  57%|#####6    | 47/83 [00:21<00:16,  2.23it/s]

Evaluating policy:  58%|#####7    | 48/83 [00:22<00:18,  1.87it/s]

Evaluating policy:  59%|#####9    | 49/83 [00:22<00:20,  1.68it/s]

Evaluating policy:  60%|######    | 50/83 [00:23<00:18,  1.80it/s]

Evaluating policy:  61%|######1   | 51/83 [00:23<00:16,  1.95it/s]

Evaluating policy:  63%|######2   | 52/83 [00:24<00:14,  2.19it/s]

Evaluating policy:  64%|######3   | 53/83 [00:24<00:17,  1.75it/s]

Evaluating policy:  65%|######5   | 54/83 [00:25<00:16,  1.72it/s]

Evaluating policy:  66%|######6   | 55/83 [00:25<00:14,  1.99it/s]

Evaluating policy:  67%|######7   | 56/83 [00:26<00:14,  1.89it/s]

Evaluating policy:  69%|######8   | 57/83 [00:26<00:14,  1.83it/s]

Evaluating policy:  70%|######9   | 58/83 [00:27<00:13,  1.83it/s]

Evaluating policy:  71%|#######1  | 59/83 [00:27<00:11,  2.08it/s]

Evaluating policy:  72%|#######2  | 60/83 [00:28<00:10,  2.28it/s]

Evaluating policy:  73%|#######3  | 61/83 [00:29<00:12,  1.76it/s]

Evaluating policy:  75%|#######4  | 62/83 [00:29<00:11,  1.89it/s]

Evaluating policy:  76%|#######5  | 63/83 [00:29<00:09,  2.16it/s]

Evaluating policy:  77%|#######7  | 64/83 [00:30<00:08,  2.16it/s]

Evaluating policy:  78%|#######8  | 65/83 [00:30<00:07,  2.32it/s]

Evaluating policy:  80%|#######9  | 66/83 [00:31<00:08,  1.96it/s]

Evaluating policy:  81%|########  | 67/83 [00:31<00:07,  2.23it/s]

Evaluating policy:  82%|########1 | 68/83 [00:32<00:06,  2.16it/s]

Evaluating policy:  83%|########3 | 69/83 [00:32<00:05,  2.41it/s]

Evaluating policy:  84%|########4 | 70/83 [00:32<00:04,  2.60it/s]

Evaluating policy:  86%|########5 | 71/83 [00:32<00:04,  2.86it/s]

Evaluating policy:  87%|########6 | 72/83 [00:33<00:03,  2.97it/s]

Evaluating policy:  88%|########7 | 73/83 [00:33<00:03,  2.99it/s]

Evaluating policy:  89%|########9 | 74/83 [00:34<00:03,  2.70it/s]

Evaluating policy:  90%|######### | 75/83 [00:34<00:03,  2.35it/s]

Evaluating policy:  92%|#########1| 76/83 [00:35<00:02,  2.37it/s]

Evaluating policy:  93%|#########2| 77/83 [00:35<00:02,  2.15it/s]

Evaluating policy:  94%|#########3| 78/83 [00:36<00:02,  2.19it/s]

Evaluating policy:  95%|#########5| 79/83 [00:36<00:02,  1.93it/s]

Evaluating policy:  96%|#########6| 80/83 [00:37<00:01,  2.09it/s]

Evaluating policy:  98%|#########7| 81/83 [00:37<00:01,  1.91it/s]

Evaluating policy:  99%|#########8| 82/83 [00:38<00:00,  1.77it/s]

Evaluating policy: 100%|##########| 83/83 [00:38<00:00,  2.01it/s]

Outer training loop:   1%|1         | 501/50000 [01:27<25:16:15,  1.84s/it]
Outer training loop:   1%|1         | 567/50000 [01:28<17:40:20,  1.29s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:01,  1.97it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  2.01it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.11it/s]

Outer training loop:   1%|1         | 601/50000 [01:29<12:32:21,  1.09it/s]
Outer training loop:   1%|1         | 664/50000 [01:29<8:46:22,  1.56it/s] 

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:00,  2.62it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  2.67it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.88it/s]

Outer training loop:   1%|1         | 701/50000 [01:30<6:15:25,  2.19it/s]
Outer training loop:   2%|1         | 768/50000 [01:30<4:22:48,  3.12it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:00,  2.56it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  2.22it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  1.97it/s]

Outer training loop:   2%|1         | 803/50000 [01:32<3:15:39,  4.19it/s]
Outer training loop:   2%|1         | 867/50000 [01:32<2:17:09,  5.97it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:01,  1.76it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  1.93it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.01it/s]

Outer training loop:   2%|1         | 904/50000 [01:34<1:45:49,  7.73it/s]
Outer training loop:   2%|1         | 968/50000 [01:34<1:14:21, 10.99it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:01,  1.99it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  2.14it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.20it/s]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:00<00:53,  1.53it/s]

Evaluating policy:   2%|2         | 2/83 [00:01<00:46,  1.75it/s]

Evaluating policy:   4%|3         | 3/83 [00:01<00:43,  1.83it/s]

Evaluating policy:   5%|4         | 4/83 [00:01<00:39,  1.98it/s]

Evaluating policy:   6%|6         | 5/83 [00:02<00:38,  2.03it/s]

Evaluating policy:   7%|7         | 6/83 [00:02<00:33,  2.32it/s]

Evaluating policy:   8%|8         | 7/83 [00:03<00:36,  2.08it/s]

Evaluating policy:  10%|9         | 8/83 [00:03<00:34,  2.18it/s]

Evaluating policy:  11%|#         | 9/83 [00:04<00:33,  2.22it/s]

Evaluating policy:  12%|#2        | 10/83 [00:04<00:37,  1.95it/s]

Evaluating policy:  13%|#3        | 11/83 [00:05<00:35,  2.01it/s]

Evaluating policy:  14%|#4        | 12/83 [00:05<00:33,  2.13it/s]

Evaluating policy:  16%|#5        | 13/83 [00:05<00:28,  2.46it/s]

Evaluating policy:  17%|#6        | 14/83 [00:06<00:26,  2.58it/s]

Evaluating policy:  18%|#8        | 15/83 [00:06<00:26,  2.60it/s]

Evaluating policy:  19%|#9        | 16/83 [00:07<00:29,  2.31it/s]

Evaluating policy:  20%|##        | 17/83 [00:07<00:28,  2.28it/s]

Evaluating policy:  22%|##1       | 18/83 [00:07<00:26,  2.44it/s]

Evaluating policy:  23%|##2       | 19/83 [00:08<00:32,  1.97it/s]

Evaluating policy:  24%|##4       | 20/83 [00:09<00:29,  2.17it/s]

Evaluating policy:  25%|##5       | 21/83 [00:09<00:27,  2.28it/s]

Evaluating policy:  27%|##6       | 22/83 [00:09<00:25,  2.37it/s]

Evaluating policy:  28%|##7       | 23/83 [00:10<00:23,  2.57it/s]

Evaluating policy:  29%|##8       | 24/83 [00:10<00:21,  2.73it/s]

Evaluating policy:  30%|###       | 25/83 [00:10<00:21,  2.65it/s]

Evaluating policy:  31%|###1      | 26/83 [00:11<00:21,  2.59it/s]

Evaluating policy:  33%|###2      | 27/83 [00:11<00:21,  2.55it/s]

Evaluating policy:  34%|###3      | 28/83 [00:12<00:24,  2.29it/s]

Evaluating policy:  35%|###4      | 29/83 [00:12<00:22,  2.41it/s]

Evaluating policy:  36%|###6      | 30/83 [00:12<00:19,  2.76it/s]

Evaluating policy:  37%|###7      | 31/83 [00:13<00:21,  2.42it/s]

Evaluating policy:  39%|###8      | 32/83 [00:13<00:22,  2.25it/s]

Evaluating policy:  40%|###9      | 33/83 [00:14<00:22,  2.26it/s]

Evaluating policy:  41%|####      | 34/83 [00:14<00:20,  2.36it/s]

Evaluating policy:  42%|####2     | 35/83 [00:15<00:20,  2.36it/s]

Evaluating policy:  43%|####3     | 36/83 [00:15<00:21,  2.21it/s]

Evaluating policy:  45%|####4     | 37/83 [00:15<00:19,  2.35it/s]

Evaluating policy:  46%|####5     | 38/83 [00:16<00:17,  2.55it/s]

Evaluating policy:  47%|####6     | 39/83 [00:16<00:16,  2.68it/s]

Evaluating policy:  48%|####8     | 40/83 [00:17<00:17,  2.42it/s]

Evaluating policy:  49%|####9     | 41/83 [00:17<00:18,  2.26it/s]
          
Outer training loop:   2%|1         | 968/50000 [01:53<1:35:43,  8.54it/s]

Evaluating policy:  51%|#####     | 42/83 [00:18<00:17,  2.37it/s]

Evaluating policy:  52%|#####1    | 43/83 [00:18<00:16,  2.44it/s]

Evaluating policy:  53%|#####3    | 44/83 [00:18<00:15,  2.47it/s]

Evaluating policy:  54%|#####4    | 45/83 [00:19<00:17,  2.22it/s]

Evaluating policy:  55%|#####5    | 46/83 [00:19<00:15,  2.36it/s]

Evaluating policy:  57%|#####6    | 47/83 [00:20<00:15,  2.37it/s]

Evaluating policy:  58%|#####7    | 48/83 [00:20<00:14,  2.43it/s]

Evaluating policy:  59%|#####9    | 49/83 [00:20<00:14,  2.37it/s]

Evaluating policy:  60%|######    | 50/83 [00:21<00:14,  2.33it/s]

Evaluating policy:  61%|######1   | 51/83 [00:21<00:14,  2.14it/s]

Evaluating policy:  63%|######2   | 52/83 [00:22<00:15,  2.01it/s]

Evaluating policy:  64%|######3   | 53/83 [00:22<00:13,  2.27it/s]

Evaluating policy:  65%|######5   | 54/83 [00:23<00:12,  2.33it/s]

Evaluating policy:  66%|######6   | 55/83 [00:23<00:13,  2.15it/s]

Evaluating policy:  67%|######7   | 56/83 [00:24<00:12,  2.13it/s]

Evaluating policy:  69%|######8   | 57/83 [00:24<00:13,  1.89it/s]

Evaluating policy:  70%|######9   | 58/83 [00:25<00:13,  1.86it/s]

Evaluating policy:  71%|#######1  | 59/83 [00:25<00:11,  2.12it/s]

Evaluating policy:  72%|#######2  | 60/83 [00:26<00:12,  1.90it/s]

Evaluating policy:  73%|#######3  | 61/83 [00:26<00:11,  1.93it/s]

Evaluating policy:  75%|#######4  | 62/83 [00:27<00:11,  1.83it/s]

Evaluating policy:  76%|#######5  | 63/83 [00:28<00:10,  1.85it/s]

Evaluating policy:  77%|#######7  | 64/83 [00:28<00:10,  1.74it/s]

Evaluating policy:  78%|#######8  | 65/83 [00:29<00:08,  2.04it/s]

Evaluating policy:  80%|#######9  | 66/83 [00:29<00:08,  2.05it/s]

Evaluating policy:  81%|########  | 67/83 [00:30<00:07,  2.04it/s]

Evaluating policy:  82%|########1 | 68/83 [00:30<00:06,  2.19it/s]

Evaluating policy:  83%|########3 | 69/83 [00:30<00:05,  2.43it/s]

Evaluating policy:  84%|########4 | 70/83 [00:31<00:04,  2.60it/s]

Evaluating policy:  86%|########5 | 71/83 [00:31<00:05,  2.38it/s]

Evaluating policy:  87%|########6 | 72/83 [00:31<00:04,  2.54it/s]

Evaluating policy:  88%|########7 | 73/83 [00:32<00:03,  2.78it/s]

Evaluating policy:  89%|########9 | 74/83 [00:32<00:03,  2.41it/s]

Evaluating policy:  90%|######### | 75/83 [00:33<00:03,  2.29it/s]

Evaluating policy:  92%|#########1| 76/83 [00:33<00:03,  2.09it/s]

Evaluating policy:  93%|#########2| 77/83 [00:33<00:02,  2.46it/s]

Evaluating policy:  94%|#########3| 78/83 [00:34<00:02,  2.15it/s]

Evaluating policy:  95%|#########5| 79/83 [00:34<00:01,  2.37it/s]

Evaluating policy:  96%|#########6| 80/83 [00:35<00:01,  2.70it/s]

Evaluating policy:  98%|#########7| 81/83 [00:35<00:00,  2.43it/s]

Evaluating policy:  99%|#########8| 82/83 [00:36<00:00,  2.53it/s]

Evaluating policy: 100%|##########| 83/83 [00:36<00:00,  2.35it/s]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:00<00:44,  1.86it/s]

Evaluating policy:   2%|2         | 2/83 [00:00<00:40,  1.98it/s]

Evaluating policy:   4%|3         | 3/83 [00:01<00:35,  2.24it/s]

Evaluating policy:   5%|4         | 4/83 [00:02<00:42,  1.87it/s]

Evaluating policy:   6%|6         | 5/83 [00:02<00:37,  2.06it/s]

Evaluating policy:   7%|7         | 6/83 [00:02<00:34,  2.23it/s]

Evaluating policy:   8%|8         | 7/83 [00:03<00:35,  2.12it/s]

Evaluating policy:  10%|9         | 8/83 [00:03<00:31,  2.35it/s]

Evaluating policy:  11%|#         | 9/83 [00:04<00:32,  2.30it/s]

Evaluating policy:  12%|#2        | 10/83 [00:04<00:30,  2.36it/s]

Evaluating policy:  13%|#3        | 11/83 [00:04<00:30,  2.38it/s]

Evaluating policy:  14%|#4        | 12/83 [00:05<00:29,  2.42it/s]

Evaluating policy:  16%|#5        | 13/83 [00:05<00:34,  2.03it/s]

Evaluating policy:  17%|#6        | 14/83 [00:06<00:34,  2.00it/s]

Evaluating policy:  18%|#8        | 15/83 [00:06<00:30,  2.20it/s]

Evaluating policy:  19%|#9        | 16/83 [00:07<00:29,  2.29it/s]

Evaluating policy:  20%|##        | 17/83 [00:07<00:27,  2.37it/s]

Evaluating policy:  22%|##1       | 18/83 [00:08<00:29,  2.23it/s]

Evaluating policy:  23%|##2       | 19/83 [00:08<00:34,  1.86it/s]

Evaluating policy:  24%|##4       | 20/83 [00:09<00:34,  1.85it/s]

Evaluating policy:  25%|##5       | 21/83 [00:09<00:32,  1.89it/s]

Evaluating policy:  27%|##6       | 22/83 [00:10<00:28,  2.17it/s]

Evaluating policy:  28%|##7       | 23/83 [00:10<00:23,  2.54it/s]

Evaluating policy:  29%|##8       | 24/83 [00:10<00:22,  2.65it/s]

Evaluating policy:  30%|###       | 25/83 [00:11<00:29,  1.94it/s]

Evaluating policy:  31%|###1      | 26/83 [00:12<00:27,  2.08it/s]

Evaluating policy:  33%|###2      | 27/83 [00:12<00:26,  2.14it/s]

Evaluating policy:  34%|###3      | 28/83 [00:12<00:26,  2.07it/s]

Evaluating policy:  35%|###4      | 29/83 [00:13<00:28,  1.87it/s]

Evaluating policy:  36%|###6      | 30/83 [00:14<00:26,  2.01it/s]

Evaluating policy:  37%|###7      | 31/83 [00:14<00:28,  1.80it/s]

Evaluating policy:  39%|###8      | 32/83 [00:15<00:24,  2.07it/s]

Evaluating policy:  40%|###9      | 33/83 [00:15<00:24,  2.05it/s]

Evaluating policy:  41%|####      | 34/83 [00:15<00:22,  2.17it/s]

Evaluating policy:  42%|####2     | 35/83 [00:16<00:23,  2.01it/s]

Evaluating policy:  43%|####3     | 36/83 [00:16<00:22,  2.07it/s]

Evaluating policy:  45%|####4     | 37/83 [00:17<00:21,  2.13it/s]

Evaluating policy:  46%|####5     | 38/83 [00:17<00:20,  2.19it/s]

Evaluating policy:  47%|####6     | 39/83 [00:18<00:19,  2.24it/s]

Evaluating policy:  48%|####8     | 40/83 [00:19<00:23,  1.79it/s]

Evaluating policy:  49%|####9     | 41/83 [00:19<00:20,  2.02it/s]

Evaluating policy:  51%|#####     | 42/83 [00:19<00:21,  1.94it/s]

Evaluating policy:  52%|#####1    | 43/83 [00:20<00:18,  2.15it/s]

Evaluating policy:  53%|#####3    | 44/83 [00:20<00:16,  2.40it/s]

Evaluating policy:  54%|#####4    | 45/83 [00:21<00:15,  2.38it/s]

Evaluating policy:  55%|#####5    | 46/83 [00:21<00:17,  2.08it/s]

Evaluating policy:  57%|#####6    | 47/83 [00:22<00:20,  1.77it/s]

Evaluating policy:  58%|#####7    | 48/83 [00:22<00:17,  2.04it/s]

Evaluating policy:  59%|#####9    | 49/83 [00:23<00:15,  2.21it/s]

Evaluating policy:  60%|######    | 50/83 [00:23<00:16,  2.04it/s]

Evaluating policy:  61%|######1   | 51/83 [00:24<00:17,  1.85it/s]

Evaluating policy:  63%|######2   | 52/83 [00:25<00:17,  1.75it/s]

Evaluating policy:  64%|######3   | 53/83 [00:25<00:18,  1.61it/s]

Evaluating policy:  65%|######5   | 54/83 [00:26<00:16,  1.72it/s]

Evaluating policy:  66%|######6   | 55/83 [00:26<00:14,  1.91it/s]

Evaluating policy:  67%|######7   | 56/83 [00:27<00:13,  2.05it/s]

Evaluating policy:  69%|######8   | 57/83 [00:27<00:12,  2.11it/s]

Evaluating policy:  70%|######9   | 58/83 [00:27<00:10,  2.36it/s]

Evaluating policy:  71%|#######1  | 59/83 [00:28<00:12,  1.86it/s]

Evaluating policy:  72%|#######2  | 60/83 [00:28<00:11,  2.03it/s]

Evaluating policy:  73%|#######3  | 61/83 [00:29<00:10,  2.17it/s]

Evaluating policy:  75%|#######4  | 62/83 [00:29<00:10,  2.07it/s]

Evaluating policy:  76%|#######5  | 63/83 [00:30<00:09,  2.02it/s]

Evaluating policy:  77%|#######7  | 64/83 [00:31<00:10,  1.83it/s]

Evaluating policy:  78%|#######8  | 65/83 [00:31<00:09,  1.90it/s]

Evaluating policy:  80%|#######9  | 66/83 [00:31<00:08,  2.05it/s]

Evaluating policy:  81%|########  | 67/83 [00:32<00:07,  2.22it/s]

Evaluating policy:  82%|########1 | 68/83 [00:33<00:08,  1.84it/s]

Evaluating policy:  83%|########3 | 69/83 [00:33<00:07,  1.80it/s]

Evaluating policy:  84%|########4 | 70/83 [00:34<00:07,  1.77it/s]

Evaluating policy:  86%|########5 | 71/83 [00:34<00:06,  1.78it/s]

Evaluating policy:  87%|########6 | 72/83 [00:35<00:06,  1.62it/s]

Evaluating policy:  88%|########7 | 73/83 [00:35<00:05,  1.83it/s]

Evaluating policy:  89%|########9 | 74/83 [00:36<00:04,  2.09it/s]

Evaluating policy:  90%|######### | 75/83 [00:36<00:03,  2.26it/s]

Evaluating policy:  92%|#########1| 76/83 [00:36<00:02,  2.41it/s]

Evaluating policy:  93%|#########2| 77/83 [00:37<00:02,  2.49it/s]

Evaluating policy:  94%|#########3| 78/83 [00:37<00:02,  2.14it/s]

Evaluating policy:  95%|#########5| 79/83 [00:38<00:02,  1.87it/s]

Evaluating policy:  96%|#########6| 80/83 [00:39<00:01,  2.02it/s]

Evaluating policy:  98%|#########7| 81/83 [00:39<00:00,  2.01it/s]

Evaluating policy:  99%|#########8| 82/83 [00:39<00:00,  2.09it/s]

Evaluating policy: 100%|##########| 83/83 [00:40<00:00,  2.25it/s]

Outer training loop:   2%|2         | 1001/50000 [02:52<10:33:15,  1.29it/s]
Outer training loop:   2%|2         | 1065/50000 [02:52<7:23:05,  1.84it/s] 

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:01,  1.36it/s]

Evaluating policy:  67%|######6   | 2/3 [00:01<00:00,  1.64it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  1.79it/s]

Outer training loop:   2%|2         | 1101/50000 [02:54<5:20:29,  2.54it/s]
Outer training loop:   2%|2         | 1164/50000 [02:54<3:44:26,  3.63it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:00,  2.74it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  2.37it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.31it/s]

Outer training loop:   2%|2         | 1201/50000 [02:55<2:46:31,  4.88it/s]
Outer training loop:   3%|2         | 1262/50000 [02:55<1:56:49,  6.95it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:00,  2.90it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  2.72it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.49it/s]

Outer training loop:   3%|2         | 1301/50000 [02:57<1:29:54,  9.03it/s]
Outer training loop:   3%|2         | 1366/50000 [02:57<1:03:14, 12.82it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:00,  2.52it/s]

Evaluating policy:  67%|######6   | 2/3 [00:00<00:00,  2.41it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  2.01it/s]

Outer training loop:   3%|2         | 1402/50000 [02:58<55:05, 14.70it/s]  
Outer training loop:   3%|2         | 1465/50000 [02:58<38:54, 20.79it/s]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:00<00:01,  1.14it/s]

Evaluating policy:  67%|######6   | 2/3 [00:01<00:00,  1.40it/s]

Evaluating policy: 100%|##########| 3/3 [00:01<00:00,  1.70it/s]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:00<00:47,  1.74it/s]

Evaluating policy:   2%|2         | 2/83 [00:00<00:40,  2.00it/s]

Evaluating policy:   4%|3         | 3/83 [00:01<00:37,  2.12it/s]

Evaluating policy:   5%|4         | 4/83 [00:01<00:38,  2.07it/s]

Evaluating policy:   6%|6         | 5/83 [00:02<00:36,  2.14it/s]

Evaluating policy:   7%|7         | 6/83 [00:02<00:34,  2.23it/s]

Evaluating policy:   8%|8         | 7/83 [00:03<00:32,  2.32it/s]

Evaluating policy:  10%|9         | 8/83 [00:03<00:32,  2.32it/s]

Evaluating policy:  11%|#         | 9/83 [00:04<00:35,  2.10it/s]

Evaluating policy:  12%|#2        | 10/83 [00:04<00:33,  2.20it/s]

Evaluating policy:  13%|#3        | 11/83 [00:04<00:33,  2.13it/s]

Evaluating policy:  14%|#4        | 12/83 [00:05<00:30,  2.31it/s]

Evaluating policy:  16%|#5        | 13/83 [00:05<00:32,  2.13it/s]

Evaluating policy:  17%|#6        | 14/83 [00:06<00:34,  2.02it/s]

Evaluating policy:  18%|#8        | 15/83 [00:06<00:30,  2.26it/s]

Evaluating policy:  19%|#9        | 16/83 [00:07<00:30,  2.17it/s]

Evaluating policy:  20%|##        | 17/83 [00:07<00:35,  1.89it/s]

Evaluating policy:  22%|##1       | 18/83 [00:08<00:31,  2.08it/s]

Evaluating policy:  23%|##2       | 19/83 [00:08<00:29,  2.16it/s]

Evaluating policy:  24%|##4       | 20/83 [00:09<00:30,  2.10it/s]

Evaluating policy:  25%|##5       | 21/83 [00:09<00:29,  2.11it/s]

Evaluating policy:  27%|##6       | 22/83 [00:10<00:27,  2.23it/s]

Evaluating policy:  28%|##7       | 23/83 [00:10<00:25,  2.34it/s]

Evaluating policy:  29%|##8       | 24/83 [00:10<00:23,  2.54it/s]

Evaluating policy:  30%|###       | 25/83 [00:11<00:23,  2.46it/s]

Evaluating policy:  31%|###1      | 26/83 [00:11<00:22,  2.52it/s]

Evaluating policy:  33%|###2      | 27/83 [00:11<00:22,  2.54it/s]

Evaluating policy:  34%|###3      | 28/83 [00:12<00:21,  2.62it/s]

Evaluating policy:  35%|###4      | 29/83 [00:12<00:21,  2.56it/s]
          
Outer training loop:   3%|2         | 1465/50000 [03:13<1:46:49,  7.57it/s]

Evaluating policy:  36%|###6      | 30/83 [00:13<00:19,  2.73it/s]

Evaluating policy:  37%|###7      | 31/83 [00:13<00:21,  2.48it/s]

Evaluating policy:  39%|###8      | 32/83 [00:13<00:21,  2.42it/s]

Evaluating policy:  40%|###9      | 33/83 [00:14<00:20,  2.48it/s]

Evaluating policy:  41%|####      | 34/83 [00:14<00:20,  2.38it/s]

Evaluating policy:  42%|####2     | 35/83 [00:15<00:19,  2.40it/s]

Evaluating policy:  43%|####3     | 36/83 [00:15<00:22,  2.11it/s]

Evaluating policy:  45%|####4     | 37/83 [00:16<00:24,  1.86it/s]

Evaluating policy:  46%|####5     | 38/83 [00:16<00:22,  2.03it/s]

Evaluating policy:  47%|####6     | 39/83 [00:17<00:20,  2.12it/s]

Evaluating policy:  48%|####8     | 40/83 [00:17<00:19,  2.25it/s]

Evaluating policy:  49%|####9     | 41/83 [00:18<00:21,  1.97it/s]

Evaluating policy:  51%|#####     | 42/83 [00:18<00:20,  2.01it/s]

Evaluating policy:  52%|#####1    | 43/83 [00:19<00:24,  1.66it/s]

Evaluating policy:  53%|#####3    | 44/83 [00:20<00:21,  1.79it/s]

Evaluating policy:  54%|#####4    | 45/83 [00:20<00:20,  1.87it/s]

Evaluating policy:  55%|#####5    | 46/83 [00:20<00:17,  2.13it/s]

Evaluating policy:  57%|#####6    | 47/83 [00:21<00:16,  2.18it/s]

Evaluating policy:  58%|#####7    | 48/83 [00:21<00:15,  2.33it/s]

Evaluating policy:  59%|#####9    | 49/83 [00:22<00:15,  2.23it/s]

Evaluating policy:  60%|######    | 50/83 [00:22<00:14,  2.23it/s]

Evaluating policy:  61%|######1   | 51/83 [00:23<00:14,  2.25it/s]

Evaluating policy:  63%|######2   | 52/83 [00:23<00:12,  2.49it/s]

Evaluating policy:  64%|######3   | 53/83 [00:24<00:13,  2.16it/s]

Evaluating policy:  65%|######5   | 54/83 [00:24<00:13,  2.14it/s]

Evaluating policy:  66%|######6   | 55/83 [00:24<00:11,  2.34it/s]

Evaluating policy:  67%|######7   | 56/83 [00:25<00:10,  2.46it/s]

Evaluating policy:  69%|######8   | 57/83 [00:25<00:10,  2.40it/s]

Evaluating policy:  70%|######9   | 58/83 [00:26<00:10,  2.44it/s]

Evaluating policy:  71%|#######1  | 59/83 [00:26<00:09,  2.56it/s]

Evaluating policy:  72%|#######2  | 60/83 [00:27<00:11,  2.06it/s]

Evaluating policy:  73%|#######3  | 61/83 [00:27<00:11,  1.91it/s]

Evaluating policy:  75%|#######4  | 62/83 [00:28<00:11,  1.89it/s]

Evaluating policy:  76%|#######5  | 63/83 [00:29<00:12,  1.59it/s]

Evaluating policy:  77%|#######7  | 64/83 [00:29<00:12,  1.51it/s]

Evaluating policy:  78%|#######8  | 65/83 [00:30<00:11,  1.53it/s]

Evaluating policy:  80%|#######9  | 66/83 [00:30<00:10,  1.68it/s]

Evaluating policy:  81%|########  | 67/83 [00:31<00:08,  1.79it/s]

Evaluating policy:  82%|########1 | 68/83 [00:31<00:07,  1.98it/s]

Evaluating policy:  83%|########3 | 69/83 [00:32<00:06,  2.12it/s]

Evaluating policy:  84%|########4 | 70/83 [00:32<00:06,  1.86it/s]

Evaluating policy:  86%|########5 | 71/83 [00:33<00:05,  2.05it/s]

Evaluating policy:  87%|########6 | 72/83 [00:33<00:04,  2.21it/s]

Evaluating policy:  88%|########7 | 73/83 [00:34<00:04,  2.12it/s]

Evaluating policy:  89%|########9 | 74/83 [00:34<00:05,  1.69it/s]

Evaluating policy:  90%|######### | 75/83 [00:35<00:04,  1.96it/s]

Evaluating policy:  92%|#########1| 76/83 [00:35<00:03,  2.24it/s]

Evaluating policy:  93%|#########2| 77/83 [00:35<00:02,  2.41it/s]

Evaluating policy:  94%|#########3| 78/83 [00:36<00:02,  2.46it/s]

Evaluating policy:  95%|#########5| 79/83 [00:36<00:01,  2.68it/s]

Evaluating policy:  96%|#########6| 80/83 [00:36<00:01,  2.74it/s]

Evaluating policy:  98%|#########7| 81/83 [00:37<00:00,  2.83it/s]

Evaluating policy:  99%|#########8| 82/83 [00:37<00:00,  2.37it/s]

Evaluating policy: 100%|##########| 83/83 [00:38<00:00,  2.55it/s]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:00<00:38,  2.12it/s]

Evaluating policy:   2%|2         | 2/83 [00:01<00:43,  1.85it/s]

Evaluating policy:   4%|3         | 3/83 [00:01<00:38,  2.08it/s]

Evaluating policy:   5%|4         | 4/83 [00:02<00:41,  1.91it/s]

Evaluating policy:   6%|6         | 5/83 [00:02<00:36,  2.16it/s]

Evaluating policy:   7%|7         | 6/83 [00:02<00:33,  2.29it/s]

Evaluating policy:   8%|8         | 7/83 [00:03<00:31,  2.40it/s]

Evaluating policy:  10%|9         | 8/83 [00:03<00:29,  2.55it/s]

Evaluating policy:  11%|#         | 9/83 [00:04<00:31,  2.34it/s]

Evaluating policy:  12%|#2        | 10/83 [00:04<00:31,  2.31it/s]

Evaluating policy:  13%|#3        | 11/83 [00:04<00:31,  2.29it/s]

Evaluating policy:  14%|#4        | 12/83 [00:05<00:36,  1.94it/s]

Evaluating policy:  16%|#5        | 13/83 [00:06<00:33,  2.10it/s]

Evaluating policy:  17%|#6        | 14/83 [00:06<00:38,  1.78it/s]

Evaluating policy:  18%|#8        | 15/83 [00:07<00:43,  1.57it/s]

Evaluating policy:  19%|#9        | 16/83 [00:07<00:36,  1.86it/s]

Evaluating policy:  20%|##        | 17/83 [00:08<00:37,  1.76it/s]

Evaluating policy:  22%|##1       | 18/83 [00:08<00:34,  1.88it/s]

Evaluating policy:  23%|##2       | 19/83 [00:09<00:35,  1.81it/s]

Evaluating policy:  24%|##4       | 20/83 [00:10<00:39,  1.62it/s]

Evaluating policy:  25%|##5       | 21/83 [00:10<00:34,  1.80it/s]

Evaluating policy:  27%|##6       | 22/83 [00:11<00:33,  1.80it/s]

Evaluating policy:  28%|##7       | 23/83 [00:11<00:30,  1.98it/s]

Evaluating policy:  29%|##8       | 24/83 [00:12<00:30,  1.92it/s]

Evaluating policy:  30%|###       | 25/83 [00:12<00:27,  2.07it/s]

Evaluating policy:  31%|###1      | 26/83 [00:13<00:28,  2.03it/s]

Evaluating policy:  33%|###2      | 27/83 [00:13<00:26,  2.11it/s]

Evaluating policy:  34%|###3      | 28/83 [00:13<00:23,  2.35it/s]

Evaluating policy:  35%|###4      | 29/83 [00:14<00:21,  2.57it/s]

Evaluating policy:  36%|###6      | 30/83 [00:14<00:22,  2.32it/s]

Evaluating policy:  37%|###7      | 31/83 [00:15<00:21,  2.37it/s]

Evaluating policy:  39%|###8      | 32/83 [00:15<00:22,  2.23it/s]

Evaluating policy:  40%|###9      | 33/83 [00:16<00:27,  1.79it/s]

Evaluating policy:  41%|####      | 34/83 [00:16<00:25,  1.91it/s]

Evaluating policy:  42%|####2     | 35/83 [00:17<00:26,  1.79it/s]

Evaluating policy:  43%|####3     | 36/83 [00:18<00:26,  1.77it/s]

Evaluating policy:  45%|####4     | 37/83 [00:18<00:26,  1.72it/s]

Evaluating policy:  46%|####5     | 38/83 [00:19<00:23,  1.95it/s]

Evaluating policy:  47%|####6     | 39/83 [00:19<00:20,  2.15it/s]

Evaluating policy:  48%|####8     | 40/83 [00:19<00:18,  2.28it/s]

Evaluating policy:  49%|####9     | 41/83 [00:20<00:17,  2.36it/s]

Evaluating policy:  51%|#####     | 42/83 [00:20<00:17,  2.32it/s]

Evaluating policy:  52%|#####1    | 43/83 [00:21<00:16,  2.39it/s]

Evaluating policy:  53%|#####3    | 44/83 [00:21<00:18,  2.06it/s]

Evaluating policy:  54%|#####4    | 45/83 [00:22<00:16,  2.32it/s]

Evaluating policy:  55%|#####5    | 46/83 [00:22<00:20,  1.80it/s]

Evaluating policy:  57%|#####6    | 47/83 [00:23<00:18,  1.99it/s]

Evaluating policy:  58%|#####7    | 48/83 [00:23<00:15,  2.23it/s]

Evaluating policy:  59%|#####9    | 49/83 [00:24<00:16,  2.12it/s]

Evaluating policy:  60%|######    | 50/83 [00:24<00:14,  2.27it/s]

Evaluating policy:  61%|######1   | 51/83 [00:24<00:13,  2.31it/s]

Evaluating policy:  63%|######2   | 52/83 [00:25<00:12,  2.45it/s]

Evaluating policy:  64%|######3   | 53/83 [00:25<00:11,  2.58it/s]

Evaluating policy:  65%|######5   | 54/83 [00:26<00:13,  2.21it/s]

Evaluating policy:  66%|######6   | 55/83 [00:26<00:12,  2.24it/s]

Evaluating policy:  67%|######7   | 56/83 [00:27<00:16,  1.61it/s]

Evaluating policy:  69%|######8   | 57/83 [00:28<00:16,  1.59it/s]

Evaluating policy:  70%|######9   | 58/83 [00:28<00:13,  1.90it/s]

Evaluating policy:  71%|#######1  | 59/83 [00:29<00:12,  1.93it/s]

Evaluating policy:  72%|#######2  | 60/83 [00:29<00:11,  1.99it/s]

Evaluating policy:  73%|#######3  | 61/83 [00:30<00:11,  1.93it/s]

Evaluating policy:  75%|#######4  | 62/83 [00:30<00:11,  1.91it/s]

Evaluating policy:  76%|#######5  | 63/83 [00:31<00:09,  2.01it/s]

Evaluating policy:  77%|#######7  | 64/83 [00:31<00:09,  2.02it/s]

Evaluating policy:  78%|#######8  | 65/83 [00:31<00:08,  2.12it/s]

Evaluating policy:  80%|#######9  | 66/83 [00:32<00:08,  2.05it/s]

Evaluating policy:  81%|########  | 67/83 [00:33<00:07,  2.00it/s]

Evaluating policy:  82%|########1 | 68/83 [00:33<00:07,  2.01it/s]

Evaluating policy:  83%|########3 | 69/83 [00:33<00:06,  2.26it/s]

Evaluating policy:  84%|########4 | 70/83 [00:34<00:05,  2.38it/s]

Evaluating policy:  86%|########5 | 71/83 [00:34<00:05,  2.31it/s]

Evaluating policy:  87%|########6 | 72/83 [00:35<00:05,  2.15it/s]

Evaluating policy:  88%|########7 | 73/83 [00:35<00:05,  1.84it/s]

Evaluating policy:  89%|########9 | 74/83 [00:36<00:04,  1.95it/s]

Evaluating policy:  90%|######### | 75/83 [00:36<00:04,  1.95it/s]

Evaluating policy:  92%|#########1| 76/83 [00:37<00:03,  2.00it/s]

Evaluating policy:  93%|#########2| 77/83 [00:37<00:02,  2.08it/s]

Evaluating policy:  94%|#########3| 78/83 [00:38<00:02,  2.07it/s]

Evaluating policy:  95%|#########5| 79/83 [00:38<00:01,  2.08it/s]

Evaluating policy:  96%|#########6| 80/83 [00:39<00:01,  2.14it/s]

Evaluating policy:  98%|#########7| 81/83 [00:39<00:00,  2.21it/s]

Evaluating policy:  99%|#########8| 82/83 [00:40<00:00,  2.13it/s]

Evaluating policy: 100%|##########| 83/83 [00:40<00:00,  2.33it/s]

Outer training loop:   3%|3         | 1501/50000 [04:19<9:27:54,  1.42it/s]
Outer training loop:   3%|3         | 1502/50000 [04:20<10:46:16,  1.25it/s]
Outer training loop:   3%|3         | 1503/50000 [04:20<9:59:50,  1.35it/s] 
Outer training loop:   3%|3         | 1504/50000 [04:21<8:37:36,  1.56it/s]
Outer training loop:   3%|3         | 1505/50000 [04:22<9:15:17,  1.46it/s]
Outer training loop:   3%|3         | 1506/50000 [04:22<8:39:44,  1.56it/s]
Outer training loop:   3%|3         | 1507/50000 [04:23<8:25:53,  1.60it/s]
Outer training loop:   3%|3         | 1508/50000 [04:23<9:04:35,  1.48it/s]
Outer training loop:   3%|3         | 1509/50000 [04:24<8:32:13,  1.58it/s]
Outer training loop:   3%|3         | 1510/50000 [04:24<7:48:04,  1.73it/s]
Outer training loop:   3%|3         | 1511/50000 [04:25<9:22:00,  1.44it/s]
Outer training loop:   3%|3         | 1512/50000 [04:26<8:15:09,  1.63it/s]
Outer training loop:   3%|3         | 1513/50000 [04:26<8:24:24,  1.60it/s]
Outer training loop:   3%|3         | 1514/50000 [04:27<8:45:04,  1.54it/s]
Outer training loop:   3%|3         | 1515/50000 [04:28<7:54:49,  1.70it/s]
Outer training loop:   3%|3         | 1516/50000 [04:28<8:56:27,  1.51it/s]
Outer training loop:   3%|3         | 1517/50000 [04:29<8:54:28,  1.51it/s]
Outer training loop:   3%|3         | 1518/50000 [04:30<8:15:33,  1.63it/s]
Outer training loop:   3%|3         | 1519/50000 [04:30<7:44:19,  1.74it/s]
Outer training loop:   3%|3         | 1520/50000 [04:31<7:10:13,  1.88it/s]
Outer training loop:   3%|3         | 1521/50000 [04:31<6:36:35,  2.04it/s]
Outer training loop:   3%|3         | 1522/50000 [04:31<6:08:59,  2.19it/s]
Outer training loop:   3%|3         | 1523/50000 [04:32<5:42:05,  2.36it/s]
Outer training loop:   3%|3         | 1524/50000 [04:32<6:14:44,  2.16it/s]
Outer training loop:   3%|3         | 1525/50000 [04:33<5:54:44,  2.28it/s]
Outer training loop:   3%|3         | 1526/50000 [04:33<6:00:13,  2.24it/s]
Outer training loop:   3%|3         | 1527/50000 [04:33<5:50:50,  2.30it/s]
Outer training loop:   3%|3         | 1528/50000 [04:34<5:50:29,  2.30it/s]
Outer training loop:   3%|3         | 1529/50000 [04:34<5:43:25,  2.35it/s]
Outer training loop:   3%|3         | 1530/50000 [04:35<5:58:19,  2.25it/s]
Outer training loop:   3%|3         | 1531/50000 [04:35<5:53:09,  2.29it/s]
Outer training loop:   3%|3         | 1532/50000 [04:36<5:45:12,  2.34it/s]
Outer training loop:   3%|3         | 1533/50000 [04:36<5:42:33,  2.36it/s]
Outer training loop:   3%|3         | 1534/50000 [04:36<5:41:01,  2.37it/s]
Outer training loop:   3%|3         | 1535/50000 [04:37<5:34:31,  2.41it/s]
Outer training loop:   3%|3         | 1536/50000 [04:37<5:45:09,  2.34it/s]
Outer training loop:   3%|3         | 1537/50000 [04:38<6:29:44,  2.07it/s]
Outer training loop:   3%|3         | 1538/50000 [04:38<6:22:03,  2.11it/s]
Outer training loop:   3%|3         | 1539/50000 [04:39<6:42:07,  2.01it/s]
Outer training loop:   3%|3         | 1540/50000 [04:39<6:56:25,  1.94it/s]
Outer training loop:   3%|3         | 1541/50000 [04:40<7:45:25,  1.74it/s]
Outer training loop:   3%|3         | 1542/50000 [04:41<7:25:31,  1.81it/s]
Outer training loop:   3%|3         | 1543/50000 [04:41<6:49:01,  1.97it/s]
Outer training loop:   3%|3         | 1544/50000 [04:42<8:28:24,  1.59it/s]
Outer training loop:   3%|3         | 1545/50000 [04:43<8:42:54,  1.54it/s]
Outer training loop:   3%|3         | 1546/50000 [04:44<10:05:40,  1.33it/s]
Outer training loop:   3%|3         | 1547/50000 [04:44<10:18:09,  1.31it/s]
Outer training loop:   3%|3         | 1548/50000 [04:46<12:26:56,  1.08it/s]
Outer training loop:   3%|3         | 1549/50000 [04:46<11:05:16,  1.21it/s]
Outer training loop:   3%|3         | 1550/50000 [04:48<14:21:23,  1.07s/it]
Outer training loop:   3%|3         | 1551/50000 [04:49<15:26:47,  1.15s/it]
Outer training loop:   3%|3         | 1552/50000 [04:51<17:35:45,  1.31s/it]
Outer training loop:   3%|3         | 1553/50000 [04:53<18:36:28,  1.38s/it]
Outer training loop:   3%|3         | 1554/50000 [04:54<18:25:23,  1.37s/it]
Outer training loop:   3%|3         | 1555/50000 [04:56<20:05:27,  1.49s/it]
Outer training loop:   3%|3         | 1556/50000 [04:57<20:56:42,  1.56s/it]
Outer training loop:   3%|3         | 1557/50000 [04:59<20:17:59,  1.51s/it]
Outer training loop:   3%|3         | 1558/50000 [05:01<21:26:04,  1.59s/it]
Outer training loop:   3%|3         | 1559/50000 [05:02<20:54:05,  1.55s/it]
Outer training loop:   3%|3         | 1560/50000 [05:04<21:39:04,  1.61s/it]
Outer training loop:   3%|3         | 1561/50000 [05:05<21:28:18,  1.60s/it]
Outer training loop:   3%|3         | 1562/50000 [05:07<21:47:07,  1.62s/it]
Outer training loop:   3%|3         | 1563/50000 [05:08<20:33:07,  1.53s/it]
Outer training loop:   3%|3         | 1564/50000 [05:10<19:43:54,  1.47s/it]
Outer training loop:   3%|3         | 1565/50000 [05:11<19:03:08,  1.42s/it]
Outer training loop:   3%|3         | 1566/50000 [05:13<19:33:39,  1.45s/it]
Outer training loop:   3%|3         | 1567/50000 [05:14<19:10:17,  1.43s/it]
Outer training loop:   3%|3         | 1568/50000 [05:15<18:30:06,  1.38s/it]
Outer training loop:   3%|3         | 1569/50000 [05:17<19:38:28,  1.46s/it]
Outer training loop:   3%|3         | 1570/50000 [05:18<20:15:53,  1.51s/it]
Outer training loop:   3%|3         | 1571/50000 [05:20<19:38:56,  1.46s/it]
Outer training loop:   3%|3         | 1572/50000 [05:21<20:18:00,  1.51s/it]
Outer training loop:   3%|3         | 1573/50000 [05:23<20:55:26,  1.56s/it]
Outer training loop:   3%|3         | 1574/50000 [05:25<20:55:27,  1.56s/it]
Outer training loop:   3%|3         | 1575/50000 [05:26<20:32:03,  1.53s/it]
Outer training loop:   3%|3         | 1576/50000 [05:28<21:05:11,  1.57s/it]
Outer training loop:   3%|3         | 1577/50000 [05:29<20:12:05,  1.50s/it]
Outer training loop:   3%|3         | 1578/50000 [05:31<20:00:17,  1.49s/it]
Outer training loop:   3%|3         | 1579/50000 [05:32<20:59:26,  1.56s/it]
Outer training loop:   3%|3         | 1580/50000 [05:34<20:53:39,  1.55s/it]
Outer training loop:   3%|3         | 1581/50000 [05:36<21:45:16,  1.62s/it]
Outer training loop:   3%|3         | 1582/50000 [05:37<20:50:01,  1.55s/it]
Outer training loop:   3%|3         | 1583/50000 [05:39<21:37:45,  1.61s/it]
Outer training loop:   3%|3         | 1584/50000 [05:40<22:08:39,  1.65s/it]
Outer training loop:   3%|3         | 1585/50000 [05:42<22:44:08,  1.69s/it]
Outer training loop:   3%|3         | 1586/50000 [05:44<22:32:19,  1.68s/it]
Outer training loop:   3%|3         | 1587/50000 [05:46<22:51:43,  1.70s/it]
Outer training loop:   3%|3         | 1588/50000 [05:47<22:15:07,  1.65s/it]
Outer training loop:   3%|3         | 1589/50000 [05:49<22:16:50,  1.66s/it]
Outer training loop:   3%|3         | 1590/50000 [05:50<22:05:34,  1.64s/it]
Outer training loop:   3%|3         | 1591/50000 [05:52<22:07:54,  1.65s/it]
Outer training loop:   3%|3         | 1592/50000 [05:54<21:36:57,  1.61s/it]
Outer training loop:   3%|3         | 1593/50000 [05:55<21:58:11,  1.63s/it]
Outer training loop:   3%|3         | 1594/50000 [05:57<23:32:48,  1.75s/it]
Outer training loop:   3%|3         | 1595/50000 [05:59<24:44:23,  1.84s/it]
Outer training loop:   3%|3         | 1596/50000 [06:01<23:58:23,  1.78s/it]
Outer training loop:   3%|3         | 1597/50000 [06:03<23:36:05,  1.76s/it]
Outer training loop:   3%|3         | 1598/50000 [06:04<23:32:59,  1.75s/it]
Outer training loop:   3%|3         | 1599/50000 [06:06<23:43:21,  1.76s/it]
Outer training loop:   3%|3         | 1600/50000 [06:08<23:06:33,  1.72s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.53s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.59s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.57s/it]

Outer training loop:   3%|3         | 1601/50000 [06:14<42:48:44,  3.18s/it]
Outer training loop:   3%|3         | 1602/50000 [06:16<37:13:08,  2.77s/it]
Outer training loop:   3%|3         | 1603/50000 [06:18<32:49:50,  2.44s/it]
Outer training loop:   3%|3         | 1604/50000 [06:20<29:34:07,  2.20s/it]
Outer training loop:   3%|3         | 1605/50000 [06:21<27:43:12,  2.06s/it]
Outer training loop:   3%|3         | 1606/50000 [06:23<25:49:07,  1.92s/it]
Outer training loop:   3%|3         | 1607/50000 [06:25<25:05:15,  1.87s/it]
Outer training loop:   3%|3         | 1608/50000 [06:26<24:00:37,  1.79s/it]
Outer training loop:   3%|3         | 1609/50000 [06:28<23:23:10,  1.74s/it]
Outer training loop:   3%|3         | 1610/50000 [06:30<23:06:52,  1.72s/it]
Outer training loop:   3%|3         | 1611/50000 [06:31<22:38:51,  1.68s/it]
Outer training loop:   3%|3         | 1612/50000 [06:33<22:50:58,  1.70s/it]
Outer training loop:   3%|3         | 1613/50000 [06:34<22:13:28,  1.65s/it]
Outer training loop:   3%|3         | 1614/50000 [06:36<21:52:01,  1.63s/it]
Outer training loop:   3%|3         | 1615/50000 [06:38<21:57:38,  1.63s/it]
Outer training loop:   3%|3         | 1616/50000 [06:39<21:14:11,  1.58s/it]
Outer training loop:   3%|3         | 1617/50000 [06:41<21:53:11,  1.63s/it]
Outer training loop:   3%|3         | 1618/50000 [06:42<21:52:39,  1.63s/it]
Outer training loop:   3%|3         | 1619/50000 [06:44<22:00:50,  1.64s/it]
Outer training loop:   3%|3         | 1620/50000 [06:46<21:54:38,  1.63s/it]
Outer training loop:   3%|3         | 1621/50000 [06:47<22:05:04,  1.64s/it]
Outer training loop:   3%|3         | 1622/50000 [06:49<21:54:39,  1.63s/it]
Outer training loop:   3%|3         | 1623/50000 [06:51<22:12:37,  1.65s/it]
Outer training loop:   3%|3         | 1624/50000 [06:52<21:55:02,  1.63s/it]
Outer training loop:   3%|3         | 1625/50000 [06:54<22:08:55,  1.65s/it]
Outer training loop:   3%|3         | 1626/50000 [06:56<22:10:35,  1.65s/it]
Outer training loop:   3%|3         | 1627/50000 [06:57<22:31:25,  1.68s/it]
Outer training loop:   3%|3         | 1628/50000 [06:59<22:00:05,  1.64s/it]
Outer training loop:   3%|3         | 1629/50000 [07:01<22:04:54,  1.64s/it]
Outer training loop:   3%|3         | 1630/50000 [07:02<21:33:43,  1.60s/it]
Outer training loop:   3%|3         | 1631/50000 [07:04<22:15:26,  1.66s/it]
Outer training loop:   3%|3         | 1632/50000 [07:05<21:52:33,  1.63s/it]
Outer training loop:   3%|3         | 1633/50000 [07:07<21:53:56,  1.63s/it]
Outer training loop:   3%|3         | 1634/50000 [07:09<21:40:47,  1.61s/it]
Outer training loop:   3%|3         | 1635/50000 [07:10<21:46:55,  1.62s/it]
Outer training loop:   3%|3         | 1636/50000 [07:12<21:40:23,  1.61s/it]
Outer training loop:   3%|3         | 1637/50000 [07:14<21:38:51,  1.61s/it]
Outer training loop:   3%|3         | 1638/50000 [07:15<21:51:01,  1.63s/it]
Outer training loop:   3%|3         | 1639/50000 [07:17<22:06:42,  1.65s/it]
Outer training loop:   3%|3         | 1640/50000 [07:18<21:45:50,  1.62s/it]
Outer training loop:   3%|3         | 1641/50000 [07:20<21:37:21,  1.61s/it]
Outer training loop:   3%|3         | 1642/50000 [07:22<22:00:54,  1.64s/it]
Outer training loop:   3%|3         | 1643/50000 [07:23<22:01:13,  1.64s/it]
Outer training loop:   3%|3         | 1644/50000 [07:25<22:22:33,  1.67s/it]
Outer training loop:   3%|3         | 1645/50000 [07:27<21:57:18,  1.63s/it]
Outer training loop:   3%|3         | 1646/50000 [07:28<22:23:51,  1.67s/it]
Outer training loop:   3%|3         | 1647/50000 [07:30<21:59:39,  1.64s/it]
Outer training loop:   3%|3         | 1648/50000 [07:32<22:14:59,  1.66s/it]
Outer training loop:   3%|3         | 1649/50000 [07:33<22:07:50,  1.65s/it]
Outer training loop:   3%|3         | 1650/50000 [07:35<22:15:51,  1.66s/it]
Outer training loop:   3%|3         | 1651/50000 [07:37<21:53:45,  1.63s/it]
Outer training loop:   3%|3         | 1652/50000 [07:38<22:43:58,  1.69s/it]
Outer training loop:   3%|3         | 1653/50000 [07:40<22:21:02,  1.66s/it]
Outer training loop:   3%|3         | 1654/50000 [07:42<22:46:09,  1.70s/it]
Outer training loop:   3%|3         | 1655/50000 [07:43<22:23:46,  1.67s/it]
Outer training loop:   3%|3         | 1656/50000 [07:45<22:32:50,  1.68s/it]
Outer training loop:   3%|3         | 1657/50000 [07:47<22:14:57,  1.66s/it]
Outer training loop:   3%|3         | 1658/50000 [07:48<22:10:08,  1.65s/it]
Outer training loop:   3%|3         | 1659/50000 [07:50<22:22:12,  1.67s/it]
Outer training loop:   3%|3         | 1660/50000 [07:52<22:19:36,  1.66s/it]
Outer training loop:   3%|3         | 1661/50000 [07:53<21:40:28,  1.61s/it]
Outer training loop:   3%|3         | 1662/50000 [07:55<22:00:04,  1.64s/it]
Outer training loop:   3%|3         | 1663/50000 [07:56<21:42:19,  1.62s/it]
Outer training loop:   3%|3         | 1664/50000 [07:58<21:54:11,  1.63s/it]
Outer training loop:   3%|3         | 1665/50000 [08:00<22:00:44,  1.64s/it]
Outer training loop:   3%|3         | 1666/50000 [08:01<22:05:44,  1.65s/it]
Outer training loop:   3%|3         | 1667/50000 [08:03<21:37:43,  1.61s/it]
Outer training loop:   3%|3         | 1668/50000 [08:04<21:26:33,  1.60s/it]
Outer training loop:   3%|3         | 1669/50000 [08:06<21:41:10,  1.62s/it]
Outer training loop:   3%|3         | 1670/50000 [08:08<21:33:23,  1.61s/it]
Outer training loop:   3%|3         | 1671/50000 [08:09<21:51:47,  1.63s/it]
Outer training loop:   3%|3         | 1672/50000 [08:11<21:29:51,  1.60s/it]
Outer training loop:   3%|3         | 1673/50000 [08:13<21:34:03,  1.61s/it]
Outer training loop:   3%|3         | 1674/50000 [08:14<21:27:21,  1.60s/it]
Outer training loop:   3%|3         | 1675/50000 [08:16<21:14:05,  1.58s/it]
Outer training loop:   3%|3         | 1676/50000 [08:17<20:53:50,  1.56s/it]
Outer training loop:   3%|3         | 1677/50000 [08:19<21:24:40,  1.60s/it]
Outer training loop:   3%|3         | 1678/50000 [08:20<21:21:35,  1.59s/it]
Outer training loop:   3%|3         | 1679/50000 [08:22<21:40:28,  1.61s/it]
Outer training loop:   3%|3         | 1680/50000 [08:24<21:33:09,  1.61s/it]
Outer training loop:   3%|3         | 1681/50000 [08:25<21:59:51,  1.64s/it]
Outer training loop:   3%|3         | 1682/50000 [08:27<21:39:03,  1.61s/it]
Outer training loop:   3%|3         | 1683/50000 [08:29<21:50:40,  1.63s/it]
Outer training loop:   3%|3         | 1684/50000 [08:30<21:54:48,  1.63s/it]
Outer training loop:   3%|3         | 1685/50000 [08:32<21:58:54,  1.64s/it]
Outer training loop:   3%|3         | 1686/50000 [08:34<22:05:08,  1.65s/it]
Outer training loop:   3%|3         | 1687/50000 [08:35<22:08:34,  1.65s/it]
Outer training loop:   3%|3         | 1688/50000 [08:37<21:50:50,  1.63s/it]
Outer training loop:   3%|3         | 1689/50000 [08:38<21:51:54,  1.63s/it]
Outer training loop:   3%|3         | 1690/50000 [08:40<21:54:57,  1.63s/it]
Outer training loop:   3%|3         | 1691/50000 [08:42<22:05:13,  1.65s/it]
Outer training loop:   3%|3         | 1692/50000 [08:43<22:01:43,  1.64s/it]
Outer training loop:   3%|3         | 1693/50000 [08:45<21:47:29,  1.62s/it]
Outer training loop:   3%|3         | 1694/50000 [08:47<22:08:01,  1.65s/it]
Outer training loop:   3%|3         | 1695/50000 [08:48<21:57:48,  1.64s/it]
Outer training loop:   3%|3         | 1696/50000 [08:50<22:12:16,  1.65s/it]
Outer training loop:   3%|3         | 1697/50000 [08:52<21:52:39,  1.63s/it]
Outer training loop:   3%|3         | 1698/50000 [08:53<22:03:03,  1.64s/it]
Outer training loop:   3%|3         | 1699/50000 [08:55<21:50:30,  1.63s/it]
Outer training loop:   3%|3         | 1700/50000 [08:57<22:22:36,  1.67s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.69s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.63s/it]

Outer training loop:   3%|3         | 1701/50000 [09:03<41:18:42,  3.08s/it]
Outer training loop:   3%|3         | 1702/50000 [09:05<35:50:18,  2.67s/it]
Outer training loop:   3%|3         | 1703/50000 [09:06<31:26:55,  2.34s/it]
Outer training loop:   3%|3         | 1704/50000 [09:08<28:30:13,  2.12s/it]
Outer training loop:   3%|3         | 1705/50000 [09:09<26:19:49,  1.96s/it]
Outer training loop:   3%|3         | 1706/50000 [09:11<25:26:11,  1.90s/it]
Outer training loop:   3%|3         | 1707/50000 [09:13<24:19:02,  1.81s/it]
Outer training loop:   3%|3         | 1708/50000 [09:15<23:46:21,  1.77s/it]
Outer training loop:   3%|3         | 1709/50000 [09:16<23:03:17,  1.72s/it]
Outer training loop:   3%|3         | 1710/50000 [09:18<22:54:55,  1.71s/it]
Outer training loop:   3%|3         | 1711/50000 [09:19<22:30:15,  1.68s/it]
Outer training loop:   3%|3         | 1712/50000 [09:21<22:30:35,  1.68s/it]
Outer training loop:   3%|3         | 1713/50000 [09:23<22:11:30,  1.65s/it]
Outer training loop:   3%|3         | 1714/50000 [09:24<22:43:05,  1.69s/it]
Outer training loop:   3%|3         | 1715/50000 [09:26<22:19:38,  1.66s/it]
Outer training loop:   3%|3         | 1716/50000 [09:28<22:17:37,  1.66s/it]
Outer training loop:   3%|3         | 1717/50000 [09:29<22:13:26,  1.66s/it]
Outer training loop:   3%|3         | 1718/50000 [09:31<22:01:06,  1.64s/it]
Outer training loop:   3%|3         | 1719/50000 [09:33<21:51:50,  1.63s/it]
Outer training loop:   3%|3         | 1720/50000 [09:34<22:04:48,  1.65s/it]
Outer training loop:   3%|3         | 1721/50000 [09:36<21:58:36,  1.64s/it]
Outer training loop:   3%|3         | 1722/50000 [09:38<22:04:43,  1.65s/it]
Outer training loop:   3%|3         | 1723/50000 [09:39<22:15:28,  1.66s/it]
Outer training loop:   3%|3         | 1724/50000 [09:41<22:34:44,  1.68s/it]
Outer training loop:   3%|3         | 1725/50000 [09:43<22:19:12,  1.66s/it]
Outer training loop:   3%|3         | 1726/50000 [09:44<22:37:24,  1.69s/it]
Outer training loop:   3%|3         | 1727/50000 [09:46<22:13:39,  1.66s/it]
Outer training loop:   3%|3         | 1728/50000 [09:48<22:25:03,  1.67s/it]
Outer training loop:   3%|3         | 1729/50000 [09:49<22:39:55,  1.69s/it]
Outer training loop:   3%|3         | 1730/50000 [09:51<22:08:51,  1.65s/it]
Outer training loop:   3%|3         | 1731/50000 [09:53<22:25:34,  1.67s/it]
Outer training loop:   3%|3         | 1732/50000 [09:54<21:52:29,  1.63s/it]
Outer training loop:   3%|3         | 1733/50000 [09:56<21:59:12,  1.64s/it]
Outer training loop:   3%|3         | 1734/50000 [09:57<21:44:18,  1.62s/it]
Outer training loop:   3%|3         | 1735/50000 [09:59<22:01:29,  1.64s/it]
Outer training loop:   3%|3         | 1736/50000 [10:01<21:42:39,  1.62s/it]
Outer training loop:   3%|3         | 1737/50000 [10:02<21:54:53,  1.63s/it]
Outer training loop:   3%|3         | 1738/50000 [10:04<21:40:57,  1.62s/it]
Outer training loop:   3%|3         | 1739/50000 [10:05<21:24:14,  1.60s/it]
Outer training loop:   3%|3         | 1740/50000 [10:07<21:43:51,  1.62s/it]
Outer training loop:   3%|3         | 1741/50000 [10:09<21:57:04,  1.64s/it]
Outer training loop:   3%|3         | 1742/50000 [10:10<22:06:12,  1.65s/it]
Outer training loop:   3%|3         | 1743/50000 [10:12<22:12:07,  1.66s/it]
Outer training loop:   3%|3         | 1744/50000 [10:14<21:38:49,  1.61s/it]
Outer training loop:   3%|3         | 1745/50000 [10:15<21:44:16,  1.62s/it]
Outer training loop:   3%|3         | 1746/50000 [10:17<21:38:22,  1.61s/it]
Outer training loop:   3%|3         | 1747/50000 [10:19<21:39:31,  1.62s/it]
Outer training loop:   3%|3         | 1748/50000 [10:20<21:21:38,  1.59s/it]
Outer training loop:   3%|3         | 1749/50000 [10:22<22:25:27,  1.67s/it]
Outer training loop:   4%|3         | 1750/50000 [10:24<22:15:31,  1.66s/it]
Outer training loop:   4%|3         | 1751/50000 [10:25<21:28:42,  1.60s/it]
Outer training loop:   4%|3         | 1752/50000 [10:27<21:45:33,  1.62s/it]
Outer training loop:   4%|3         | 1753/50000 [10:28<22:01:22,  1.64s/it]
Outer training loop:   4%|3         | 1754/50000 [10:30<22:04:31,  1.65s/it]
Outer training loop:   4%|3         | 1755/50000 [10:32<21:40:37,  1.62s/it]
Outer training loop:   4%|3         | 1756/50000 [10:33<22:01:38,  1.64s/it]
Outer training loop:   4%|3         | 1757/50000 [10:35<22:13:28,  1.66s/it]
Outer training loop:   4%|3         | 1758/50000 [10:37<22:10:56,  1.66s/it]
Outer training loop:   4%|3         | 1759/50000 [10:38<22:04:05,  1.65s/it]
Outer training loop:   4%|3         | 1760/50000 [10:40<22:12:59,  1.66s/it]
Outer training loop:   4%|3         | 1761/50000 [10:42<21:48:43,  1.63s/it]
Outer training loop:   4%|3         | 1762/50000 [10:43<22:30:39,  1.68s/it]
Outer training loop:   4%|3         | 1763/50000 [10:45<22:14:38,  1.66s/it]
Outer training loop:   4%|3         | 1764/50000 [10:47<21:58:08,  1.64s/it]
Outer training loop:   4%|3         | 1765/50000 [10:48<22:11:32,  1.66s/it]
Outer training loop:   4%|3         | 1766/50000 [10:50<22:34:07,  1.68s/it]
Outer training loop:   4%|3         | 1767/50000 [10:52<22:27:48,  1.68s/it]
Outer training loop:   4%|3         | 1768/50000 [10:53<22:04:58,  1.65s/it]
Outer training loop:   4%|3         | 1769/50000 [10:55<22:03:37,  1.65s/it]
Outer training loop:   4%|3         | 1770/50000 [10:56<21:44:10,  1.62s/it]
Outer training loop:   4%|3         | 1771/50000 [10:58<21:21:40,  1.59s/it]
Outer training loop:   4%|3         | 1772/50000 [11:00<21:57:16,  1.64s/it]
Outer training loop:   4%|3         | 1773/50000 [11:01<21:35:53,  1.61s/it]
Outer training loop:   4%|3         | 1774/50000 [11:03<21:41:59,  1.62s/it]
Outer training loop:   4%|3         | 1775/50000 [11:04<21:31:25,  1.61s/it]
Outer training loop:   4%|3         | 1776/50000 [11:06<21:53:30,  1.63s/it]
Outer training loop:   4%|3         | 1777/50000 [11:08<21:32:22,  1.61s/it]
Outer training loop:   4%|3         | 1778/50000 [11:09<21:47:33,  1.63s/it]
Outer training loop:   4%|3         | 1779/50000 [11:11<21:23:21,  1.60s/it]
Outer training loop:   4%|3         | 1780/50000 [11:13<21:37:15,  1.61s/it]
Outer training loop:   4%|3         | 1781/50000 [11:14<21:24:39,  1.60s/it]
Outer training loop:   4%|3         | 1782/50000 [11:16<21:40:05,  1.62s/it]
Outer training loop:   4%|3         | 1783/50000 [11:17<21:54:30,  1.64s/it]
Outer training loop:   4%|3         | 1784/50000 [11:19<22:09:45,  1.65s/it]
Outer training loop:   4%|3         | 1785/50000 [11:21<21:49:05,  1.63s/it]
Outer training loop:   4%|3         | 1786/50000 [11:22<21:49:37,  1.63s/it]
Outer training loop:   4%|3         | 1787/50000 [11:24<21:44:24,  1.62s/it]
Outer training loop:   4%|3         | 1788/50000 [11:26<21:51:59,  1.63s/it]
Outer training loop:   4%|3         | 1789/50000 [11:27<21:34:24,  1.61s/it]
Outer training loop:   4%|3         | 1790/50000 [11:29<21:35:58,  1.61s/it]
Outer training loop:   4%|3         | 1791/50000 [11:30<21:26:28,  1.60s/it]
Outer training loop:   4%|3         | 1792/50000 [11:32<21:36:00,  1.61s/it]
Outer training loop:   4%|3         | 1793/50000 [11:34<21:17:34,  1.59s/it]
Outer training loop:   4%|3         | 1794/50000 [11:35<22:01:56,  1.65s/it]
Outer training loop:   4%|3         | 1795/50000 [11:37<21:43:09,  1.62s/it]
Outer training loop:   4%|3         | 1796/50000 [11:39<21:47:59,  1.63s/it]
Outer training loop:   4%|3         | 1797/50000 [11:40<21:53:17,  1.63s/it]
Outer training loop:   4%|3         | 1798/50000 [11:42<21:55:52,  1.64s/it]
Outer training loop:   4%|3         | 1799/50000 [11:44<22:22:22,  1.67s/it]
Outer training loop:   4%|3         | 1800/50000 [11:45<22:20:17,  1.67s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.61s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.62s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.62s/it]

Outer training loop:   4%|3         | 1801/50000 [11:52<41:44:04,  3.12s/it]
Outer training loop:   4%|3         | 1802/50000 [11:53<35:37:22,  2.66s/it]
Outer training loop:   4%|3         | 1803/50000 [11:55<31:21:05,  2.34s/it]
Outer training loop:   4%|3         | 1804/50000 [11:57<28:24:22,  2.12s/it]
Outer training loop:   4%|3         | 1805/50000 [11:58<26:14:35,  1.96s/it]
Outer training loop:   4%|3         | 1806/50000 [12:00<25:01:56,  1.87s/it]
Outer training loop:   4%|3         | 1807/50000 [12:02<24:29:35,  1.83s/it]
Outer training loop:   4%|3         | 1808/50000 [12:03<23:20:41,  1.74s/it]
Outer training loop:   4%|3         | 1809/50000 [12:05<23:43:44,  1.77s/it]
Outer training loop:   4%|3         | 1810/50000 [12:06<22:56:17,  1.71s/it]
Outer training loop:   4%|3         | 1811/50000 [12:08<22:04:52,  1.65s/it]
Outer training loop:   4%|3         | 1812/50000 [12:10<22:37:37,  1.69s/it]
Outer training loop:   4%|3         | 1813/50000 [12:11<22:11:03,  1.66s/it]
Outer training loop:   4%|3         | 1814/50000 [12:13<22:04:00,  1.65s/it]
Outer training loop:   4%|3         | 1815/50000 [12:15<21:36:35,  1.61s/it]
Outer training loop:   4%|3         | 1816/50000 [12:16<21:56:14,  1.64s/it]
Outer training loop:   4%|3         | 1817/50000 [12:18<21:41:34,  1.62s/it]
Outer training loop:   4%|3         | 1818/50000 [12:19<21:51:56,  1.63s/it]
Outer training loop:   4%|3         | 1819/50000 [12:21<21:42:03,  1.62s/it]
Outer training loop:   4%|3         | 1820/50000 [12:23<21:51:51,  1.63s/it]
Outer training loop:   4%|3         | 1821/50000 [12:24<21:44:38,  1.62s/it]
Outer training loop:   4%|3         | 1822/50000 [12:26<21:45:14,  1.63s/it]
Outer training loop:   4%|3         | 1823/50000 [12:28<21:34:28,  1.61s/it]
Outer training loop:   4%|3         | 1824/50000 [12:29<21:36:39,  1.61s/it]
Outer training loop:   4%|3         | 1825/50000 [12:31<21:25:32,  1.60s/it]
Outer training loop:   4%|3         | 1826/50000 [12:32<21:35:40,  1.61s/it]
Outer training loop:   4%|3         | 1827/50000 [12:34<21:41:12,  1.62s/it]
Outer training loop:   4%|3         | 1828/50000 [12:36<21:43:04,  1.62s/it]
Outer training loop:   4%|3         | 1829/50000 [12:37<21:30:07,  1.61s/it]
Outer training loop:   4%|3         | 1830/50000 [12:39<21:56:36,  1.64s/it]
Outer training loop:   4%|3         | 1831/50000 [12:41<22:05:54,  1.65s/it]
Outer training loop:   4%|3         | 1832/50000 [12:42<22:09:29,  1.66s/it]
Outer training loop:   4%|3         | 1833/50000 [12:44<21:41:39,  1.62s/it]
Outer training loop:   4%|3         | 1834/50000 [12:45<21:58:30,  1.64s/it]
Outer training loop:   4%|3         | 1835/50000 [12:47<21:37:14,  1.62s/it]
Outer training loop:   4%|3         | 1836/50000 [12:49<22:11:10,  1.66s/it]
Outer training loop:   4%|3         | 1837/50000 [12:50<21:55:04,  1.64s/it]
Outer training loop:   4%|3         | 1838/50000 [12:52<22:18:01,  1.67s/it]
Outer training loop:   4%|3         | 1839/50000 [12:54<21:44:21,  1.62s/it]
Outer training loop:   4%|3         | 1840/50000 [12:55<22:01:37,  1.65s/it]
Outer training loop:   4%|3         | 1841/50000 [12:57<21:36:30,  1.62s/it]
Outer training loop:   4%|3         | 1842/50000 [12:59<21:53:58,  1.64s/it]
Outer training loop:   4%|3         | 1843/50000 [13:00<21:38:55,  1.62s/it]
Outer training loop:   4%|3         | 1844/50000 [13:02<21:58:39,  1.64s/it]
Outer training loop:   4%|3         | 1845/50000 [13:03<21:39:05,  1.62s/it]
Outer training loop:   4%|3         | 1846/50000 [13:05<22:38:23,  1.69s/it]
Outer training loop:   4%|3         | 1847/50000 [13:07<22:36:34,  1.69s/it]
Outer training loop:   4%|3         | 1848/50000 [13:09<22:47:06,  1.70s/it]
Outer training loop:   4%|3         | 1849/50000 [13:10<22:19:22,  1.67s/it]
Outer training loop:   4%|3         | 1850/50000 [13:12<22:32:46,  1.69s/it]
Outer training loop:   4%|3         | 1851/50000 [13:14<21:56:41,  1.64s/it]
Outer training loop:   4%|3         | 1852/50000 [13:15<22:01:19,  1.65s/it]
Outer training loop:   4%|3         | 1853/50000 [13:17<21:58:12,  1.64s/it]
Outer training loop:   4%|3         | 1854/50000 [13:18<21:42:21,  1.62s/it]
Outer training loop:   4%|3         | 1855/50000 [13:20<21:59:51,  1.64s/it]
Outer training loop:   4%|3         | 1856/50000 [13:22<21:41:33,  1.62s/it]
Outer training loop:   4%|3         | 1857/50000 [13:23<22:04:01,  1.65s/it]
Outer training loop:   4%|3         | 1858/50000 [13:25<22:24:04,  1.68s/it]
Outer training loop:   4%|3         | 1859/50000 [13:27<21:46:01,  1.63s/it]
Outer training loop:   4%|3         | 1860/50000 [13:28<21:46:24,  1.63s/it]
Outer training loop:   4%|3         | 1861/50000 [13:30<21:39:52,  1.62s/it]
Outer training loop:   4%|3         | 1862/50000 [13:32<21:53:21,  1.64s/it]
Outer training loop:   4%|3         | 1863/50000 [13:33<21:52:43,  1.64s/it]
Outer training loop:   4%|3         | 1864/50000 [13:35<22:04:30,  1.65s/it]
Outer training loop:   4%|3         | 1865/50000 [13:37<22:10:39,  1.66s/it]
Outer training loop:   4%|3         | 1866/50000 [13:38<21:42:30,  1.62s/it]
Outer training loop:   4%|3         | 1867/50000 [13:40<21:51:29,  1.63s/it]
Outer training loop:   4%|3         | 1868/50000 [13:41<21:32:57,  1.61s/it]
Outer training loop:   4%|3         | 1869/50000 [13:43<21:55:51,  1.64s/it]
Outer training loop:   4%|3         | 1870/50000 [13:45<21:31:33,  1.61s/it]
Outer training loop:   4%|3         | 1871/50000 [13:46<21:24:00,  1.60s/it]
Outer training loop:   4%|3         | 1872/50000 [13:48<21:08:28,  1.58s/it]
Outer training loop:   4%|3         | 1873/50000 [13:49<21:39:19,  1.62s/it]
Outer training loop:   4%|3         | 1874/50000 [13:51<21:44:16,  1.63s/it]
Outer training loop:   4%|3         | 1875/50000 [13:53<22:08:43,  1.66s/it]
Outer training loop:   4%|3         | 1876/50000 [13:54<22:22:16,  1.67s/it]
Outer training loop:   4%|3         | 1877/50000 [13:56<21:51:40,  1.64s/it]
Outer training loop:   4%|3         | 1878/50000 [13:58<21:35:29,  1.62s/it]
Outer training loop:   4%|3         | 1879/50000 [13:59<21:25:37,  1.60s/it]
Outer training loop:   4%|3         | 1880/50000 [14:01<21:09:11,  1.58s/it]
Outer training loop:   4%|3         | 1881/50000 [14:02<21:40:08,  1.62s/it]
Outer training loop:   4%|3         | 1882/50000 [14:04<21:30:52,  1.61s/it]
Outer training loop:   4%|3         | 1883/50000 [14:06<21:53:09,  1.64s/it]
Outer training loop:   4%|3         | 1884/50000 [14:07<21:42:44,  1.62s/it]
Outer training loop:   4%|3         | 1885/50000 [14:09<22:00:37,  1.65s/it]
Outer training loop:   4%|3         | 1886/50000 [14:11<21:53:53,  1.64s/it]
Outer training loop:   4%|3         | 1887/50000 [14:12<21:52:17,  1.64s/it]
Outer training loop:   4%|3         | 1888/50000 [14:14<22:27:13,  1.68s/it]
Outer training loop:   4%|3         | 1889/50000 [14:16<21:59:49,  1.65s/it]
Outer training loop:   4%|3         | 1890/50000 [14:17<21:34:16,  1.61s/it]
Outer training loop:   4%|3         | 1891/50000 [14:19<22:22:40,  1.67s/it]
Outer training loop:   4%|3         | 1892/50000 [14:20<21:47:30,  1.63s/it]
Outer training loop:   4%|3         | 1893/50000 [14:22<21:50:43,  1.63s/it]
Outer training loop:   4%|3         | 1894/50000 [14:24<21:17:56,  1.59s/it]
Outer training loop:   4%|3         | 1895/50000 [14:25<21:48:57,  1.63s/it]
Outer training loop:   4%|3         | 1896/50000 [14:27<22:16:24,  1.67s/it]
Outer training loop:   4%|3         | 1897/50000 [14:29<22:28:57,  1.68s/it]
Outer training loop:   4%|3         | 1898/50000 [14:30<22:07:04,  1.66s/it]
Outer training loop:   4%|3         | 1899/50000 [14:32<22:06:48,  1.66s/it]
Outer training loop:   4%|3         | 1900/50000 [14:34<21:56:52,  1.64s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.63s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.67s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.64s/it]

Outer training loop:   4%|3         | 1901/50000 [14:40<42:36:46,  3.19s/it]
Outer training loop:   4%|3         | 1902/50000 [14:42<36:23:00,  2.72s/it]
Outer training loop:   4%|3         | 1903/50000 [14:44<31:37:23,  2.37s/it]
Outer training loop:   4%|3         | 1904/50000 [14:45<28:53:07,  2.16s/it]
Outer training loop:   4%|3         | 1905/50000 [14:47<26:30:43,  1.98s/it]
Outer training loop:   4%|3         | 1906/50000 [14:49<25:54:09,  1.94s/it]
Outer training loop:   4%|3         | 1907/50000 [14:50<24:49:37,  1.86s/it]
Outer training loop:   4%|3         | 1908/50000 [14:52<24:08:33,  1.81s/it]
Outer training loop:   4%|3         | 1909/50000 [14:54<23:02:55,  1.73s/it]
Outer training loop:   4%|3         | 1910/50000 [14:55<22:52:47,  1.71s/it]
Outer training loop:   4%|3         | 1911/50000 [14:57<22:04:07,  1.65s/it]
Outer training loop:   4%|3         | 1912/50000 [14:58<21:55:33,  1.64s/it]
Outer training loop:   4%|3         | 1913/50000 [15:00<21:45:12,  1.63s/it]
Outer training loop:   4%|3         | 1914/50000 [15:02<21:45:27,  1.63s/it]
Outer training loop:   4%|3         | 1915/50000 [15:03<22:04:55,  1.65s/it]
Outer training loop:   4%|3         | 1916/50000 [15:05<22:40:14,  1.70s/it]
Outer training loop:   4%|3         | 1917/50000 [15:07<21:47:31,  1.63s/it]
Outer training loop:   4%|3         | 1918/50000 [15:08<21:52:05,  1.64s/it]
Outer training loop:   4%|3         | 1919/50000 [15:10<21:50:17,  1.64s/it]
Outer training loop:   4%|3         | 1920/50000 [15:11<21:41:29,  1.62s/it]
Outer training loop:   4%|3         | 1921/50000 [15:13<21:21:32,  1.60s/it]
Outer training loop:   4%|3         | 1922/50000 [15:15<21:37:11,  1.62s/it]
Outer training loop:   4%|3         | 1923/50000 [15:16<21:34:15,  1.62s/it]
Outer training loop:   4%|3         | 1924/50000 [15:18<21:51:10,  1.64s/it]
Outer training loop:   4%|3         | 1925/50000 [15:20<21:27:11,  1.61s/it]
Outer training loop:   4%|3         | 1926/50000 [15:21<21:52:54,  1.64s/it]
Outer training loop:   4%|3         | 1927/50000 [15:23<21:36:12,  1.62s/it]
Outer training loop:   4%|3         | 1928/50000 [15:25<22:02:44,  1.65s/it]
Outer training loop:   4%|3         | 1929/50000 [15:26<21:58:37,  1.65s/it]
Outer training loop:   4%|3         | 1930/50000 [15:28<22:09:27,  1.66s/it]
Outer training loop:   4%|3         | 1931/50000 [15:29<21:55:50,  1.64s/it]
Outer training loop:   4%|3         | 1932/50000 [15:31<21:48:56,  1.63s/it]
Outer training loop:   4%|3         | 1933/50000 [15:33<21:58:00,  1.65s/it]
Outer training loop:   4%|3         | 1934/50000 [15:34<21:43:13,  1.63s/it]
Outer training loop:   4%|3         | 1935/50000 [15:36<21:51:14,  1.64s/it]
Outer training loop:   4%|3         | 1936/50000 [15:38<21:41:11,  1.62s/it]
Outer training loop:   4%|3         | 1937/50000 [15:39<22:01:42,  1.65s/it]
Outer training loop:   4%|3         | 1938/50000 [15:41<21:50:00,  1.64s/it]
Outer training loop:   4%|3         | 1939/50000 [15:43<21:58:54,  1.65s/it]
Outer training loop:   4%|3         | 1940/50000 [15:44<21:21:02,  1.60s/it]
Outer training loop:   4%|3         | 1941/50000 [15:46<21:43:03,  1.63s/it]
Outer training loop:   4%|3         | 1942/50000 [15:47<21:29:04,  1.61s/it]
Outer training loop:   4%|3         | 1943/50000 [15:49<22:37:03,  1.69s/it]
Outer training loop:   4%|3         | 1944/50000 [15:51<22:07:51,  1.66s/it]
Outer training loop:   4%|3         | 1945/50000 [15:52<22:01:51,  1.65s/it]
Outer training loop:   4%|3         | 1946/50000 [15:54<21:37:40,  1.62s/it]
Outer training loop:   4%|3         | 1947/50000 [15:56<21:36:48,  1.62s/it]
Outer training loop:   4%|3         | 1948/50000 [15:57<21:19:12,  1.60s/it]
Outer training loop:   4%|3         | 1949/50000 [15:59<21:31:29,  1.61s/it]
Outer training loop:   4%|3         | 1950/50000 [16:00<21:43:21,  1.63s/it]
Outer training loop:   4%|3         | 1951/50000 [16:02<22:01:41,  1.65s/it]
Outer training loop:   4%|3         | 1952/50000 [16:04<21:53:40,  1.64s/it]
Outer training loop:   4%|3         | 1953/50000 [16:05<21:59:44,  1.65s/it]
Outer training loop:   4%|3         | 1954/50000 [16:07<21:51:19,  1.64s/it]
Outer training loop:   4%|3         | 1955/50000 [16:09<21:53:07,  1.64s/it]
Outer training loop:   4%|3         | 1956/50000 [16:10<21:53:29,  1.64s/it]
Outer training loop:   4%|3         | 1957/50000 [16:12<21:47:08,  1.63s/it]
Outer training loop:   4%|3         | 1958/50000 [16:13<21:23:04,  1.60s/it]
Outer training loop:   4%|3         | 1959/50000 [16:15<21:22:45,  1.60s/it]
Outer training loop:   4%|3         | 1960/50000 [16:17<21:11:11,  1.59s/it]
Outer training loop:   4%|3         | 1961/50000 [16:18<21:51:48,  1.64s/it]
Outer training loop:   4%|3         | 1962/50000 [16:20<21:33:18,  1.62s/it]
Outer training loop:   4%|3         | 1963/50000 [16:22<21:36:48,  1.62s/it]
Outer training loop:   4%|3         | 1964/50000 [16:23<21:23:53,  1.60s/it]
Outer training loop:   4%|3         | 1965/50000 [16:25<21:48:39,  1.63s/it]
Outer training loop:   4%|3         | 1966/50000 [16:26<21:38:28,  1.62s/it]
Outer training loop:   4%|3         | 1967/50000 [16:28<22:07:32,  1.66s/it]
Outer training loop:   4%|3         | 1968/50000 [16:30<21:48:43,  1.63s/it]
Outer training loop:   4%|3         | 1969/50000 [16:31<21:06:58,  1.58s/it]
Outer training loop:   4%|3         | 1970/50000 [16:33<21:31:10,  1.61s/it]
Outer training loop:   4%|3         | 1971/50000 [16:35<21:42:05,  1.63s/it]
Outer training loop:   4%|3         | 1972/50000 [16:36<21:25:55,  1.61s/it]
Outer training loop:   4%|3         | 1973/50000 [16:38<21:35:13,  1.62s/it]
Outer training loop:   4%|3         | 1974/50000 [16:39<21:22:18,  1.60s/it]
Outer training loop:   4%|3         | 1975/50000 [16:41<21:35:10,  1.62s/it]
Outer training loop:   4%|3         | 1976/50000 [16:43<21:27:26,  1.61s/it]
Outer training loop:   4%|3         | 1977/50000 [16:44<21:35:51,  1.62s/it]
Outer training loop:   4%|3         | 1978/50000 [16:46<21:07:11,  1.58s/it]
Outer training loop:   4%|3         | 1979/50000 [16:47<21:19:27,  1.60s/it]
Outer training loop:   4%|3         | 1980/50000 [16:49<21:10:35,  1.59s/it]
Outer training loop:   4%|3         | 1981/50000 [16:51<21:26:01,  1.61s/it]
Outer training loop:   4%|3         | 1982/50000 [16:52<21:05:44,  1.58s/it]
Outer training loop:   4%|3         | 1983/50000 [16:54<21:16:52,  1.60s/it]
Outer training loop:   4%|3         | 1984/50000 [16:55<21:16:29,  1.60s/it]
Outer training loop:   4%|3         | 1985/50000 [16:57<21:37:25,  1.62s/it]
Outer training loop:   4%|3         | 1986/50000 [16:59<21:30:35,  1.61s/it]
Outer training loop:   4%|3         | 1987/50000 [17:00<21:49:05,  1.64s/it]
Outer training loop:   4%|3         | 1988/50000 [17:02<21:34:12,  1.62s/it]
Outer training loop:   4%|3         | 1989/50000 [17:04<22:29:54,  1.69s/it]
Outer training loop:   4%|3         | 1990/50000 [17:05<22:03:00,  1.65s/it]
Outer training loop:   4%|3         | 1991/50000 [17:07<22:14:13,  1.67s/it]
Outer training loop:   4%|3         | 1992/50000 [17:09<21:54:17,  1.64s/it]
Outer training loop:   4%|3         | 1993/50000 [17:10<22:28:13,  1.69s/it]
Outer training loop:   4%|3         | 1994/50000 [17:12<21:37:44,  1.62s/it]
Outer training loop:   4%|3         | 1995/50000 [17:13<21:39:21,  1.62s/it]
Outer training loop:   4%|3         | 1996/50000 [17:15<21:22:31,  1.60s/it]
Outer training loop:   4%|3         | 1997/50000 [17:17<21:34:08,  1.62s/it]
Outer training loop:   4%|3         | 1998/50000 [17:18<21:31:38,  1.61s/it]
Outer training loop:   4%|3         | 1999/50000 [17:20<21:36:42,  1.62s/it]
Outer training loop:   4%|4         | 2000/50000 [17:21<21:13:25,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.59s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.58s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.57s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:05,  1.53s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:06,  1.56s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:05,  1.57s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:09,  1.63s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:06,  1.63s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:07,  1.65s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<02:05,  1.65s/it]

Evaluating policy:  10%|9         | 8/83 [00:13<02:03,  1.65s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<02:04,  1.68s/it]

Evaluating policy:  12%|#2        | 10/83 [00:16<01:59,  1.63s/it]

Evaluating policy:  13%|#3        | 11/83 [00:18<01:59,  1.65s/it]

Evaluating policy:  14%|#4        | 12/83 [00:19<01:55,  1.63s/it]

Evaluating policy:  16%|#5        | 13/83 [00:21<01:53,  1.62s/it]

Evaluating policy:  17%|#6        | 14/83 [00:23<01:54,  1.65s/it]

Evaluating policy:  18%|#8        | 15/83 [00:24<01:50,  1.62s/it]

Evaluating policy:  19%|#9        | 16/83 [00:26<01:46,  1.59s/it]

Evaluating policy:  20%|##        | 17/83 [00:27<01:46,  1.61s/it]

Evaluating policy:  22%|##1       | 18/83 [00:29<01:42,  1.58s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:41,  1.58s/it]

Evaluating policy:  24%|##4       | 20/83 [00:32<01:40,  1.59s/it]

Evaluating policy:  25%|##5       | 21/83 [00:34<01:40,  1.63s/it]

Evaluating policy:  27%|##6       | 22/83 [00:35<01:37,  1.60s/it]

Evaluating policy:  28%|##7       | 23/83 [00:37<01:36,  1.60s/it]

Evaluating policy:  29%|##8       | 24/83 [00:39<01:37,  1.66s/it]

Evaluating policy:  30%|###       | 25/83 [00:40<01:34,  1.63s/it]

Evaluating policy:  31%|###1      | 26/83 [00:42<01:31,  1.61s/it]

Evaluating policy:  33%|###2      | 27/83 [00:43<01:31,  1.63s/it]

Evaluating policy:  34%|###3      | 28/83 [00:45<01:31,  1.66s/it]

Evaluating policy:  35%|###4      | 29/83 [00:47<01:27,  1.63s/it]

Evaluating policy:  36%|###6      | 30/83 [00:48<01:25,  1.61s/it]

Evaluating policy:  37%|###7      | 31/83 [00:50<01:24,  1.62s/it]

Evaluating policy:  39%|###8      | 32/83 [00:51<01:21,  1.60s/it]

Evaluating policy:  40%|###9      | 33/83 [00:53<01:23,  1.68s/it]

Evaluating policy:  41%|####      | 34/83 [00:55<01:19,  1.62s/it]

Evaluating policy:  42%|####2     | 35/83 [00:56<01:16,  1.60s/it]

Evaluating policy:  43%|####3     | 36/83 [00:58<01:15,  1.60s/it]

Evaluating policy:  45%|####4     | 37/83 [01:00<01:16,  1.65s/it]

Evaluating policy:  46%|####5     | 38/83 [01:01<01:12,  1.61s/it]

Evaluating policy:  47%|####6     | 39/83 [01:03<01:13,  1.67s/it]

Evaluating policy:  48%|####8     | 40/83 [01:05<01:10,  1.63s/it]

Evaluating policy:  49%|####9     | 41/83 [01:06<01:07,  1.60s/it]

Evaluating policy:  51%|#####     | 42/83 [01:08<01:08,  1.68s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:09<01:04,  1.62s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:11<01:02,  1.60s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:13<00:59,  1.57s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:14<00:58,  1.57s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:16<00:55,  1.55s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:17<00:57,  1.65s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:19<00:54,  1.61s/it]

Evaluating policy:  60%|######    | 50/83 [01:21<00:53,  1.61s/it]

Evaluating policy:  61%|######1   | 51/83 [01:22<00:50,  1.59s/it]

Evaluating policy:  63%|######2   | 52/83 [01:24<00:48,  1.58s/it]

Evaluating policy:  64%|######3   | 53/83 [01:25<00:46,  1.56s/it]

Evaluating policy:  65%|######5   | 54/83 [01:27<00:48,  1.67s/it]

Evaluating policy:  66%|######6   | 55/83 [01:29<00:45,  1.63s/it]

Evaluating policy:  67%|######7   | 56/83 [01:30<00:43,  1.60s/it]

Evaluating policy:  69%|######8   | 57/83 [01:32<00:41,  1.61s/it]

Evaluating policy:  70%|######9   | 58/83 [01:33<00:40,  1.62s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:35<00:38,  1.60s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:37<00:39,  1.72s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:39<00:36,  1.67s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:40<00:34,  1.63s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:42<00:32,  1.61s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:43<00:30,  1.60s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:45<00:28,  1.59s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:46<00:26,  1.58s/it]

Evaluating policy:  81%|########  | 67/83 [01:48<00:26,  1.69s/it]

Evaluating policy:  82%|########1 | 68/83 [01:50<00:25,  1.67s/it]

Evaluating policy:  83%|########3 | 69/83 [01:51<00:22,  1.64s/it]

Evaluating policy:  84%|########4 | 70/83 [01:53<00:21,  1.63s/it]

Evaluating policy:  86%|########5 | 71/83 [01:55<00:19,  1.60s/it]

Evaluating policy:  87%|########6 | 72/83 [01:56<00:17,  1.59s/it]

Evaluating policy:  88%|########7 | 73/83 [01:58<00:15,  1.58s/it]

Evaluating policy:  89%|########9 | 74/83 [02:00<00:15,  1.71s/it]

Evaluating policy:  90%|######### | 75/83 [02:01<00:13,  1.66s/it]

Evaluating policy:  92%|#########1| 76/83 [02:03<00:11,  1.63s/it]

Evaluating policy:  93%|#########2| 77/83 [02:04<00:09,  1.62s/it]

Evaluating policy:  94%|#########3| 78/83 [02:06<00:07,  1.59s/it]

Evaluating policy:  95%|#########5| 79/83 [02:08<00:06,  1.60s/it]

Evaluating policy:  96%|#########6| 80/83 [02:09<00:04,  1.58s/it]

Evaluating policy:  98%|#########7| 81/83 [02:11<00:03,  1.55s/it]

Evaluating policy:  99%|#########8| 82/83 [02:12<00:01,  1.56s/it]

Evaluating policy: 100%|##########| 83/83 [02:14<00:00,  1.65s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:00,  1.47s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:00,  1.49s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<01:59,  1.49s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<01:59,  1.51s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<01:57,  1.51s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:00,  1.57s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:59,  1.57s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:57,  1.57s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<02:07,  1.72s/it]

Evaluating policy:  12%|#2        | 10/83 [00:16<02:02,  1.68s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<02:03,  1.72s/it]

Evaluating policy:  14%|#4        | 12/83 [00:19<01:59,  1.69s/it]

Evaluating policy:  16%|#5        | 13/83 [00:21<01:55,  1.65s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:55,  1.67s/it]

Evaluating policy:  18%|#8        | 15/83 [00:24<01:52,  1.66s/it]

Evaluating policy:  19%|#9        | 16/83 [00:26<01:49,  1.64s/it]

Evaluating policy:  20%|##        | 17/83 [00:27<01:51,  1.69s/it]

Evaluating policy:  22%|##1       | 18/83 [00:29<01:47,  1.66s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:44,  1.63s/it]

Evaluating policy:  24%|##4       | 20/83 [00:32<01:40,  1.60s/it]

Evaluating policy:  25%|##5       | 21/83 [00:34<01:42,  1.65s/it]

Evaluating policy:  27%|##6       | 22/83 [00:35<01:39,  1.63s/it]

Evaluating policy:  28%|##7       | 23/83 [00:37<01:38,  1.64s/it]

Evaluating policy:  29%|##8       | 24/83 [00:39<01:39,  1.68s/it]

Evaluating policy:  30%|###       | 25/83 [00:40<01:35,  1.64s/it]

Evaluating policy:  31%|###1      | 26/83 [00:42<01:33,  1.63s/it]

Evaluating policy:  33%|###2      | 27/83 [00:43<01:28,  1.58s/it]

Evaluating policy:  34%|###3      | 28/83 [00:45<01:30,  1.65s/it]

Evaluating policy:  35%|###4      | 29/83 [00:47<01:27,  1.62s/it]

Evaluating policy:  36%|###6      | 30/83 [00:48<01:24,  1.60s/it]

Evaluating policy:  37%|###7      | 31/83 [00:50<01:23,  1.61s/it]

Evaluating policy:  39%|###8      | 32/83 [00:52<01:25,  1.68s/it]

Evaluating policy:  40%|###9      | 33/83 [00:53<01:21,  1.64s/it]

Evaluating policy:  41%|####      | 34/83 [00:55<01:18,  1.60s/it]

Evaluating policy:  42%|####2     | 35/83 [00:56<01:15,  1.58s/it]

Evaluating policy:  43%|####3     | 36/83 [00:58<01:18,  1.66s/it]

Evaluating policy:  45%|####4     | 37/83 [01:00<01:14,  1.62s/it]

Evaluating policy:  46%|####5     | 38/83 [01:01<01:11,  1.59s/it]

Evaluating policy:  47%|####6     | 39/83 [01:03<01:08,  1.56s/it]

Evaluating policy:  48%|####8     | 40/83 [01:04<01:07,  1.58s/it]

Evaluating policy:  49%|####9     | 41/83 [01:06<01:09,  1.66s/it]

Evaluating policy:  51%|#####     | 42/83 [01:08<01:07,  1.65s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:09<01:04,  1.62s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:11<01:02,  1.61s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:13<01:00,  1.60s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:14<01:02,  1.68s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:16<01:00,  1.67s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:18<00:56,  1.62s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:19<00:54,  1.60s/it]

Evaluating policy:  60%|######    | 50/83 [01:21<00:52,  1.60s/it]

Evaluating policy:  61%|######1   | 51/83 [01:22<00:51,  1.61s/it]

Evaluating policy:  63%|######2   | 52/83 [01:24<00:53,  1.72s/it]

Evaluating policy:  64%|######3   | 53/83 [01:26<00:49,  1.66s/it]

Evaluating policy:  65%|######5   | 54/83 [01:27<00:46,  1.61s/it]

Evaluating policy:  66%|######6   | 55/83 [01:29<00:44,  1.60s/it]

Evaluating policy:  67%|######7   | 56/83 [01:31<00:43,  1.63s/it]

Evaluating policy:  69%|######8   | 57/83 [01:32<00:41,  1.60s/it]

Evaluating policy:  70%|######9   | 58/83 [01:34<00:43,  1.73s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:36<00:40,  1.67s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:37<00:37,  1.62s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:39<00:35,  1.61s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:40<00:33,  1.59s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:42<00:31,  1.58s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:44<00:29,  1.57s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:45<00:29,  1.67s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:47<00:27,  1.61s/it]

Evaluating policy:  81%|########  | 67/83 [01:48<00:25,  1.59s/it]

Evaluating policy:  82%|########1 | 68/83 [01:50<00:23,  1.58s/it]

Evaluating policy:  83%|########3 | 69/83 [01:52<00:22,  1.63s/it]

Evaluating policy:  84%|########4 | 70/83 [01:53<00:20,  1.59s/it]

Evaluating policy:  86%|########5 | 71/83 [01:55<00:19,  1.61s/it]

Evaluating policy:  87%|########6 | 72/83 [01:57<00:18,  1.71s/it]

Evaluating policy:  88%|########7 | 73/83 [01:58<00:16,  1.64s/it]

Evaluating policy:  89%|########9 | 74/83 [02:00<00:14,  1.61s/it]

Evaluating policy:  90%|######### | 75/83 [02:01<00:12,  1.59s/it]

Evaluating policy:  92%|#########1| 76/83 [02:03<00:10,  1.57s/it]

Evaluating policy:  93%|#########2| 77/83 [02:05<00:09,  1.59s/it]

Evaluating policy:  94%|#########3| 78/83 [02:06<00:07,  1.57s/it]

Evaluating policy:  95%|#########5| 79/83 [02:08<00:06,  1.58s/it]

Evaluating policy:  96%|#########6| 80/83 [02:10<00:05,  1.71s/it]

Evaluating policy:  98%|#########7| 81/83 [02:11<00:03,  1.65s/it]

Evaluating policy:  99%|#########8| 82/83 [02:13<00:01,  1.62s/it]

Evaluating policy: 100%|##########| 83/83 [02:14<00:00,  1.60s/it]

Outer training loop:   4%|4         | 2001/50000 [21:57<1118:36:06, 83.90s/it]
Outer training loop:   4%|4         | 2002/50000 [21:59<789:20:08, 59.20s/it] 
Outer training loop:   4%|4         | 2003/50000 [22:01<558:53:32, 41.92s/it]
Outer training loop:   4%|4         | 2004/50000 [22:02<397:14:24, 29.80s/it]
Outer training loop:   4%|4         | 2005/50000 [22:04<284:16:20, 21.32s/it]
Outer training loop:   4%|4         | 2006/50000 [22:06<207:30:03, 15.56s/it]
Outer training loop:   4%|4         | 2007/50000 [22:07<151:21:20, 11.35s/it]
Outer training loop:   4%|4         | 2008/50000 [22:09<112:37:07,  8.45s/it]
Outer training loop:   4%|4         | 2009/50000 [22:11<85:06:39,  6.38s/it] 
Outer training loop:   4%|4         | 2010/50000 [22:12<66:08:47,  4.96s/it]
Outer training loop:   4%|4         | 2011/50000 [22:14<52:19:44,  3.93s/it]
Outer training loop:   4%|4         | 2012/50000 [22:15<43:17:53,  3.25s/it]
Outer training loop:   4%|4         | 2013/50000 [22:17<36:27:24,  2.74s/it]
Outer training loop:   4%|4         | 2014/50000 [22:19<32:13:07,  2.42s/it]
Outer training loop:   4%|4         | 2015/50000 [22:20<28:39:02,  2.15s/it]
Outer training loop:   4%|4         | 2016/50000 [22:22<26:11:31,  1.97s/it]
Outer training loop:   4%|4         | 2017/50000 [22:23<24:28:58,  1.84s/it]
Outer training loop:   4%|4         | 2018/50000 [22:25<23:42:08,  1.78s/it]
Outer training loop:   4%|4         | 2019/50000 [22:26<22:43:21,  1.70s/it]
Outer training loop:   4%|4         | 2020/50000 [22:28<21:53:22,  1.64s/it]
Outer training loop:   4%|4         | 2021/50000 [22:29<21:48:37,  1.64s/it]
Outer training loop:   4%|4         | 2022/50000 [22:31<21:29:12,  1.61s/it]
Outer training loop:   4%|4         | 2023/50000 [22:33<21:52:44,  1.64s/it]
Outer training loop:   4%|4         | 2024/50000 [22:34<22:21:16,  1.68s/it]
Outer training loop:   4%|4         | 2025/50000 [22:36<22:26:06,  1.68s/it]
Outer training loop:   4%|4         | 2026/50000 [22:38<22:16:38,  1.67s/it]
Outer training loop:   4%|4         | 2027/50000 [22:39<21:50:33,  1.64s/it]
Outer training loop:   4%|4         | 2028/50000 [22:41<21:46:22,  1.63s/it]
Outer training loop:   4%|4         | 2029/50000 [22:43<21:45:20,  1.63s/it]
Outer training loop:   4%|4         | 2030/50000 [22:44<21:11:00,  1.59s/it]
Outer training loop:   4%|4         | 2031/50000 [22:46<21:40:47,  1.63s/it]
Outer training loop:   4%|4         | 2032/50000 [22:47<21:11:51,  1.59s/it]
Outer training loop:   4%|4         | 2033/50000 [22:49<21:23:37,  1.61s/it]
Outer training loop:   4%|4         | 2034/50000 [22:51<21:19:25,  1.60s/it]
Outer training loop:   4%|4         | 2035/50000 [22:52<21:33:06,  1.62s/it]
Outer training loop:   4%|4         | 2036/50000 [22:54<21:28:22,  1.61s/it]
Outer training loop:   4%|4         | 2037/50000 [22:55<21:44:48,  1.63s/it]
Outer training loop:   4%|4         | 2038/50000 [22:57<21:18:56,  1.60s/it]
Outer training loop:   4%|4         | 2039/50000 [22:59<21:27:03,  1.61s/it]
Outer training loop:   4%|4         | 2040/50000 [23:00<21:05:43,  1.58s/it]
Outer training loop:   4%|4         | 2041/50000 [23:02<20:46:36,  1.56s/it]
Outer training loop:   4%|4         | 2042/50000 [23:03<21:31:26,  1.62s/it]
Outer training loop:   4%|4         | 2043/50000 [23:05<21:30:29,  1.61s/it]
Outer training loop:   4%|4         | 2044/50000 [23:07<21:17:21,  1.60s/it]
Outer training loop:   4%|4         | 2045/50000 [23:08<21:09:48,  1.59s/it]
Outer training loop:   4%|4         | 2046/50000 [23:10<21:27:32,  1.61s/it]
Outer training loop:   4%|4         | 2047/50000 [23:11<20:58:43,  1.57s/it]
Outer training loop:   4%|4         | 2048/50000 [23:13<21:20:49,  1.60s/it]
Outer training loop:   4%|4         | 2049/50000 [23:14<20:45:04,  1.56s/it]
Outer training loop:   4%|4         | 2050/50000 [23:16<21:02:13,  1.58s/it]
Outer training loop:   4%|4         | 2051/50000 [23:18<20:45:54,  1.56s/it]
Outer training loop:   4%|4         | 2052/50000 [23:19<20:54:05,  1.57s/it]
Outer training loop:   4%|4         | 2053/50000 [23:21<21:10:28,  1.59s/it]
Outer training loop:   4%|4         | 2054/50000 [23:23<21:38:54,  1.63s/it]
Outer training loop:   4%|4         | 2055/50000 [23:24<21:03:16,  1.58s/it]
Outer training loop:   4%|4         | 2056/50000 [23:26<21:14:54,  1.60s/it]
Outer training loop:   4%|4         | 2057/50000 [23:27<20:46:17,  1.56s/it]
Outer training loop:   4%|4         | 2058/50000 [23:29<21:00:10,  1.58s/it]
Outer training loop:   4%|4         | 2059/50000 [23:30<21:28:38,  1.61s/it]
Outer training loop:   4%|4         | 2060/50000 [23:32<21:36:26,  1.62s/it]
Outer training loop:   4%|4         | 2061/50000 [23:34<21:24:29,  1.61s/it]
Outer training loop:   4%|4         | 2062/50000 [23:35<21:22:51,  1.61s/it]
Outer training loop:   4%|4         | 2063/50000 [23:37<21:08:35,  1.59s/it]
Outer training loop:   4%|4         | 2064/50000 [23:38<21:00:54,  1.58s/it]
Outer training loop:   4%|4         | 2065/50000 [23:40<20:50:57,  1.57s/it]
Outer training loop:   4%|4         | 2066/50000 [23:41<21:00:55,  1.58s/it]
Outer training loop:   4%|4         | 2067/50000 [23:43<20:44:32,  1.56s/it]
Outer training loop:   4%|4         | 2068/50000 [23:45<21:07:03,  1.59s/it]
Outer training loop:   4%|4         | 2069/50000 [23:46<21:35:05,  1.62s/it]
Outer training loop:   4%|4         | 2070/50000 [23:48<21:16:21,  1.60s/it]
Outer training loop:   4%|4         | 2071/50000 [23:50<21:37:20,  1.62s/it]
Outer training loop:   4%|4         | 2072/50000 [23:51<21:25:52,  1.61s/it]
Outer training loop:   4%|4         | 2073/50000 [23:53<21:13:59,  1.59s/it]
Outer training loop:   4%|4         | 2074/50000 [23:54<21:35:57,  1.62s/it]
Outer training loop:   4%|4         | 2075/50000 [23:56<21:41:57,  1.63s/it]
Outer training loop:   4%|4         | 2076/50000 [23:58<21:22:28,  1.61s/it]
Outer training loop:   4%|4         | 2077/50000 [23:59<21:28:57,  1.61s/it]
Outer training loop:   4%|4         | 2078/50000 [24:01<21:42:08,  1.63s/it]
Outer training loop:   4%|4         | 2079/50000 [24:02<21:04:25,  1.58s/it]
Outer training loop:   4%|4         | 2080/50000 [24:04<21:25:41,  1.61s/it]
Outer training loop:   4%|4         | 2081/50000 [24:06<21:16:04,  1.60s/it]
Outer training loop:   4%|4         | 2082/50000 [24:07<21:26:30,  1.61s/it]
Outer training loop:   4%|4         | 2083/50000 [24:09<21:11:59,  1.59s/it]
Outer training loop:   4%|4         | 2084/50000 [24:10<21:14:37,  1.60s/it]
Outer training loop:   4%|4         | 2085/50000 [24:12<21:15:53,  1.60s/it]
Outer training loop:   4%|4         | 2086/50000 [24:14<21:24:47,  1.61s/it]
Outer training loop:   4%|4         | 2087/50000 [24:15<21:09:46,  1.59s/it]
Outer training loop:   4%|4         | 2088/50000 [24:17<21:10:06,  1.59s/it]
Outer training loop:   4%|4         | 2089/50000 [24:18<20:56:05,  1.57s/it]
Outer training loop:   4%|4         | 2090/50000 [24:20<21:11:39,  1.59s/it]
Outer training loop:   4%|4         | 2091/50000 [24:21<20:52:37,  1.57s/it]
Outer training loop:   4%|4         | 2092/50000 [24:23<21:10:08,  1.59s/it]
Outer training loop:   4%|4         | 2093/50000 [24:25<21:12:35,  1.59s/it]
Outer training loop:   4%|4         | 2094/50000 [24:26<21:31:01,  1.62s/it]
Outer training loop:   4%|4         | 2095/50000 [24:28<21:04:44,  1.58s/it]
Outer training loop:   4%|4         | 2096/50000 [24:29<21:12:24,  1.59s/it]
Outer training loop:   4%|4         | 2097/50000 [24:31<20:49:27,  1.56s/it]
Outer training loop:   4%|4         | 2098/50000 [24:33<21:18:31,  1.60s/it]
Outer training loop:   4%|4         | 2099/50000 [24:34<21:31:09,  1.62s/it]
Outer training loop:   4%|4         | 2100/50000 [24:36<21:02:58,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.60s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.61s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.55s/it]

Outer training loop:   4%|4         | 2101/50000 [24:42<39:51:13,  3.00s/it]
Outer training loop:   4%|4         | 2102/50000 [24:44<34:29:07,  2.59s/it]
Outer training loop:   4%|4         | 2103/50000 [24:45<30:15:34,  2.27s/it]
Outer training loop:   4%|4         | 2104/50000 [24:47<27:33:29,  2.07s/it]
Outer training loop:   4%|4         | 2105/50000 [24:48<25:27:25,  1.91s/it]
Outer training loop:   4%|4         | 2106/50000 [24:50<23:59:34,  1.80s/it]
Outer training loop:   4%|4         | 2107/50000 [24:52<23:15:22,  1.75s/it]
Outer training loop:   4%|4         | 2108/50000 [24:53<22:24:39,  1.68s/it]
Outer training loop:   4%|4         | 2109/50000 [24:55<22:13:55,  1.67s/it]
Outer training loop:   4%|4         | 2110/50000 [24:56<21:20:43,  1.60s/it]
Outer training loop:   4%|4         | 2111/50000 [24:58<21:25:55,  1.61s/it]
Outer training loop:   4%|4         | 2112/50000 [24:59<21:06:10,  1.59s/it]
Outer training loop:   4%|4         | 2113/50000 [25:01<21:17:59,  1.60s/it]
Outer training loop:   4%|4         | 2114/50000 [25:03<20:54:54,  1.57s/it]
Outer training loop:   4%|4         | 2115/50000 [25:04<20:54:29,  1.57s/it]
Outer training loop:   4%|4         | 2116/50000 [25:06<21:00:20,  1.58s/it]
Outer training loop:   4%|4         | 2117/50000 [25:07<21:21:23,  1.61s/it]
Outer training loop:   4%|4         | 2118/50000 [25:09<21:17:17,  1.60s/it]
Outer training loop:   4%|4         | 2119/50000 [25:11<21:14:24,  1.60s/it]
Outer training loop:   4%|4         | 2120/50000 [25:12<21:00:51,  1.58s/it]
Outer training loop:   4%|4         | 2121/50000 [25:14<21:50:20,  1.64s/it]
Outer training loop:   4%|4         | 2122/50000 [25:15<21:06:53,  1.59s/it]
Outer training loop:   4%|4         | 2123/50000 [25:17<21:18:32,  1.60s/it]
Outer training loop:   4%|4         | 2124/50000 [25:19<21:00:39,  1.58s/it]
Outer training loop:   4%|4         | 2125/50000 [25:20<21:14:24,  1.60s/it]
Outer training loop:   4%|4         | 2126/50000 [25:22<20:58:22,  1.58s/it]
Outer training loop:   4%|4         | 2127/50000 [25:23<21:16:54,  1.60s/it]
Outer training loop:   4%|4         | 2128/50000 [25:25<21:00:38,  1.58s/it]
Outer training loop:   4%|4         | 2129/50000 [25:26<21:10:24,  1.59s/it]
Outer training loop:   4%|4         | 2130/50000 [25:28<20:54:07,  1.57s/it]
Outer training loop:   4%|4         | 2131/50000 [25:30<21:21:00,  1.61s/it]
Outer training loop:   4%|4         | 2132/50000 [25:31<21:06:16,  1.59s/it]
Outer training loop:   4%|4         | 2133/50000 [25:33<21:24:23,  1.61s/it]
Outer training loop:   4%|4         | 2134/50000 [25:35<21:24:13,  1.61s/it]
Outer training loop:   4%|4         | 2135/50000 [25:36<21:18:55,  1.60s/it]
Outer training loop:   4%|4         | 2136/50000 [25:38<20:52:54,  1.57s/it]
Outer training loop:   4%|4         | 2137/50000 [25:39<21:09:59,  1.59s/it]
Outer training loop:   4%|4         | 2138/50000 [25:41<20:47:30,  1.56s/it]
Outer training loop:   4%|4         | 2139/50000 [25:42<21:02:57,  1.58s/it]
Outer training loop:   4%|4         | 2140/50000 [25:44<20:45:13,  1.56s/it]
Outer training loop:   4%|4         | 2141/50000 [25:46<21:03:32,  1.58s/it]
Outer training loop:   4%|4         | 2142/50000 [25:47<21:05:08,  1.59s/it]
Outer training loop:   4%|4         | 2143/50000 [25:49<21:30:47,  1.62s/it]
Outer training loop:   4%|4         | 2144/50000 [25:50<21:14:52,  1.60s/it]
Outer training loop:   4%|4         | 2145/50000 [25:52<21:05:37,  1.59s/it]
Outer training loop:   4%|4         | 2146/50000 [25:53<20:52:08,  1.57s/it]
Outer training loop:   4%|4         | 2147/50000 [25:55<20:54:15,  1.57s/it]
Outer training loop:   4%|4         | 2148/50000 [25:56<20:30:10,  1.54s/it]
Outer training loop:   4%|4         | 2149/50000 [25:58<20:33:28,  1.55s/it]
Outer training loop:   4%|4         | 2150/50000 [26:00<20:26:16,  1.54s/it]
Outer training loop:   4%|4         | 2151/50000 [26:01<20:48:58,  1.57s/it]
Outer training loop:   4%|4         | 2152/50000 [26:03<20:48:55,  1.57s/it]
Outer training loop:   4%|4         | 2153/50000 [26:04<21:09:49,  1.59s/it]
Outer training loop:   4%|4         | 2154/50000 [26:06<20:51:39,  1.57s/it]
Outer training loop:   4%|4         | 2155/50000 [26:08<21:02:13,  1.58s/it]
Outer training loop:   4%|4         | 2156/50000 [26:09<20:29:59,  1.54s/it]
Outer training loop:   4%|4         | 2157/50000 [26:11<20:40:21,  1.56s/it]
Outer training loop:   4%|4         | 2158/50000 [26:12<21:00:42,  1.58s/it]
Outer training loop:   4%|4         | 2159/50000 [26:14<21:02:00,  1.58s/it]
Outer training loop:   4%|4         | 2160/50000 [26:15<20:33:03,  1.55s/it]
Outer training loop:   4%|4         | 2161/50000 [26:17<20:58:37,  1.58s/it]
Outer training loop:   4%|4         | 2162/50000 [26:18<20:59:10,  1.58s/it]
Outer training loop:   4%|4         | 2163/50000 [26:20<20:49:45,  1.57s/it]
Outer training loop:   4%|4         | 2164/50000 [26:22<20:55:51,  1.58s/it]
Outer training loop:   4%|4         | 2165/50000 [26:23<20:39:55,  1.56s/it]
Outer training loop:   4%|4         | 2166/50000 [26:25<21:01:54,  1.58s/it]
Outer training loop:   4%|4         | 2167/50000 [26:26<20:30:08,  1.54s/it]
Outer training loop:   4%|4         | 2168/50000 [26:28<20:49:36,  1.57s/it]
Outer training loop:   4%|4         | 2169/50000 [26:29<20:42:44,  1.56s/it]
Outer training loop:   4%|4         | 2170/50000 [26:31<20:51:29,  1.57s/it]
Outer training loop:   4%|4         | 2171/50000 [26:33<21:40:41,  1.63s/it]
Outer training loop:   4%|4         | 2172/50000 [26:34<21:15:42,  1.60s/it]
Outer training loop:   4%|4         | 2173/50000 [26:36<21:02:00,  1.58s/it]
Outer training loop:   4%|4         | 2174/50000 [26:37<20:50:04,  1.57s/it]
Outer training loop:   4%|4         | 2175/50000 [26:39<20:27:00,  1.54s/it]
Outer training loop:   4%|4         | 2176/50000 [26:40<20:42:18,  1.56s/it]
Outer training loop:   4%|4         | 2177/50000 [26:42<20:45:14,  1.56s/it]
Outer training loop:   4%|4         | 2178/50000 [26:44<20:42:31,  1.56s/it]
Outer training loop:   4%|4         | 2179/50000 [26:45<21:14:24,  1.60s/it]
Outer training loop:   4%|4         | 2180/50000 [26:47<21:14:49,  1.60s/it]
Outer training loop:   4%|4         | 2181/50000 [26:48<21:17:44,  1.60s/it]
Outer training loop:   4%|4         | 2182/50000 [26:50<21:07:33,  1.59s/it]
Outer training loop:   4%|4         | 2183/50000 [26:52<20:52:41,  1.57s/it]
Outer training loop:   4%|4         | 2184/50000 [26:53<21:07:05,  1.59s/it]
Outer training loop:   4%|4         | 2185/50000 [26:55<21:20:47,  1.61s/it]
Outer training loop:   4%|4         | 2186/50000 [26:56<20:56:20,  1.58s/it]
Outer training loop:   4%|4         | 2187/50000 [26:58<20:54:57,  1.57s/it]
Outer training loop:   4%|4         | 2188/50000 [26:59<20:43:52,  1.56s/it]
Outer training loop:   4%|4         | 2189/50000 [27:01<21:20:59,  1.61s/it]
Outer training loop:   4%|4         | 2190/50000 [27:03<21:21:10,  1.61s/it]
Outer training loop:   4%|4         | 2191/50000 [27:04<21:39:11,  1.63s/it]
Outer training loop:   4%|4         | 2192/50000 [27:06<21:09:56,  1.59s/it]
Outer training loop:   4%|4         | 2193/50000 [27:08<21:07:42,  1.59s/it]
Outer training loop:   4%|4         | 2194/50000 [27:09<20:59:44,  1.58s/it]
Outer training loop:   4%|4         | 2195/50000 [27:11<21:25:03,  1.61s/it]
Outer training loop:   4%|4         | 2196/50000 [27:12<20:57:03,  1.58s/it]
Outer training loop:   4%|4         | 2197/50000 [27:14<21:09:15,  1.59s/it]
Outer training loop:   4%|4         | 2198/50000 [27:15<20:53:38,  1.57s/it]
Outer training loop:   4%|4         | 2199/50000 [27:17<21:06:05,  1.59s/it]
Outer training loop:   4%|4         | 2200/50000 [27:19<20:59:39,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.53s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.55s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.53s/it]

Outer training loop:   4%|4         | 2201/50000 [27:25<40:02:13,  3.02s/it]
Outer training loop:   4%|4         | 2202/50000 [27:27<34:22:56,  2.59s/it]
Outer training loop:   4%|4         | 2203/50000 [27:28<30:05:32,  2.27s/it]
Outer training loop:   4%|4         | 2204/50000 [27:30<27:37:35,  2.08s/it]
Outer training loop:   4%|4         | 2205/50000 [27:31<25:12:41,  1.90s/it]
Outer training loop:   4%|4         | 2206/50000 [27:33<24:34:45,  1.85s/it]
Outer training loop:   4%|4         | 2207/50000 [27:35<23:24:44,  1.76s/it]
Outer training loop:   4%|4         | 2208/50000 [27:36<22:34:42,  1.70s/it]
Outer training loop:   4%|4         | 2209/50000 [27:38<21:52:43,  1.65s/it]
Outer training loop:   4%|4         | 2210/50000 [27:39<22:06:09,  1.66s/it]
Outer training loop:   4%|4         | 2211/50000 [27:41<21:25:32,  1.61s/it]
Outer training loop:   4%|4         | 2212/50000 [27:42<21:10:29,  1.60s/it]
Outer training loop:   4%|4         | 2213/50000 [27:44<20:43:47,  1.56s/it]
Outer training loop:   4%|4         | 2214/50000 [27:45<20:58:47,  1.58s/it]
Outer training loop:   4%|4         | 2215/50000 [27:47<20:40:11,  1.56s/it]
Outer training loop:   4%|4         | 2216/50000 [27:49<21:01:16,  1.58s/it]
Outer training loop:   4%|4         | 2217/50000 [27:50<20:40:36,  1.56s/it]
Outer training loop:   4%|4         | 2218/50000 [27:52<21:25:44,  1.61s/it]
Outer training loop:   4%|4         | 2219/50000 [27:53<21:18:50,  1.61s/it]
Outer training loop:   4%|4         | 2220/50000 [27:55<21:34:24,  1.63s/it]
Outer training loop:   4%|4         | 2221/50000 [27:57<21:25:56,  1.61s/it]
Outer training loop:   4%|4         | 2222/50000 [27:58<21:07:37,  1.59s/it]
Outer training loop:   4%|4         | 2223/50000 [28:00<21:11:27,  1.60s/it]
Outer training loop:   4%|4         | 2224/50000 [28:02<21:27:58,  1.62s/it]
Outer training loop:   4%|4         | 2225/50000 [28:03<21:08:41,  1.59s/it]
Outer training loop:   4%|4         | 2226/50000 [28:05<21:17:24,  1.60s/it]
Outer training loop:   4%|4         | 2227/50000 [28:06<21:04:58,  1.59s/it]
Outer training loop:   4%|4         | 2228/50000 [28:08<21:20:47,  1.61s/it]
Outer training loop:   4%|4         | 2229/50000 [28:09<21:00:31,  1.58s/it]
Outer training loop:   4%|4         | 2230/50000 [28:11<21:02:15,  1.59s/it]
Outer training loop:   4%|4         | 2231/50000 [28:13<20:41:20,  1.56s/it]
Outer training loop:   4%|4         | 2232/50000 [28:14<20:53:42,  1.57s/it]
Outer training loop:   4%|4         | 2233/50000 [28:16<21:13:24,  1.60s/it]
Outer training loop:   4%|4         | 2234/50000 [28:17<21:31:16,  1.62s/it]
Outer training loop:   4%|4         | 2235/50000 [28:19<21:33:08,  1.62s/it]
Outer training loop:   4%|4         | 2236/50000 [28:21<21:22:39,  1.61s/it]
Outer training loop:   4%|4         | 2237/50000 [28:22<21:07:12,  1.59s/it]
Outer training loop:   4%|4         | 2238/50000 [28:24<21:16:40,  1.60s/it]
Outer training loop:   4%|4         | 2239/50000 [28:25<21:01:41,  1.59s/it]
Outer training loop:   4%|4         | 2240/50000 [28:27<20:58:49,  1.58s/it]
Outer training loop:   4%|4         | 2241/50000 [28:29<21:08:31,  1.59s/it]
Outer training loop:   4%|4         | 2242/50000 [28:30<21:28:27,  1.62s/it]
Outer training loop:   4%|4         | 2243/50000 [28:32<21:17:47,  1.61s/it]
Outer training loop:   4%|4         | 2244/50000 [28:33<21:30:25,  1.62s/it]
Outer training loop:   4%|4         | 2245/50000 [28:35<21:13:12,  1.60s/it]
Outer training loop:   4%|4         | 2246/50000 [28:37<21:23:08,  1.61s/it]
Outer training loop:   4%|4         | 2247/50000 [28:38<20:50:42,  1.57s/it]
Outer training loop:   4%|4         | 2248/50000 [28:40<20:59:32,  1.58s/it]
Outer training loop:   4%|4         | 2249/50000 [28:41<20:46:24,  1.57s/it]
Outer training loop:   4%|4         | 2250/50000 [28:43<20:58:13,  1.58s/it]
Outer training loop:   5%|4         | 2251/50000 [28:44<20:12:26,  1.52s/it]
Outer training loop:   5%|4         | 2252/50000 [28:46<20:44:09,  1.56s/it]
Outer training loop:   5%|4         | 2253/50000 [28:47<20:30:24,  1.55s/it]
Outer training loop:   5%|4         | 2254/50000 [28:49<20:44:37,  1.56s/it]
Outer training loop:   5%|4         | 2255/50000 [28:51<20:41:52,  1.56s/it]
Outer training loop:   5%|4         | 2256/50000 [28:52<20:51:22,  1.57s/it]
Outer training loop:   5%|4         | 2257/50000 [28:54<20:30:56,  1.55s/it]
Outer training loop:   5%|4         | 2258/50000 [28:55<20:44:27,  1.56s/it]
Outer training loop:   5%|4         | 2259/50000 [28:57<20:40:57,  1.56s/it]
Outer training loop:   5%|4         | 2260/50000 [28:58<20:54:08,  1.58s/it]
Outer training loop:   5%|4         | 2261/50000 [29:00<20:28:10,  1.54s/it]
Outer training loop:   5%|4         | 2262/50000 [29:02<20:38:07,  1.56s/it]
Outer training loop:   5%|4         | 2263/50000 [29:03<20:35:14,  1.55s/it]
Outer training loop:   5%|4         | 2264/50000 [29:05<21:04:16,  1.59s/it]
Outer training loop:   5%|4         | 2265/50000 [29:06<20:25:39,  1.54s/it]
Outer training loop:   5%|4         | 2266/50000 [29:08<20:51:15,  1.57s/it]
Outer training loop:   5%|4         | 2267/50000 [29:09<20:52:50,  1.57s/it]
Outer training loop:   5%|4         | 2268/50000 [29:11<21:14:32,  1.60s/it]
Outer training loop:   5%|4         | 2269/50000 [29:13<21:17:55,  1.61s/it]
Outer training loop:   5%|4         | 2270/50000 [29:14<21:08:48,  1.59s/it]
Outer training loop:   5%|4         | 2271/50000 [29:16<20:40:43,  1.56s/it]
Outer training loop:   5%|4         | 2272/50000 [29:17<20:41:01,  1.56s/it]
Outer training loop:   5%|4         | 2273/50000 [29:19<20:37:04,  1.56s/it]
Outer training loop:   5%|4         | 2274/50000 [29:20<21:04:32,  1.59s/it]
Outer training loop:   5%|4         | 2275/50000 [29:22<21:01:00,  1.59s/it]
Outer training loop:   5%|4         | 2276/50000 [29:24<21:24:22,  1.61s/it]
Outer training loop:   5%|4         | 2277/50000 [29:25<21:18:56,  1.61s/it]
Outer training loop:   5%|4         | 2278/50000 [29:27<21:20:51,  1.61s/it]
Outer training loop:   5%|4         | 2279/50000 [29:28<20:42:46,  1.56s/it]
Outer training loop:   5%|4         | 2280/50000 [29:30<20:44:59,  1.57s/it]
Outer training loop:   5%|4         | 2281/50000 [29:32<20:45:01,  1.57s/it]
Outer training loop:   5%|4         | 2282/50000 [29:33<21:18:35,  1.61s/it]
Outer training loop:   5%|4         | 2283/50000 [29:35<20:51:28,  1.57s/it]
Outer training loop:   5%|4         | 2284/50000 [29:36<20:41:11,  1.56s/it]
Outer training loop:   5%|4         | 2285/50000 [29:38<21:02:41,  1.59s/it]
Outer training loop:   5%|4         | 2286/50000 [29:40<20:58:59,  1.58s/it]
Outer training loop:   5%|4         | 2287/50000 [29:41<21:05:37,  1.59s/it]
Outer training loop:   5%|4         | 2288/50000 [29:43<20:57:10,  1.58s/it]
Outer training loop:   5%|4         | 2289/50000 [29:44<21:13:34,  1.60s/it]
Outer training loop:   5%|4         | 2290/50000 [29:46<21:17:45,  1.61s/it]
Outer training loop:   5%|4         | 2291/50000 [29:48<21:12:53,  1.60s/it]
Outer training loop:   5%|4         | 2292/50000 [29:49<20:46:41,  1.57s/it]
Outer training loop:   5%|4         | 2293/50000 [29:51<20:46:47,  1.57s/it]
Outer training loop:   5%|4         | 2294/50000 [29:52<20:36:09,  1.55s/it]
Outer training loop:   5%|4         | 2295/50000 [29:54<21:00:31,  1.59s/it]
Outer training loop:   5%|4         | 2296/50000 [29:55<20:33:28,  1.55s/it]
Outer training loop:   5%|4         | 2297/50000 [29:57<20:42:59,  1.56s/it]
Outer training loop:   5%|4         | 2298/50000 [29:58<20:24:38,  1.54s/it]
Outer training loop:   5%|4         | 2299/50000 [30:00<20:42:10,  1.56s/it]
Outer training loop:   5%|4         | 2300/50000 [30:01<20:29:22,  1.55s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.48s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.55s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.52s/it]

Outer training loop:   5%|4         | 2301/50000 [30:08<39:22:11,  2.97s/it]
Outer training loop:   5%|4         | 2302/50000 [30:09<34:18:23,  2.59s/it]
Outer training loop:   5%|4         | 2303/50000 [30:11<29:44:39,  2.24s/it]
Outer training loop:   5%|4         | 2304/50000 [30:12<27:14:40,  2.06s/it]
Outer training loop:   5%|4         | 2305/50000 [30:14<25:15:11,  1.91s/it]
Outer training loop:   5%|4         | 2306/50000 [30:16<24:03:55,  1.82s/it]
Outer training loop:   5%|4         | 2307/50000 [30:17<22:37:05,  1.71s/it]
Outer training loop:   5%|4         | 2308/50000 [30:19<21:56:41,  1.66s/it]
Outer training loop:   5%|4         | 2309/50000 [30:20<21:54:23,  1.65s/it]
Outer training loop:   5%|4         | 2310/50000 [30:22<22:02:17,  1.66s/it]
Outer training loop:   5%|4         | 2311/50000 [30:23<21:24:13,  1.62s/it]
Outer training loop:   5%|4         | 2312/50000 [30:25<20:52:23,  1.58s/it]
Outer training loop:   5%|4         | 2313/50000 [30:27<20:57:42,  1.58s/it]
Outer training loop:   5%|4         | 2314/50000 [30:28<21:08:34,  1.60s/it]
Outer training loop:   5%|4         | 2315/50000 [30:30<21:34:37,  1.63s/it]
Outer training loop:   5%|4         | 2316/50000 [30:31<21:08:07,  1.60s/it]
Outer training loop:   5%|4         | 2317/50000 [30:33<21:23:29,  1.62s/it]
Outer training loop:   5%|4         | 2318/50000 [30:35<20:55:35,  1.58s/it]
Outer training loop:   5%|4         | 2319/50000 [30:36<21:02:35,  1.59s/it]
Outer training loop:   5%|4         | 2320/50000 [30:38<20:45:02,  1.57s/it]
Outer training loop:   5%|4         | 2321/50000 [30:39<20:52:02,  1.58s/it]
Outer training loop:   5%|4         | 2322/50000 [30:41<20:31:31,  1.55s/it]
Outer training loop:   5%|4         | 2323/50000 [30:42<20:45:46,  1.57s/it]
Outer training loop:   5%|4         | 2324/50000 [30:44<20:59:08,  1.58s/it]
Outer training loop:   5%|4         | 2325/50000 [30:46<20:53:36,  1.58s/it]
Outer training loop:   5%|4         | 2326/50000 [30:47<20:39:46,  1.56s/it]
Outer training loop:   5%|4         | 2327/50000 [30:49<20:53:59,  1.58s/it]
Outer training loop:   5%|4         | 2328/50000 [30:50<20:59:32,  1.59s/it]
Outer training loop:   5%|4         | 2329/50000 [30:52<21:16:59,  1.61s/it]
Outer training loop:   5%|4         | 2330/50000 [30:54<21:25:08,  1.62s/it]
Outer training loop:   5%|4         | 2331/50000 [30:55<21:00:45,  1.59s/it]
Outer training loop:   5%|4         | 2332/50000 [30:57<21:10:09,  1.60s/it]
Outer training loop:   5%|4         | 2333/50000 [30:58<20:39:22,  1.56s/it]
Outer training loop:   5%|4         | 2334/50000 [31:00<20:54:26,  1.58s/it]
Outer training loop:   5%|4         | 2335/50000 [31:01<20:30:20,  1.55s/it]
Outer training loop:   5%|4         | 2336/50000 [31:03<20:40:12,  1.56s/it]
Outer training loop:   5%|4         | 2337/50000 [31:04<20:28:53,  1.55s/it]
Outer training loop:   5%|4         | 2338/50000 [31:06<20:32:29,  1.55s/it]
Outer training loop:   5%|4         | 2339/50000 [31:07<20:19:55,  1.54s/it]
Outer training loop:   5%|4         | 2340/50000 [31:09<20:26:18,  1.54s/it]
Outer training loop:   5%|4         | 2341/50000 [31:11<20:27:47,  1.55s/it]
Outer training loop:   5%|4         | 2342/50000 [31:12<20:41:42,  1.56s/it]
Outer training loop:   5%|4         | 2343/50000 [31:14<20:35:36,  1.56s/it]
Outer training loop:   5%|4         | 2344/50000 [31:15<20:58:27,  1.58s/it]
Outer training loop:   5%|4         | 2345/50000 [31:17<20:29:44,  1.55s/it]
Outer training loop:   5%|4         | 2346/50000 [31:19<20:51:31,  1.58s/it]
Outer training loop:   5%|4         | 2347/50000 [31:20<20:56:13,  1.58s/it]
Outer training loop:   5%|4         | 2348/50000 [31:22<20:59:36,  1.59s/it]
Outer training loop:   5%|4         | 2349/50000 [31:23<20:44:48,  1.57s/it]
Outer training loop:   5%|4         | 2350/50000 [31:25<21:13:35,  1.60s/it]
Outer training loop:   5%|4         | 2351/50000 [31:26<20:56:02,  1.58s/it]
Outer training loop:   5%|4         | 2352/50000 [31:28<20:34:53,  1.56s/it]
Outer training loop:   5%|4         | 2353/50000 [31:30<20:50:17,  1.57s/it]
Outer training loop:   5%|4         | 2354/50000 [31:31<20:25:52,  1.54s/it]
Outer training loop:   5%|4         | 2355/50000 [31:33<20:51:12,  1.58s/it]
Outer training loop:   5%|4         | 2356/50000 [31:34<20:50:58,  1.58s/it]
Outer training loop:   5%|4         | 2357/50000 [31:36<20:47:51,  1.57s/it]
Outer training loop:   5%|4         | 2358/50000 [31:37<20:49:30,  1.57s/it]
Outer training loop:   5%|4         | 2359/50000 [31:39<20:51:44,  1.58s/it]
Outer training loop:   5%|4         | 2360/50000 [31:41<20:47:35,  1.57s/it]
Outer training loop:   5%|4         | 2361/50000 [31:42<20:53:07,  1.58s/it]
Outer training loop:   5%|4         | 2362/50000 [31:44<20:47:36,  1.57s/it]
Outer training loop:   5%|4         | 2363/50000 [31:45<20:50:05,  1.57s/it]
Outer training loop:   5%|4         | 2364/50000 [31:47<20:27:06,  1.55s/it]
Outer training loop:   5%|4         | 2365/50000 [31:49<21:33:33,  1.63s/it]
Outer training loop:   5%|4         | 2366/50000 [31:50<21:20:34,  1.61s/it]
Outer training loop:   5%|4         | 2367/50000 [31:52<21:20:39,  1.61s/it]
Outer training loop:   5%|4         | 2368/50000 [31:53<20:48:46,  1.57s/it]
Outer training loop:   5%|4         | 2369/50000 [31:55<20:36:24,  1.56s/it]
Outer training loop:   5%|4         | 2370/50000 [31:56<20:21:59,  1.54s/it]
Outer training loop:   5%|4         | 2371/50000 [31:58<20:57:19,  1.58s/it]
Outer training loop:   5%|4         | 2372/50000 [32:00<21:02:15,  1.59s/it]
Outer training loop:   5%|4         | 2373/50000 [32:01<21:09:37,  1.60s/it]
Outer training loop:   5%|4         | 2374/50000 [32:03<20:52:09,  1.58s/it]
Outer training loop:   5%|4         | 2375/50000 [32:04<21:26:40,  1.62s/it]
Outer training loop:   5%|4         | 2376/50000 [32:06<21:40:54,  1.64s/it]
Outer training loop:   5%|4         | 2377/50000 [32:08<21:05:07,  1.59s/it]
Outer training loop:   5%|4         | 2378/50000 [32:09<21:35:25,  1.63s/it]
Outer training loop:   5%|4         | 2379/50000 [32:11<21:07:50,  1.60s/it]
Outer training loop:   5%|4         | 2380/50000 [32:12<20:39:02,  1.56s/it]
Outer training loop:   5%|4         | 2381/50000 [32:14<20:57:19,  1.58s/it]
Outer training loop:   5%|4         | 2382/50000 [32:15<20:38:42,  1.56s/it]
Outer training loop:   5%|4         | 2383/50000 [32:17<20:57:48,  1.58s/it]
Outer training loop:   5%|4         | 2384/50000 [32:19<20:56:50,  1.58s/it]
Outer training loop:   5%|4         | 2385/50000 [32:20<21:05:24,  1.59s/it]
Outer training loop:   5%|4         | 2386/50000 [32:22<20:35:08,  1.56s/it]
Outer training loop:   5%|4         | 2387/50000 [32:23<20:18:12,  1.54s/it]
Outer training loop:   5%|4         | 2388/50000 [32:25<21:12:08,  1.60s/it]
Outer training loop:   5%|4         | 2389/50000 [32:27<21:10:39,  1.60s/it]
Outer training loop:   5%|4         | 2390/50000 [32:28<20:40:46,  1.56s/it]
Outer training loop:   5%|4         | 2391/50000 [32:30<20:43:26,  1.57s/it]
Outer training loop:   5%|4         | 2392/50000 [32:31<20:34:11,  1.56s/it]
Outer training loop:   5%|4         | 2393/50000 [32:33<20:53:30,  1.58s/it]
Outer training loop:   5%|4         | 2394/50000 [32:34<20:35:18,  1.56s/it]
Outer training loop:   5%|4         | 2395/50000 [32:36<20:54:04,  1.58s/it]
Outer training loop:   5%|4         | 2396/50000 [32:37<20:31:11,  1.55s/it]
Outer training loop:   5%|4         | 2397/50000 [32:39<20:30:57,  1.55s/it]
Outer training loop:   5%|4         | 2398/50000 [32:41<20:35:13,  1.56s/it]
Outer training loop:   5%|4         | 2399/50000 [32:42<20:45:07,  1.57s/it]
Outer training loop:   5%|4         | 2400/50000 [32:44<20:44:01,  1.57s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.49s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.53s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.49s/it]

Outer training loop:   5%|4         | 2401/50000 [32:50<38:47:22,  2.93s/it]
Outer training loop:   5%|4         | 2402/50000 [32:51<33:06:20,  2.50s/it]
Outer training loop:   5%|4         | 2403/50000 [32:53<29:33:01,  2.24s/it]
Outer training loop:   5%|4         | 2404/50000 [32:55<26:57:46,  2.04s/it]
Outer training loop:   5%|4         | 2405/50000 [32:56<24:58:16,  1.89s/it]
Outer training loop:   5%|4         | 2406/50000 [32:58<23:41:06,  1.79s/it]
Outer training loop:   5%|4         | 2407/50000 [32:59<23:15:56,  1.76s/it]
Outer training loop:   5%|4         | 2408/50000 [33:01<22:39:03,  1.71s/it]
Outer training loop:   5%|4         | 2409/50000 [33:02<21:55:22,  1.66s/it]
Outer training loop:   5%|4         | 2410/50000 [33:04<21:12:51,  1.60s/it]
Outer training loop:   5%|4         | 2411/50000 [33:06<21:38:49,  1.64s/it]
Outer training loop:   5%|4         | 2412/50000 [33:07<22:06:05,  1.67s/it]
Outer training loop:   5%|4         | 2413/50000 [33:09<21:42:13,  1.64s/it]
Outer training loop:   5%|4         | 2414/50000 [33:11<21:46:04,  1.65s/it]
Outer training loop:   5%|4         | 2415/50000 [33:12<21:09:52,  1.60s/it]
Outer training loop:   5%|4         | 2416/50000 [33:14<20:34:59,  1.56s/it]
Outer training loop:   5%|4         | 2417/50000 [33:15<20:40:28,  1.56s/it]
Outer training loop:   5%|4         | 2418/50000 [33:17<20:40:07,  1.56s/it]
Outer training loop:   5%|4         | 2419/50000 [33:18<20:24:14,  1.54s/it]
Outer training loop:   5%|4         | 2420/50000 [33:20<20:31:25,  1.55s/it]
Outer training loop:   5%|4         | 2421/50000 [33:21<20:20:49,  1.54s/it]
Outer training loop:   5%|4         | 2422/50000 [33:23<21:03:13,  1.59s/it]
Outer training loop:   5%|4         | 2423/50000 [33:25<20:37:26,  1.56s/it]
Outer training loop:   5%|4         | 2424/50000 [33:26<20:48:22,  1.57s/it]
Outer training loop:   5%|4         | 2425/50000 [33:28<20:33:44,  1.56s/it]
Outer training loop:   5%|4         | 2426/50000 [33:29<20:44:40,  1.57s/it]
Outer training loop:   5%|4         | 2427/50000 [33:31<20:28:47,  1.55s/it]
Outer training loop:   5%|4         | 2428/50000 [33:32<20:20:29,  1.54s/it]
Outer training loop:   5%|4         | 2429/50000 [33:34<20:48:34,  1.57s/it]
Outer training loop:   5%|4         | 2430/50000 [33:35<20:51:45,  1.58s/it]
Outer training loop:   5%|4         | 2431/50000 [33:37<20:25:07,  1.55s/it]
Outer training loop:   5%|4         | 2432/50000 [33:39<20:42:42,  1.57s/it]
Outer training loop:   5%|4         | 2433/50000 [33:40<20:46:48,  1.57s/it]
Outer training loop:   5%|4         | 2434/50000 [33:42<20:45:25,  1.57s/it]
Outer training loop:   5%|4         | 2435/50000 [33:43<21:22:28,  1.62s/it]
Outer training loop:   5%|4         | 2436/50000 [33:45<21:27:54,  1.62s/it]
Outer training loop:   5%|4         | 2437/50000 [33:47<21:37:03,  1.64s/it]
Outer training loop:   5%|4         | 2438/50000 [33:48<21:45:20,  1.65s/it]
Outer training loop:   5%|4         | 2439/50000 [33:50<21:12:23,  1.61s/it]
Outer training loop:   5%|4         | 2440/50000 [33:51<20:39:51,  1.56s/it]
Outer training loop:   5%|4         | 2441/50000 [33:53<20:51:44,  1.58s/it]
Outer training loop:   5%|4         | 2442/50000 [33:55<20:55:20,  1.58s/it]
Outer training loop:   5%|4         | 2443/50000 [33:56<20:45:41,  1.57s/it]
Outer training loop:   5%|4         | 2444/50000 [33:58<20:55:08,  1.58s/it]
Outer training loop:   5%|4         | 2445/50000 [33:59<20:56:03,  1.58s/it]
Outer training loop:   5%|4         | 2446/50000 [34:01<20:18:34,  1.54s/it]
Outer training loop:   5%|4         | 2447/50000 [34:02<20:42:44,  1.57s/it]
Outer training loop:   5%|4         | 2448/50000 [34:04<20:25:17,  1.55s/it]
Outer training loop:   5%|4         | 2449/50000 [34:06<20:43:07,  1.57s/it]
Outer training loop:   5%|4         | 2450/50000 [34:07<20:35:40,  1.56s/it]
Outer training loop:   5%|4         | 2451/50000 [34:09<21:10:34,  1.60s/it]
Outer training loop:   5%|4         | 2452/50000 [34:10<21:05:28,  1.60s/it]
Outer training loop:   5%|4         | 2453/50000 [34:12<21:00:07,  1.59s/it]
Outer training loop:   5%|4         | 2454/50000 [34:13<20:44:15,  1.57s/it]
Outer training loop:   5%|4         | 2455/50000 [34:15<21:11:45,  1.60s/it]
Outer training loop:   5%|4         | 2456/50000 [34:17<20:51:41,  1.58s/it]
Outer training loop:   5%|4         | 2457/50000 [34:18<20:55:07,  1.58s/it]
Outer training loop:   5%|4         | 2458/50000 [34:20<21:12:33,  1.61s/it]
Outer training loop:   5%|4         | 2459/50000 [34:22<21:31:19,  1.63s/it]
Outer training loop:   5%|4         | 2460/50000 [34:23<21:31:52,  1.63s/it]
Outer training loop:   5%|4         | 2461/50000 [34:25<21:37:44,  1.64s/it]
Outer training loop:   5%|4         | 2462/50000 [34:27<22:33:53,  1.71s/it]
Outer training loop:   5%|4         | 2463/50000 [34:28<21:49:38,  1.65s/it]
Outer training loop:   5%|4         | 2464/50000 [34:30<21:35:03,  1.63s/it]
Outer training loop:   5%|4         | 2465/50000 [34:31<21:00:11,  1.59s/it]
Outer training loop:   5%|4         | 2466/50000 [34:33<21:11:38,  1.61s/it]
Outer training loop:   5%|4         | 2467/50000 [34:35<21:06:35,  1.60s/it]
Outer training loop:   5%|4         | 2468/50000 [34:36<21:05:33,  1.60s/it]
Outer training loop:   5%|4         | 2469/50000 [34:38<20:46:48,  1.57s/it]
Outer training loop:   5%|4         | 2470/50000 [34:39<20:21:01,  1.54s/it]
Outer training loop:   5%|4         | 2471/50000 [34:41<20:50:12,  1.58s/it]
Outer training loop:   5%|4         | 2472/50000 [34:42<21:00:07,  1.59s/it]
Outer training loop:   5%|4         | 2473/50000 [34:44<20:53:44,  1.58s/it]
Outer training loop:   5%|4         | 2474/50000 [34:46<20:44:53,  1.57s/it]
Outer training loop:   5%|4         | 2475/50000 [34:47<20:15:29,  1.53s/it]
Outer training loop:   5%|4         | 2476/50000 [34:49<20:47:11,  1.57s/it]
Outer training loop:   5%|4         | 2477/50000 [34:50<20:41:15,  1.57s/it]
Outer training loop:   5%|4         | 2478/50000 [34:52<20:39:25,  1.56s/it]
Outer training loop:   5%|4         | 2479/50000 [34:53<20:26:04,  1.55s/it]
Outer training loop:   5%|4         | 2480/50000 [34:55<20:44:49,  1.57s/it]
Outer training loop:   5%|4         | 2481/50000 [34:56<20:24:47,  1.55s/it]
Outer training loop:   5%|4         | 2482/50000 [34:58<20:33:31,  1.56s/it]
Outer training loop:   5%|4         | 2483/50000 [35:00<20:32:54,  1.56s/it]
Outer training loop:   5%|4         | 2484/50000 [35:01<20:50:18,  1.58s/it]
Outer training loop:   5%|4         | 2485/50000 [35:03<21:00:08,  1.59s/it]
Outer training loop:   5%|4         | 2486/50000 [35:04<20:58:54,  1.59s/it]
Outer training loop:   5%|4         | 2487/50000 [35:06<20:31:46,  1.56s/it]
Outer training loop:   5%|4         | 2488/50000 [35:07<20:37:04,  1.56s/it]
Outer training loop:   5%|4         | 2489/50000 [35:09<20:27:46,  1.55s/it]
Outer training loop:   5%|4         | 2490/50000 [35:11<20:43:38,  1.57s/it]
Outer training loop:   5%|4         | 2491/50000 [35:12<20:29:15,  1.55s/it]
Outer training loop:   5%|4         | 2492/50000 [35:14<20:49:48,  1.58s/it]
Outer training loop:   5%|4         | 2493/50000 [35:15<20:34:49,  1.56s/it]
Outer training loop:   5%|4         | 2494/50000 [35:17<20:39:35,  1.57s/it]
Outer training loop:   5%|4         | 2495/50000 [35:18<20:50:19,  1.58s/it]
Outer training loop:   5%|4         | 2496/50000 [35:20<21:14:19,  1.61s/it]
Outer training loop:   5%|4         | 2497/50000 [35:22<20:54:30,  1.58s/it]
Outer training loop:   5%|4         | 2498/50000 [35:23<20:50:34,  1.58s/it]
Outer training loop:   5%|4         | 2499/50000 [35:25<20:39:24,  1.57s/it]
Outer training loop:   5%|5         | 2500/50000 [35:26<20:55:51,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.56s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.58s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.58s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:02,  1.50s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:02,  1.51s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:04,  1.56s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:02,  1.55s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:03,  1.59s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:01,  1.58s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:58,  1.56s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<02:02,  1.64s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:59,  1.61s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:58,  1.62s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:55,  1.61s/it]

Evaluating policy:  14%|#4        | 12/83 [00:19<01:54,  1.61s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:50,  1.58s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:48,  1.57s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:48,  1.60s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:44,  1.56s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:41,  1.54s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:37,  1.51s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:40,  1.58s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:39,  1.58s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:37,  1.57s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:38,  1.61s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:36,  1.60s/it]

Evaluating policy:  29%|##8       | 24/83 [00:38<01:33,  1.58s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:31,  1.58s/it]

Evaluating policy:  31%|###1      | 26/83 [00:41<01:32,  1.62s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:29,  1.60s/it]

Evaluating policy:  34%|###3      | 28/83 [00:44<01:27,  1.59s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:24,  1.57s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:26,  1.64s/it]

Evaluating policy:  37%|###7      | 31/83 [00:49<01:24,  1.62s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:20,  1.58s/it]

Evaluating policy:  40%|###9      | 33/83 [00:52<01:20,  1.60s/it]

Evaluating policy:  41%|####      | 34/83 [00:54<01:20,  1.64s/it]

Evaluating policy:  42%|####2     | 35/83 [00:55<01:15,  1.58s/it]

Evaluating policy:  43%|####3     | 36/83 [00:57<01:13,  1.57s/it]

Evaluating policy:  45%|####4     | 37/83 [00:58<01:12,  1.57s/it]

Evaluating policy:  46%|####5     | 38/83 [01:00<01:10,  1.56s/it]

Evaluating policy:  47%|####6     | 39/83 [01:02<01:11,  1.63s/it]

Evaluating policy:  48%|####8     | 40/83 [01:03<01:08,  1.59s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:04,  1.55s/it]

Evaluating policy:  51%|#####     | 42/83 [01:06<01:04,  1.56s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:08<01:02,  1.56s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:09<01:00,  1.55s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:11<01:00,  1.60s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<00:58,  1.58s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:14<00:56,  1.57s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:16<00:54,  1.57s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:17<00:52,  1.54s/it]

Evaluating policy:  60%|######    | 50/83 [01:19<00:50,  1.54s/it]

Evaluating policy:  61%|######1   | 51/83 [01:20<00:51,  1.61s/it]

Evaluating policy:  63%|######2   | 52/83 [01:22<00:49,  1.61s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:47,  1.59s/it]

Evaluating policy:  65%|######5   | 54/83 [01:25<00:45,  1.57s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:42,  1.53s/it]

Evaluating policy:  67%|######7   | 56/83 [01:28<00:41,  1.54s/it]

Evaluating policy:  69%|######8   | 57/83 [01:30<00:43,  1.68s/it]

Evaluating policy:  70%|######9   | 58/83 [01:32<00:42,  1.71s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:33<00:39,  1.65s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:35<00:37,  1.64s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:36<00:35,  1.61s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:38<00:33,  1.58s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:39<00:31,  1.57s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:41<00:31,  1.68s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:43<00:29,  1.63s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:44<00:27,  1.61s/it]

Evaluating policy:  81%|########  | 67/83 [01:46<00:25,  1.57s/it]

Evaluating policy:  82%|########1 | 68/83 [01:48<00:23,  1.59s/it]

Evaluating policy:  83%|########3 | 69/83 [01:49<00:21,  1.57s/it]

Evaluating policy:  84%|########4 | 70/83 [01:51<00:20,  1.55s/it]

Evaluating policy:  86%|########5 | 71/83 [01:53<00:20,  1.67s/it]

Evaluating policy:  87%|########6 | 72/83 [01:54<00:17,  1.63s/it]

Evaluating policy:  88%|########7 | 73/83 [01:56<00:15,  1.59s/it]

Evaluating policy:  89%|########9 | 74/83 [01:57<00:14,  1.58s/it]

Evaluating policy:  90%|######### | 75/83 [01:59<00:12,  1.56s/it]

Evaluating policy:  92%|#########1| 76/83 [02:00<00:11,  1.57s/it]

Evaluating policy:  93%|#########2| 77/83 [02:02<00:09,  1.58s/it]

Evaluating policy:  94%|#########3| 78/83 [02:03<00:07,  1.57s/it]

Evaluating policy:  95%|#########5| 79/83 [02:05<00:06,  1.72s/it]

Evaluating policy:  96%|#########6| 80/83 [02:07<00:05,  1.67s/it]

Evaluating policy:  98%|#########7| 81/83 [02:09<00:03,  1.64s/it]

Evaluating policy:  99%|#########8| 82/83 [02:10<00:01,  1.61s/it]

Evaluating policy: 100%|##########| 83/83 [02:12<00:00,  1.56s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:01,  1.48s/it]

Evaluating policy:   2%|2         | 2/83 [00:02<02:00,  1.49s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:01,  1.52s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<01:59,  1.51s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:09,  1.66s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:05,  1.63s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<02:06,  1.66s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<02:01,  1.62s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:59,  1.61s/it]

Evaluating policy:  12%|#2        | 10/83 [00:16<01:58,  1.62s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:55,  1.60s/it]

Evaluating policy:  14%|#4        | 12/83 [00:19<01:55,  1.63s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:52,  1.61s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:50,  1.60s/it]

Evaluating policy:  18%|#8        | 15/83 [00:24<01:51,  1.64s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:49,  1.64s/it]

Evaluating policy:  20%|##        | 17/83 [00:27<01:45,  1.60s/it]

Evaluating policy:  22%|##1       | 18/83 [00:29<01:47,  1.65s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:43,  1.62s/it]

Evaluating policy:  24%|##4       | 20/83 [00:32<01:40,  1.59s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:37,  1.58s/it]

Evaluating policy:  27%|##6       | 22/83 [00:35<01:38,  1.62s/it]

Evaluating policy:  28%|##7       | 23/83 [00:37<01:36,  1.62s/it]

Evaluating policy:  29%|##8       | 24/83 [00:38<01:32,  1.57s/it]

Evaluating policy:  30%|###       | 25/83 [00:40<01:38,  1.70s/it]

Evaluating policy:  31%|###1      | 26/83 [00:42<01:35,  1.67s/it]

Evaluating policy:  33%|###2      | 27/83 [00:43<01:30,  1.62s/it]

Evaluating policy:  34%|###3      | 28/83 [00:45<01:27,  1.59s/it]

Evaluating policy:  35%|###4      | 29/83 [00:46<01:28,  1.65s/it]

Evaluating policy:  36%|###6      | 30/83 [00:48<01:26,  1.63s/it]

Evaluating policy:  37%|###7      | 31/83 [00:50<01:23,  1.60s/it]

Evaluating policy:  39%|###8      | 32/83 [00:51<01:19,  1.56s/it]

Evaluating policy:  40%|###9      | 33/83 [00:53<01:21,  1.63s/it]

Evaluating policy:  41%|####      | 34/83 [00:54<01:18,  1.61s/it]

Evaluating policy:  42%|####2     | 35/83 [00:56<01:15,  1.58s/it]

Evaluating policy:  43%|####3     | 36/83 [00:57<01:13,  1.57s/it]

Evaluating policy:  45%|####4     | 37/83 [00:59<01:11,  1.56s/it]

Evaluating policy:  46%|####5     | 38/83 [01:01<01:13,  1.64s/it]

Evaluating policy:  47%|####6     | 39/83 [01:02<01:10,  1.60s/it]

Evaluating policy:  48%|####8     | 40/83 [01:04<01:07,  1.57s/it]

Evaluating policy:  49%|####9     | 41/83 [01:05<01:06,  1.58s/it]

Evaluating policy:  51%|#####     | 42/83 [01:07<01:04,  1.57s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:09<01:06,  1.66s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:10<01:02,  1.61s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:12<01:00,  1.61s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:13<00:58,  1.59s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:15<00:57,  1.59s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:17<00:58,  1.67s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:19<00:56,  1.65s/it]

Evaluating policy:  60%|######    | 50/83 [01:20<00:52,  1.60s/it]

Evaluating policy:  61%|######1   | 51/83 [01:22<00:50,  1.59s/it]

Evaluating policy:  63%|######2   | 52/83 [01:23<00:48,  1.56s/it]

Evaluating policy:  64%|######3   | 53/83 [01:25<00:46,  1.56s/it]

Evaluating policy:  65%|######5   | 54/83 [01:26<00:47,  1.64s/it]

Evaluating policy:  66%|######6   | 55/83 [01:28<00:44,  1.61s/it]

Evaluating policy:  67%|######7   | 56/83 [01:30<00:42,  1.58s/it]

Evaluating policy:  69%|######8   | 57/83 [01:31<00:40,  1.57s/it]

Evaluating policy:  70%|######9   | 58/83 [01:33<00:39,  1.56s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:34<00:36,  1.53s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:36<00:35,  1.54s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:38<00:36,  1.67s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:39<00:34,  1.63s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:41<00:32,  1.61s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:42<00:30,  1.58s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:44<00:28,  1.56s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:45<00:26,  1.55s/it]

Evaluating policy:  81%|########  | 67/83 [01:47<00:24,  1.55s/it]

Evaluating policy:  82%|########1 | 68/83 [01:49<00:25,  1.69s/it]

Evaluating policy:  83%|########3 | 69/83 [01:50<00:23,  1.66s/it]

Evaluating policy:  84%|########4 | 70/83 [01:52<00:20,  1.61s/it]

Evaluating policy:  86%|########5 | 71/83 [01:53<00:19,  1.59s/it]

Evaluating policy:  87%|########6 | 72/83 [01:55<00:17,  1.57s/it]

Evaluating policy:  88%|########7 | 73/83 [01:57<00:15,  1.58s/it]

Evaluating policy:  89%|########9 | 74/83 [01:58<00:14,  1.57s/it]

Evaluating policy:  90%|######### | 75/83 [02:00<00:12,  1.61s/it]

Evaluating policy:  92%|#########1| 76/83 [02:02<00:12,  1.72s/it]

Evaluating policy:  93%|#########2| 77/83 [02:03<00:10,  1.69s/it]

Evaluating policy:  94%|#########3| 78/83 [02:05<00:08,  1.64s/it]

Evaluating policy:  95%|#########5| 79/83 [02:07<00:06,  1.62s/it]

Evaluating policy:  96%|#########6| 80/83 [02:08<00:04,  1.60s/it]

Evaluating policy:  98%|#########7| 81/83 [02:10<00:03,  1.58s/it]

Evaluating policy:  99%|#########8| 82/83 [02:11<00:01,  1.59s/it]

Evaluating policy: 100%|##########| 83/83 [02:13<00:00,  1.58s/it]

Outer training loop:   5%|5         | 2501/50000 [39:58<1091:10:07, 82.70s/it]
Outer training loop:   5%|5         | 2502/50000 [40:00<772:03:20, 58.52s/it] 
Outer training loop:   5%|5         | 2503/50000 [40:02<546:35:11, 41.43s/it]
Outer training loop:   5%|5         | 2504/50000 [40:04<388:52:35, 29.48s/it]
Outer training loop:   5%|5         | 2505/50000 [40:05<278:49:38, 21.13s/it]
Outer training loop:   5%|5         | 2506/50000 [40:07<201:08:49, 15.25s/it]
Outer training loop:   5%|5         | 2507/50000 [40:08<147:17:01, 11.16s/it]
Outer training loop:   5%|5         | 2508/50000 [40:10<109:02:55,  8.27s/it]
Outer training loop:   5%|5         | 2509/50000 [40:11<82:32:45,  6.26s/it] 
Outer training loop:   5%|5         | 2510/50000 [40:13<63:57:52,  4.85s/it]
Outer training loop:   5%|5         | 2511/50000 [40:15<51:02:27,  3.87s/it]
Outer training loop:   5%|5         | 2512/50000 [40:16<41:36:46,  3.15s/it]
Outer training loop:   5%|5         | 2513/50000 [40:18<35:53:40,  2.72s/it]
Outer training loop:   5%|5         | 2514/50000 [40:19<31:17:54,  2.37s/it]
Outer training loop:   5%|5         | 2515/50000 [40:21<28:24:00,  2.15s/it]
Outer training loop:   5%|5         | 2516/50000 [40:23<26:01:23,  1.97s/it]
Outer training loop:   5%|5         | 2517/50000 [40:24<24:42:20,  1.87s/it]
Outer training loop:   5%|5         | 2518/50000 [40:26<23:29:49,  1.78s/it]
Outer training loop:   5%|5         | 2519/50000 [40:27<22:46:11,  1.73s/it]
Outer training loop:   5%|5         | 2520/50000 [40:29<22:09:42,  1.68s/it]
Outer training loop:   5%|5         | 2521/50000 [40:31<22:10:21,  1.68s/it]
Outer training loop:   5%|5         | 2522/50000 [40:32<21:37:15,  1.64s/it]
Outer training loop:   5%|5         | 2523/50000 [40:34<21:51:04,  1.66s/it]
Outer training loop:   5%|5         | 2524/50000 [40:35<21:24:56,  1.62s/it]
Outer training loop:   5%|5         | 2525/50000 [40:37<21:19:31,  1.62s/it]
Outer training loop:   5%|5         | 2526/50000 [40:39<20:57:39,  1.59s/it]
Outer training loop:   5%|5         | 2527/50000 [40:40<21:01:29,  1.59s/it]
Outer training loop:   5%|5         | 2528/50000 [40:42<20:24:41,  1.55s/it]
Outer training loop:   5%|5         | 2529/50000 [40:43<20:39:20,  1.57s/it]
Outer training loop:   5%|5         | 2530/50000 [40:45<20:11:54,  1.53s/it]
Outer training loop:   5%|5         | 2531/50000 [40:46<20:25:27,  1.55s/it]
Outer training loop:   5%|5         | 2532/50000 [40:48<20:19:54,  1.54s/it]
Outer training loop:   5%|5         | 2533/50000 [40:49<20:46:09,  1.58s/it]
Outer training loop:   5%|5         | 2534/50000 [40:51<20:28:23,  1.55s/it]
Outer training loop:   5%|5         | 2535/50000 [40:52<20:27:56,  1.55s/it]
Outer training loop:   5%|5         | 2536/50000 [40:54<20:05:26,  1.52s/it]
Outer training loop:   5%|5         | 2537/50000 [40:56<20:44:02,  1.57s/it]
Outer training loop:   5%|5         | 2538/50000 [40:57<20:33:12,  1.56s/it]
Outer training loop:   5%|5         | 2539/50000 [40:59<20:49:48,  1.58s/it]
Outer training loop:   5%|5         | 2540/50000 [41:00<20:31:15,  1.56s/it]
Outer training loop:   5%|5         | 2541/50000 [41:02<20:36:58,  1.56s/it]
Outer training loop:   5%|5         | 2542/50000 [41:04<20:57:48,  1.59s/it]
Outer training loop:   5%|5         | 2543/50000 [41:05<21:26:00,  1.63s/it]
Outer training loop:   5%|5         | 2544/50000 [41:07<21:03:44,  1.60s/it]
Outer training loop:   5%|5         | 2545/50000 [41:08<21:02:29,  1.60s/it]
Outer training loop:   5%|5         | 2546/50000 [41:10<20:41:52,  1.57s/it]
Outer training loop:   5%|5         | 2547/50000 [41:12<21:09:10,  1.60s/it]
Outer training loop:   5%|5         | 2548/50000 [41:13<20:54:55,  1.59s/it]
Outer training loop:   5%|5         | 2549/50000 [41:15<21:18:39,  1.62s/it]
Outer training loop:   5%|5         | 2550/50000 [41:16<20:55:01,  1.59s/it]
Outer training loop:   5%|5         | 2551/50000 [41:18<21:02:37,  1.60s/it]
Outer training loop:   5%|5         | 2552/50000 [41:19<20:38:39,  1.57s/it]
Outer training loop:   5%|5         | 2553/50000 [41:21<20:55:04,  1.59s/it]
Outer training loop:   5%|5         | 2554/50000 [41:23<20:40:18,  1.57s/it]
Outer training loop:   5%|5         | 2555/50000 [41:24<20:37:06,  1.56s/it]
Outer training loop:   5%|5         | 2556/50000 [41:26<20:54:09,  1.59s/it]
Outer training loop:   5%|5         | 2557/50000 [41:27<20:59:58,  1.59s/it]
Outer training loop:   5%|5         | 2558/50000 [41:29<20:52:37,  1.58s/it]
Outer training loop:   5%|5         | 2559/50000 [41:31<20:58:17,  1.59s/it]
Outer training loop:   5%|5         | 2560/50000 [41:32<20:27:03,  1.55s/it]
Outer training loop:   5%|5         | 2561/50000 [41:34<20:45:23,  1.58s/it]
Outer training loop:   5%|5         | 2562/50000 [41:35<20:42:36,  1.57s/it]
Outer training loop:   5%|5         | 2563/50000 [41:37<20:51:38,  1.58s/it]
Outer training loop:   5%|5         | 2564/50000 [41:38<20:29:03,  1.55s/it]
Outer training loop:   5%|5         | 2565/50000 [41:40<20:51:06,  1.58s/it]
Outer training loop:   5%|5         | 2566/50000 [41:42<20:52:28,  1.58s/it]
Outer training loop:   5%|5         | 2567/50000 [41:43<21:28:35,  1.63s/it]
Outer training loop:   5%|5         | 2568/50000 [41:45<21:11:46,  1.61s/it]
Outer training loop:   5%|5         | 2569/50000 [41:46<21:22:57,  1.62s/it]
Outer training loop:   5%|5         | 2570/50000 [41:48<20:37:30,  1.57s/it]
Outer training loop:   5%|5         | 2571/50000 [41:50<21:07:29,  1.60s/it]
Outer training loop:   5%|5         | 2572/50000 [41:51<20:49:39,  1.58s/it]
Outer training loop:   5%|5         | 2573/50000 [41:53<21:05:27,  1.60s/it]
Outer training loop:   5%|5         | 2574/50000 [41:54<21:01:33,  1.60s/it]
Outer training loop:   5%|5         | 2575/50000 [41:56<21:19:34,  1.62s/it]
Outer training loop:   5%|5         | 2576/50000 [41:58<21:16:55,  1.62s/it]
Outer training loop:   5%|5         | 2577/50000 [41:59<21:19:21,  1.62s/it]
Outer training loop:   5%|5         | 2578/50000 [42:01<20:48:55,  1.58s/it]
Outer training loop:   5%|5         | 2579/50000 [42:02<21:02:13,  1.60s/it]
Outer training loop:   5%|5         | 2580/50000 [42:04<21:00:12,  1.59s/it]
Outer training loop:   5%|5         | 2581/50000 [42:06<21:25:02,  1.63s/it]
Outer training loop:   5%|5         | 2582/50000 [42:07<20:50:39,  1.58s/it]
Outer training loop:   5%|5         | 2583/50000 [42:09<20:46:47,  1.58s/it]
Outer training loop:   5%|5         | 2584/50000 [42:10<21:14:11,  1.61s/it]
Outer training loop:   5%|5         | 2585/50000 [42:12<20:41:59,  1.57s/it]
Outer training loop:   5%|5         | 2586/50000 [42:14<21:06:58,  1.60s/it]
Outer training loop:   5%|5         | 2587/50000 [42:15<20:46:55,  1.58s/it]
Outer training loop:   5%|5         | 2588/50000 [42:17<21:05:24,  1.60s/it]
Outer training loop:   5%|5         | 2589/50000 [42:18<20:49:36,  1.58s/it]
Outer training loop:   5%|5         | 2590/50000 [42:20<20:48:58,  1.58s/it]
Outer training loop:   5%|5         | 2591/50000 [42:21<20:25:18,  1.55s/it]
Outer training loop:   5%|5         | 2592/50000 [42:23<20:49:23,  1.58s/it]
Outer training loop:   5%|5         | 2593/50000 [42:25<21:08:35,  1.61s/it]
Outer training loop:   5%|5         | 2594/50000 [42:26<21:17:37,  1.62s/it]
Outer training loop:   5%|5         | 2595/50000 [42:28<20:47:05,  1.58s/it]
Outer training loop:   5%|5         | 2596/50000 [42:29<21:00:18,  1.60s/it]
Outer training loop:   5%|5         | 2597/50000 [42:31<20:48:46,  1.58s/it]
Outer training loop:   5%|5         | 2598/50000 [42:33<21:02:40,  1.60s/it]
Outer training loop:   5%|5         | 2599/50000 [42:34<21:05:37,  1.60s/it]
Outer training loop:   5%|5         | 2600/50000 [42:36<21:14:08,  1.61s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.58s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.57s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.55s/it]

Outer training loop:   5%|5         | 2601/50000 [42:42<39:24:48,  2.99s/it]
Outer training loop:   5%|5         | 2602/50000 [42:44<33:38:57,  2.56s/it]
Outer training loop:   5%|5         | 2603/50000 [42:45<29:52:02,  2.27s/it]
Outer training loop:   5%|5         | 2604/50000 [42:47<27:19:21,  2.08s/it]
Outer training loop:   5%|5         | 2605/50000 [42:48<25:03:05,  1.90s/it]
Outer training loop:   5%|5         | 2606/50000 [42:50<23:46:48,  1.81s/it]
Outer training loop:   5%|5         | 2607/50000 [42:51<22:49:38,  1.73s/it]
Outer training loop:   5%|5         | 2608/50000 [42:53<22:14:04,  1.69s/it]
Outer training loop:   5%|5         | 2609/50000 [42:55<21:44:24,  1.65s/it]
Outer training loop:   5%|5         | 2610/50000 [42:56<21:28:51,  1.63s/it]
Outer training loop:   5%|5         | 2611/50000 [42:58<21:08:16,  1.61s/it]
Outer training loop:   5%|5         | 2612/50000 [42:59<21:08:59,  1.61s/it]
Outer training loop:   5%|5         | 2613/50000 [43:01<21:00:22,  1.60s/it]
Outer training loop:   5%|5         | 2614/50000 [43:03<21:18:22,  1.62s/it]
Outer training loop:   5%|5         | 2615/50000 [43:04<21:11:27,  1.61s/it]
Outer training loop:   5%|5         | 2616/50000 [43:06<21:19:52,  1.62s/it]
Outer training loop:   5%|5         | 2617/50000 [43:07<20:49:18,  1.58s/it]
Outer training loop:   5%|5         | 2618/50000 [43:09<21:09:16,  1.61s/it]
Outer training loop:   5%|5         | 2619/50000 [43:11<21:08:23,  1.61s/it]
Outer training loop:   5%|5         | 2620/50000 [43:12<20:58:44,  1.59s/it]
Outer training loop:   5%|5         | 2621/50000 [43:14<20:45:53,  1.58s/it]
Outer training loop:   5%|5         | 2622/50000 [43:15<20:53:00,  1.59s/it]
Outer training loop:   5%|5         | 2623/50000 [43:17<20:35:46,  1.57s/it]
Outer training loop:   5%|5         | 2624/50000 [43:18<20:59:38,  1.60s/it]
Outer training loop:   5%|5         | 2625/50000 [43:20<20:59:48,  1.60s/it]
Outer training loop:   5%|5         | 2626/50000 [43:22<20:59:38,  1.60s/it]
Outer training loop:   5%|5         | 2627/50000 [43:23<20:46:28,  1.58s/it]
Outer training loop:   5%|5         | 2628/50000 [43:25<21:24:39,  1.63s/it]
Outer training loop:   5%|5         | 2629/50000 [43:26<21:00:56,  1.60s/it]
Outer training loop:   5%|5         | 2630/50000 [43:28<20:50:17,  1.58s/it]
Outer training loop:   5%|5         | 2631/50000 [43:30<20:44:30,  1.58s/it]
Outer training loop:   5%|5         | 2632/50000 [43:31<20:22:51,  1.55s/it]
Outer training loop:   5%|5         | 2633/50000 [43:33<21:00:45,  1.60s/it]
Outer training loop:   5%|5         | 2634/50000 [43:34<21:14:30,  1.61s/it]
Outer training loop:   5%|5         | 2635/50000 [43:36<20:44:33,  1.58s/it]
Outer training loop:   5%|5         | 2636/50000 [43:38<20:55:36,  1.59s/it]
Outer training loop:   5%|5         | 2637/50000 [43:39<20:41:36,  1.57s/it]
Outer training loop:   5%|5         | 2638/50000 [43:41<20:48:56,  1.58s/it]
Outer training loop:   5%|5         | 2639/50000 [43:42<20:27:33,  1.56s/it]
Outer training loop:   5%|5         | 2640/50000 [43:44<21:05:01,  1.60s/it]
Outer training loop:   5%|5         | 2641/50000 [43:45<20:41:05,  1.57s/it]
Outer training loop:   5%|5         | 2642/50000 [43:47<20:53:09,  1.59s/it]
Outer training loop:   5%|5         | 2643/50000 [43:49<20:50:33,  1.58s/it]
Outer training loop:   5%|5         | 2644/50000 [43:50<20:48:22,  1.58s/it]
Outer training loop:   5%|5         | 2645/50000 [43:52<20:45:04,  1.58s/it]
Outer training loop:   5%|5         | 2646/50000 [43:53<20:37:06,  1.57s/it]
Outer training loop:   5%|5         | 2647/50000 [43:55<21:17:11,  1.62s/it]
Outer training loop:   5%|5         | 2648/50000 [43:57<21:09:33,  1.61s/it]
Outer training loop:   5%|5         | 2649/50000 [43:58<21:19:23,  1.62s/it]
Outer training loop:   5%|5         | 2650/50000 [44:00<20:57:16,  1.59s/it]
Outer training loop:   5%|5         | 2651/50000 [44:01<21:14:32,  1.62s/it]
Outer training loop:   5%|5         | 2652/50000 [44:03<21:00:37,  1.60s/it]
Outer training loop:   5%|5         | 2653/50000 [44:05<21:17:05,  1.62s/it]
Outer training loop:   5%|5         | 2654/50000 [44:06<21:04:18,  1.60s/it]
Outer training loop:   5%|5         | 2655/50000 [44:08<20:58:25,  1.59s/it]
Outer training loop:   5%|5         | 2656/50000 [44:09<20:21:40,  1.55s/it]
Outer training loop:   5%|5         | 2657/50000 [44:11<20:52:11,  1.59s/it]
Outer training loop:   5%|5         | 2658/50000 [44:12<20:36:36,  1.57s/it]
Outer training loop:   5%|5         | 2659/50000 [44:14<20:55:46,  1.59s/it]
Outer training loop:   5%|5         | 2660/50000 [44:16<20:30:54,  1.56s/it]
Outer training loop:   5%|5         | 2661/50000 [44:17<20:46:34,  1.58s/it]
Outer training loop:   5%|5         | 2662/50000 [44:19<20:31:48,  1.56s/it]
Outer training loop:   5%|5         | 2663/50000 [44:20<20:53:26,  1.59s/it]
Outer training loop:   5%|5         | 2664/50000 [44:22<20:41:51,  1.57s/it]
Outer training loop:   5%|5         | 2665/50000 [44:24<20:41:30,  1.57s/it]
Outer training loop:   5%|5         | 2666/50000 [44:25<20:55:18,  1.59s/it]
Outer training loop:   5%|5         | 2667/50000 [44:27<20:50:01,  1.58s/it]
Outer training loop:   5%|5         | 2668/50000 [44:28<20:38:37,  1.57s/it]
Outer training loop:   5%|5         | 2669/50000 [44:30<20:48:12,  1.58s/it]
Outer training loop:   5%|5         | 2670/50000 [44:31<20:13:14,  1.54s/it]
Outer training loop:   5%|5         | 2671/50000 [44:33<20:25:29,  1.55s/it]
Outer training loop:   5%|5         | 2672/50000 [44:34<20:35:16,  1.57s/it]
Outer training loop:   5%|5         | 2673/50000 [44:36<20:55:15,  1.59s/it]
Outer training loop:   5%|5         | 2674/50000 [44:38<20:40:41,  1.57s/it]
Outer training loop:   5%|5         | 2675/50000 [44:39<20:31:00,  1.56s/it]
Outer training loop:   5%|5         | 2676/50000 [44:41<20:51:52,  1.59s/it]
Outer training loop:   5%|5         | 2677/50000 [44:42<20:32:20,  1.56s/it]
Outer training loop:   5%|5         | 2678/50000 [44:44<20:38:23,  1.57s/it]
Outer training loop:   5%|5         | 2679/50000 [44:45<20:36:28,  1.57s/it]
Outer training loop:   5%|5         | 2680/50000 [44:47<20:46:22,  1.58s/it]
Outer training loop:   5%|5         | 2681/50000 [44:49<20:39:21,  1.57s/it]
Outer training loop:   5%|5         | 2682/50000 [44:50<20:47:45,  1.58s/it]
Outer training loop:   5%|5         | 2683/50000 [44:52<20:30:42,  1.56s/it]
Outer training loop:   5%|5         | 2684/50000 [44:53<20:40:12,  1.57s/it]
Outer training loop:   5%|5         | 2685/50000 [44:55<20:22:14,  1.55s/it]
Outer training loop:   5%|5         | 2686/50000 [44:56<20:10:37,  1.54s/it]
Outer training loop:   5%|5         | 2687/50000 [44:58<20:43:23,  1.58s/it]
Outer training loop:   5%|5         | 2688/50000 [45:00<20:50:36,  1.59s/it]
Outer training loop:   5%|5         | 2689/50000 [45:01<20:40:17,  1.57s/it]
Outer training loop:   5%|5         | 2690/50000 [45:03<21:21:30,  1.63s/it]
Outer training loop:   5%|5         | 2691/50000 [45:05<21:31:23,  1.64s/it]
Outer training loop:   5%|5         | 2692/50000 [45:06<21:02:24,  1.60s/it]
Outer training loop:   5%|5         | 2693/50000 [45:08<20:33:57,  1.57s/it]
Outer training loop:   5%|5         | 2694/50000 [45:09<20:45:46,  1.58s/it]
Outer training loop:   5%|5         | 2695/50000 [45:11<20:44:01,  1.58s/it]
Outer training loop:   5%|5         | 2696/50000 [45:12<20:25:20,  1.55s/it]
Outer training loop:   5%|5         | 2697/50000 [45:14<20:58:17,  1.60s/it]
Outer training loop:   5%|5         | 2698/50000 [45:16<21:19:21,  1.62s/it]
Outer training loop:   5%|5         | 2699/50000 [45:17<21:04:27,  1.60s/it]
Outer training loop:   5%|5         | 2700/50000 [45:19<20:44:24,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.59s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.58s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.58s/it]

Outer training loop:   5%|5         | 2701/50000 [45:25<39:42:33,  3.02s/it]
Outer training loop:   5%|5         | 2702/50000 [45:27<33:49:39,  2.57s/it]
Outer training loop:   5%|5         | 2703/50000 [45:28<29:24:18,  2.24s/it]
Outer training loop:   5%|5         | 2704/50000 [45:30<26:52:08,  2.05s/it]
Outer training loop:   5%|5         | 2705/50000 [45:31<24:38:26,  1.88s/it]
Outer training loop:   5%|5         | 2706/50000 [45:33<24:01:24,  1.83s/it]
Outer training loop:   5%|5         | 2707/50000 [45:35<23:10:26,  1.76s/it]
Outer training loop:   5%|5         | 2708/50000 [45:36<22:36:37,  1.72s/it]
Outer training loop:   5%|5         | 2709/50000 [45:38<21:31:18,  1.64s/it]
Outer training loop:   5%|5         | 2710/50000 [45:39<21:37:06,  1.65s/it]
Outer training loop:   5%|5         | 2711/50000 [45:41<21:24:21,  1.63s/it]
Outer training loop:   5%|5         | 2712/50000 [45:42<21:26:01,  1.63s/it]
Outer training loop:   5%|5         | 2713/50000 [45:44<21:38:48,  1.65s/it]
Outer training loop:   5%|5         | 2714/50000 [45:46<21:21:52,  1.63s/it]
Outer training loop:   5%|5         | 2715/50000 [45:47<20:57:52,  1.60s/it]
Outer training loop:   5%|5         | 2716/50000 [45:49<21:07:08,  1.61s/it]
Outer training loop:   5%|5         | 2717/50000 [45:51<21:09:06,  1.61s/it]
Outer training loop:   5%|5         | 2718/50000 [45:52<20:40:17,  1.57s/it]
Outer training loop:   5%|5         | 2719/50000 [45:54<20:56:12,  1.59s/it]
Outer training loop:   5%|5         | 2720/50000 [45:55<20:42:53,  1.58s/it]
Outer training loop:   5%|5         | 2721/50000 [45:57<20:27:55,  1.56s/it]
Outer training loop:   5%|5         | 2722/50000 [45:58<20:36:07,  1.57s/it]
Outer training loop:   5%|5         | 2723/50000 [46:00<20:42:13,  1.58s/it]
Outer training loop:   5%|5         | 2724/50000 [46:02<21:03:06,  1.60s/it]
Outer training loop:   5%|5         | 2725/50000 [46:03<21:20:35,  1.63s/it]
Outer training loop:   5%|5         | 2726/50000 [46:05<21:18:41,  1.62s/it]
Outer training loop:   5%|5         | 2727/50000 [46:06<21:08:13,  1.61s/it]
Outer training loop:   5%|5         | 2728/50000 [46:08<20:58:48,  1.60s/it]
Outer training loop:   5%|5         | 2729/50000 [46:10<20:56:59,  1.60s/it]
Outer training loop:   5%|5         | 2730/50000 [46:11<20:45:49,  1.58s/it]
Outer training loop:   5%|5         | 2731/50000 [46:13<20:58:03,  1.60s/it]
Outer training loop:   5%|5         | 2732/50000 [46:14<20:22:52,  1.55s/it]
Outer training loop:   5%|5         | 2733/50000 [46:16<20:17:39,  1.55s/it]
Outer training loop:   5%|5         | 2734/50000 [46:17<20:31:16,  1.56s/it]
Outer training loop:   5%|5         | 2735/50000 [46:19<20:20:29,  1.55s/it]
Outer training loop:   5%|5         | 2736/50000 [46:21<20:43:53,  1.58s/it]
Outer training loop:   5%|5         | 2737/50000 [46:22<21:35:37,  1.64s/it]
Outer training loop:   5%|5         | 2738/50000 [46:24<21:10:17,  1.61s/it]
Outer training loop:   5%|5         | 2739/50000 [46:25<21:07:14,  1.61s/it]
Outer training loop:   5%|5         | 2740/50000 [46:27<20:59:06,  1.60s/it]
Outer training loop:   5%|5         | 2741/50000 [46:29<20:56:48,  1.60s/it]
Outer training loop:   5%|5         | 2742/50000 [46:30<20:49:27,  1.59s/it]
Outer training loop:   5%|5         | 2743/50000 [46:32<20:36:37,  1.57s/it]
Outer training loop:   5%|5         | 2744/50000 [46:33<21:02:36,  1.60s/it]
Outer training loop:   5%|5         | 2745/50000 [46:35<20:33:21,  1.57s/it]
Outer training loop:   5%|5         | 2746/50000 [46:36<20:35:54,  1.57s/it]
Outer training loop:   5%|5         | 2747/50000 [46:38<20:21:35,  1.55s/it]
Outer training loop:   5%|5         | 2748/50000 [46:40<20:40:46,  1.58s/it]
Outer training loop:   5%|5         | 2749/50000 [46:41<20:28:27,  1.56s/it]
Outer training loop:   6%|5         | 2750/50000 [46:43<20:54:22,  1.59s/it]
Outer training loop:   6%|5         | 2751/50000 [46:44<21:11:58,  1.62s/it]
Outer training loop:   6%|5         | 2752/50000 [46:46<20:53:41,  1.59s/it]
Outer training loop:   6%|5         | 2753/50000 [46:48<21:22:14,  1.63s/it]
Outer training loop:   6%|5         | 2754/50000 [46:49<20:52:52,  1.59s/it]
Outer training loop:   6%|5         | 2755/50000 [46:51<21:16:22,  1.62s/it]
Outer training loop:   6%|5         | 2756/50000 [46:52<20:39:03,  1.57s/it]
Outer training loop:   6%|5         | 2757/50000 [46:54<20:53:12,  1.59s/it]
Outer training loop:   6%|5         | 2758/50000 [46:56<20:30:45,  1.56s/it]
Outer training loop:   6%|5         | 2759/50000 [46:57<20:23:38,  1.55s/it]
Outer training loop:   6%|5         | 2760/50000 [46:59<20:33:04,  1.57s/it]
Outer training loop:   6%|5         | 2761/50000 [47:00<20:34:18,  1.57s/it]
Outer training loop:   6%|5         | 2762/50000 [47:02<20:57:19,  1.60s/it]
Outer training loop:   6%|5         | 2763/50000 [47:03<21:00:04,  1.60s/it]
Outer training loop:   6%|5         | 2764/50000 [47:05<20:34:55,  1.57s/it]
Outer training loop:   6%|5         | 2765/50000 [47:07<20:45:17,  1.58s/it]
Outer training loop:   6%|5         | 2766/50000 [47:08<20:34:08,  1.57s/it]
Outer training loop:   6%|5         | 2767/50000 [47:10<20:13:58,  1.54s/it]
Outer training loop:   6%|5         | 2768/50000 [47:11<20:17:31,  1.55s/it]
Outer training loop:   6%|5         | 2769/50000 [47:13<19:53:23,  1.52s/it]
Outer training loop:   6%|5         | 2770/50000 [47:14<20:12:50,  1.54s/it]
Outer training loop:   6%|5         | 2771/50000 [47:16<20:15:01,  1.54s/it]
Outer training loop:   6%|5         | 2772/50000 [47:17<20:17:56,  1.55s/it]
Outer training loop:   6%|5         | 2773/50000 [47:19<20:25:05,  1.56s/it]
Outer training loop:   6%|5         | 2774/50000 [47:21<20:44:29,  1.58s/it]
Outer training loop:   6%|5         | 2775/50000 [47:22<20:32:37,  1.57s/it]
Outer training loop:   6%|5         | 2776/50000 [47:24<20:43:06,  1.58s/it]
Outer training loop:   6%|5         | 2777/50000 [47:25<20:45:09,  1.58s/it]
Outer training loop:   6%|5         | 2778/50000 [47:27<20:56:53,  1.60s/it]
Outer training loop:   6%|5         | 2779/50000 [47:28<20:40:39,  1.58s/it]
Outer training loop:   6%|5         | 2780/50000 [47:30<20:54:29,  1.59s/it]
Outer training loop:   6%|5         | 2781/50000 [47:32<20:44:12,  1.58s/it]
Outer training loop:   6%|5         | 2782/50000 [47:33<20:52:56,  1.59s/it]
Outer training loop:   6%|5         | 2783/50000 [47:35<20:36:15,  1.57s/it]
Outer training loop:   6%|5         | 2784/50000 [47:36<20:41:38,  1.58s/it]
Outer training loop:   6%|5         | 2785/50000 [47:38<20:09:59,  1.54s/it]
Outer training loop:   6%|5         | 2786/50000 [47:39<20:29:23,  1.56s/it]
Outer training loop:   6%|5         | 2787/50000 [47:41<20:56:31,  1.60s/it]
Outer training loop:   6%|5         | 2788/50000 [47:43<20:52:40,  1.59s/it]
Outer training loop:   6%|5         | 2789/50000 [47:44<20:37:19,  1.57s/it]
Outer training loop:   6%|5         | 2790/50000 [47:46<20:37:28,  1.57s/it]
Outer training loop:   6%|5         | 2791/50000 [47:47<20:27:23,  1.56s/it]
Outer training loop:   6%|5         | 2792/50000 [47:49<21:07:04,  1.61s/it]
Outer training loop:   6%|5         | 2793/50000 [47:51<20:48:35,  1.59s/it]
Outer training loop:   6%|5         | 2794/50000 [47:52<21:02:24,  1.60s/it]
Outer training loop:   6%|5         | 2795/50000 [47:54<20:49:27,  1.59s/it]
Outer training loop:   6%|5         | 2796/50000 [47:55<21:00:11,  1.60s/it]
Outer training loop:   6%|5         | 2797/50000 [47:57<20:47:56,  1.59s/it]
Outer training loop:   6%|5         | 2798/50000 [47:59<20:58:48,  1.60s/it]
Outer training loop:   6%|5         | 2799/50000 [48:00<20:40:19,  1.58s/it]
Outer training loop:   6%|5         | 2800/50000 [48:02<20:42:20,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.59s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.57s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.52s/it]

Outer training loop:   6%|5         | 2801/50000 [48:08<38:29:02,  2.94s/it]
Outer training loop:   6%|5         | 2802/50000 [48:09<33:20:05,  2.54s/it]
Outer training loop:   6%|5         | 2803/50000 [48:11<29:03:02,  2.22s/it]
Outer training loop:   6%|5         | 2804/50000 [48:12<26:49:00,  2.05s/it]
Outer training loop:   6%|5         | 2805/50000 [48:14<24:52:56,  1.90s/it]
Outer training loop:   6%|5         | 2806/50000 [48:16<23:41:13,  1.81s/it]
Outer training loop:   6%|5         | 2807/50000 [48:17<23:02:42,  1.76s/it]
Outer training loop:   6%|5         | 2808/50000 [48:19<22:08:34,  1.69s/it]
Outer training loop:   6%|5         | 2809/50000 [48:20<21:57:16,  1.67s/it]
Outer training loop:   6%|5         | 2810/50000 [48:22<21:52:28,  1.67s/it]
Outer training loop:   6%|5         | 2811/50000 [48:24<22:07:19,  1.69s/it]
Outer training loop:   6%|5         | 2812/50000 [48:25<21:29:07,  1.64s/it]
Outer training loop:   6%|5         | 2813/50000 [48:27<21:19:30,  1.63s/it]
Outer training loop:   6%|5         | 2814/50000 [48:29<21:25:14,  1.63s/it]
Outer training loop:   6%|5         | 2815/50000 [48:30<21:35:00,  1.65s/it]
Outer training loop:   6%|5         | 2816/50000 [48:32<21:08:57,  1.61s/it]
Outer training loop:   6%|5         | 2817/50000 [48:34<21:25:37,  1.63s/it]
Outer training loop:   6%|5         | 2818/50000 [48:35<20:47:51,  1.59s/it]
Outer training loop:   6%|5         | 2819/50000 [48:36<20:16:52,  1.55s/it]
Outer training loop:   6%|5         | 2820/50000 [48:38<20:30:39,  1.57s/it]
Outer training loop:   6%|5         | 2821/50000 [48:40<20:25:08,  1.56s/it]
Outer training loop:   6%|5         | 2822/50000 [48:41<20:31:01,  1.57s/it]
Outer training loop:   6%|5         | 2823/50000 [48:43<20:35:14,  1.57s/it]
Outer training loop:   6%|5         | 2824/50000 [48:44<21:09:58,  1.62s/it]
Outer training loop:   6%|5         | 2825/50000 [48:46<20:54:07,  1.60s/it]
Outer training loop:   6%|5         | 2826/50000 [48:48<21:03:27,  1.61s/it]
Outer training loop:   6%|5         | 2827/50000 [48:49<20:32:05,  1.57s/it]
Outer training loop:   6%|5         | 2828/50000 [48:51<20:17:12,  1.55s/it]
Outer training loop:   6%|5         | 2829/50000 [48:52<20:13:04,  1.54s/it]
Outer training loop:   6%|5         | 2830/50000 [48:54<20:04:02,  1.53s/it]
Outer training loop:   6%|5         | 2831/50000 [48:55<20:34:29,  1.57s/it]
Outer training loop:   6%|5         | 2832/50000 [48:57<20:38:03,  1.57s/it]
Outer training loop:   6%|5         | 2833/50000 [48:58<20:10:29,  1.54s/it]
Outer training loop:   6%|5         | 2834/50000 [49:00<20:58:40,  1.60s/it]
Outer training loop:   6%|5         | 2835/50000 [49:02<20:35:56,  1.57s/it]
Outer training loop:   6%|5         | 2836/50000 [49:03<20:17:01,  1.55s/it]
Outer training loop:   6%|5         | 2837/50000 [49:05<20:22:14,  1.55s/it]
Outer training loop:   6%|5         | 2838/50000 [49:06<20:29:43,  1.56s/it]
Outer training loop:   6%|5         | 2839/50000 [49:08<20:19:07,  1.55s/it]
Outer training loop:   6%|5         | 2840/50000 [49:09<20:44:08,  1.58s/it]
Outer training loop:   6%|5         | 2841/50000 [49:11<20:40:33,  1.58s/it]
Outer training loop:   6%|5         | 2842/50000 [49:13<20:51:04,  1.59s/it]
Outer training loop:   6%|5         | 2843/50000 [49:14<21:00:54,  1.60s/it]
Outer training loop:   6%|5         | 2844/50000 [49:16<21:11:08,  1.62s/it]
Outer training loop:   6%|5         | 2845/50000 [49:17<20:45:16,  1.58s/it]
Outer training loop:   6%|5         | 2846/50000 [49:19<20:38:02,  1.58s/it]
Outer training loop:   6%|5         | 2847/50000 [49:21<20:58:29,  1.60s/it]
Outer training loop:   6%|5         | 2848/50000 [49:22<20:50:42,  1.59s/it]
Outer training loop:   6%|5         | 2849/50000 [49:24<21:00:10,  1.60s/it]
Outer training loop:   6%|5         | 2850/50000 [49:25<20:49:46,  1.59s/it]
Outer training loop:   6%|5         | 2851/50000 [49:27<21:40:14,  1.65s/it]
Outer training loop:   6%|5         | 2852/50000 [49:29<21:24:44,  1.63s/it]
Outer training loop:   6%|5         | 2853/50000 [49:30<20:54:32,  1.60s/it]
Outer training loop:   6%|5         | 2854/50000 [49:32<20:40:10,  1.58s/it]
Outer training loop:   6%|5         | 2855/50000 [49:33<20:53:17,  1.60s/it]
Outer training loop:   6%|5         | 2856/50000 [49:35<20:39:28,  1.58s/it]
Outer training loop:   6%|5         | 2857/50000 [49:37<20:46:14,  1.59s/it]
Outer training loop:   6%|5         | 2858/50000 [49:38<20:30:57,  1.57s/it]
Outer training loop:   6%|5         | 2859/50000 [49:40<21:00:03,  1.60s/it]
Outer training loop:   6%|5         | 2860/50000 [49:41<20:28:07,  1.56s/it]
Outer training loop:   6%|5         | 2861/50000 [49:43<20:53:31,  1.60s/it]
Outer training loop:   6%|5         | 2862/50000 [49:44<20:36:56,  1.57s/it]
Outer training loop:   6%|5         | 2863/50000 [49:46<20:54:41,  1.60s/it]
Outer training loop:   6%|5         | 2864/50000 [49:48<20:27:18,  1.56s/it]
Outer training loop:   6%|5         | 2865/50000 [49:49<20:40:48,  1.58s/it]
Outer training loop:   6%|5         | 2866/50000 [49:51<20:25:26,  1.56s/it]
Outer training loop:   6%|5         | 2867/50000 [49:52<20:19:19,  1.55s/it]
Outer training loop:   6%|5         | 2868/50000 [49:54<20:09:11,  1.54s/it]
Outer training loop:   6%|5         | 2869/50000 [49:55<20:36:11,  1.57s/it]
Outer training loop:   6%|5         | 2870/50000 [49:57<20:27:50,  1.56s/it]
Outer training loop:   6%|5         | 2871/50000 [49:59<20:55:09,  1.60s/it]
Outer training loop:   6%|5         | 2872/50000 [50:00<20:34:26,  1.57s/it]
Outer training loop:   6%|5         | 2873/50000 [50:02<20:50:18,  1.59s/it]
Outer training loop:   6%|5         | 2874/50000 [50:03<20:35:04,  1.57s/it]
Outer training loop:   6%|5         | 2875/50000 [50:05<20:50:39,  1.59s/it]
Outer training loop:   6%|5         | 2876/50000 [50:06<20:29:40,  1.57s/it]
Outer training loop:   6%|5         | 2877/50000 [50:08<20:40:48,  1.58s/it]
Outer training loop:   6%|5         | 2878/50000 [50:10<20:54:28,  1.60s/it]
Outer training loop:   6%|5         | 2879/50000 [50:11<20:41:31,  1.58s/it]
Outer training loop:   6%|5         | 2880/50000 [50:13<20:18:17,  1.55s/it]
Outer training loop:   6%|5         | 2881/50000 [50:14<20:10:00,  1.54s/it]
Outer training loop:   6%|5         | 2882/50000 [50:16<20:14:00,  1.55s/it]
Outer training loop:   6%|5         | 2883/50000 [50:17<20:22:21,  1.56s/it]
Outer training loop:   6%|5         | 2884/50000 [50:19<21:18:03,  1.63s/it]
Outer training loop:   6%|5         | 2885/50000 [50:21<20:57:36,  1.60s/it]
Outer training loop:   6%|5         | 2886/50000 [50:22<20:50:56,  1.59s/it]
Outer training loop:   6%|5         | 2887/50000 [50:24<20:52:49,  1.60s/it]
Outer training loop:   6%|5         | 2888/50000 [50:26<20:58:58,  1.60s/it]
Outer training loop:   6%|5         | 2889/50000 [50:27<20:51:32,  1.59s/it]
Outer training loop:   6%|5         | 2890/50000 [50:29<20:50:35,  1.59s/it]
Outer training loop:   6%|5         | 2891/50000 [50:30<20:44:36,  1.59s/it]
Outer training loop:   6%|5         | 2892/50000 [50:32<20:47:34,  1.59s/it]
Outer training loop:   6%|5         | 2893/50000 [50:33<20:38:53,  1.58s/it]
Outer training loop:   6%|5         | 2894/50000 [50:35<20:57:40,  1.60s/it]
Outer training loop:   6%|5         | 2895/50000 [50:37<20:45:08,  1.59s/it]
Outer training loop:   6%|5         | 2896/50000 [50:38<20:52:31,  1.60s/it]
Outer training loop:   6%|5         | 2897/50000 [50:40<20:33:19,  1.57s/it]
Outer training loop:   6%|5         | 2898/50000 [50:41<20:28:00,  1.56s/it]
Outer training loop:   6%|5         | 2899/50000 [50:43<20:27:45,  1.56s/it]
Outer training loop:   6%|5         | 2900/50000 [50:45<20:40:35,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.59s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.57s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.56s/it]

Outer training loop:   6%|5         | 2901/50000 [50:51<38:58:29,  2.98s/it]
Outer training loop:   6%|5         | 2902/50000 [50:52<33:08:50,  2.53s/it]
Outer training loop:   6%|5         | 2903/50000 [50:54<29:28:21,  2.25s/it]
Outer training loop:   6%|5         | 2904/50000 [50:55<26:50:39,  2.05s/it]
Outer training loop:   6%|5         | 2905/50000 [50:57<24:50:52,  1.90s/it]
Outer training loop:   6%|5         | 2906/50000 [50:58<23:22:37,  1.79s/it]
Outer training loop:   6%|5         | 2907/50000 [51:00<22:51:10,  1.75s/it]
Outer training loop:   6%|5         | 2908/50000 [51:02<21:54:00,  1.67s/it]
Outer training loop:   6%|5         | 2909/50000 [51:03<21:37:02,  1.65s/it]
Outer training loop:   6%|5         | 2910/50000 [51:05<21:33:13,  1.65s/it]
Outer training loop:   6%|5         | 2911/50000 [51:07<21:33:38,  1.65s/it]
Outer training loop:   6%|5         | 2912/50000 [51:08<21:30:34,  1.64s/it]
Outer training loop:   6%|5         | 2913/50000 [51:10<21:04:24,  1.61s/it]
Outer training loop:   6%|5         | 2914/50000 [51:11<20:39:14,  1.58s/it]
Outer training loop:   6%|5         | 2915/50000 [51:13<20:52:16,  1.60s/it]
Outer training loop:   6%|5         | 2916/50000 [51:14<20:31:19,  1.57s/it]
Outer training loop:   6%|5         | 2917/50000 [51:16<20:32:19,  1.57s/it]
Outer training loop:   6%|5         | 2918/50000 [51:17<20:25:43,  1.56s/it]
Outer training loop:   6%|5         | 2919/50000 [51:19<20:37:39,  1.58s/it]
Outer training loop:   6%|5         | 2920/50000 [51:21<20:17:56,  1.55s/it]
Outer training loop:   6%|5         | 2921/50000 [51:22<20:28:32,  1.57s/it]
Outer training loop:   6%|5         | 2922/50000 [51:24<20:19:09,  1.55s/it]
Outer training loop:   6%|5         | 2923/50000 [51:25<20:26:01,  1.56s/it]
Outer training loop:   6%|5         | 2924/50000 [51:27<20:20:13,  1.56s/it]
Outer training loop:   6%|5         | 2925/50000 [51:28<20:22:08,  1.56s/it]
Outer training loop:   6%|5         | 2926/50000 [51:30<20:25:53,  1.56s/it]
Outer training loop:   6%|5         | 2927/50000 [51:32<20:24:30,  1.56s/it]
Outer training loop:   6%|5         | 2928/50000 [51:33<20:41:36,  1.58s/it]
Outer training loop:   6%|5         | 2929/50000 [51:35<20:53:26,  1.60s/it]
Outer training loop:   6%|5         | 2930/50000 [51:36<20:48:49,  1.59s/it]
Outer training loop:   6%|5         | 2931/50000 [51:38<21:33:35,  1.65s/it]
Outer training loop:   6%|5         | 2932/50000 [51:40<21:41:03,  1.66s/it]
Outer training loop:   6%|5         | 2933/50000 [51:41<21:08:47,  1.62s/it]
Outer training loop:   6%|5         | 2934/50000 [51:43<20:39:32,  1.58s/it]
Outer training loop:   6%|5         | 2935/50000 [51:44<20:56:00,  1.60s/it]
Outer training loop:   6%|5         | 2936/50000 [51:46<20:54:10,  1.60s/it]
Outer training loop:   6%|5         | 2937/50000 [51:48<20:43:35,  1.59s/it]
Outer training loop:   6%|5         | 2938/50000 [51:49<20:55:44,  1.60s/it]
Outer training loop:   6%|5         | 2939/50000 [51:51<20:34:49,  1.57s/it]
Outer training loop:   6%|5         | 2940/50000 [51:52<19:59:34,  1.53s/it]
Outer training loop:   6%|5         | 2941/50000 [51:54<20:25:35,  1.56s/it]
Outer training loop:   6%|5         | 2942/50000 [51:55<20:16:35,  1.55s/it]
Outer training loop:   6%|5         | 2943/50000 [51:57<20:29:13,  1.57s/it]
Outer training loop:   6%|5         | 2944/50000 [51:58<20:12:15,  1.55s/it]
Outer training loop:   6%|5         | 2945/50000 [52:00<20:23:30,  1.56s/it]
Outer training loop:   6%|5         | 2946/50000 [52:02<20:17:35,  1.55s/it]
Outer training loop:   6%|5         | 2947/50000 [52:03<20:16:41,  1.55s/it]
Outer training loop:   6%|5         | 2948/50000 [52:05<20:09:35,  1.54s/it]
Outer training loop:   6%|5         | 2949/50000 [52:06<20:36:57,  1.58s/it]
Outer training loop:   6%|5         | 2950/50000 [52:08<20:06:19,  1.54s/it]
Outer training loop:   6%|5         | 2951/50000 [52:09<20:06:31,  1.54s/it]
Outer training loop:   6%|5         | 2952/50000 [52:11<20:17:17,  1.55s/it]
Outer training loop:   6%|5         | 2953/50000 [52:12<20:23:08,  1.56s/it]
Outer training loop:   6%|5         | 2954/50000 [52:14<20:33:05,  1.57s/it]
Outer training loop:   6%|5         | 2955/50000 [52:16<20:50:06,  1.59s/it]
Outer training loop:   6%|5         | 2956/50000 [52:17<20:26:29,  1.56s/it]
Outer training loop:   6%|5         | 2957/50000 [52:19<20:24:32,  1.56s/it]
Outer training loop:   6%|5         | 2958/50000 [52:20<20:29:42,  1.57s/it]
Outer training loop:   6%|5         | 2959/50000 [52:22<20:52:32,  1.60s/it]
Outer training loop:   6%|5         | 2960/50000 [52:23<20:17:02,  1.55s/it]
Outer training loop:   6%|5         | 2961/50000 [52:25<20:40:01,  1.58s/it]
Outer training loop:   6%|5         | 2962/50000 [52:27<20:30:25,  1.57s/it]
Outer training loop:   6%|5         | 2963/50000 [52:28<20:10:43,  1.54s/it]
Outer training loop:   6%|5         | 2964/50000 [52:30<20:21:16,  1.56s/it]
Outer training loop:   6%|5         | 2965/50000 [52:31<20:07:25,  1.54s/it]
Outer training loop:   6%|5         | 2966/50000 [52:33<20:26:30,  1.56s/it]
Outer training loop:   6%|5         | 2967/50000 [52:34<20:00:39,  1.53s/it]
Outer training loop:   6%|5         | 2968/50000 [52:36<20:29:59,  1.57s/it]
Outer training loop:   6%|5         | 2969/50000 [52:38<20:35:40,  1.58s/it]
Outer training loop:   6%|5         | 2970/50000 [52:39<20:47:35,  1.59s/it]
Outer training loop:   6%|5         | 2971/50000 [52:41<20:14:24,  1.55s/it]
Outer training loop:   6%|5         | 2972/50000 [52:42<20:39:06,  1.58s/it]
Outer training loop:   6%|5         | 2973/50000 [52:44<20:39:23,  1.58s/it]
Outer training loop:   6%|5         | 2974/50000 [52:46<21:03:15,  1.61s/it]
Outer training loop:   6%|5         | 2975/50000 [52:47<20:47:39,  1.59s/it]
Outer training loop:   6%|5         | 2976/50000 [52:49<21:04:43,  1.61s/it]
Outer training loop:   6%|5         | 2977/50000 [52:50<20:40:12,  1.58s/it]
Outer training loop:   6%|5         | 2978/50000 [52:52<21:13:44,  1.63s/it]
Outer training loop:   6%|5         | 2979/50000 [52:54<20:44:27,  1.59s/it]
Outer training loop:   6%|5         | 2980/50000 [52:55<20:59:15,  1.61s/it]
Outer training loop:   6%|5         | 2981/50000 [52:57<21:42:53,  1.66s/it]
Outer training loop:   6%|5         | 2982/50000 [52:58<20:46:16,  1.59s/it]
Outer training loop:   6%|5         | 2983/50000 [53:00<20:56:05,  1.60s/it]
Outer training loop:   6%|5         | 2984/50000 [53:02<20:36:32,  1.58s/it]
Outer training loop:   6%|5         | 2985/50000 [53:03<20:31:36,  1.57s/it]
Outer training loop:   6%|5         | 2986/50000 [53:05<20:38:17,  1.58s/it]
Outer training loop:   6%|5         | 2987/50000 [53:06<20:23:12,  1.56s/it]
Outer training loop:   6%|5         | 2988/50000 [53:08<20:41:47,  1.58s/it]
Outer training loop:   6%|5         | 2989/50000 [53:09<20:54:10,  1.60s/it]
Outer training loop:   6%|5         | 2990/50000 [53:11<20:36:00,  1.58s/it]
Outer training loop:   6%|5         | 2991/50000 [53:12<20:15:05,  1.55s/it]
Outer training loop:   6%|5         | 2992/50000 [53:14<20:42:08,  1.59s/it]
Outer training loop:   6%|5         | 2993/50000 [53:16<20:25:37,  1.56s/it]
Outer training loop:   6%|5         | 2994/50000 [53:17<20:32:17,  1.57s/it]
Outer training loop:   6%|5         | 2995/50000 [53:19<20:27:49,  1.57s/it]
Outer training loop:   6%|5         | 2996/50000 [53:20<20:38:40,  1.58s/it]
Outer training loop:   6%|5         | 2997/50000 [53:22<20:19:02,  1.56s/it]
Outer training loop:   6%|5         | 2998/50000 [53:24<20:23:08,  1.56s/it]
Outer training loop:   6%|5         | 2999/50000 [53:25<20:32:15,  1.57s/it]
Outer training loop:   6%|6         | 3000/50000 [53:27<20:46:07,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.54s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.52s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.54s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:07,  1.56s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:07,  1.58s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:04,  1.56s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:04,  1.58s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:02,  1.58s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<01:59,  1.55s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<01:59,  1.57s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:56,  1.55s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:57,  1.59s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:55,  1.58s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:53,  1.58s/it]

Evaluating policy:  14%|#4        | 12/83 [00:19<01:55,  1.63s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:51,  1.59s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:49,  1.59s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:50,  1.62s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:46,  1.59s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:43,  1.57s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:43,  1.59s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:41,  1.59s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:40,  1.60s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:42,  1.65s/it]

Evaluating policy:  27%|##6       | 22/83 [00:35<01:38,  1.62s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:37,  1.62s/it]

Evaluating policy:  29%|##8       | 24/83 [00:38<01:34,  1.60s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:34,  1.64s/it]

Evaluating policy:  31%|###1      | 26/83 [00:41<01:31,  1.60s/it]

Evaluating policy:  33%|###2      | 27/83 [00:43<01:33,  1.66s/it]

Evaluating policy:  34%|###3      | 28/83 [00:44<01:30,  1.64s/it]

Evaluating policy:  35%|###4      | 29/83 [00:46<01:30,  1.68s/it]

Evaluating policy:  36%|###6      | 30/83 [00:48<01:27,  1.65s/it]

Evaluating policy:  37%|###7      | 31/83 [00:49<01:25,  1.64s/it]

Evaluating policy:  39%|###8      | 32/83 [00:51<01:22,  1.62s/it]

Evaluating policy:  40%|###9      | 33/83 [00:53<01:24,  1.69s/it]

Evaluating policy:  41%|####      | 34/83 [00:54<01:20,  1.63s/it]

Evaluating policy:  42%|####2     | 35/83 [00:56<01:16,  1.59s/it]

Evaluating policy:  43%|####3     | 36/83 [00:57<01:14,  1.59s/it]

Evaluating policy:  45%|####4     | 37/83 [00:59<01:16,  1.66s/it]

Evaluating policy:  46%|####5     | 38/83 [01:01<01:13,  1.64s/it]

Evaluating policy:  47%|####6     | 39/83 [01:02<01:12,  1.64s/it]

Evaluating policy:  48%|####8     | 40/83 [01:04<01:07,  1.58s/it]

Evaluating policy:  49%|####9     | 41/83 [01:05<01:05,  1.57s/it]

Evaluating policy:  51%|#####     | 42/83 [01:07<01:03,  1.55s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:09<01:04,  1.61s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:10<01:01,  1.59s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:12<01:00,  1.59s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:13<00:58,  1.57s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:15<00:56,  1.56s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:16<00:54,  1.56s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:18<00:55,  1.64s/it]

Evaluating policy:  60%|######    | 50/83 [01:20<00:52,  1.60s/it]

Evaluating policy:  61%|######1   | 51/83 [01:21<00:50,  1.58s/it]

Evaluating policy:  63%|######2   | 52/83 [01:23<00:48,  1.55s/it]

Evaluating policy:  64%|######3   | 53/83 [01:24<00:46,  1.55s/it]

Evaluating policy:  65%|######5   | 54/83 [01:26<00:44,  1.52s/it]

Evaluating policy:  66%|######6   | 55/83 [01:28<00:45,  1.61s/it]

Evaluating policy:  67%|######7   | 56/83 [01:29<00:43,  1.60s/it]

Evaluating policy:  69%|######8   | 57/83 [01:31<00:41,  1.59s/it]

Evaluating policy:  70%|######9   | 58/83 [01:32<00:39,  1.59s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:34<00:37,  1.54s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:35<00:35,  1.53s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:37<00:35,  1.64s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:39<00:33,  1.61s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:40<00:31,  1.58s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:42<00:29,  1.56s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:43<00:27,  1.55s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:45<00:26,  1.53s/it]

Evaluating policy:  81%|########  | 67/83 [01:46<00:24,  1.54s/it]

Evaluating policy:  82%|########1 | 68/83 [01:48<00:24,  1.66s/it]

Evaluating policy:  83%|########3 | 69/83 [01:50<00:22,  1.62s/it]

Evaluating policy:  84%|########4 | 70/83 [01:51<00:20,  1.61s/it]

Evaluating policy:  86%|########5 | 71/83 [01:53<00:18,  1.58s/it]

Evaluating policy:  87%|########6 | 72/83 [01:54<00:17,  1.57s/it]

Evaluating policy:  88%|########7 | 73/83 [01:56<00:15,  1.55s/it]

Evaluating policy:  89%|########9 | 74/83 [01:57<00:13,  1.54s/it]

Evaluating policy:  90%|######### | 75/83 [01:59<00:12,  1.56s/it]

Evaluating policy:  92%|#########1| 76/83 [02:01<00:11,  1.71s/it]

Evaluating policy:  93%|#########2| 77/83 [02:03<00:10,  1.72s/it]

Evaluating policy:  94%|#########3| 78/83 [02:04<00:08,  1.66s/it]

Evaluating policy:  95%|#########5| 79/83 [02:06<00:06,  1.65s/it]

Evaluating policy:  96%|#########6| 80/83 [02:08<00:04,  1.64s/it]

Evaluating policy:  98%|#########7| 81/83 [02:09<00:03,  1.59s/it]

Evaluating policy:  99%|#########8| 82/83 [02:11<00:01,  1.58s/it]

Evaluating policy: 100%|##########| 83/83 [02:12<00:00,  1.56s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:48,  2.06s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:33,  1.90s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:24,  1.80s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:16,  1.73s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:11,  1.69s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:08,  1.67s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<02:05,  1.65s/it]

Evaluating policy:  10%|9         | 8/83 [00:13<02:04,  1.65s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:58,  1.61s/it]

Evaluating policy:  12%|#2        | 10/83 [00:16<01:57,  1.61s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:53,  1.57s/it]

Evaluating policy:  14%|#4        | 12/83 [00:19<01:52,  1.58s/it]

Evaluating policy:  16%|#5        | 13/83 [00:21<01:52,  1.61s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:49,  1.58s/it]

Evaluating policy:  18%|#8        | 15/83 [00:24<01:50,  1.62s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:47,  1.61s/it]

Evaluating policy:  20%|##        | 17/83 [00:27<01:44,  1.58s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:41,  1.56s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:41,  1.59s/it]

Evaluating policy:  24%|##4       | 20/83 [00:32<01:39,  1.58s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:37,  1.57s/it]

Evaluating policy:  27%|##6       | 22/83 [00:35<01:38,  1.61s/it]

Evaluating policy:  28%|##7       | 23/83 [00:37<01:36,  1.61s/it]

Evaluating policy:  29%|##8       | 24/83 [00:38<01:33,  1.58s/it]

Evaluating policy:  30%|###       | 25/83 [00:40<01:31,  1.58s/it]

Evaluating policy:  31%|###1      | 26/83 [00:41<01:32,  1.63s/it]

Evaluating policy:  33%|###2      | 27/83 [00:43<01:28,  1.59s/it]

Evaluating policy:  34%|###3      | 28/83 [00:44<01:27,  1.58s/it]

Evaluating policy:  35%|###4      | 29/83 [00:46<01:24,  1.56s/it]

Evaluating policy:  36%|###6      | 30/83 [00:48<01:26,  1.64s/it]

Evaluating policy:  37%|###7      | 31/83 [00:49<01:23,  1.61s/it]

Evaluating policy:  39%|###8      | 32/83 [00:51<01:21,  1.59s/it]

Evaluating policy:  40%|###9      | 33/83 [00:52<01:18,  1.56s/it]

Evaluating policy:  41%|####      | 34/83 [00:54<01:20,  1.65s/it]

Evaluating policy:  42%|####2     | 35/83 [00:56<01:17,  1.61s/it]

Evaluating policy:  43%|####3     | 36/83 [00:57<01:15,  1.60s/it]

Evaluating policy:  45%|####4     | 37/83 [00:59<01:12,  1.57s/it]

Evaluating policy:  46%|####5     | 38/83 [01:00<01:10,  1.56s/it]

Evaluating policy:  47%|####6     | 39/83 [01:02<01:11,  1.62s/it]

Evaluating policy:  48%|####8     | 40/83 [01:04<01:09,  1.61s/it]

Evaluating policy:  49%|####9     | 41/83 [01:05<01:06,  1.58s/it]

Evaluating policy:  51%|#####     | 42/83 [01:07<01:04,  1.57s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:08<01:02,  1.57s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:10<01:06,  1.71s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:12<01:02,  1.65s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:13<00:59,  1.60s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:15<00:56,  1.58s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:16<00:54,  1.54s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:18<00:55,  1.65s/it]

Evaluating policy:  60%|######    | 50/83 [01:20<00:53,  1.62s/it]

Evaluating policy:  61%|######1   | 51/83 [01:21<00:51,  1.60s/it]

Evaluating policy:  63%|######2   | 52/83 [01:23<00:48,  1.57s/it]

Evaluating policy:  64%|######3   | 53/83 [01:24<00:46,  1.56s/it]

Evaluating policy:  65%|######5   | 54/83 [01:26<00:44,  1.54s/it]

Evaluating policy:  66%|######6   | 55/83 [01:28<00:45,  1.64s/it]

Evaluating policy:  67%|######7   | 56/83 [01:29<00:43,  1.63s/it]

Evaluating policy:  69%|######8   | 57/83 [01:31<00:41,  1.61s/it]

Evaluating policy:  70%|######9   | 58/83 [01:32<00:39,  1.59s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:34<00:37,  1.57s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:36<00:35,  1.56s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:37<00:36,  1.67s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:39<00:34,  1.63s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:40<00:31,  1.59s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:42<00:29,  1.56s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:43<00:27,  1.52s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:45<00:25,  1.52s/it]

Evaluating policy:  81%|########  | 67/83 [01:46<00:24,  1.54s/it]

Evaluating policy:  82%|########1 | 68/83 [01:48<00:24,  1.66s/it]

Evaluating policy:  83%|########3 | 69/83 [01:50<00:23,  1.66s/it]

Evaluating policy:  84%|########4 | 70/83 [01:52<00:20,  1.61s/it]

Evaluating policy:  86%|########5 | 71/83 [01:53<00:19,  1.59s/it]

Evaluating policy:  87%|########6 | 72/83 [01:55<00:17,  1.55s/it]

Evaluating policy:  88%|########7 | 73/83 [01:56<00:15,  1.54s/it]

Evaluating policy:  89%|########9 | 74/83 [01:58<00:13,  1.53s/it]

Evaluating policy:  90%|######### | 75/83 [02:00<00:13,  1.64s/it]

Evaluating policy:  92%|#########1| 76/83 [02:01<00:11,  1.60s/it]

Evaluating policy:  93%|#########2| 77/83 [02:03<00:09,  1.58s/it]

Evaluating policy:  94%|#########3| 78/83 [02:04<00:07,  1.56s/it]

Evaluating policy:  95%|#########5| 79/83 [02:06<00:06,  1.55s/it]

Evaluating policy:  96%|#########6| 80/83 [02:07<00:04,  1.54s/it]

Evaluating policy:  98%|#########7| 81/83 [02:09<00:03,  1.53s/it]

Evaluating policy:  99%|#########8| 82/83 [02:10<00:01,  1.51s/it]

Evaluating policy: 100%|##########| 83/83 [02:12<00:00,  1.65s/it]

Outer training loop:   6%|6         | 3001/50000 [57:58<1077:49:51, 82.56s/it]
Outer training loop:   6%|6         | 3002/50000 [58:00<760:35:56, 58.26s/it] 
Outer training loop:   6%|6         | 3003/50000 [58:01<538:30:27, 41.25s/it]
Outer training loop:   6%|6         | 3004/50000 [58:03<383:36:08, 29.38s/it]
Outer training loop:   6%|6         | 3005/50000 [58:05<274:31:52, 21.03s/it]
Outer training loop:   6%|6         | 3006/50000 [58:06<198:19:15, 15.19s/it]
Outer training loop:   6%|6         | 3007/50000 [58:08<144:39:38, 11.08s/it]
Outer training loop:   6%|6         | 3008/50000 [58:09<107:09:09,  8.21s/it]
Outer training loop:   6%|6         | 3009/50000 [58:11<82:53:21,  6.35s/it] 
Outer training loop:   6%|6         | 3010/50000 [58:13<64:09:36,  4.92s/it]
Outer training loop:   6%|6         | 3011/50000 [58:14<51:22:13,  3.94s/it]
Outer training loop:   6%|6         | 3012/50000 [58:16<42:44:21,  3.27s/it]
Outer training loop:   6%|6         | 3013/50000 [58:18<36:30:54,  2.80s/it]
Outer training loop:   6%|6         | 3014/50000 [58:19<31:24:50,  2.41s/it]
Outer training loop:   6%|6         | 3015/50000 [58:21<28:08:12,  2.16s/it]
Outer training loop:   6%|6         | 3016/50000 [58:23<26:12:08,  2.01s/it]
Outer training loop:   6%|6         | 3017/50000 [58:24<24:46:53,  1.90s/it]
Outer training loop:   6%|6         | 3018/50000 [58:26<24:07:36,  1.85s/it]
Outer training loop:   6%|6         | 3019/50000 [58:27<23:04:34,  1.77s/it]
Outer training loop:   6%|6         | 3020/50000 [58:29<22:09:00,  1.70s/it]
Outer training loop:   6%|6         | 3021/50000 [58:31<21:31:41,  1.65s/it]
Outer training loop:   6%|6         | 3022/50000 [58:32<21:11:37,  1.62s/it]
Outer training loop:   6%|6         | 3023/50000 [58:34<21:16:31,  1.63s/it]
Outer training loop:   6%|6         | 3024/50000 [58:35<21:04:47,  1.62s/it]
Outer training loop:   6%|6         | 3025/50000 [58:37<20:55:34,  1.60s/it]
Outer training loop:   6%|6         | 3026/50000 [58:38<20:26:47,  1.57s/it]
Outer training loop:   6%|6         | 3027/50000 [58:40<20:39:50,  1.58s/it]
Outer training loop:   6%|6         | 3028/50000 [58:42<20:42:27,  1.59s/it]
Outer training loop:   6%|6         | 3029/50000 [58:43<20:48:45,  1.60s/it]
Outer training loop:   6%|6         | 3030/50000 [58:45<20:52:33,  1.60s/it]
Outer training loop:   6%|6         | 3031/50000 [58:46<20:50:44,  1.60s/it]
Outer training loop:   6%|6         | 3032/50000 [58:48<20:31:15,  1.57s/it]
Outer training loop:   6%|6         | 3033/50000 [58:50<20:51:35,  1.60s/it]
Outer training loop:   6%|6         | 3034/50000 [58:51<20:26:52,  1.57s/it]
Outer training loop:   6%|6         | 3035/50000 [58:53<20:18:41,  1.56s/it]
Outer training loop:   6%|6         | 3036/50000 [58:54<19:55:22,  1.53s/it]
Outer training loop:   6%|6         | 3037/50000 [58:56<20:22:59,  1.56s/it]
Outer training loop:   6%|6         | 3038/50000 [58:57<20:15:17,  1.55s/it]
Outer training loop:   6%|6         | 3039/50000 [58:59<20:41:54,  1.59s/it]
Outer training loop:   6%|6         | 3040/50000 [59:00<20:31:15,  1.57s/it]
Outer training loop:   6%|6         | 3041/50000 [59:02<20:27:20,  1.57s/it]
Outer training loop:   6%|6         | 3042/50000 [59:04<20:26:43,  1.57s/it]
Outer training loop:   6%|6         | 3043/50000 [59:05<20:44:40,  1.59s/it]
Outer training loop:   6%|6         | 3044/50000 [59:07<20:44:05,  1.59s/it]
Outer training loop:   6%|6         | 3045/50000 [59:08<20:41:33,  1.59s/it]
Outer training loop:   6%|6         | 3046/50000 [59:10<20:26:53,  1.57s/it]
Outer training loop:   6%|6         | 3047/50000 [59:12<20:34:06,  1.58s/it]
Outer training loop:   6%|6         | 3048/50000 [59:13<20:14:30,  1.55s/it]
Outer training loop:   6%|6         | 3049/50000 [59:15<20:17:39,  1.56s/it]
Outer training loop:   6%|6         | 3050/50000 [59:16<20:11:42,  1.55s/it]
Outer training loop:   6%|6         | 3051/50000 [59:18<20:29:45,  1.57s/it]
Outer training loop:   6%|6         | 3052/50000 [59:19<20:36:36,  1.58s/it]
Outer training loop:   6%|6         | 3053/50000 [59:21<20:39:42,  1.58s/it]
Outer training loop:   6%|6         | 3054/50000 [59:22<20:28:25,  1.57s/it]
Outer training loop:   6%|6         | 3055/50000 [59:24<20:09:21,  1.55s/it]
Outer training loop:   6%|6         | 3056/50000 [59:26<20:16:17,  1.55s/it]
Outer training loop:   6%|6         | 3057/50000 [59:27<20:01:29,  1.54s/it]
Outer training loop:   6%|6         | 3058/50000 [59:29<19:55:32,  1.53s/it]
Outer training loop:   6%|6         | 3059/50000 [59:30<19:41:50,  1.51s/it]
Outer training loop:   6%|6         | 3060/50000 [59:32<20:00:41,  1.53s/it]
Outer training loop:   6%|6         | 3061/50000 [59:33<20:27:34,  1.57s/it]
Outer training loop:   6%|6         | 3062/50000 [59:35<21:09:39,  1.62s/it]
Outer training loop:   6%|6         | 3063/50000 [59:37<20:56:42,  1.61s/it]
Outer training loop:   6%|6         | 3064/50000 [59:38<20:46:34,  1.59s/it]
Outer training loop:   6%|6         | 3065/50000 [59:40<20:30:23,  1.57s/it]
Outer training loop:   6%|6         | 3066/50000 [59:41<20:45:36,  1.59s/it]
Outer training loop:   6%|6         | 3067/50000 [59:43<21:09:23,  1.62s/it]
Outer training loop:   6%|6         | 3068/50000 [59:44<20:36:34,  1.58s/it]
Outer training loop:   6%|6         | 3069/50000 [59:46<20:59:37,  1.61s/it]
Outer training loop:   6%|6         | 3070/50000 [59:48<20:30:02,  1.57s/it]
Outer training loop:   6%|6         | 3071/50000 [59:49<20:18:08,  1.56s/it]
Outer training loop:   6%|6         | 3072/50000 [59:51<20:37:56,  1.58s/it]
Outer training loop:   6%|6         | 3073/50000 [59:52<20:19:03,  1.56s/it]
Outer training loop:   6%|6         | 3074/50000 [59:54<20:38:57,  1.58s/it]
Outer training loop:   6%|6         | 3075/50000 [59:56<20:57:14,  1.61s/it]
Outer training loop:   6%|6         | 3076/50000 [59:57<20:14:59,  1.55s/it]
Outer training loop:   6%|6         | 3077/50000 [59:59<20:24:36,  1.57s/it]
Outer training loop:   6%|6         | 3078/50000 [1:00:00<20:14:08,  1.55s/it]
Outer training loop:   6%|6         | 3079/50000 [1:00:02<20:01:41,  1.54s/it]
Outer training loop:   6%|6         | 3080/50000 [1:00:03<20:20:48,  1.56s/it]
Outer training loop:   6%|6         | 3081/50000 [1:00:05<20:13:31,  1.55s/it]
Outer training loop:   6%|6         | 3082/50000 [1:00:06<20:11:45,  1.55s/it]
Outer training loop:   6%|6         | 3083/50000 [1:00:08<20:16:04,  1.56s/it]
Outer training loop:   6%|6         | 3084/50000 [1:00:09<20:09:40,  1.55s/it]
Outer training loop:   6%|6         | 3085/50000 [1:00:11<20:31:04,  1.57s/it]
Outer training loop:   6%|6         | 3086/50000 [1:00:13<20:07:36,  1.54s/it]
Outer training loop:   6%|6         | 3087/50000 [1:00:14<19:59:18,  1.53s/it]
Outer training loop:   6%|6         | 3088/50000 [1:00:16<20:14:06,  1.55s/it]
Outer training loop:   6%|6         | 3089/50000 [1:00:17<19:49:05,  1.52s/it]
Outer training loop:   6%|6         | 3090/50000 [1:00:19<20:28:07,  1.57s/it]
Outer training loop:   6%|6         | 3091/50000 [1:00:20<20:00:17,  1.54s/it]
Outer training loop:   6%|6         | 3092/50000 [1:00:22<20:10:52,  1.55s/it]
Outer training loop:   6%|6         | 3093/50000 [1:00:23<20:31:45,  1.58s/it]
Outer training loop:   6%|6         | 3094/50000 [1:00:25<20:45:24,  1.59s/it]
Outer training loop:   6%|6         | 3095/50000 [1:00:27<20:25:41,  1.57s/it]
Outer training loop:   6%|6         | 3096/50000 [1:00:28<20:34:28,  1.58s/it]
Outer training loop:   6%|6         | 3097/50000 [1:00:30<20:44:31,  1.59s/it]
Outer training loop:   6%|6         | 3098/50000 [1:00:31<20:38:30,  1.58s/it]
Outer training loop:   6%|6         | 3099/50000 [1:00:33<20:50:52,  1.60s/it]
Outer training loop:   6%|6         | 3100/50000 [1:00:35<20:31:52,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.44s/it]

Evaluating policy:  67%|######6   | 2/3 [00:02<00:01,  1.47s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.48s/it]

Outer training loop:   6%|6         | 3101/50000 [1:00:41<38:17:41,  2.94s/it]
Outer training loop:   6%|6         | 3102/50000 [1:00:42<32:35:29,  2.50s/it]
Outer training loop:   6%|6         | 3103/50000 [1:00:44<29:04:46,  2.23s/it]
Outer training loop:   6%|6         | 3104/50000 [1:00:45<26:10:39,  2.01s/it]
Outer training loop:   6%|6         | 3105/50000 [1:00:47<24:54:12,  1.91s/it]
Outer training loop:   6%|6         | 3106/50000 [1:00:49<23:38:01,  1.81s/it]
Outer training loop:   6%|6         | 3107/50000 [1:00:50<22:58:42,  1.76s/it]
Outer training loop:   6%|6         | 3108/50000 [1:00:52<21:48:45,  1.67s/it]
Outer training loop:   6%|6         | 3109/50000 [1:00:53<22:11:01,  1.70s/it]
Outer training loop:   6%|6         | 3110/50000 [1:00:55<21:39:15,  1.66s/it]
Outer training loop:   6%|6         | 3111/50000 [1:00:57<21:36:05,  1.66s/it]
Outer training loop:   6%|6         | 3112/50000 [1:00:58<20:55:27,  1.61s/it]
Outer training loop:   6%|6         | 3113/50000 [1:01:00<21:03:31,  1.62s/it]
Outer training loop:   6%|6         | 3114/50000 [1:01:01<20:49:21,  1.60s/it]
Outer training loop:   6%|6         | 3115/50000 [1:01:03<20:47:48,  1.60s/it]
Outer training loop:   6%|6         | 3116/50000 [1:01:04<20:43:10,  1.59s/it]
Outer training loop:   6%|6         | 3117/50000 [1:01:06<20:40:32,  1.59s/it]
Outer training loop:   6%|6         | 3118/50000 [1:01:08<20:23:01,  1.57s/it]
Outer training loop:   6%|6         | 3119/50000 [1:01:09<20:40:33,  1.59s/it]
Outer training loop:   6%|6         | 3120/50000 [1:01:11<20:37:41,  1.58s/it]
Outer training loop:   6%|6         | 3121/50000 [1:01:12<20:23:15,  1.57s/it]
Outer training loop:   6%|6         | 3122/50000 [1:01:14<20:38:17,  1.58s/it]
Outer training loop:   6%|6         | 3123/50000 [1:01:15<20:12:31,  1.55s/it]
Outer training loop:   6%|6         | 3124/50000 [1:01:17<20:21:24,  1.56s/it]
Outer training loop:   6%|6         | 3125/50000 [1:01:19<20:09:16,  1.55s/it]
Outer training loop:   6%|6         | 3126/50000 [1:01:20<20:37:01,  1.58s/it]
Outer training loop:   6%|6         | 3127/50000 [1:01:22<20:51:30,  1.60s/it]
Outer training loop:   6%|6         | 3128/50000 [1:01:23<20:22:13,  1.56s/it]
Outer training loop:   6%|6         | 3129/50000 [1:01:25<20:36:13,  1.58s/it]
Outer training loop:   6%|6         | 3130/50000 [1:01:26<20:17:30,  1.56s/it]
Outer training loop:   6%|6         | 3131/50000 [1:01:28<20:01:11,  1.54s/it]
Outer training loop:   6%|6         | 3132/50000 [1:01:29<20:06:45,  1.54s/it]
Outer training loop:   6%|6         | 3133/50000 [1:01:31<19:57:49,  1.53s/it]
Outer training loop:   6%|6         | 3134/50000 [1:01:33<20:27:13,  1.57s/it]
Outer training loop:   6%|6         | 3135/50000 [1:01:34<20:27:34,  1.57s/it]
Outer training loop:   6%|6         | 3136/50000 [1:01:36<20:22:38,  1.57s/it]
Outer training loop:   6%|6         | 3137/50000 [1:01:37<20:05:56,  1.54s/it]
Outer training loop:   6%|6         | 3138/50000 [1:01:39<20:23:30,  1.57s/it]
Outer training loop:   6%|6         | 3139/50000 [1:01:40<20:09:12,  1.55s/it]
Outer training loop:   6%|6         | 3140/50000 [1:01:42<20:08:05,  1.55s/it]
Outer training loop:   6%|6         | 3141/50000 [1:01:43<19:48:26,  1.52s/it]
Outer training loop:   6%|6         | 3142/50000 [1:01:45<20:10:27,  1.55s/it]
Outer training loop:   6%|6         | 3143/50000 [1:01:47<20:06:00,  1.54s/it]
Outer training loop:   6%|6         | 3144/50000 [1:01:48<20:28:08,  1.57s/it]
Outer training loop:   6%|6         | 3145/50000 [1:01:50<20:10:37,  1.55s/it]
Outer training loop:   6%|6         | 3146/50000 [1:01:51<20:11:02,  1.55s/it]
Outer training loop:   6%|6         | 3147/50000 [1:01:53<20:14:59,  1.56s/it]
Outer training loop:   6%|6         | 3148/50000 [1:01:54<20:43:40,  1.59s/it]
Outer training loop:   6%|6         | 3149/50000 [1:01:56<20:16:03,  1.56s/it]
Outer training loop:   6%|6         | 3150/50000 [1:01:58<20:29:37,  1.57s/it]
Outer training loop:   6%|6         | 3151/50000 [1:01:59<20:26:43,  1.57s/it]
Outer training loop:   6%|6         | 3152/50000 [1:02:01<20:40:20,  1.59s/it]
Outer training loop:   6%|6         | 3153/50000 [1:02:02<20:09:04,  1.55s/it]
Outer training loop:   6%|6         | 3154/50000 [1:02:04<20:34:12,  1.58s/it]
Outer training loop:   6%|6         | 3155/50000 [1:02:05<20:16:26,  1.56s/it]
Outer training loop:   6%|6         | 3156/50000 [1:02:07<20:31:18,  1.58s/it]
Outer training loop:   6%|6         | 3157/50000 [1:02:09<20:15:32,  1.56s/it]
Outer training loop:   6%|6         | 3158/50000 [1:02:10<20:39:25,  1.59s/it]
Outer training loop:   6%|6         | 3159/50000 [1:02:12<20:40:43,  1.59s/it]
Outer training loop:   6%|6         | 3160/50000 [1:02:13<20:26:15,  1.57s/it]
Outer training loop:   6%|6         | 3161/50000 [1:02:15<20:50:02,  1.60s/it]
Outer training loop:   6%|6         | 3162/50000 [1:02:17<20:45:09,  1.60s/it]
Outer training loop:   6%|6         | 3163/50000 [1:02:18<20:17:25,  1.56s/it]
Outer training loop:   6%|6         | 3164/50000 [1:02:20<20:44:15,  1.59s/it]
Outer training loop:   6%|6         | 3165/50000 [1:02:21<20:26:05,  1.57s/it]
Outer training loop:   6%|6         | 3166/50000 [1:02:23<20:52:57,  1.61s/it]
Outer training loop:   6%|6         | 3167/50000 [1:02:24<20:35:39,  1.58s/it]
Outer training loop:   6%|6         | 3168/50000 [1:02:26<20:18:49,  1.56s/it]
Outer training loop:   6%|6         | 3169/50000 [1:02:27<20:10:45,  1.55s/it]
Outer training loop:   6%|6         | 3170/50000 [1:02:29<19:52:28,  1.53s/it]
Outer training loop:   6%|6         | 3171/50000 [1:02:31<20:19:09,  1.56s/it]
Outer training loop:   6%|6         | 3172/50000 [1:02:32<19:57:00,  1.53s/it]
Outer training loop:   6%|6         | 3173/50000 [1:02:34<20:25:53,  1.57s/it]
Outer training loop:   6%|6         | 3174/50000 [1:02:35<19:59:35,  1.54s/it]
Outer training loop:   6%|6         | 3175/50000 [1:02:37<20:14:36,  1.56s/it]
Outer training loop:   6%|6         | 3176/50000 [1:02:38<20:07:39,  1.55s/it]
Outer training loop:   6%|6         | 3177/50000 [1:02:40<20:19:16,  1.56s/it]
Outer training loop:   6%|6         | 3178/50000 [1:02:41<20:05:45,  1.55s/it]
Outer training loop:   6%|6         | 3179/50000 [1:02:43<20:32:08,  1.58s/it]
Outer training loop:   6%|6         | 3180/50000 [1:02:45<20:03:31,  1.54s/it]
Outer training loop:   6%|6         | 3181/50000 [1:02:46<20:10:58,  1.55s/it]
Outer training loop:   6%|6         | 3182/50000 [1:02:48<20:03:56,  1.54s/it]
Outer training loop:   6%|6         | 3183/50000 [1:02:49<20:20:17,  1.56s/it]
Outer training loop:   6%|6         | 3184/50000 [1:02:51<20:32:58,  1.58s/it]
Outer training loop:   6%|6         | 3185/50000 [1:02:52<20:38:31,  1.59s/it]
Outer training loop:   6%|6         | 3186/50000 [1:02:54<20:09:25,  1.55s/it]
Outer training loop:   6%|6         | 3187/50000 [1:02:56<20:26:01,  1.57s/it]
Outer training loop:   6%|6         | 3188/50000 [1:02:57<20:27:19,  1.57s/it]
Outer training loop:   6%|6         | 3189/50000 [1:02:59<20:18:53,  1.56s/it]
Outer training loop:   6%|6         | 3190/50000 [1:03:00<20:00:59,  1.54s/it]
Outer training loop:   6%|6         | 3191/50000 [1:03:02<20:18:17,  1.56s/it]
Outer training loop:   6%|6         | 3192/50000 [1:03:03<20:05:52,  1.55s/it]
Outer training loop:   6%|6         | 3193/50000 [1:03:05<20:10:41,  1.55s/it]
Outer training loop:   6%|6         | 3194/50000 [1:03:06<19:59:56,  1.54s/it]
Outer training loop:   6%|6         | 3195/50000 [1:03:08<20:31:18,  1.58s/it]
Outer training loop:   6%|6         | 3196/50000 [1:03:10<20:43:40,  1.59s/it]
Outer training loop:   6%|6         | 3197/50000 [1:03:11<20:49:52,  1.60s/it]
Outer training loop:   6%|6         | 3198/50000 [1:03:13<20:18:29,  1.56s/it]
Outer training loop:   6%|6         | 3199/50000 [1:03:14<20:34:03,  1.58s/it]
Outer training loop:   6%|6         | 3200/50000 [1:03:16<20:19:07,  1.56s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.50s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.50s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.51s/it]

Outer training loop:   6%|6         | 3201/50000 [1:03:22<38:10:02,  2.94s/it]
Outer training loop:   6%|6         | 3202/50000 [1:03:23<32:17:40,  2.48s/it]
Outer training loop:   6%|6         | 3203/50000 [1:03:25<28:48:21,  2.22s/it]
Outer training loop:   6%|6         | 3204/50000 [1:03:27<25:58:43,  2.00s/it]
Outer training loop:   6%|6         | 3205/50000 [1:03:28<24:15:22,  1.87s/it]
Outer training loop:   6%|6         | 3206/50000 [1:03:30<23:21:00,  1.80s/it]
Outer training loop:   6%|6         | 3207/50000 [1:03:31<22:29:56,  1.73s/it]
Outer training loop:   6%|6         | 3208/50000 [1:03:33<22:10:08,  1.71s/it]
Outer training loop:   6%|6         | 3209/50000 [1:03:35<21:44:59,  1.67s/it]
Outer training loop:   6%|6         | 3210/50000 [1:03:36<21:10:21,  1.63s/it]
Outer training loop:   6%|6         | 3211/50000 [1:03:38<21:06:25,  1.62s/it]
Outer training loop:   6%|6         | 3212/50000 [1:03:39<20:39:04,  1.59s/it]
Outer training loop:   6%|6         | 3213/50000 [1:03:41<20:32:58,  1.58s/it]
Outer training loop:   6%|6         | 3214/50000 [1:03:42<20:22:15,  1.57s/it]
Outer training loop:   6%|6         | 3215/50000 [1:03:44<20:35:19,  1.58s/it]
Outer training loop:   6%|6         | 3216/50000 [1:03:45<20:27:36,  1.57s/it]
Outer training loop:   6%|6         | 3217/50000 [1:03:47<20:50:06,  1.60s/it]
Outer training loop:   6%|6         | 3218/50000 [1:03:49<20:39:07,  1.59s/it]
Outer training loop:   6%|6         | 3219/50000 [1:03:50<20:46:24,  1.60s/it]
Outer training loop:   6%|6         | 3220/50000 [1:03:52<20:41:52,  1.59s/it]
Outer training loop:   6%|6         | 3221/50000 [1:03:54<20:55:34,  1.61s/it]
Outer training loop:   6%|6         | 3222/50000 [1:03:55<20:28:58,  1.58s/it]
Outer training loop:   6%|6         | 3223/50000 [1:03:57<20:36:46,  1.59s/it]
Outer training loop:   6%|6         | 3224/50000 [1:03:58<20:24:38,  1.57s/it]
Outer training loop:   6%|6         | 3225/50000 [1:04:00<20:23:32,  1.57s/it]
Outer training loop:   6%|6         | 3226/50000 [1:04:01<19:59:38,  1.54s/it]
Outer training loop:   6%|6         | 3227/50000 [1:04:03<19:51:03,  1.53s/it]
Outer training loop:   6%|6         | 3228/50000 [1:04:04<20:03:06,  1.54s/it]
Outer training loop:   6%|6         | 3229/50000 [1:04:06<20:00:37,  1.54s/it]
Outer training loop:   6%|6         | 3230/50000 [1:04:07<20:22:51,  1.57s/it]
Outer training loop:   6%|6         | 3231/50000 [1:04:09<20:21:42,  1.57s/it]
Outer training loop:   6%|6         | 3232/50000 [1:04:11<20:16:02,  1.56s/it]
Outer training loop:   6%|6         | 3233/50000 [1:04:12<19:53:33,  1.53s/it]
Outer training loop:   6%|6         | 3234/50000 [1:04:14<20:15:27,  1.56s/it]
Outer training loop:   6%|6         | 3235/50000 [1:04:15<20:10:02,  1.55s/it]
Outer training loop:   6%|6         | 3236/50000 [1:04:17<20:13:20,  1.56s/it]
Outer training loop:   6%|6         | 3237/50000 [1:04:18<20:07:06,  1.55s/it]
Outer training loop:   6%|6         | 3238/50000 [1:04:20<20:17:12,  1.56s/it]
Outer training loop:   6%|6         | 3239/50000 [1:04:21<19:56:42,  1.54s/it]
Outer training loop:   6%|6         | 3240/50000 [1:04:23<20:10:42,  1.55s/it]
Outer training loop:   6%|6         | 3241/50000 [1:04:25<20:31:31,  1.58s/it]
Outer training loop:   6%|6         | 3242/50000 [1:04:26<20:46:34,  1.60s/it]
Outer training loop:   6%|6         | 3243/50000 [1:04:28<20:14:35,  1.56s/it]
Outer training loop:   6%|6         | 3244/50000 [1:04:29<20:11:40,  1.55s/it]
Outer training loop:   6%|6         | 3245/50000 [1:04:31<19:51:07,  1.53s/it]
Outer training loop:   6%|6         | 3246/50000 [1:04:32<20:07:17,  1.55s/it]
Outer training loop:   6%|6         | 3247/50000 [1:04:34<20:05:28,  1.55s/it]
Outer training loop:   6%|6         | 3248/50000 [1:04:36<20:38:19,  1.59s/it]
Outer training loop:   6%|6         | 3249/50000 [1:04:37<20:10:36,  1.55s/it]
Outer training loop:   6%|6         | 3250/50000 [1:04:39<20:33:09,  1.58s/it]
Outer training loop:   7%|6         | 3251/50000 [1:04:40<20:22:56,  1.57s/it]
Outer training loop:   7%|6         | 3252/50000 [1:04:42<20:31:59,  1.58s/it]
Outer training loop:   7%|6         | 3253/50000 [1:04:43<20:33:44,  1.58s/it]
Outer training loop:   7%|6         | 3254/50000 [1:04:45<19:54:30,  1.53s/it]
Outer training loop:   7%|6         | 3255/50000 [1:04:46<19:34:34,  1.51s/it]
Outer training loop:   7%|6         | 3256/50000 [1:04:48<20:27:18,  1.58s/it]
Outer training loop:   7%|6         | 3257/50000 [1:04:50<20:20:32,  1.57s/it]
Outer training loop:   7%|6         | 3258/50000 [1:04:51<20:14:50,  1.56s/it]
Outer training loop:   7%|6         | 3259/50000 [1:04:53<20:10:50,  1.55s/it]
Outer training loop:   7%|6         | 3260/50000 [1:04:54<20:27:59,  1.58s/it]
Outer training loop:   7%|6         | 3261/50000 [1:04:56<20:21:16,  1.57s/it]
Outer training loop:   7%|6         | 3262/50000 [1:04:57<20:24:15,  1.57s/it]
Outer training loop:   7%|6         | 3263/50000 [1:04:59<20:19:19,  1.57s/it]
Outer training loop:   7%|6         | 3264/50000 [1:05:01<20:32:45,  1.58s/it]
Outer training loop:   7%|6         | 3265/50000 [1:05:02<20:04:54,  1.55s/it]
Outer training loop:   7%|6         | 3266/50000 [1:05:04<20:45:11,  1.60s/it]
Outer training loop:   7%|6         | 3267/50000 [1:05:05<20:33:05,  1.58s/it]
Outer training loop:   7%|6         | 3268/50000 [1:05:07<20:43:55,  1.60s/it]
Outer training loop:   7%|6         | 3269/50000 [1:05:08<20:36:28,  1.59s/it]
Outer training loop:   7%|6         | 3270/50000 [1:05:10<20:30:06,  1.58s/it]
Outer training loop:   7%|6         | 3271/50000 [1:05:12<20:30:32,  1.58s/it]
Outer training loop:   7%|6         | 3272/50000 [1:05:13<20:29:04,  1.58s/it]
Outer training loop:   7%|6         | 3273/50000 [1:05:15<19:56:42,  1.54s/it]
Outer training loop:   7%|6         | 3274/50000 [1:05:16<20:10:39,  1.55s/it]
Outer training loop:   7%|6         | 3275/50000 [1:05:18<20:00:38,  1.54s/it]
Outer training loop:   7%|6         | 3276/50000 [1:05:19<20:15:08,  1.56s/it]
Outer training loop:   7%|6         | 3277/50000 [1:05:21<20:12:15,  1.56s/it]
Outer training loop:   7%|6         | 3278/50000 [1:05:23<20:29:19,  1.58s/it]
Outer training loop:   7%|6         | 3279/50000 [1:05:24<20:16:56,  1.56s/it]
Outer training loop:   7%|6         | 3280/50000 [1:05:26<20:19:42,  1.57s/it]
Outer training loop:   7%|6         | 3281/50000 [1:05:27<20:02:20,  1.54s/it]
Outer training loop:   7%|6         | 3282/50000 [1:05:29<20:05:45,  1.55s/it]
Outer training loop:   7%|6         | 3283/50000 [1:05:30<19:41:29,  1.52s/it]
Outer training loop:   7%|6         | 3284/50000 [1:05:32<20:02:31,  1.54s/it]
Outer training loop:   7%|6         | 3285/50000 [1:05:33<19:53:00,  1.53s/it]
Outer training loop:   7%|6         | 3286/50000 [1:05:35<20:31:07,  1.58s/it]
Outer training loop:   7%|6         | 3287/50000 [1:05:36<20:25:26,  1.57s/it]
Outer training loop:   7%|6         | 3288/50000 [1:05:38<20:15:20,  1.56s/it]
Outer training loop:   7%|6         | 3289/50000 [1:05:40<20:06:06,  1.55s/it]
Outer training loop:   7%|6         | 3290/50000 [1:05:41<20:14:30,  1.56s/it]
Outer training loop:   7%|6         | 3291/50000 [1:05:43<19:57:38,  1.54s/it]
Outer training loop:   7%|6         | 3292/50000 [1:05:44<20:11:35,  1.56s/it]
Outer training loop:   7%|6         | 3293/50000 [1:05:46<19:43:16,  1.52s/it]
Outer training loop:   7%|6         | 3294/50000 [1:05:47<20:00:27,  1.54s/it]
Outer training loop:   7%|6         | 3295/50000 [1:05:49<19:54:33,  1.53s/it]
Outer training loop:   7%|6         | 3296/50000 [1:05:50<20:23:24,  1.57s/it]
Outer training loop:   7%|6         | 3297/50000 [1:05:52<20:07:59,  1.55s/it]
Outer training loop:   7%|6         | 3298/50000 [1:05:53<20:14:14,  1.56s/it]
Outer training loop:   7%|6         | 3299/50000 [1:05:55<20:10:57,  1.56s/it]
Outer training loop:   7%|6         | 3300/50000 [1:05:57<20:44:26,  1.60s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.56s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.56s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.55s/it]

Outer training loop:   7%|6         | 3301/50000 [1:06:03<38:13:39,  2.95s/it]
Outer training loop:   7%|6         | 3302/50000 [1:06:04<32:24:13,  2.50s/it]
Outer training loop:   7%|6         | 3303/50000 [1:06:06<29:38:54,  2.29s/it]
Outer training loop:   7%|6         | 3304/50000 [1:06:08<26:24:11,  2.04s/it]
Outer training loop:   7%|6         | 3305/50000 [1:06:09<24:46:32,  1.91s/it]
Outer training loop:   7%|6         | 3306/50000 [1:06:11<23:25:52,  1.81s/it]
Outer training loop:   7%|6         | 3307/50000 [1:06:12<22:26:15,  1.73s/it]
Outer training loop:   7%|6         | 3308/50000 [1:06:14<21:17:53,  1.64s/it]
Outer training loop:   7%|6         | 3309/50000 [1:06:15<21:17:02,  1.64s/it]
Outer training loop:   7%|6         | 3310/50000 [1:06:17<20:44:29,  1.60s/it]
Outer training loop:   7%|6         | 3311/50000 [1:06:18<20:46:36,  1.60s/it]
Outer training loop:   7%|6         | 3312/50000 [1:06:20<20:25:36,  1.58s/it]
Outer training loop:   7%|6         | 3313/50000 [1:06:22<20:35:33,  1.59s/it]
Outer training loop:   7%|6         | 3314/50000 [1:06:23<20:03:11,  1.55s/it]
Outer training loop:   7%|6         | 3315/50000 [1:06:24<19:44:46,  1.52s/it]
Outer training loop:   7%|6         | 3316/50000 [1:06:26<19:47:39,  1.53s/it]
Outer training loop:   7%|6         | 3317/50000 [1:06:28<19:43:41,  1.52s/it]
Outer training loop:   7%|6         | 3318/50000 [1:06:29<19:52:01,  1.53s/it]
Outer training loop:   7%|6         | 3319/50000 [1:06:31<20:12:25,  1.56s/it]
Outer training loop:   7%|6         | 3320/50000 [1:06:32<20:00:00,  1.54s/it]
Outer training loop:   7%|6         | 3321/50000 [1:06:34<20:05:34,  1.55s/it]
Outer training loop:   7%|6         | 3322/50000 [1:06:35<20:19:02,  1.57s/it]
Outer training loop:   7%|6         | 3323/50000 [1:06:37<19:58:27,  1.54s/it]
Outer training loop:   7%|6         | 3324/50000 [1:06:38<20:20:40,  1.57s/it]
Outer training loop:   7%|6         | 3325/50000 [1:06:40<20:16:05,  1.56s/it]
Outer training loop:   7%|6         | 3326/50000 [1:06:42<20:41:14,  1.60s/it]
Outer training loop:   7%|6         | 3327/50000 [1:06:43<20:40:44,  1.60s/it]
Outer training loop:   7%|6         | 3328/50000 [1:06:45<20:34:32,  1.59s/it]
Outer training loop:   7%|6         | 3329/50000 [1:06:46<20:12:02,  1.56s/it]
Outer training loop:   7%|6         | 3330/50000 [1:06:48<20:19:02,  1.57s/it]
Outer training loop:   7%|6         | 3331/50000 [1:06:50<20:33:05,  1.59s/it]
Outer training loop:   7%|6         | 3332/50000 [1:06:51<20:04:37,  1.55s/it]
Outer training loop:   7%|6         | 3333/50000 [1:06:53<20:24:39,  1.57s/it]
Outer training loop:   7%|6         | 3334/50000 [1:06:54<19:57:17,  1.54s/it]
Outer training loop:   7%|6         | 3335/50000 [1:06:56<20:17:32,  1.57s/it]
Outer training loop:   7%|6         | 3336/50000 [1:06:57<20:04:41,  1.55s/it]
Outer training loop:   7%|6         | 3337/50000 [1:06:59<20:28:13,  1.58s/it]
Outer training loop:   7%|6         | 3338/50000 [1:07:00<20:08:38,  1.55s/it]
Outer training loop:   7%|6         | 3339/50000 [1:07:02<20:25:31,  1.58s/it]
Outer training loop:   7%|6         | 3340/50000 [1:07:04<20:01:40,  1.55s/it]
Outer training loop:   7%|6         | 3341/50000 [1:07:05<19:52:08,  1.53s/it]
Outer training loop:   7%|6         | 3342/50000 [1:07:07<20:04:24,  1.55s/it]
Outer training loop:   7%|6         | 3343/50000 [1:07:08<20:05:00,  1.55s/it]
Outer training loop:   7%|6         | 3344/50000 [1:07:10<20:16:48,  1.56s/it]
Outer training loop:   7%|6         | 3345/50000 [1:07:11<20:23:34,  1.57s/it]
Outer training loop:   7%|6         | 3346/50000 [1:07:13<20:44:57,  1.60s/it]
Outer training loop:   7%|6         | 3347/50000 [1:07:15<20:22:00,  1.57s/it]
Outer training loop:   7%|6         | 3348/50000 [1:07:16<20:15:28,  1.56s/it]
Outer training loop:   7%|6         | 3349/50000 [1:07:18<20:09:16,  1.56s/it]
Outer training loop:   7%|6         | 3350/50000 [1:07:19<20:12:08,  1.56s/it]
Outer training loop:   7%|6         | 3351/50000 [1:07:21<20:21:26,  1.57s/it]
Outer training loop:   7%|6         | 3352/50000 [1:07:22<20:37:53,  1.59s/it]
Outer training loop:   7%|6         | 3353/50000 [1:07:24<20:58:23,  1.62s/it]
Outer training loop:   7%|6         | 3354/50000 [1:07:26<20:35:22,  1.59s/it]
Outer training loop:   7%|6         | 3355/50000 [1:07:27<20:36:37,  1.59s/it]
Outer training loop:   7%|6         | 3356/50000 [1:07:29<20:56:15,  1.62s/it]
Outer training loop:   7%|6         | 3357/50000 [1:07:30<20:20:23,  1.57s/it]
Outer training loop:   7%|6         | 3358/50000 [1:07:32<20:37:34,  1.59s/it]
Outer training loop:   7%|6         | 3359/50000 [1:07:34<20:34:17,  1.59s/it]
Outer training loop:   7%|6         | 3360/50000 [1:07:35<20:39:35,  1.59s/it]
Outer training loop:   7%|6         | 3361/50000 [1:07:37<20:27:47,  1.58s/it]
Outer training loop:   7%|6         | 3362/50000 [1:07:38<20:28:34,  1.58s/it]
Outer training loop:   7%|6         | 3363/50000 [1:07:40<20:31:49,  1.58s/it]
Outer training loop:   7%|6         | 3364/50000 [1:07:42<20:43:08,  1.60s/it]
Outer training loop:   7%|6         | 3365/50000 [1:07:43<20:13:19,  1.56s/it]
Outer training loop:   7%|6         | 3366/50000 [1:07:45<20:25:13,  1.58s/it]
Outer training loop:   7%|6         | 3367/50000 [1:07:46<20:18:38,  1.57s/it]
Outer training loop:   7%|6         | 3368/50000 [1:07:48<20:41:35,  1.60s/it]
Outer training loop:   7%|6         | 3369/50000 [1:07:49<20:41:23,  1.60s/it]
Outer training loop:   7%|6         | 3370/50000 [1:07:51<20:43:27,  1.60s/it]
Outer training loop:   7%|6         | 3371/50000 [1:07:53<20:33:04,  1.59s/it]
Outer training loop:   7%|6         | 3372/50000 [1:07:54<20:34:28,  1.59s/it]
Outer training loop:   7%|6         | 3373/50000 [1:07:56<20:18:52,  1.57s/it]
Outer training loop:   7%|6         | 3374/50000 [1:07:57<20:21:56,  1.57s/it]
Outer training loop:   7%|6         | 3375/50000 [1:07:59<19:58:29,  1.54s/it]
Outer training loop:   7%|6         | 3376/50000 [1:08:00<19:55:22,  1.54s/it]
Outer training loop:   7%|6         | 3377/50000 [1:08:02<20:01:05,  1.55s/it]
Outer training loop:   7%|6         | 3378/50000 [1:08:04<20:31:08,  1.58s/it]
Outer training loop:   7%|6         | 3379/50000 [1:08:05<20:37:18,  1.59s/it]
Outer training loop:   7%|6         | 3380/50000 [1:08:07<20:13:23,  1.56s/it]
Outer training loop:   7%|6         | 3381/50000 [1:08:08<19:52:19,  1.53s/it]
Outer training loop:   7%|6         | 3382/50000 [1:08:10<19:59:25,  1.54s/it]
Outer training loop:   7%|6         | 3383/50000 [1:08:11<19:38:32,  1.52s/it]
Outer training loop:   7%|6         | 3384/50000 [1:08:13<19:52:43,  1.54s/it]
Outer training loop:   7%|6         | 3385/50000 [1:08:14<19:33:40,  1.51s/it]
Outer training loop:   7%|6         | 3386/50000 [1:08:16<19:26:45,  1.50s/it]
Outer training loop:   7%|6         | 3387/50000 [1:08:17<19:53:12,  1.54s/it]
Outer training loop:   7%|6         | 3388/50000 [1:08:19<19:51:42,  1.53s/it]
Outer training loop:   7%|6         | 3389/50000 [1:08:20<19:51:51,  1.53s/it]
Outer training loop:   7%|6         | 3390/50000 [1:08:22<19:53:09,  1.54s/it]
Outer training loop:   7%|6         | 3391/50000 [1:08:24<20:22:01,  1.57s/it]
Outer training loop:   7%|6         | 3392/50000 [1:08:25<20:39:23,  1.60s/it]
Outer training loop:   7%|6         | 3393/50000 [1:08:27<20:38:42,  1.59s/it]
Outer training loop:   7%|6         | 3394/50000 [1:08:28<20:37:55,  1.59s/it]
Outer training loop:   7%|6         | 3395/50000 [1:08:30<20:11:05,  1.56s/it]
Outer training loop:   7%|6         | 3396/50000 [1:08:31<19:47:18,  1.53s/it]
Outer training loop:   7%|6         | 3397/50000 [1:08:33<20:40:03,  1.60s/it]
Outer training loop:   7%|6         | 3398/50000 [1:08:35<20:35:09,  1.59s/it]
Outer training loop:   7%|6         | 3399/50000 [1:08:36<20:32:02,  1.59s/it]
Outer training loop:   7%|6         | 3400/50000 [1:08:38<20:48:47,  1.61s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.45s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.51s/it]

Outer training loop:   7%|6         | 3401/50000 [1:08:44<38:27:18,  2.97s/it]
Outer training loop:   7%|6         | 3402/50000 [1:08:46<33:05:43,  2.56s/it]
Outer training loop:   7%|6         | 3403/50000 [1:08:47<28:54:29,  2.23s/it]
Outer training loop:   7%|6         | 3404/50000 [1:08:49<26:56:11,  2.08s/it]
Outer training loop:   7%|6         | 3405/50000 [1:08:50<24:40:43,  1.91s/it]
Outer training loop:   7%|6         | 3406/50000 [1:08:52<23:26:01,  1.81s/it]
Outer training loop:   7%|6         | 3407/50000 [1:08:53<22:20:31,  1.73s/it]
Outer training loop:   7%|6         | 3408/50000 [1:08:55<21:50:58,  1.69s/it]
Outer training loop:   7%|6         | 3409/50000 [1:08:56<21:02:01,  1.63s/it]
Outer training loop:   7%|6         | 3410/50000 [1:08:58<20:42:52,  1.60s/it]
Outer training loop:   7%|6         | 3411/50000 [1:08:59<20:10:00,  1.56s/it]
Outer training loop:   7%|6         | 3412/50000 [1:09:01<20:07:56,  1.56s/it]
Outer training loop:   7%|6         | 3413/50000 [1:09:03<20:12:30,  1.56s/it]
Outer training loop:   7%|6         | 3414/50000 [1:09:04<20:17:38,  1.57s/it]
Outer training loop:   7%|6         | 3415/50000 [1:09:06<20:04:10,  1.55s/it]
Outer training loop:   7%|6         | 3416/50000 [1:09:07<20:25:13,  1.58s/it]
Outer training loop:   7%|6         | 3417/50000 [1:09:09<20:06:34,  1.55s/it]
Outer training loop:   7%|6         | 3418/50000 [1:09:10<20:20:21,  1.57s/it]
Outer training loop:   7%|6         | 3419/50000 [1:09:12<19:51:35,  1.53s/it]
Outer training loop:   7%|6         | 3420/50000 [1:09:13<19:56:11,  1.54s/it]
Outer training loop:   7%|6         | 3421/50000 [1:09:15<19:39:23,  1.52s/it]
Outer training loop:   7%|6         | 3422/50000 [1:09:16<19:40:49,  1.52s/it]
Outer training loop:   7%|6         | 3423/50000 [1:09:18<20:15:29,  1.57s/it]
Outer training loop:   7%|6         | 3424/50000 [1:09:20<20:26:42,  1.58s/it]
Outer training loop:   7%|6         | 3425/50000 [1:09:21<20:25:27,  1.58s/it]
Outer training loop:   7%|6         | 3426/50000 [1:09:23<20:10:50,  1.56s/it]
Outer training loop:   7%|6         | 3427/50000 [1:09:24<20:27:34,  1.58s/it]
Outer training loop:   7%|6         | 3428/50000 [1:09:26<20:19:51,  1.57s/it]
Outer training loop:   7%|6         | 3429/50000 [1:09:28<20:09:00,  1.56s/it]
Outer training loop:   7%|6         | 3430/50000 [1:09:29<19:51:19,  1.53s/it]
Outer training loop:   7%|6         | 3431/50000 [1:09:31<20:00:41,  1.55s/it]
Outer training loop:   7%|6         | 3432/50000 [1:09:32<19:41:23,  1.52s/it]
Outer training loop:   7%|6         | 3433/50000 [1:09:34<20:21:06,  1.57s/it]
Outer training loop:   7%|6         | 3434/50000 [1:09:35<20:01:46,  1.55s/it]
Outer training loop:   7%|6         | 3435/50000 [1:09:37<20:07:51,  1.56s/it]
Outer training loop:   7%|6         | 3436/50000 [1:09:38<19:56:47,  1.54s/it]
Outer training loop:   7%|6         | 3437/50000 [1:09:40<20:01:58,  1.55s/it]
Outer training loop:   7%|6         | 3438/50000 [1:09:41<20:01:24,  1.55s/it]
Outer training loop:   7%|6         | 3439/50000 [1:09:43<20:02:16,  1.55s/it]
Outer training loop:   7%|6         | 3440/50000 [1:09:45<19:56:16,  1.54s/it]
Outer training loop:   7%|6         | 3441/50000 [1:09:46<20:14:03,  1.56s/it]
Outer training loop:   7%|6         | 3442/50000 [1:09:48<20:13:13,  1.56s/it]
Outer training loop:   7%|6         | 3443/50000 [1:09:49<20:14:55,  1.57s/it]
Outer training loop:   7%|6         | 3444/50000 [1:09:51<20:02:01,  1.55s/it]
Outer training loop:   7%|6         | 3445/50000 [1:09:52<20:08:52,  1.56s/it]
Outer training loop:   7%|6         | 3446/50000 [1:09:54<19:43:03,  1.52s/it]
Outer training loop:   7%|6         | 3447/50000 [1:09:55<20:01:59,  1.55s/it]
Outer training loop:   7%|6         | 3448/50000 [1:09:57<19:51:24,  1.54s/it]
Outer training loop:   7%|6         | 3449/50000 [1:09:59<20:06:36,  1.56s/it]
Outer training loop:   7%|6         | 3450/50000 [1:10:00<20:39:01,  1.60s/it]
Outer training loop:   7%|6         | 3451/50000 [1:10:02<20:26:29,  1.58s/it]
Outer training loop:   7%|6         | 3452/50000 [1:10:03<20:06:30,  1.56s/it]
Outer training loop:   7%|6         | 3453/50000 [1:10:05<20:25:09,  1.58s/it]
Outer training loop:   7%|6         | 3454/50000 [1:10:06<20:08:39,  1.56s/it]
Outer training loop:   7%|6         | 3455/50000 [1:10:08<20:16:16,  1.57s/it]
Outer training loop:   7%|6         | 3456/50000 [1:10:10<20:07:58,  1.56s/it]
Outer training loop:   7%|6         | 3457/50000 [1:10:11<20:14:31,  1.57s/it]
Outer training loop:   7%|6         | 3458/50000 [1:10:13<20:17:35,  1.57s/it]
Outer training loop:   7%|6         | 3459/50000 [1:10:14<19:37:24,  1.52s/it]
Outer training loop:   7%|6         | 3460/50000 [1:10:16<19:22:02,  1.50s/it]
Outer training loop:   7%|6         | 3461/50000 [1:10:17<19:47:04,  1.53s/it]
Outer training loop:   7%|6         | 3462/50000 [1:10:19<19:53:16,  1.54s/it]
Outer training loop:   7%|6         | 3463/50000 [1:10:20<20:21:24,  1.57s/it]
Outer training loop:   7%|6         | 3464/50000 [1:10:22<20:16:44,  1.57s/it]
Outer training loop:   7%|6         | 3465/50000 [1:10:23<20:22:11,  1.58s/it]
Outer training loop:   7%|6         | 3466/50000 [1:10:25<20:32:09,  1.59s/it]
Outer training loop:   7%|6         | 3467/50000 [1:10:27<19:56:22,  1.54s/it]
Outer training loop:   7%|6         | 3468/50000 [1:10:28<20:23:37,  1.58s/it]
Outer training loop:   7%|6         | 3469/50000 [1:10:30<19:54:43,  1.54s/it]
Outer training loop:   7%|6         | 3470/50000 [1:10:31<20:10:40,  1.56s/it]
Outer training loop:   7%|6         | 3471/50000 [1:10:33<20:11:37,  1.56s/it]
Outer training loop:   7%|6         | 3472/50000 [1:10:34<20:23:36,  1.58s/it]
Outer training loop:   7%|6         | 3473/50000 [1:10:36<20:00:05,  1.55s/it]
Outer training loop:   7%|6         | 3474/50000 [1:10:37<19:45:59,  1.53s/it]
Outer training loop:   7%|6         | 3475/50000 [1:10:39<20:00:51,  1.55s/it]
Outer training loop:   7%|6         | 3476/50000 [1:10:41<19:50:02,  1.53s/it]
Outer training loop:   7%|6         | 3477/50000 [1:10:42<20:09:11,  1.56s/it]
Outer training loop:   7%|6         | 3478/50000 [1:10:44<19:43:10,  1.53s/it]
Outer training loop:   7%|6         | 3479/50000 [1:10:45<20:00:23,  1.55s/it]
Outer training loop:   7%|6         | 3480/50000 [1:10:47<19:52:42,  1.54s/it]
Outer training loop:   7%|6         | 3481/50000 [1:10:48<20:04:26,  1.55s/it]
Outer training loop:   7%|6         | 3482/50000 [1:10:50<19:51:47,  1.54s/it]
Outer training loop:   7%|6         | 3483/50000 [1:10:51<19:50:00,  1.53s/it]
Outer training loop:   7%|6         | 3484/50000 [1:10:53<19:54:50,  1.54s/it]
Outer training loop:   7%|6         | 3485/50000 [1:10:54<20:09:52,  1.56s/it]
Outer training loop:   7%|6         | 3486/50000 [1:10:56<19:56:27,  1.54s/it]
Outer training loop:   7%|6         | 3487/50000 [1:10:58<20:03:27,  1.55s/it]
Outer training loop:   7%|6         | 3488/50000 [1:10:59<19:43:43,  1.53s/it]
Outer training loop:   7%|6         | 3489/50000 [1:11:01<20:11:23,  1.56s/it]
Outer training loop:   7%|6         | 3490/50000 [1:11:02<19:53:17,  1.54s/it]
Outer training loop:   7%|6         | 3491/50000 [1:11:04<20:14:49,  1.57s/it]
Outer training loop:   7%|6         | 3492/50000 [1:11:05<19:56:38,  1.54s/it]
Outer training loop:   7%|6         | 3493/50000 [1:11:07<20:10:53,  1.56s/it]
Outer training loop:   7%|6         | 3494/50000 [1:11:08<19:59:06,  1.55s/it]
Outer training loop:   7%|6         | 3495/50000 [1:11:10<20:05:28,  1.56s/it]
Outer training loop:   7%|6         | 3496/50000 [1:11:12<20:04:16,  1.55s/it]
Outer training loop:   7%|6         | 3497/50000 [1:11:13<20:08:58,  1.56s/it]
Outer training loop:   7%|6         | 3498/50000 [1:11:15<20:05:43,  1.56s/it]
Outer training loop:   7%|6         | 3499/50000 [1:11:16<20:17:40,  1.57s/it]
Outer training loop:   7%|7         | 3500/50000 [1:11:18<20:35:23,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.50s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.51s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.51s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:04,  1.52s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:05,  1.55s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:03,  1.54s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:03,  1.57s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:00,  1.55s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:01,  1.58s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<02:01,  1.59s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:57,  1.57s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:58,  1.60s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:55,  1.58s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:55,  1.61s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:52,  1.59s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:48,  1.55s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:46,  1.55s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:46,  1.57s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:43,  1.55s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:40,  1.53s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:41,  1.56s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:39,  1.55s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:37,  1.55s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:38,  1.59s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:35,  1.57s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:33,  1.56s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:30,  1.53s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:31,  1.58s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:29,  1.56s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:26,  1.55s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:23,  1.52s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:24,  1.56s/it]

Evaluating policy:  36%|###6      | 30/83 [00:46<01:22,  1.55s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:19,  1.52s/it]

Evaluating policy:  39%|###8      | 32/83 [00:49<01:17,  1.51s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:19,  1.60s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:16,  1.55s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:12,  1.50s/it]

Evaluating policy:  43%|####3     | 36/83 [00:55<01:11,  1.51s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:09,  1.51s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:10,  1.56s/it]

Evaluating policy:  47%|####6     | 39/83 [01:00<01:08,  1.55s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:05,  1.51s/it]

Evaluating policy:  49%|####9     | 41/83 [01:03<01:03,  1.51s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:01,  1.50s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:06<01:00,  1.50s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<01:01,  1.58s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:09<00:59,  1.56s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:11<00:58,  1.58s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:12<00:55,  1.54s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:14<00:53,  1.52s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:54,  1.61s/it]

Evaluating policy:  60%|######    | 50/83 [01:17<00:52,  1.58s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:50,  1.57s/it]

Evaluating policy:  63%|######2   | 52/83 [01:20<00:47,  1.54s/it]

Evaluating policy:  64%|######3   | 53/83 [01:22<00:45,  1.52s/it]

Evaluating policy:  65%|######5   | 54/83 [01:23<00:43,  1.51s/it]

Evaluating policy:  66%|######6   | 55/83 [01:25<00:42,  1.51s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:44,  1.64s/it]

Evaluating policy:  69%|######8   | 57/83 [01:28<00:41,  1.60s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:39,  1.60s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:31<00:37,  1.55s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:33<00:35,  1.55s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:34<00:33,  1.52s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:36<00:33,  1.61s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:31,  1.56s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:39<00:29,  1.55s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:27,  1.54s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:42<00:25,  1.51s/it]

Evaluating policy:  81%|########  | 67/83 [01:43<00:24,  1.50s/it]

Evaluating policy:  82%|########1 | 68/83 [01:45<00:23,  1.54s/it]

Evaluating policy:  83%|########3 | 69/83 [01:47<00:23,  1.65s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:21,  1.64s/it]

Evaluating policy:  86%|########5 | 71/83 [01:50<00:18,  1.58s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:17,  1.56s/it]

Evaluating policy:  88%|########7 | 73/83 [01:53<00:15,  1.55s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:13,  1.55s/it]

Evaluating policy:  90%|######### | 75/83 [01:56<00:12,  1.57s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:10,  1.54s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.65s/it]

Evaluating policy:  94%|#########3| 78/83 [02:01<00:08,  1.61s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.62s/it]

Evaluating policy:  96%|#########6| 80/83 [02:04<00:04,  1.58s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.55s/it]

Evaluating policy:  99%|#########8| 82/83 [02:07<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.52s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:02,  1.50s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:10,  1.61s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:06,  1.59s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:05,  1.58s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:04,  1.59s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:00,  1.56s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<01:58,  1.56s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:55,  1.54s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:54,  1.54s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:53,  1.56s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:51,  1.55s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:52,  1.58s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:52,  1.61s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:48,  1.57s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:49,  1.60s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:45,  1.58s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:42,  1.56s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:40,  1.55s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:41,  1.58s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:38,  1.57s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:36,  1.56s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:37,  1.59s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:35,  1.58s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:31,  1.55s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:32,  1.60s/it]

Evaluating policy:  31%|###1      | 26/83 [00:41<01:30,  1.58s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:26,  1.54s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:23,  1.51s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:26,  1.60s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:24,  1.59s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:20,  1.56s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:18,  1.54s/it]

Evaluating policy:  40%|###9      | 33/83 [00:52<01:19,  1.60s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:16,  1.56s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:14,  1.55s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:11,  1.53s/it]

Evaluating policy:  45%|####4     | 37/83 [00:58<01:10,  1.53s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:12,  1.62s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:09,  1.59s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:07,  1.56s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:05,  1.55s/it]

Evaluating policy:  51%|#####     | 42/83 [01:06<01:04,  1.58s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:05,  1.65s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:09<01:02,  1.60s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<00:59,  1.56s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<00:57,  1.56s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:13<00:56,  1.56s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:56,  1.62s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:17<00:54,  1.60s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:51,  1.55s/it]

Evaluating policy:  61%|######1   | 51/83 [01:20<00:49,  1.53s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:47,  1.53s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:45,  1.50s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:46,  1.59s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:44,  1.57s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:41,  1.54s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:39,  1.53s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:38,  1.54s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:36,  1.52s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:33<00:34,  1.49s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:35,  1.60s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:37<00:33,  1.59s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:32,  1.60s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:29,  1.56s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:27,  1.52s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:25,  1.52s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:24,  1.51s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:24,  1.62s/it]

Evaluating policy:  83%|########3 | 69/83 [01:48<00:22,  1.61s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:20,  1.58s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:18,  1.57s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:16,  1.53s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:15,  1.50s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:13,  1.49s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:11,  1.50s/it]

Evaluating policy:  92%|#########1| 76/83 [01:59<00:11,  1.62s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.60s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.58s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.58s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.55s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.54s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.52s/it]

Outer training loop:   7%|7         | 3501/50000 [1:15:43<1042:42:32, 80.73s/it]
Outer training loop:   7%|7         | 3502/50000 [1:15:45<737:23:22, 57.09s/it] 
Outer training loop:   7%|7         | 3503/50000 [1:15:47<521:50:04, 40.40s/it]
Outer training loop:   7%|7         | 3504/50000 [1:15:48<371:11:59, 28.74s/it]
Outer training loop:   7%|7         | 3505/50000 [1:15:50<265:47:08, 20.58s/it]
Outer training loop:   7%|7         | 3506/50000 [1:15:51<192:25:05, 14.90s/it]
Outer training loop:   7%|7         | 3507/50000 [1:15:53<140:47:09, 10.90s/it]
Outer training loop:   7%|7         | 3508/50000 [1:15:54<104:23:43,  8.08s/it]
Outer training loop:   7%|7         | 3509/50000 [1:15:56<79:21:32,  6.15s/it] 
Outer training loop:   7%|7         | 3510/50000 [1:15:58<61:21:02,  4.75s/it]
Outer training loop:   7%|7         | 3511/50000 [1:15:59<48:41:56,  3.77s/it]
Outer training loop:   7%|7         | 3512/50000 [1:16:01<40:14:59,  3.12s/it]
Outer training loop:   7%|7         | 3513/50000 [1:16:02<33:46:47,  2.62s/it]
Outer training loop:   7%|7         | 3514/50000 [1:16:04<29:50:08,  2.31s/it]
Outer training loop:   7%|7         | 3515/50000 [1:16:05<26:33:29,  2.06s/it]
Outer training loop:   7%|7         | 3516/50000 [1:16:07<24:51:48,  1.93s/it]
Outer training loop:   7%|7         | 3517/50000 [1:16:08<23:07:34,  1.79s/it]
Outer training loop:   7%|7         | 3518/50000 [1:16:10<22:35:12,  1.75s/it]
Outer training loop:   7%|7         | 3519/50000 [1:16:11<21:32:21,  1.67s/it]
Outer training loop:   7%|7         | 3520/50000 [1:16:13<21:15:51,  1.65s/it]
Outer training loop:   7%|7         | 3521/50000 [1:16:15<20:53:56,  1.62s/it]
Outer training loop:   7%|7         | 3522/50000 [1:16:16<20:41:29,  1.60s/it]
Outer training loop:   7%|7         | 3523/50000 [1:16:18<20:20:51,  1.58s/it]
Outer training loop:   7%|7         | 3524/50000 [1:16:19<19:54:38,  1.54s/it]
Outer training loop:   7%|7         | 3525/50000 [1:16:21<20:17:17,  1.57s/it]
Outer training loop:   7%|7         | 3526/50000 [1:16:22<20:06:57,  1.56s/it]
Outer training loop:   7%|7         | 3527/50000 [1:16:24<20:24:14,  1.58s/it]
Outer training loop:   7%|7         | 3528/50000 [1:16:25<20:08:28,  1.56s/it]
Outer training loop:   7%|7         | 3529/50000 [1:16:27<20:26:27,  1.58s/it]
Outer training loop:   7%|7         | 3530/50000 [1:16:29<20:26:57,  1.58s/it]
Outer training loop:   7%|7         | 3531/50000 [1:16:30<21:03:42,  1.63s/it]
Outer training loop:   7%|7         | 3532/50000 [1:16:32<20:26:04,  1.58s/it]
Outer training loop:   7%|7         | 3533/50000 [1:16:33<20:39:05,  1.60s/it]
Outer training loop:   7%|7         | 3534/50000 [1:16:35<20:20:57,  1.58s/it]
Outer training loop:   7%|7         | 3535/50000 [1:16:37<20:42:17,  1.60s/it]
Outer training loop:   7%|7         | 3536/50000 [1:16:38<20:56:58,  1.62s/it]
Outer training loop:   7%|7         | 3537/50000 [1:16:40<20:46:22,  1.61s/it]
Outer training loop:   7%|7         | 3538/50000 [1:16:41<20:10:23,  1.56s/it]
Outer training loop:   7%|7         | 3539/50000 [1:16:43<20:22:21,  1.58s/it]
Outer training loop:   7%|7         | 3540/50000 [1:16:44<20:09:05,  1.56s/it]
Outer training loop:   7%|7         | 3541/50000 [1:16:46<19:57:24,  1.55s/it]
Outer training loop:   7%|7         | 3542/50000 [1:16:48<20:36:12,  1.60s/it]
Outer training loop:   7%|7         | 3543/50000 [1:16:49<20:42:38,  1.60s/it]
Outer training loop:   7%|7         | 3544/50000 [1:16:51<20:33:56,  1.59s/it]
Outer training loop:   7%|7         | 3545/50000 [1:16:52<19:59:54,  1.55s/it]
Outer training loop:   7%|7         | 3546/50000 [1:16:54<20:22:31,  1.58s/it]
Outer training loop:   7%|7         | 3547/50000 [1:16:56<20:10:50,  1.56s/it]
Outer training loop:   7%|7         | 3548/50000 [1:16:57<20:12:11,  1.57s/it]
Outer training loop:   7%|7         | 3549/50000 [1:16:59<19:55:43,  1.54s/it]
Outer training loop:   7%|7         | 3550/50000 [1:17:00<19:49:59,  1.54s/it]
Outer training loop:   7%|7         | 3551/50000 [1:17:02<20:06:58,  1.56s/it]
Outer training loop:   7%|7         | 3552/50000 [1:17:03<19:54:42,  1.54s/it]
Outer training loop:   7%|7         | 3553/50000 [1:17:05<20:12:27,  1.57s/it]
Outer training loop:   7%|7         | 3554/50000 [1:17:06<19:52:55,  1.54s/it]
Outer training loop:   7%|7         | 3555/50000 [1:17:08<20:10:31,  1.56s/it]
Outer training loop:   7%|7         | 3556/50000 [1:17:09<20:05:21,  1.56s/it]
Outer training loop:   7%|7         | 3557/50000 [1:17:11<20:01:01,  1.55s/it]
Outer training loop:   7%|7         | 3558/50000 [1:17:13<19:52:24,  1.54s/it]
Outer training loop:   7%|7         | 3559/50000 [1:17:14<20:10:56,  1.56s/it]
Outer training loop:   7%|7         | 3560/50000 [1:17:16<19:45:30,  1.53s/it]
Outer training loop:   7%|7         | 3561/50000 [1:17:17<20:07:35,  1.56s/it]
Outer training loop:   7%|7         | 3562/50000 [1:17:19<20:01:35,  1.55s/it]
Outer training loop:   7%|7         | 3563/50000 [1:17:20<20:23:51,  1.58s/it]
Outer training loop:   7%|7         | 3564/50000 [1:17:22<20:09:59,  1.56s/it]
Outer training loop:   7%|7         | 3565/50000 [1:17:24<20:39:02,  1.60s/it]
Outer training loop:   7%|7         | 3566/50000 [1:17:25<20:31:14,  1.59s/it]
Outer training loop:   7%|7         | 3567/50000 [1:17:27<20:43:55,  1.61s/it]
Outer training loop:   7%|7         | 3568/50000 [1:17:28<20:08:57,  1.56s/it]
Outer training loop:   7%|7         | 3569/50000 [1:17:30<20:31:28,  1.59s/it]
Outer training loop:   7%|7         | 3570/50000 [1:17:31<20:09:34,  1.56s/it]
Outer training loop:   7%|7         | 3571/50000 [1:17:33<20:22:35,  1.58s/it]
Outer training loop:   7%|7         | 3572/50000 [1:17:35<20:36:25,  1.60s/it]
Outer training loop:   7%|7         | 3573/50000 [1:17:36<20:30:58,  1.59s/it]
Outer training loop:   7%|7         | 3574/50000 [1:17:38<20:00:42,  1.55s/it]
Outer training loop:   7%|7         | 3575/50000 [1:17:39<20:15:31,  1.57s/it]
Outer training loop:   7%|7         | 3576/50000 [1:17:41<19:46:39,  1.53s/it]
Outer training loop:   7%|7         | 3577/50000 [1:17:42<20:12:58,  1.57s/it]
Outer training loop:   7%|7         | 3578/50000 [1:17:44<19:47:07,  1.53s/it]
Outer training loop:   7%|7         | 3579/50000 [1:17:46<20:01:02,  1.55s/it]
Outer training loop:   7%|7         | 3580/50000 [1:17:47<20:02:37,  1.55s/it]
Outer training loop:   7%|7         | 3581/50000 [1:17:49<21:08:05,  1.64s/it]
Outer training loop:   7%|7         | 3582/50000 [1:17:50<20:40:53,  1.60s/it]
Outer training loop:   7%|7         | 3583/50000 [1:17:52<20:35:22,  1.60s/it]
Outer training loop:   7%|7         | 3584/50000 [1:17:54<20:23:43,  1.58s/it]
Outer training loop:   7%|7         | 3585/50000 [1:17:55<20:25:51,  1.58s/it]
Outer training loop:   7%|7         | 3586/50000 [1:17:57<19:53:47,  1.54s/it]
Outer training loop:   7%|7         | 3587/50000 [1:17:58<20:09:43,  1.56s/it]
Outer training loop:   7%|7         | 3588/50000 [1:18:00<19:55:37,  1.55s/it]
Outer training loop:   7%|7         | 3589/50000 [1:18:01<20:17:10,  1.57s/it]
Outer training loop:   7%|7         | 3590/50000 [1:18:03<19:56:39,  1.55s/it]
Outer training loop:   7%|7         | 3591/50000 [1:18:04<20:15:11,  1.57s/it]
Outer training loop:   7%|7         | 3592/50000 [1:18:06<20:30:26,  1.59s/it]
Outer training loop:   7%|7         | 3593/50000 [1:18:08<20:37:45,  1.60s/it]
Outer training loop:   7%|7         | 3594/50000 [1:18:09<20:08:31,  1.56s/it]
Outer training loop:   7%|7         | 3595/50000 [1:18:11<20:24:37,  1.58s/it]
Outer training loop:   7%|7         | 3596/50000 [1:18:12<20:03:09,  1.56s/it]
Outer training loop:   7%|7         | 3597/50000 [1:18:14<19:57:09,  1.55s/it]
Outer training loop:   7%|7         | 3598/50000 [1:18:15<19:57:45,  1.55s/it]
Outer training loop:   7%|7         | 3599/50000 [1:18:17<20:11:12,  1.57s/it]
Outer training loop:   7%|7         | 3600/50000 [1:18:19<20:07:59,  1.56s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.45s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.49s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.47s/it]

Outer training loop:   7%|7         | 3601/50000 [1:18:25<37:44:42,  2.93s/it]
Outer training loop:   7%|7         | 3602/50000 [1:18:26<32:45:36,  2.54s/it]
Outer training loop:   7%|7         | 3603/50000 [1:18:28<28:50:18,  2.24s/it]
Outer training loop:   7%|7         | 3604/50000 [1:18:29<25:56:52,  2.01s/it]
Outer training loop:   7%|7         | 3605/50000 [1:18:31<24:07:08,  1.87s/it]
Outer training loop:   7%|7         | 3606/50000 [1:18:32<22:44:40,  1.76s/it]
Outer training loop:   7%|7         | 3607/50000 [1:18:34<22:10:36,  1.72s/it]
Outer training loop:   7%|7         | 3608/50000 [1:18:36<21:16:44,  1.65s/it]
Outer training loop:   7%|7         | 3609/50000 [1:18:37<21:00:41,  1.63s/it]
Outer training loop:   7%|7         | 3610/50000 [1:18:39<20:20:16,  1.58s/it]
Outer training loop:   7%|7         | 3611/50000 [1:18:40<20:36:27,  1.60s/it]
Outer training loop:   7%|7         | 3612/50000 [1:18:42<20:38:02,  1.60s/it]
Outer training loop:   7%|7         | 3613/50000 [1:18:43<20:20:34,  1.58s/it]
Outer training loop:   7%|7         | 3614/50000 [1:18:45<20:10:02,  1.57s/it]
Outer training loop:   7%|7         | 3615/50000 [1:18:46<19:56:59,  1.55s/it]
Outer training loop:   7%|7         | 3616/50000 [1:18:48<19:51:35,  1.54s/it]
Outer training loop:   7%|7         | 3617/50000 [1:18:50<20:10:06,  1.57s/it]
Outer training loop:   7%|7         | 3618/50000 [1:18:51<19:52:10,  1.54s/it]
Outer training loop:   7%|7         | 3619/50000 [1:18:53<20:27:04,  1.59s/it]
Outer training loop:   7%|7         | 3620/50000 [1:18:54<20:43:06,  1.61s/it]
Outer training loop:   7%|7         | 3621/50000 [1:18:56<20:19:04,  1.58s/it]
Outer training loop:   7%|7         | 3622/50000 [1:18:57<20:05:54,  1.56s/it]
Outer training loop:   7%|7         | 3623/50000 [1:18:59<20:20:56,  1.58s/it]
Outer training loop:   7%|7         | 3624/50000 [1:19:01<20:12:13,  1.57s/it]
Outer training loop:   7%|7         | 3625/50000 [1:19:02<20:15:33,  1.57s/it]
Outer training loop:   7%|7         | 3626/50000 [1:19:04<20:05:42,  1.56s/it]
Outer training loop:   7%|7         | 3627/50000 [1:19:05<20:02:40,  1.56s/it]
Outer training loop:   7%|7         | 3628/50000 [1:19:07<20:19:47,  1.58s/it]
Outer training loop:   7%|7         | 3629/50000 [1:19:08<20:22:32,  1.58s/it]
Outer training loop:   7%|7         | 3630/50000 [1:19:10<20:08:48,  1.56s/it]
Outer training loop:   7%|7         | 3631/50000 [1:19:12<20:10:04,  1.57s/it]
Outer training loop:   7%|7         | 3632/50000 [1:19:13<20:08:03,  1.56s/it]
Outer training loop:   7%|7         | 3633/50000 [1:19:15<20:15:19,  1.57s/it]
Outer training loop:   7%|7         | 3634/50000 [1:19:16<20:16:13,  1.57s/it]
Outer training loop:   7%|7         | 3635/50000 [1:19:18<20:11:12,  1.57s/it]
Outer training loop:   7%|7         | 3636/50000 [1:19:19<19:58:57,  1.55s/it]
Outer training loop:   7%|7         | 3637/50000 [1:19:21<20:09:31,  1.57s/it]
Outer training loop:   7%|7         | 3638/50000 [1:19:23<20:17:26,  1.58s/it]
Outer training loop:   7%|7         | 3639/50000 [1:19:24<20:33:55,  1.60s/it]
Outer training loop:   7%|7         | 3640/50000 [1:19:26<20:26:00,  1.59s/it]
Outer training loop:   7%|7         | 3641/50000 [1:19:27<20:42:45,  1.61s/it]
Outer training loop:   7%|7         | 3642/50000 [1:19:29<20:14:36,  1.57s/it]
Outer training loop:   7%|7         | 3643/50000 [1:19:31<20:34:06,  1.60s/it]
Outer training loop:   7%|7         | 3644/50000 [1:19:32<20:10:14,  1.57s/it]
Outer training loop:   7%|7         | 3645/50000 [1:19:34<20:22:09,  1.58s/it]
Outer training loop:   7%|7         | 3646/50000 [1:19:35<20:37:04,  1.60s/it]
Outer training loop:   7%|7         | 3647/50000 [1:19:37<19:59:02,  1.55s/it]
Outer training loop:   7%|7         | 3648/50000 [1:19:38<19:48:21,  1.54s/it]
Outer training loop:   7%|7         | 3649/50000 [1:19:40<19:59:11,  1.55s/it]
Outer training loop:   7%|7         | 3650/50000 [1:19:41<20:09:41,  1.57s/it]
Outer training loop:   7%|7         | 3651/50000 [1:19:43<20:22:12,  1.58s/it]
Outer training loop:   7%|7         | 3652/50000 [1:19:44<19:52:21,  1.54s/it]
Outer training loop:   7%|7         | 3653/50000 [1:19:46<19:56:39,  1.55s/it]
Outer training loop:   7%|7         | 3654/50000 [1:19:48<19:58:19,  1.55s/it]
Outer training loop:   7%|7         | 3655/50000 [1:19:49<20:21:23,  1.58s/it]
Outer training loop:   7%|7         | 3656/50000 [1:19:51<20:16:32,  1.58s/it]
Outer training loop:   7%|7         | 3657/50000 [1:19:52<20:27:16,  1.59s/it]
Outer training loop:   7%|7         | 3658/50000 [1:19:54<20:02:43,  1.56s/it]
Outer training loop:   7%|7         | 3659/50000 [1:19:56<20:12:19,  1.57s/it]
Outer training loop:   7%|7         | 3660/50000 [1:19:57<19:54:22,  1.55s/it]
Outer training loop:   7%|7         | 3661/50000 [1:19:59<20:08:10,  1.56s/it]
Outer training loop:   7%|7         | 3662/50000 [1:20:00<20:01:36,  1.56s/it]
Outer training loop:   7%|7         | 3663/50000 [1:20:02<20:25:52,  1.59s/it]
Outer training loop:   7%|7         | 3664/50000 [1:20:03<20:19:39,  1.58s/it]
Outer training loop:   7%|7         | 3665/50000 [1:20:05<20:26:39,  1.59s/it]
Outer training loop:   7%|7         | 3666/50000 [1:20:06<19:48:45,  1.54s/it]
Outer training loop:   7%|7         | 3667/50000 [1:20:08<20:04:30,  1.56s/it]
Outer training loop:   7%|7         | 3668/50000 [1:20:10<19:57:58,  1.55s/it]
Outer training loop:   7%|7         | 3669/50000 [1:20:11<20:08:50,  1.57s/it]
Outer training loop:   7%|7         | 3670/50000 [1:20:13<19:49:19,  1.54s/it]
Outer training loop:   7%|7         | 3671/50000 [1:20:14<20:02:48,  1.56s/it]
Outer training loop:   7%|7         | 3672/50000 [1:20:16<19:57:01,  1.55s/it]
Outer training loop:   7%|7         | 3673/50000 [1:20:17<19:51:33,  1.54s/it]
Outer training loop:   7%|7         | 3674/50000 [1:20:19<19:44:48,  1.53s/it]
Outer training loop:   7%|7         | 3675/50000 [1:20:20<20:04:05,  1.56s/it]
Outer training loop:   7%|7         | 3676/50000 [1:20:22<19:45:12,  1.54s/it]
Outer training loop:   7%|7         | 3677/50000 [1:20:23<19:55:35,  1.55s/it]
Outer training loop:   7%|7         | 3678/50000 [1:20:25<20:33:31,  1.60s/it]
Outer training loop:   7%|7         | 3679/50000 [1:20:27<20:21:11,  1.58s/it]
Outer training loop:   7%|7         | 3680/50000 [1:20:28<19:41:30,  1.53s/it]
Outer training loop:   7%|7         | 3681/50000 [1:20:30<19:57:27,  1.55s/it]
Outer training loop:   7%|7         | 3682/50000 [1:20:31<19:48:10,  1.54s/it]
Outer training loop:   7%|7         | 3683/50000 [1:20:33<19:57:03,  1.55s/it]
Outer training loop:   7%|7         | 3684/50000 [1:20:34<19:51:15,  1.54s/it]
Outer training loop:   7%|7         | 3685/50000 [1:20:36<20:08:37,  1.57s/it]
Outer training loop:   7%|7         | 3686/50000 [1:20:38<20:07:56,  1.56s/it]
Outer training loop:   7%|7         | 3687/50000 [1:20:39<20:08:05,  1.57s/it]
Outer training loop:   7%|7         | 3688/50000 [1:20:41<19:55:27,  1.55s/it]
Outer training loop:   7%|7         | 3689/50000 [1:20:42<20:06:28,  1.56s/it]
Outer training loop:   7%|7         | 3690/50000 [1:20:44<19:48:21,  1.54s/it]
Outer training loop:   7%|7         | 3691/50000 [1:20:45<19:58:02,  1.55s/it]
Outer training loop:   7%|7         | 3692/50000 [1:20:47<19:49:06,  1.54s/it]
Outer training loop:   7%|7         | 3693/50000 [1:20:48<20:08:26,  1.57s/it]
Outer training loop:   7%|7         | 3694/50000 [1:20:50<19:43:57,  1.53s/it]
Outer training loop:   7%|7         | 3695/50000 [1:20:51<19:56:08,  1.55s/it]
Outer training loop:   7%|7         | 3696/50000 [1:20:53<19:30:13,  1.52s/it]
Outer training loop:   7%|7         | 3697/50000 [1:20:54<19:25:54,  1.51s/it]
Outer training loop:   7%|7         | 3698/50000 [1:20:56<19:33:34,  1.52s/it]
Outer training loop:   7%|7         | 3699/50000 [1:20:58<19:58:29,  1.55s/it]
Outer training loop:   7%|7         | 3700/50000 [1:20:59<19:46:22,  1.54s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.54s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.55s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.52s/it]

Outer training loop:   7%|7         | 3701/50000 [1:21:05<37:56:18,  2.95s/it]
Outer training loop:   7%|7         | 3702/50000 [1:21:07<32:29:53,  2.53s/it]
Outer training loop:   7%|7         | 3703/50000 [1:21:08<28:58:17,  2.25s/it]
Outer training loop:   7%|7         | 3704/50000 [1:21:10<26:39:47,  2.07s/it]
Outer training loop:   7%|7         | 3705/50000 [1:21:12<24:11:10,  1.88s/it]
Outer training loop:   7%|7         | 3706/50000 [1:21:13<22:32:12,  1.75s/it]
Outer training loop:   7%|7         | 3707/50000 [1:21:15<21:58:26,  1.71s/it]
Outer training loop:   7%|7         | 3708/50000 [1:21:16<20:52:35,  1.62s/it]
Outer training loop:   7%|7         | 3709/50000 [1:21:18<20:47:23,  1.62s/it]
Outer training loop:   7%|7         | 3710/50000 [1:21:19<20:53:41,  1.63s/it]
Outer training loop:   7%|7         | 3711/50000 [1:21:21<20:29:29,  1.59s/it]
Outer training loop:   7%|7         | 3712/50000 [1:21:22<20:19:09,  1.58s/it]
Outer training loop:   7%|7         | 3713/50000 [1:21:24<19:52:55,  1.55s/it]
Outer training loop:   7%|7         | 3714/50000 [1:21:26<20:25:27,  1.59s/it]
Outer training loop:   7%|7         | 3715/50000 [1:21:27<20:11:12,  1.57s/it]
Outer training loop:   7%|7         | 3716/50000 [1:21:29<19:51:12,  1.54s/it]
Outer training loop:   7%|7         | 3717/50000 [1:21:30<20:08:13,  1.57s/it]
Outer training loop:   7%|7         | 3718/50000 [1:21:32<19:54:58,  1.55s/it]
Outer training loop:   7%|7         | 3719/50000 [1:21:33<20:23:51,  1.59s/it]
Outer training loop:   7%|7         | 3720/50000 [1:21:35<20:09:15,  1.57s/it]
Outer training loop:   7%|7         | 3721/50000 [1:21:36<20:18:14,  1.58s/it]
Outer training loop:   7%|7         | 3722/50000 [1:21:38<20:07:21,  1.57s/it]
Outer training loop:   7%|7         | 3723/50000 [1:21:40<20:07:54,  1.57s/it]
Outer training loop:   7%|7         | 3724/50000 [1:21:41<19:53:06,  1.55s/it]
Outer training loop:   7%|7         | 3725/50000 [1:21:43<20:46:22,  1.62s/it]
Outer training loop:   7%|7         | 3726/50000 [1:21:44<20:12:00,  1.57s/it]
Outer training loop:   7%|7         | 3727/50000 [1:21:46<20:11:59,  1.57s/it]
Outer training loop:   7%|7         | 3728/50000 [1:21:47<20:05:15,  1.56s/it]
Outer training loop:   7%|7         | 3729/50000 [1:21:49<20:20:05,  1.58s/it]
Outer training loop:   7%|7         | 3730/50000 [1:21:51<20:07:38,  1.57s/it]
Outer training loop:   7%|7         | 3731/50000 [1:21:52<20:02:40,  1.56s/it]
Outer training loop:   7%|7         | 3732/50000 [1:21:54<20:13:20,  1.57s/it]
Outer training loop:   7%|7         | 3733/50000 [1:21:55<20:11:35,  1.57s/it]
Outer training loop:   7%|7         | 3734/50000 [1:21:57<20:21:52,  1.58s/it]
Outer training loop:   7%|7         | 3735/50000 [1:21:58<19:50:07,  1.54s/it]
Outer training loop:   7%|7         | 3736/50000 [1:22:00<20:14:25,  1.57s/it]
Outer training loop:   7%|7         | 3737/50000 [1:22:02<20:08:21,  1.57s/it]
Outer training loop:   7%|7         | 3738/50000 [1:22:03<19:59:20,  1.56s/it]
Outer training loop:   7%|7         | 3739/50000 [1:22:05<19:54:58,  1.55s/it]
Outer training loop:   7%|7         | 3740/50000 [1:22:06<20:12:50,  1.57s/it]
Outer training loop:   7%|7         | 3741/50000 [1:22:08<19:37:04,  1.53s/it]
Outer training loop:   7%|7         | 3742/50000 [1:22:09<20:09:49,  1.57s/it]
Outer training loop:   7%|7         | 3743/50000 [1:22:11<20:30:58,  1.60s/it]
Outer training loop:   7%|7         | 3744/50000 [1:22:12<19:55:35,  1.55s/it]
Outer training loop:   7%|7         | 3745/50000 [1:22:14<19:41:12,  1.53s/it]
Outer training loop:   7%|7         | 3746/50000 [1:22:16<20:18:41,  1.58s/it]
Outer training loop:   7%|7         | 3747/50000 [1:22:17<20:24:06,  1.59s/it]
Outer training loop:   7%|7         | 3748/50000 [1:22:19<20:11:46,  1.57s/it]
Outer training loop:   7%|7         | 3749/50000 [1:22:20<20:04:07,  1.56s/it]
Outer training loop:   8%|7         | 3750/50000 [1:22:22<19:59:17,  1.56s/it]
Outer training loop:   8%|7         | 3751/50000 [1:22:23<19:45:13,  1.54s/it]
Outer training loop:   8%|7         | 3752/50000 [1:22:25<20:03:27,  1.56s/it]
Outer training loop:   8%|7         | 3753/50000 [1:22:26<19:39:21,  1.53s/it]
Outer training loop:   8%|7         | 3754/50000 [1:22:28<19:53:01,  1.55s/it]
Outer training loop:   8%|7         | 3755/50000 [1:22:29<19:41:50,  1.53s/it]
Outer training loop:   8%|7         | 3756/50000 [1:22:31<19:40:40,  1.53s/it]
Outer training loop:   8%|7         | 3757/50000 [1:22:33<20:02:52,  1.56s/it]
Outer training loop:   8%|7         | 3758/50000 [1:22:34<19:55:31,  1.55s/it]
Outer training loop:   8%|7         | 3759/50000 [1:22:36<20:05:48,  1.56s/it]
Outer training loop:   8%|7         | 3760/50000 [1:22:37<19:49:44,  1.54s/it]
Outer training loop:   8%|7         | 3761/50000 [1:22:39<19:50:08,  1.54s/it]
Outer training loop:   8%|7         | 3762/50000 [1:22:40<19:59:49,  1.56s/it]
Outer training loop:   8%|7         | 3763/50000 [1:22:42<20:10:57,  1.57s/it]
Outer training loop:   8%|7         | 3764/50000 [1:22:44<20:17:22,  1.58s/it]
Outer training loop:   8%|7         | 3765/50000 [1:22:45<20:21:21,  1.58s/it]
Outer training loop:   8%|7         | 3766/50000 [1:22:47<19:59:28,  1.56s/it]
Outer training loop:   8%|7         | 3767/50000 [1:22:48<20:15:27,  1.58s/it]
Outer training loop:   8%|7         | 3768/50000 [1:22:50<20:01:53,  1.56s/it]
Outer training loop:   8%|7         | 3769/50000 [1:22:51<20:18:36,  1.58s/it]
Outer training loop:   8%|7         | 3770/50000 [1:22:53<20:03:49,  1.56s/it]
Outer training loop:   8%|7         | 3771/50000 [1:22:55<20:31:40,  1.60s/it]
Outer training loop:   8%|7         | 3772/50000 [1:22:56<20:11:24,  1.57s/it]
Outer training loop:   8%|7         | 3773/50000 [1:22:58<20:10:55,  1.57s/it]
Outer training loop:   8%|7         | 3774/50000 [1:22:59<19:59:25,  1.56s/it]
Outer training loop:   8%|7         | 3775/50000 [1:23:01<20:44:06,  1.61s/it]
Outer training loop:   8%|7         | 3776/50000 [1:23:02<20:08:19,  1.57s/it]
Outer training loop:   8%|7         | 3777/50000 [1:23:04<19:47:20,  1.54s/it]
Outer training loop:   8%|7         | 3778/50000 [1:23:06<19:51:38,  1.55s/it]
Outer training loop:   8%|7         | 3779/50000 [1:23:07<19:32:36,  1.52s/it]
Outer training loop:   8%|7         | 3780/50000 [1:23:09<19:58:20,  1.56s/it]
Outer training loop:   8%|7         | 3781/50000 [1:23:10<19:38:11,  1.53s/it]
Outer training loop:   8%|7         | 3782/50000 [1:23:12<19:47:14,  1.54s/it]
Outer training loop:   8%|7         | 3783/50000 [1:23:13<19:37:50,  1.53s/it]
Outer training loop:   8%|7         | 3784/50000 [1:23:15<20:01:42,  1.56s/it]
Outer training loop:   8%|7         | 3785/50000 [1:23:16<19:52:10,  1.55s/it]
Outer training loop:   8%|7         | 3786/50000 [1:23:18<20:03:08,  1.56s/it]
Outer training loop:   8%|7         | 3787/50000 [1:23:19<19:33:32,  1.52s/it]
Outer training loop:   8%|7         | 3788/50000 [1:23:21<20:07:51,  1.57s/it]
Outer training loop:   8%|7         | 3789/50000 [1:23:23<19:55:19,  1.55s/it]
Outer training loop:   8%|7         | 3790/50000 [1:23:24<20:08:44,  1.57s/it]
Outer training loop:   8%|7         | 3791/50000 [1:23:26<20:18:49,  1.58s/it]
Outer training loop:   8%|7         | 3792/50000 [1:23:27<20:15:23,  1.58s/it]
Outer training loop:   8%|7         | 3793/50000 [1:23:29<20:06:14,  1.57s/it]
Outer training loop:   8%|7         | 3794/50000 [1:23:30<20:04:38,  1.56s/it]
Outer training loop:   8%|7         | 3795/50000 [1:23:32<19:56:58,  1.55s/it]
Outer training loop:   8%|7         | 3796/50000 [1:23:34<20:11:47,  1.57s/it]
Outer training loop:   8%|7         | 3797/50000 [1:23:35<19:57:56,  1.56s/it]
Outer training loop:   8%|7         | 3798/50000 [1:23:37<20:08:57,  1.57s/it]
Outer training loop:   8%|7         | 3799/50000 [1:23:38<20:02:03,  1.56s/it]
Outer training loop:   8%|7         | 3800/50000 [1:23:40<20:26:20,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.52s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.52s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.52s/it]

Outer training loop:   8%|7         | 3801/50000 [1:23:46<37:43:15,  2.94s/it]
Outer training loop:   8%|7         | 3802/50000 [1:23:48<32:29:56,  2.53s/it]
Outer training loop:   8%|7         | 3803/50000 [1:23:49<28:28:57,  2.22s/it]
Outer training loop:   8%|7         | 3804/50000 [1:23:51<26:12:26,  2.04s/it]
Outer training loop:   8%|7         | 3805/50000 [1:23:52<23:55:44,  1.86s/it]
Outer training loop:   8%|7         | 3806/50000 [1:23:54<22:38:27,  1.76s/it]
Outer training loop:   8%|7         | 3807/50000 [1:23:55<21:56:47,  1.71s/it]
Outer training loop:   8%|7         | 3808/50000 [1:23:57<21:28:54,  1.67s/it]
Outer training loop:   8%|7         | 3809/50000 [1:23:58<20:47:18,  1.62s/it]
Outer training loop:   8%|7         | 3810/50000 [1:24:00<20:42:01,  1.61s/it]
Outer training loop:   8%|7         | 3811/50000 [1:24:02<20:57:41,  1.63s/it]
Outer training loop:   8%|7         | 3812/50000 [1:24:03<20:40:03,  1.61s/it]
Outer training loop:   8%|7         | 3813/50000 [1:24:05<20:17:45,  1.58s/it]
Outer training loop:   8%|7         | 3814/50000 [1:24:06<20:16:59,  1.58s/it]
Outer training loop:   8%|7         | 3815/50000 [1:24:08<20:39:17,  1.61s/it]
Outer training loop:   8%|7         | 3816/50000 [1:24:09<20:28:43,  1.60s/it]
Outer training loop:   8%|7         | 3817/50000 [1:24:11<20:24:07,  1.59s/it]
Outer training loop:   8%|7         | 3818/50000 [1:24:13<19:57:56,  1.56s/it]
Outer training loop:   8%|7         | 3819/50000 [1:24:14<19:44:42,  1.54s/it]
Outer training loop:   8%|7         | 3820/50000 [1:24:16<19:56:34,  1.55s/it]
Outer training loop:   8%|7         | 3821/50000 [1:24:17<20:02:11,  1.56s/it]
Outer training loop:   8%|7         | 3822/50000 [1:24:19<20:32:04,  1.60s/it]
Outer training loop:   8%|7         | 3823/50000 [1:24:20<20:30:11,  1.60s/it]
Outer training loop:   8%|7         | 3824/50000 [1:24:22<20:18:46,  1.58s/it]
Outer training loop:   8%|7         | 3825/50000 [1:24:24<20:15:15,  1.58s/it]
Outer training loop:   8%|7         | 3826/50000 [1:24:25<19:53:26,  1.55s/it]
Outer training loop:   8%|7         | 3827/50000 [1:24:27<19:59:35,  1.56s/it]
Outer training loop:   8%|7         | 3828/50000 [1:24:28<20:06:21,  1.57s/it]
Outer training loop:   8%|7         | 3829/50000 [1:24:30<19:52:09,  1.55s/it]
Outer training loop:   8%|7         | 3830/50000 [1:24:31<19:50:35,  1.55s/it]
Outer training loop:   8%|7         | 3831/50000 [1:24:33<20:40:00,  1.61s/it]
Outer training loop:   8%|7         | 3832/50000 [1:24:35<20:34:06,  1.60s/it]
Outer training loop:   8%|7         | 3833/50000 [1:24:36<20:14:01,  1.58s/it]
Outer training loop:   8%|7         | 3834/50000 [1:24:38<20:06:39,  1.57s/it]
Outer training loop:   8%|7         | 3835/50000 [1:24:39<20:19:00,  1.58s/it]
Outer training loop:   8%|7         | 3836/50000 [1:24:41<19:51:28,  1.55s/it]
Outer training loop:   8%|7         | 3837/50000 [1:24:42<19:56:13,  1.55s/it]
Outer training loop:   8%|7         | 3838/50000 [1:24:44<20:15:59,  1.58s/it]
Outer training loop:   8%|7         | 3839/50000 [1:24:45<19:52:02,  1.55s/it]
Outer training loop:   8%|7         | 3840/50000 [1:24:47<20:05:19,  1.57s/it]
Outer training loop:   8%|7         | 3841/50000 [1:24:49<20:10:36,  1.57s/it]
Outer training loop:   8%|7         | 3842/50000 [1:24:50<20:15:56,  1.58s/it]
Outer training loop:   8%|7         | 3843/50000 [1:24:52<20:06:57,  1.57s/it]
Outer training loop:   8%|7         | 3844/50000 [1:24:53<20:19:17,  1.59s/it]
Outer training loop:   8%|7         | 3845/50000 [1:24:55<20:06:19,  1.57s/it]
Outer training loop:   8%|7         | 3846/50000 [1:24:57<20:24:02,  1.59s/it]
Outer training loop:   8%|7         | 3847/50000 [1:24:58<20:27:53,  1.60s/it]
Outer training loop:   8%|7         | 3848/50000 [1:25:00<20:01:07,  1.56s/it]
Outer training loop:   8%|7         | 3849/50000 [1:25:01<20:05:36,  1.57s/it]
Outer training loop:   8%|7         | 3850/50000 [1:25:03<20:22:01,  1.59s/it]
Outer training loop:   8%|7         | 3851/50000 [1:25:05<20:37:42,  1.61s/it]
Outer training loop:   8%|7         | 3852/50000 [1:25:06<20:11:38,  1.58s/it]
Outer training loop:   8%|7         | 3853/50000 [1:25:08<20:09:51,  1.57s/it]
Outer training loop:   8%|7         | 3854/50000 [1:25:09<19:49:52,  1.55s/it]
Outer training loop:   8%|7         | 3855/50000 [1:25:11<19:53:41,  1.55s/it]
Outer training loop:   8%|7         | 3856/50000 [1:25:12<19:54:30,  1.55s/it]
Outer training loop:   8%|7         | 3857/50000 [1:25:14<20:08:33,  1.57s/it]
Outer training loop:   8%|7         | 3858/50000 [1:25:15<19:32:55,  1.53s/it]
Outer training loop:   8%|7         | 3859/50000 [1:25:17<19:55:01,  1.55s/it]
Outer training loop:   8%|7         | 3860/50000 [1:25:18<19:53:23,  1.55s/it]
Outer training loop:   8%|7         | 3861/50000 [1:25:20<19:49:50,  1.55s/it]
Outer training loop:   8%|7         | 3862/50000 [1:25:22<19:53:17,  1.55s/it]
Outer training loop:   8%|7         | 3863/50000 [1:25:23<20:07:39,  1.57s/it]
Outer training loop:   8%|7         | 3864/50000 [1:25:25<20:06:55,  1.57s/it]
Outer training loop:   8%|7         | 3865/50000 [1:25:26<20:02:47,  1.56s/it]
Outer training loop:   8%|7         | 3866/50000 [1:25:28<20:12:31,  1.58s/it]
Outer training loop:   8%|7         | 3867/50000 [1:25:29<19:57:15,  1.56s/it]
Outer training loop:   8%|7         | 3868/50000 [1:25:31<20:01:47,  1.56s/it]
Outer training loop:   8%|7         | 3869/50000 [1:25:32<19:24:31,  1.51s/it]
Outer training loop:   8%|7         | 3870/50000 [1:25:34<19:40:02,  1.53s/it]
Outer training loop:   8%|7         | 3871/50000 [1:25:35<19:31:00,  1.52s/it]
Outer training loop:   8%|7         | 3872/50000 [1:25:37<20:33:08,  1.60s/it]
Outer training loop:   8%|7         | 3873/50000 [1:25:39<20:16:36,  1.58s/it]
Outer training loop:   8%|7         | 3874/50000 [1:25:40<20:36:04,  1.61s/it]
Outer training loop:   8%|7         | 3875/50000 [1:25:42<20:12:50,  1.58s/it]
Outer training loop:   8%|7         | 3876/50000 [1:25:44<20:03:35,  1.57s/it]
Outer training loop:   8%|7         | 3877/50000 [1:25:45<20:11:31,  1.58s/it]
Outer training loop:   8%|7         | 3878/50000 [1:25:47<20:20:06,  1.59s/it]
Outer training loop:   8%|7         | 3879/50000 [1:25:48<20:12:18,  1.58s/it]
Outer training loop:   8%|7         | 3880/50000 [1:25:50<20:30:24,  1.60s/it]
Outer training loop:   8%|7         | 3881/50000 [1:25:51<20:22:06,  1.59s/it]
Outer training loop:   8%|7         | 3882/50000 [1:25:53<20:37:24,  1.61s/it]
Outer training loop:   8%|7         | 3883/50000 [1:25:55<20:30:06,  1.60s/it]
Outer training loop:   8%|7         | 3884/50000 [1:25:56<20:23:04,  1.59s/it]
Outer training loop:   8%|7         | 3885/50000 [1:25:58<20:10:35,  1.58s/it]
Outer training loop:   8%|7         | 3886/50000 [1:25:59<20:15:52,  1.58s/it]
Outer training loop:   8%|7         | 3887/50000 [1:26:01<20:00:53,  1.56s/it]
Outer training loop:   8%|7         | 3888/50000 [1:26:03<19:59:38,  1.56s/it]
Outer training loop:   8%|7         | 3889/50000 [1:26:04<19:55:46,  1.56s/it]
Outer training loop:   8%|7         | 3890/50000 [1:26:06<20:20:47,  1.59s/it]
Outer training loop:   8%|7         | 3891/50000 [1:26:07<20:34:59,  1.61s/it]
Outer training loop:   8%|7         | 3892/50000 [1:26:09<20:04:10,  1.57s/it]
Outer training loop:   8%|7         | 3893/50000 [1:26:10<20:02:10,  1.56s/it]
Outer training loop:   8%|7         | 3894/50000 [1:26:12<20:10:56,  1.58s/it]
Outer training loop:   8%|7         | 3895/50000 [1:26:14<20:04:48,  1.57s/it]
Outer training loop:   8%|7         | 3896/50000 [1:26:15<19:55:55,  1.56s/it]
Outer training loop:   8%|7         | 3897/50000 [1:26:17<20:06:42,  1.57s/it]
Outer training loop:   8%|7         | 3898/50000 [1:26:18<20:17:57,  1.59s/it]
Outer training loop:   8%|7         | 3899/50000 [1:26:20<19:56:29,  1.56s/it]
Outer training loop:   8%|7         | 3900/50000 [1:26:21<19:59:51,  1.56s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.56s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.53s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.50s/it]

Outer training loop:   8%|7         | 3901/50000 [1:26:27<36:32:16,  2.85s/it]
Outer training loop:   8%|7         | 3902/50000 [1:26:29<31:42:26,  2.48s/it]
Outer training loop:   8%|7         | 3903/50000 [1:26:30<27:37:26,  2.16s/it]
Outer training loop:   8%|7         | 3904/50000 [1:26:32<25:31:04,  1.99s/it]
Outer training loop:   8%|7         | 3905/50000 [1:26:33<23:52:35,  1.86s/it]
Outer training loop:   8%|7         | 3906/50000 [1:26:35<23:08:04,  1.81s/it]
Outer training loop:   8%|7         | 3907/50000 [1:26:37<22:13:12,  1.74s/it]
Outer training loop:   8%|7         | 3908/50000 [1:26:38<21:45:00,  1.70s/it]
Outer training loop:   8%|7         | 3909/50000 [1:26:40<21:08:56,  1.65s/it]
Outer training loop:   8%|7         | 3910/50000 [1:26:41<20:53:05,  1.63s/it]
Outer training loop:   8%|7         | 3911/50000 [1:26:43<20:30:05,  1.60s/it]
Outer training loop:   8%|7         | 3912/50000 [1:26:44<20:10:45,  1.58s/it]
Outer training loop:   8%|7         | 3913/50000 [1:26:46<19:54:43,  1.56s/it]
Outer training loop:   8%|7         | 3914/50000 [1:26:48<20:08:02,  1.57s/it]
Outer training loop:   8%|7         | 3915/50000 [1:26:49<20:21:19,  1.59s/it]
Outer training loop:   8%|7         | 3916/50000 [1:26:51<20:40:37,  1.62s/it]
Outer training loop:   8%|7         | 3917/50000 [1:26:52<20:08:11,  1.57s/it]
Outer training loop:   8%|7         | 3918/50000 [1:26:54<20:22:20,  1.59s/it]
Outer training loop:   8%|7         | 3919/50000 [1:26:56<20:41:43,  1.62s/it]
Outer training loop:   8%|7         | 3920/50000 [1:26:57<20:38:57,  1.61s/it]
Outer training loop:   8%|7         | 3921/50000 [1:26:59<20:24:46,  1.59s/it]
Outer training loop:   8%|7         | 3922/50000 [1:27:00<20:36:14,  1.61s/it]
Outer training loop:   8%|7         | 3923/50000 [1:27:02<20:40:30,  1.62s/it]
Outer training loop:   8%|7         | 3924/50000 [1:27:04<21:08:46,  1.65s/it]
Outer training loop:   8%|7         | 3925/50000 [1:27:05<20:42:12,  1.62s/it]
Outer training loop:   8%|7         | 3926/50000 [1:27:07<20:43:46,  1.62s/it]
Outer training loop:   8%|7         | 3927/50000 [1:27:09<20:25:52,  1.60s/it]
Outer training loop:   8%|7         | 3928/50000 [1:27:10<20:43:06,  1.62s/it]
Outer training loop:   8%|7         | 3929/50000 [1:27:12<20:50:49,  1.63s/it]
Outer training loop:   8%|7         | 3930/50000 [1:27:13<20:17:25,  1.59s/it]
Outer training loop:   8%|7         | 3931/50000 [1:27:15<19:53:44,  1.55s/it]
Outer training loop:   8%|7         | 3932/50000 [1:27:16<20:11:55,  1.58s/it]
Outer training loop:   8%|7         | 3933/50000 [1:27:18<20:02:26,  1.57s/it]
Outer training loop:   8%|7         | 3934/50000 [1:27:20<20:03:13,  1.57s/it]
Outer training loop:   8%|7         | 3935/50000 [1:27:21<19:56:58,  1.56s/it]
Outer training loop:   8%|7         | 3936/50000 [1:27:23<20:02:51,  1.57s/it]
Outer training loop:   8%|7         | 3937/50000 [1:27:24<19:44:18,  1.54s/it]
Outer training loop:   8%|7         | 3938/50000 [1:27:26<19:43:12,  1.54s/it]
Outer training loop:   8%|7         | 3939/50000 [1:27:27<19:24:47,  1.52s/it]
Outer training loop:   8%|7         | 3940/50000 [1:27:29<20:00:57,  1.56s/it]
Outer training loop:   8%|7         | 3941/50000 [1:27:30<19:54:55,  1.56s/it]
Outer training loop:   8%|7         | 3942/50000 [1:27:32<20:02:07,  1.57s/it]
Outer training loop:   8%|7         | 3943/50000 [1:27:34<20:23:17,  1.59s/it]
Outer training loop:   8%|7         | 3944/50000 [1:27:35<20:28:27,  1.60s/it]
Outer training loop:   8%|7         | 3945/50000 [1:27:37<20:22:48,  1.59s/it]
Outer training loop:   8%|7         | 3946/50000 [1:27:38<20:05:12,  1.57s/it]
Outer training loop:   8%|7         | 3947/50000 [1:27:40<19:54:25,  1.56s/it]
Outer training loop:   8%|7         | 3948/50000 [1:27:41<19:52:50,  1.55s/it]
Outer training loop:   8%|7         | 3949/50000 [1:27:43<19:35:11,  1.53s/it]
Outer training loop:   8%|7         | 3950/50000 [1:27:45<19:59:09,  1.56s/it]
Outer training loop:   8%|7         | 3951/50000 [1:27:46<19:42:51,  1.54s/it]
Outer training loop:   8%|7         | 3952/50000 [1:27:48<19:57:42,  1.56s/it]
Outer training loop:   8%|7         | 3953/50000 [1:27:49<19:44:49,  1.54s/it]
Outer training loop:   8%|7         | 3954/50000 [1:27:51<20:10:57,  1.58s/it]
Outer training loop:   8%|7         | 3955/50000 [1:27:52<19:56:41,  1.56s/it]
Outer training loop:   8%|7         | 3956/50000 [1:27:54<20:11:04,  1.58s/it]
Outer training loop:   8%|7         | 3957/50000 [1:27:55<20:02:06,  1.57s/it]
Outer training loop:   8%|7         | 3958/50000 [1:27:57<20:15:25,  1.58s/it]
Outer training loop:   8%|7         | 3959/50000 [1:27:59<20:00:51,  1.56s/it]
Outer training loop:   8%|7         | 3960/50000 [1:28:00<20:05:30,  1.57s/it]
Outer training loop:   8%|7         | 3961/50000 [1:28:02<20:01:39,  1.57s/it]
Outer training loop:   8%|7         | 3962/50000 [1:28:03<20:05:37,  1.57s/it]
Outer training loop:   8%|7         | 3963/50000 [1:28:05<19:59:29,  1.56s/it]
Outer training loop:   8%|7         | 3964/50000 [1:28:06<20:00:42,  1.56s/it]
Outer training loop:   8%|7         | 3965/50000 [1:28:08<19:48:03,  1.55s/it]
Outer training loop:   8%|7         | 3966/50000 [1:28:10<20:29:17,  1.60s/it]
Outer training loop:   8%|7         | 3967/50000 [1:28:11<20:16:49,  1.59s/it]
Outer training loop:   8%|7         | 3968/50000 [1:28:13<20:10:24,  1.58s/it]
Outer training loop:   8%|7         | 3969/50000 [1:28:14<20:30:34,  1.60s/it]
Outer training loop:   8%|7         | 3970/50000 [1:28:16<20:47:06,  1.63s/it]
Outer training loop:   8%|7         | 3971/50000 [1:28:18<20:24:38,  1.60s/it]
Outer training loop:   8%|7         | 3972/50000 [1:28:19<20:34:45,  1.61s/it]
Outer training loop:   8%|7         | 3973/50000 [1:28:21<20:31:08,  1.60s/it]
Outer training loop:   8%|7         | 3974/50000 [1:28:22<20:16:56,  1.59s/it]
Outer training loop:   8%|7         | 3975/50000 [1:28:24<20:05:32,  1.57s/it]
Outer training loop:   8%|7         | 3976/50000 [1:28:26<20:04:17,  1.57s/it]
Outer training loop:   8%|7         | 3977/50000 [1:28:27<20:00:23,  1.56s/it]
Outer training loop:   8%|7         | 3978/50000 [1:28:29<20:14:12,  1.58s/it]
Outer training loop:   8%|7         | 3979/50000 [1:28:30<20:01:10,  1.57s/it]
Outer training loop:   8%|7         | 3980/50000 [1:28:32<20:04:04,  1.57s/it]
Outer training loop:   8%|7         | 3981/50000 [1:28:33<20:07:58,  1.57s/it]
Outer training loop:   8%|7         | 3982/50000 [1:28:35<20:26:24,  1.60s/it]
Outer training loop:   8%|7         | 3983/50000 [1:28:37<20:09:34,  1.58s/it]
Outer training loop:   8%|7         | 3984/50000 [1:28:38<20:44:51,  1.62s/it]
Outer training loop:   8%|7         | 3985/50000 [1:28:40<20:33:42,  1.61s/it]
Outer training loop:   8%|7         | 3986/50000 [1:28:42<20:34:46,  1.61s/it]
Outer training loop:   8%|7         | 3987/50000 [1:28:43<20:02:52,  1.57s/it]
Outer training loop:   8%|7         | 3988/50000 [1:28:45<20:22:55,  1.59s/it]
Outer training loop:   8%|7         | 3989/50000 [1:28:46<20:08:16,  1.58s/it]
Outer training loop:   8%|7         | 3990/50000 [1:28:48<20:19:12,  1.59s/it]
Outer training loop:   8%|7         | 3991/50000 [1:28:49<19:58:30,  1.56s/it]
Outer training loop:   8%|7         | 3992/50000 [1:28:51<20:12:04,  1.58s/it]
Outer training loop:   8%|7         | 3993/50000 [1:28:52<19:40:20,  1.54s/it]
Outer training loop:   8%|7         | 3994/50000 [1:28:54<19:51:43,  1.55s/it]
Outer training loop:   8%|7         | 3995/50000 [1:28:55<19:39:07,  1.54s/it]
Outer training loop:   8%|7         | 3996/50000 [1:28:57<19:22:11,  1.52s/it]
Outer training loop:   8%|7         | 3997/50000 [1:28:59<19:57:11,  1.56s/it]
Outer training loop:   8%|7         | 3998/50000 [1:29:00<20:11:03,  1.58s/it]
Outer training loop:   8%|7         | 3999/50000 [1:29:02<19:51:30,  1.55s/it]
Outer training loop:   8%|8         | 4000/50000 [1:29:03<20:21:46,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.44s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.50s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.47s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:06,  1.54s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:03,  1.53s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:03,  1.54s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<01:59,  1.52s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:00,  1.55s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<01:57,  1.53s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:58,  1.56s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:56,  1.55s/it]

Evaluating policy:  11%|#         | 9/83 [00:13<01:55,  1.57s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:56,  1.59s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:54,  1.59s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:53,  1.60s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:48,  1.56s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:46,  1.54s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:51,  1.64s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:46,  1.58s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:41,  1.54s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:39,  1.53s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:41,  1.58s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:38,  1.56s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:35,  1.54s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:36,  1.58s/it]

Evaluating policy:  28%|##7       | 23/83 [00:35<01:33,  1.56s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:31,  1.55s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:33,  1.62s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:30,  1.60s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:28,  1.58s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:25,  1.56s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:21,  1.51s/it]

Evaluating policy:  36%|###6      | 30/83 [00:46<01:23,  1.58s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:21,  1.56s/it]

Evaluating policy:  39%|###8      | 32/83 [00:49<01:18,  1.53s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:16,  1.52s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:16,  1.57s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:14,  1.55s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:13,  1.56s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:10,  1.52s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:08,  1.52s/it]

Evaluating policy:  47%|####6     | 39/83 [01:00<01:10,  1.60s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:06,  1.56s/it]

Evaluating policy:  49%|####9     | 41/83 [01:03<01:04,  1.53s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:02,  1.52s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:06<01:00,  1.50s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<01:02,  1.59s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<00:59,  1.57s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:11<00:56,  1.54s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:13<00:54,  1.52s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:14<00:53,  1.52s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:51,  1.51s/it]

Evaluating policy:  60%|######    | 50/83 [01:17<00:52,  1.59s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:50,  1.57s/it]

Evaluating policy:  63%|######2   | 52/83 [01:20<00:47,  1.55s/it]

Evaluating policy:  64%|######3   | 53/83 [01:22<00:45,  1.52s/it]

Evaluating policy:  65%|######5   | 54/83 [01:23<00:44,  1.52s/it]

Evaluating policy:  66%|######6   | 55/83 [01:25<00:42,  1.53s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:43,  1.62s/it]

Evaluating policy:  69%|######8   | 57/83 [01:28<00:41,  1.58s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:39,  1.58s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:31<00:37,  1.56s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:33<00:35,  1.55s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:34<00:34,  1.55s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:36<00:32,  1.54s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:32,  1.62s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:39<00:30,  1.60s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:29,  1.63s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:42<00:26,  1.59s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:24,  1.55s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:23,  1.57s/it]

Evaluating policy:  83%|########3 | 69/83 [01:47<00:21,  1.54s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:21,  1.65s/it]

Evaluating policy:  86%|########5 | 71/83 [01:50<00:19,  1.59s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:17,  1.58s/it]

Evaluating policy:  88%|########7 | 73/83 [01:53<00:15,  1.57s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:14,  1.58s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.57s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:10,  1.56s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.57s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:08,  1.67s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.63s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.60s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.56s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.55s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.51s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:00,  1.47s/it]

Evaluating policy:   2%|2         | 2/83 [00:02<01:58,  1.47s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:07,  1.59s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:02,  1.56s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:02,  1.57s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<01:59,  1.55s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:58,  1.56s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:58,  1.58s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:56,  1.57s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:55,  1.58s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:55,  1.60s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:51,  1.57s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:50,  1.59s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:47,  1.56s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:45,  1.55s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:47,  1.60s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:45,  1.59s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:42,  1.58s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:39,  1.56s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:40,  1.60s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:38,  1.59s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:36,  1.58s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:38,  1.64s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:34,  1.59s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:30,  1.56s/it]

Evaluating policy:  31%|###1      | 26/83 [00:41<01:28,  1.56s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:29,  1.60s/it]

Evaluating policy:  34%|###3      | 28/83 [00:44<01:26,  1.57s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:23,  1.55s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:21,  1.53s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:22,  1.58s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:23,  1.63s/it]

Evaluating policy:  40%|###9      | 33/83 [00:52<01:19,  1.59s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:17,  1.59s/it]

Evaluating policy:  42%|####2     | 35/83 [00:55<01:15,  1.57s/it]

Evaluating policy:  43%|####3     | 36/83 [00:57<01:16,  1.62s/it]

Evaluating policy:  45%|####4     | 37/83 [00:58<01:12,  1.57s/it]

Evaluating policy:  46%|####5     | 38/83 [01:00<01:10,  1.57s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:07,  1.54s/it]

Evaluating policy:  48%|####8     | 40/83 [01:03<01:06,  1.54s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:07,  1.61s/it]

Evaluating policy:  51%|#####     | 42/83 [01:06<01:05,  1.60s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:03,  1.58s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:09<01:00,  1.56s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:11<00:59,  1.56s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<01:00,  1.63s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:14<00:57,  1.60s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:54,  1.57s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:17<00:52,  1.55s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:49,  1.51s/it]

Evaluating policy:  61%|######1   | 51/83 [01:20<00:51,  1.62s/it]

Evaluating policy:  63%|######2   | 52/83 [01:22<00:49,  1.59s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:46,  1.56s/it]

Evaluating policy:  65%|######5   | 54/83 [01:25<00:44,  1.52s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:42,  1.50s/it]

Evaluating policy:  67%|######7   | 56/83 [01:28<00:40,  1.50s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:42,  1.62s/it]

Evaluating policy:  70%|######9   | 58/83 [01:31<00:40,  1.60s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:33<00:37,  1.57s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:34<00:35,  1.54s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:36<00:33,  1.54s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:37<00:32,  1.53s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:39<00:30,  1.52s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:30,  1.61s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:42<00:28,  1.56s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:26,  1.55s/it]

Evaluating policy:  81%|########  | 67/83 [01:45<00:24,  1.55s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:22,  1.52s/it]

Evaluating policy:  83%|########3 | 69/83 [01:48<00:21,  1.55s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:19,  1.53s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:20,  1.67s/it]

Evaluating policy:  87%|########6 | 72/83 [01:53<00:17,  1.61s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:15,  1.58s/it]

Evaluating policy:  89%|########9 | 74/83 [01:56<00:13,  1.55s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.55s/it]

Evaluating policy:  92%|#########1| 76/83 [01:59<00:10,  1.55s/it]

Evaluating policy:  93%|#########2| 77/83 [02:01<00:09,  1.55s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.51s/it]

Evaluating policy:  95%|#########5| 79/83 [02:04<00:06,  1.65s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.59s/it]

Evaluating policy:  98%|#########7| 81/83 [02:07<00:03,  1.55s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.58s/it]

Evaluating policy: 100%|##########| 83/83 [02:10<00:00,  1.55s/it]

Outer training loop:   8%|8         | 4001/50000 [1:33:30<1035:07:26, 81.01s/it]
Outer training loop:   8%|8         | 4002/50000 [1:33:31<730:08:17, 57.14s/it] 
Outer training loop:   8%|8         | 4003/50000 [1:33:33<517:12:48, 40.48s/it]
Outer training loop:   8%|8         | 4004/50000 [1:33:34<367:52:13, 28.79s/it]
Outer training loop:   8%|8         | 4005/50000 [1:33:36<265:23:01, 20.77s/it]
Outer training loop:   8%|8         | 4006/50000 [1:33:38<191:33:13, 14.99s/it]
Outer training loop:   8%|8         | 4007/50000 [1:33:39<140:02:55, 10.96s/it]
Outer training loop:   8%|8         | 4008/50000 [1:33:41<104:17:50,  8.16s/it]
Outer training loop:   8%|8         | 4009/50000 [1:33:43<79:21:10,  6.21s/it] 
Outer training loop:   8%|8         | 4010/50000 [1:33:44<61:22:36,  4.80s/it]
Outer training loop:   8%|8         | 4011/50000 [1:33:46<49:04:19,  3.84s/it]
Outer training loop:   8%|8         | 4012/50000 [1:33:47<39:57:27,  3.13s/it]
Outer training loop:   8%|8         | 4013/50000 [1:33:49<33:53:00,  2.65s/it]
Outer training loop:   8%|8         | 4014/50000 [1:33:50<29:55:31,  2.34s/it]
Outer training loop:   8%|8         | 4015/50000 [1:33:52<26:59:27,  2.11s/it]
Outer training loop:   8%|8         | 4016/50000 [1:33:53<24:38:24,  1.93s/it]
Outer training loop:   8%|8         | 4017/50000 [1:33:55<22:49:44,  1.79s/it]
Outer training loop:   8%|8         | 4018/50000 [1:33:57<22:27:14,  1.76s/it]
Outer training loop:   8%|8         | 4019/50000 [1:33:58<21:52:06,  1.71s/it]
Outer training loop:   8%|8         | 4020/50000 [1:34:00<21:16:04,  1.67s/it]
Outer training loop:   8%|8         | 4021/50000 [1:34:01<20:26:53,  1.60s/it]
Outer training loop:   8%|8         | 4022/50000 [1:34:03<20:13:43,  1.58s/it]
Outer training loop:   8%|8         | 4023/50000 [1:34:04<19:47:34,  1.55s/it]
Outer training loop:   8%|8         | 4024/50000 [1:34:06<19:55:04,  1.56s/it]
Outer training loop:   8%|8         | 4025/50000 [1:34:07<19:49:35,  1.55s/it]
Outer training loop:   8%|8         | 4026/50000 [1:34:09<20:07:00,  1.58s/it]
Outer training loop:   8%|8         | 4027/50000 [1:34:11<19:54:53,  1.56s/it]
Outer training loop:   8%|8         | 4028/50000 [1:34:12<19:59:36,  1.57s/it]
Outer training loop:   8%|8         | 4029/50000 [1:34:14<19:38:35,  1.54s/it]
Outer training loop:   8%|8         | 4030/50000 [1:34:15<19:41:04,  1.54s/it]
Outer training loop:   8%|8         | 4031/50000 [1:34:17<19:40:10,  1.54s/it]
Outer training loop:   8%|8         | 4032/50000 [1:34:18<19:51:44,  1.56s/it]
Outer training loop:   8%|8         | 4033/50000 [1:34:20<19:52:51,  1.56s/it]
Outer training loop:   8%|8         | 4034/50000 [1:34:21<19:46:03,  1.55s/it]
Outer training loop:   8%|8         | 4035/50000 [1:34:23<19:40:10,  1.54s/it]
Outer training loop:   8%|8         | 4036/50000 [1:34:25<20:01:28,  1.57s/it]
Outer training loop:   8%|8         | 4037/50000 [1:34:26<19:44:50,  1.55s/it]
Outer training loop:   8%|8         | 4038/50000 [1:34:28<20:00:43,  1.57s/it]
Outer training loop:   8%|8         | 4039/50000 [1:34:29<19:49:03,  1.55s/it]
Outer training loop:   8%|8         | 4040/50000 [1:34:31<19:46:56,  1.55s/it]
Outer training loop:   8%|8         | 4041/50000 [1:34:32<19:40:52,  1.54s/it]
Outer training loop:   8%|8         | 4042/50000 [1:34:34<19:52:12,  1.56s/it]
Outer training loop:   8%|8         | 4043/50000 [1:34:35<19:41:27,  1.54s/it]
Outer training loop:   8%|8         | 4044/50000 [1:34:37<19:52:28,  1.56s/it]
Outer training loop:   8%|8         | 4045/50000 [1:34:38<19:50:41,  1.55s/it]
Outer training loop:   8%|8         | 4046/50000 [1:34:40<20:02:36,  1.57s/it]
Outer training loop:   8%|8         | 4047/50000 [1:34:42<19:41:43,  1.54s/it]
Outer training loop:   8%|8         | 4048/50000 [1:34:43<19:51:47,  1.56s/it]
Outer training loop:   8%|8         | 4049/50000 [1:34:45<19:43:45,  1.55s/it]
Outer training loop:   8%|8         | 4050/50000 [1:34:46<20:25:55,  1.60s/it]
Outer training loop:   8%|8         | 4051/50000 [1:34:48<20:28:23,  1.60s/it]
Outer training loop:   8%|8         | 4052/50000 [1:34:50<20:11:10,  1.58s/it]
Outer training loop:   8%|8         | 4053/50000 [1:34:51<19:33:58,  1.53s/it]
Outer training loop:   8%|8         | 4054/50000 [1:34:52<19:34:24,  1.53s/it]
Outer training loop:   8%|8         | 4055/50000 [1:34:54<19:31:58,  1.53s/it]
Outer training loop:   8%|8         | 4056/50000 [1:34:56<19:39:40,  1.54s/it]
Outer training loop:   8%|8         | 4057/50000 [1:34:57<19:32:08,  1.53s/it]
Outer training loop:   8%|8         | 4058/50000 [1:34:59<19:50:01,  1.55s/it]
Outer training loop:   8%|8         | 4059/50000 [1:35:00<19:31:34,  1.53s/it]
Outer training loop:   8%|8         | 4060/50000 [1:35:02<19:35:04,  1.53s/it]
Outer training loop:   8%|8         | 4061/50000 [1:35:03<19:42:27,  1.54s/it]
Outer training loop:   8%|8         | 4062/50000 [1:35:05<20:02:29,  1.57s/it]
Outer training loop:   8%|8         | 4063/50000 [1:35:06<19:52:06,  1.56s/it]
Outer training loop:   8%|8         | 4064/50000 [1:35:08<20:00:33,  1.57s/it]
Outer training loop:   8%|8         | 4065/50000 [1:35:10<19:51:17,  1.56s/it]
Outer training loop:   8%|8         | 4066/50000 [1:35:11<19:50:35,  1.56s/it]
Outer training loop:   8%|8         | 4067/50000 [1:35:13<19:52:21,  1.56s/it]
Outer training loop:   8%|8         | 4068/50000 [1:35:14<19:58:31,  1.57s/it]
Outer training loop:   8%|8         | 4069/50000 [1:35:16<19:32:06,  1.53s/it]
Outer training loop:   8%|8         | 4070/50000 [1:35:17<19:52:43,  1.56s/it]
Outer training loop:   8%|8         | 4071/50000 [1:35:19<19:54:35,  1.56s/it]
Outer training loop:   8%|8         | 4072/50000 [1:35:20<20:06:39,  1.58s/it]
Outer training loop:   8%|8         | 4073/50000 [1:35:22<19:52:50,  1.56s/it]
Outer training loop:   8%|8         | 4074/50000 [1:35:24<20:01:44,  1.57s/it]
Outer training loop:   8%|8         | 4075/50000 [1:35:25<19:55:09,  1.56s/it]
Outer training loop:   8%|8         | 4076/50000 [1:35:27<20:09:18,  1.58s/it]
Outer training loop:   8%|8         | 4077/50000 [1:35:28<19:48:12,  1.55s/it]
Outer training loop:   8%|8         | 4078/50000 [1:35:30<19:56:28,  1.56s/it]
Outer training loop:   8%|8         | 4079/50000 [1:35:31<19:46:20,  1.55s/it]
Outer training loop:   8%|8         | 4080/50000 [1:35:33<19:54:29,  1.56s/it]
Outer training loop:   8%|8         | 4081/50000 [1:35:35<20:16:19,  1.59s/it]
Outer training loop:   8%|8         | 4082/50000 [1:35:36<19:57:30,  1.56s/it]
Outer training loop:   8%|8         | 4083/50000 [1:35:38<20:02:05,  1.57s/it]
Outer training loop:   8%|8         | 4084/50000 [1:35:39<19:57:02,  1.56s/it]
Outer training loop:   8%|8         | 4085/50000 [1:35:41<19:46:41,  1.55s/it]
Outer training loop:   8%|8         | 4086/50000 [1:35:42<20:06:50,  1.58s/it]
Outer training loop:   8%|8         | 4087/50000 [1:35:44<20:35:11,  1.61s/it]
Outer training loop:   8%|8         | 4088/50000 [1:35:46<20:18:49,  1.59s/it]
Outer training loop:   8%|8         | 4089/50000 [1:35:47<20:26:08,  1.60s/it]
Outer training loop:   8%|8         | 4090/50000 [1:35:49<20:06:11,  1.58s/it]
Outer training loop:   8%|8         | 4091/50000 [1:35:50<20:33:33,  1.61s/it]
Outer training loop:   8%|8         | 4092/50000 [1:35:52<20:00:24,  1.57s/it]
Outer training loop:   8%|8         | 4093/50000 [1:35:54<20:01:13,  1.57s/it]
Outer training loop:   8%|8         | 4094/50000 [1:35:55<20:02:03,  1.57s/it]
Outer training loop:   8%|8         | 4095/50000 [1:35:57<20:05:48,  1.58s/it]
Outer training loop:   8%|8         | 4096/50000 [1:35:58<20:00:50,  1.57s/it]
Outer training loop:   8%|8         | 4097/50000 [1:36:00<19:51:46,  1.56s/it]
Outer training loop:   8%|8         | 4098/50000 [1:36:01<19:36:15,  1.54s/it]
Outer training loop:   8%|8         | 4099/50000 [1:36:03<19:49:05,  1.55s/it]
Outer training loop:   8%|8         | 4100/50000 [1:36:05<20:14:10,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.59s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.57s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.56s/it]

Outer training loop:   8%|8         | 4101/50000 [1:36:11<37:55:43,  2.97s/it]
Outer training loop:   8%|8         | 4102/50000 [1:36:12<31:57:24,  2.51s/it]
Outer training loop:   8%|8         | 4103/50000 [1:36:14<28:37:46,  2.25s/it]
Outer training loop:   8%|8         | 4104/50000 [1:36:15<25:56:02,  2.03s/it]
Outer training loop:   8%|8         | 4105/50000 [1:36:17<24:19:54,  1.91s/it]
Outer training loop:   8%|8         | 4106/50000 [1:36:19<23:09:55,  1.82s/it]
Outer training loop:   8%|8         | 4107/50000 [1:36:20<21:58:51,  1.72s/it]
Outer training loop:   8%|8         | 4108/50000 [1:36:22<21:23:18,  1.68s/it]
Outer training loop:   8%|8         | 4109/50000 [1:36:23<20:46:43,  1.63s/it]
Outer training loop:   8%|8         | 4110/50000 [1:36:25<21:08:36,  1.66s/it]
Outer training loop:   8%|8         | 4111/50000 [1:36:26<20:23:00,  1.60s/it]
Outer training loop:   8%|8         | 4112/50000 [1:36:28<20:30:42,  1.61s/it]
Outer training loop:   8%|8         | 4113/50000 [1:36:30<20:16:21,  1.59s/it]
Outer training loop:   8%|8         | 4114/50000 [1:36:31<20:20:55,  1.60s/it]
Outer training loop:   8%|8         | 4115/50000 [1:36:33<20:24:34,  1.60s/it]
Outer training loop:   8%|8         | 4116/50000 [1:36:34<20:20:03,  1.60s/it]
Outer training loop:   8%|8         | 4117/50000 [1:36:36<19:51:09,  1.56s/it]
Outer training loop:   8%|8         | 4118/50000 [1:36:37<19:48:21,  1.55s/it]
Outer training loop:   8%|8         | 4119/50000 [1:36:39<19:42:27,  1.55s/it]
Outer training loop:   8%|8         | 4120/50000 [1:36:40<19:36:48,  1.54s/it]
Outer training loop:   8%|8         | 4121/50000 [1:36:42<20:02:20,  1.57s/it]
Outer training loop:   8%|8         | 4122/50000 [1:36:44<19:41:56,  1.55s/it]
Outer training loop:   8%|8         | 4123/50000 [1:36:45<20:06:33,  1.58s/it]
Outer training loop:   8%|8         | 4124/50000 [1:36:47<19:30:53,  1.53s/it]
Outer training loop:   8%|8         | 4125/50000 [1:36:48<20:06:04,  1.58s/it]
Outer training loop:   8%|8         | 4126/50000 [1:36:50<20:06:21,  1.58s/it]
Outer training loop:   8%|8         | 4127/50000 [1:36:51<19:47:35,  1.55s/it]
Outer training loop:   8%|8         | 4128/50000 [1:36:53<19:37:01,  1.54s/it]
Outer training loop:   8%|8         | 4129/50000 [1:36:54<19:47:24,  1.55s/it]
Outer training loop:   8%|8         | 4130/50000 [1:36:56<19:17:50,  1.51s/it]
Outer training loop:   8%|8         | 4131/50000 [1:36:57<19:28:19,  1.53s/it]
Outer training loop:   8%|8         | 4132/50000 [1:36:59<19:05:21,  1.50s/it]
Outer training loop:   8%|8         | 4133/50000 [1:37:00<19:23:05,  1.52s/it]
Outer training loop:   8%|8         | 4134/50000 [1:37:02<19:08:24,  1.50s/it]
Outer training loop:   8%|8         | 4135/50000 [1:37:04<19:54:08,  1.56s/it]
Outer training loop:   8%|8         | 4136/50000 [1:37:05<19:37:03,  1.54s/it]
Outer training loop:   8%|8         | 4137/50000 [1:37:07<19:39:40,  1.54s/it]
Outer training loop:   8%|8         | 4138/50000 [1:37:08<19:33:58,  1.54s/it]
Outer training loop:   8%|8         | 4139/50000 [1:37:10<19:59:51,  1.57s/it]
Outer training loop:   8%|8         | 4140/50000 [1:37:11<19:51:47,  1.56s/it]
Outer training loop:   8%|8         | 4141/50000 [1:37:13<20:01:37,  1.57s/it]
Outer training loop:   8%|8         | 4142/50000 [1:37:14<19:33:58,  1.54s/it]
Outer training loop:   8%|8         | 4143/50000 [1:37:16<19:47:58,  1.55s/it]
Outer training loop:   8%|8         | 4144/50000 [1:37:17<19:33:21,  1.54s/it]
Outer training loop:   8%|8         | 4145/50000 [1:37:19<19:39:15,  1.54s/it]
Outer training loop:   8%|8         | 4146/50000 [1:37:21<19:35:11,  1.54s/it]
Outer training loop:   8%|8         | 4147/50000 [1:37:22<20:15:01,  1.59s/it]
Outer training loop:   8%|8         | 4148/50000 [1:37:24<19:32:31,  1.53s/it]
Outer training loop:   8%|8         | 4149/50000 [1:37:25<19:49:42,  1.56s/it]
Outer training loop:   8%|8         | 4150/50000 [1:37:27<19:33:56,  1.54s/it]
Outer training loop:   8%|8         | 4151/50000 [1:37:28<19:57:10,  1.57s/it]
Outer training loop:   8%|8         | 4152/50000 [1:37:30<20:14:28,  1.59s/it]
Outer training loop:   8%|8         | 4153/50000 [1:37:32<20:04:10,  1.58s/it]
Outer training loop:   8%|8         | 4154/50000 [1:37:33<20:17:11,  1.59s/it]
Outer training loop:   8%|8         | 4155/50000 [1:37:35<20:39:37,  1.62s/it]
Outer training loop:   8%|8         | 4156/50000 [1:37:36<20:21:36,  1.60s/it]
Outer training loop:   8%|8         | 4157/50000 [1:37:38<20:38:09,  1.62s/it]
Outer training loop:   8%|8         | 4158/50000 [1:37:40<20:01:49,  1.57s/it]
Outer training loop:   8%|8         | 4159/50000 [1:37:41<20:07:42,  1.58s/it]
Outer training loop:   8%|8         | 4160/50000 [1:37:43<19:44:00,  1.55s/it]
Outer training loop:   8%|8         | 4161/50000 [1:37:44<19:55:00,  1.56s/it]
Outer training loop:   8%|8         | 4162/50000 [1:37:46<19:46:34,  1.55s/it]
Outer training loop:   8%|8         | 4163/50000 [1:37:47<19:46:25,  1.55s/it]
Outer training loop:   8%|8         | 4164/50000 [1:37:49<19:47:29,  1.55s/it]
Outer training loop:   8%|8         | 4165/50000 [1:37:51<20:14:38,  1.59s/it]
Outer training loop:   8%|8         | 4166/50000 [1:37:52<20:02:32,  1.57s/it]
Outer training loop:   8%|8         | 4167/50000 [1:37:54<19:45:54,  1.55s/it]
Outer training loop:   8%|8         | 4168/50000 [1:37:55<20:03:25,  1.58s/it]
Outer training loop:   8%|8         | 4169/50000 [1:37:57<19:59:52,  1.57s/it]
Outer training loop:   8%|8         | 4170/50000 [1:37:58<19:42:21,  1.55s/it]
Outer training loop:   8%|8         | 4171/50000 [1:38:00<19:38:41,  1.54s/it]
Outer training loop:   8%|8         | 4172/50000 [1:38:01<19:39:22,  1.54s/it]
Outer training loop:   8%|8         | 4173/50000 [1:38:03<19:43:05,  1.55s/it]
Outer training loop:   8%|8         | 4174/50000 [1:38:04<19:43:29,  1.55s/it]
Outer training loop:   8%|8         | 4175/50000 [1:38:06<19:52:01,  1.56s/it]
Outer training loop:   8%|8         | 4176/50000 [1:38:08<19:51:53,  1.56s/it]
Outer training loop:   8%|8         | 4177/50000 [1:38:09<19:39:09,  1.54s/it]
Outer training loop:   8%|8         | 4178/50000 [1:38:11<19:28:50,  1.53s/it]
Outer training loop:   8%|8         | 4179/50000 [1:38:12<19:21:08,  1.52s/it]
Outer training loop:   8%|8         | 4180/50000 [1:38:14<19:31:00,  1.53s/it]
Outer training loop:   8%|8         | 4181/50000 [1:38:15<19:06:46,  1.50s/it]
Outer training loop:   8%|8         | 4182/50000 [1:38:17<19:23:36,  1.52s/it]
Outer training loop:   8%|8         | 4183/50000 [1:38:18<19:15:40,  1.51s/it]
Outer training loop:   8%|8         | 4184/50000 [1:38:20<19:46:05,  1.55s/it]
Outer training loop:   8%|8         | 4185/50000 [1:38:21<19:24:25,  1.52s/it]
Outer training loop:   8%|8         | 4186/50000 [1:38:23<19:31:23,  1.53s/it]
Outer training loop:   8%|8         | 4187/50000 [1:38:24<19:13:13,  1.51s/it]
Outer training loop:   8%|8         | 4188/50000 [1:38:26<19:42:32,  1.55s/it]
Outer training loop:   8%|8         | 4189/50000 [1:38:27<19:39:05,  1.54s/it]
Outer training loop:   8%|8         | 4190/50000 [1:38:29<19:27:53,  1.53s/it]
Outer training loop:   8%|8         | 4191/50000 [1:38:30<19:12:27,  1.51s/it]
Outer training loop:   8%|8         | 4192/50000 [1:38:32<19:21:00,  1.52s/it]
Outer training loop:   8%|8         | 4193/50000 [1:38:34<19:28:31,  1.53s/it]
Outer training loop:   8%|8         | 4194/50000 [1:38:35<19:47:15,  1.56s/it]
Outer training loop:   8%|8         | 4195/50000 [1:38:37<19:40:46,  1.55s/it]
Outer training loop:   8%|8         | 4196/50000 [1:38:38<19:40:21,  1.55s/it]
Outer training loop:   8%|8         | 4197/50000 [1:38:40<20:33:18,  1.62s/it]
Outer training loop:   8%|8         | 4198/50000 [1:38:42<20:13:04,  1.59s/it]
Outer training loop:   8%|8         | 4199/50000 [1:38:43<20:17:57,  1.60s/it]
Outer training loop:   8%|8         | 4200/50000 [1:38:45<19:51:11,  1.56s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.53s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.57s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.53s/it]

Outer training loop:   8%|8         | 4201/50000 [1:38:51<37:48:37,  2.97s/it]
Outer training loop:   8%|8         | 4202/50000 [1:38:52<32:24:31,  2.55s/it]
Outer training loop:   8%|8         | 4203/50000 [1:38:54<28:30:41,  2.24s/it]
Outer training loop:   8%|8         | 4204/50000 [1:38:56<26:01:56,  2.05s/it]
Outer training loop:   8%|8         | 4205/50000 [1:38:57<23:59:34,  1.89s/it]
Outer training loop:   8%|8         | 4206/50000 [1:38:59<22:53:21,  1.80s/it]
Outer training loop:   8%|8         | 4207/50000 [1:39:00<21:49:07,  1.72s/it]
Outer training loop:   8%|8         | 4208/50000 [1:39:02<20:59:28,  1.65s/it]
Outer training loop:   8%|8         | 4209/50000 [1:39:03<20:23:11,  1.60s/it]
Outer training loop:   8%|8         | 4210/50000 [1:39:05<20:23:42,  1.60s/it]
Outer training loop:   8%|8         | 4211/50000 [1:39:06<20:01:11,  1.57s/it]
Outer training loop:   8%|8         | 4212/50000 [1:39:08<19:54:06,  1.56s/it]
Outer training loop:   8%|8         | 4213/50000 [1:39:09<19:32:46,  1.54s/it]
Outer training loop:   8%|8         | 4214/50000 [1:39:11<19:33:51,  1.54s/it]
Outer training loop:   8%|8         | 4215/50000 [1:39:12<19:47:47,  1.56s/it]
Outer training loop:   8%|8         | 4216/50000 [1:39:14<20:04:07,  1.58s/it]
Outer training loop:   8%|8         | 4217/50000 [1:39:16<19:59:03,  1.57s/it]
Outer training loop:   8%|8         | 4218/50000 [1:39:17<20:04:49,  1.58s/it]
Outer training loop:   8%|8         | 4219/50000 [1:39:19<19:54:16,  1.57s/it]
Outer training loop:   8%|8         | 4220/50000 [1:39:20<19:32:56,  1.54s/it]
Outer training loop:   8%|8         | 4221/50000 [1:39:22<20:01:32,  1.57s/it]
Outer training loop:   8%|8         | 4222/50000 [1:39:23<19:47:58,  1.56s/it]
Outer training loop:   8%|8         | 4223/50000 [1:39:25<20:00:09,  1.57s/it]
Outer training loop:   8%|8         | 4224/50000 [1:39:27<19:58:56,  1.57s/it]
Outer training loop:   8%|8         | 4225/50000 [1:39:28<19:48:30,  1.56s/it]
Outer training loop:   8%|8         | 4226/50000 [1:39:30<19:32:47,  1.54s/it]
Outer training loop:   8%|8         | 4227/50000 [1:39:31<19:38:14,  1.54s/it]
Outer training loop:   8%|8         | 4228/50000 [1:39:33<19:33:07,  1.54s/it]
Outer training loop:   8%|8         | 4229/50000 [1:39:34<20:07:07,  1.58s/it]
Outer training loop:   8%|8         | 4230/50000 [1:39:36<19:57:32,  1.57s/it]
Outer training loop:   8%|8         | 4231/50000 [1:39:38<20:10:14,  1.59s/it]
Outer training loop:   8%|8         | 4232/50000 [1:39:39<19:48:19,  1.56s/it]
Outer training loop:   8%|8         | 4233/50000 [1:39:41<19:57:24,  1.57s/it]
Outer training loop:   8%|8         | 4234/50000 [1:39:42<19:37:08,  1.54s/it]
Outer training loop:   8%|8         | 4235/50000 [1:39:44<19:46:12,  1.56s/it]
Outer training loop:   8%|8         | 4236/50000 [1:39:45<19:24:12,  1.53s/it]
Outer training loop:   8%|8         | 4237/50000 [1:39:47<19:42:15,  1.55s/it]
Outer training loop:   8%|8         | 4238/50000 [1:39:48<19:50:03,  1.56s/it]
Outer training loop:   8%|8         | 4239/50000 [1:39:50<19:38:50,  1.55s/it]
Outer training loop:   8%|8         | 4240/50000 [1:39:51<19:39:11,  1.55s/it]
Outer training loop:   8%|8         | 4241/50000 [1:39:53<19:16:33,  1.52s/it]
Outer training loop:   8%|8         | 4242/50000 [1:39:54<19:46:15,  1.56s/it]
Outer training loop:   8%|8         | 4243/50000 [1:39:56<19:52:14,  1.56s/it]
Outer training loop:   8%|8         | 4244/50000 [1:39:58<20:52:17,  1.64s/it]
Outer training loop:   8%|8         | 4245/50000 [1:40:00<20:42:39,  1.63s/it]
Outer training loop:   8%|8         | 4246/50000 [1:40:01<20:11:31,  1.59s/it]
Outer training loop:   8%|8         | 4247/50000 [1:40:03<20:12:59,  1.59s/it]
Outer training loop:   8%|8         | 4248/50000 [1:40:04<19:50:46,  1.56s/it]
Outer training loop:   8%|8         | 4249/50000 [1:40:06<19:53:22,  1.57s/it]
Outer training loop:   8%|8         | 4250/50000 [1:40:07<19:57:51,  1.57s/it]
Outer training loop:   9%|8         | 4251/50000 [1:40:09<20:04:23,  1.58s/it]
Outer training loop:   9%|8         | 4252/50000 [1:40:10<20:19:13,  1.60s/it]
Outer training loop:   9%|8         | 4253/50000 [1:40:12<19:56:31,  1.57s/it]
Outer training loop:   9%|8         | 4254/50000 [1:40:14<20:17:59,  1.60s/it]
Outer training loop:   9%|8         | 4255/50000 [1:40:15<19:59:19,  1.57s/it]
Outer training loop:   9%|8         | 4256/50000 [1:40:17<20:09:45,  1.59s/it]
Outer training loop:   9%|8         | 4257/50000 [1:40:18<19:56:50,  1.57s/it]
Outer training loop:   9%|8         | 4258/50000 [1:40:20<20:02:32,  1.58s/it]
Outer training loop:   9%|8         | 4259/50000 [1:40:21<19:28:42,  1.53s/it]
Outer training loop:   9%|8         | 4260/50000 [1:40:23<19:41:42,  1.55s/it]
Outer training loop:   9%|8         | 4261/50000 [1:40:24<19:22:47,  1.53s/it]
Outer training loop:   9%|8         | 4262/50000 [1:40:26<19:44:24,  1.55s/it]
Outer training loop:   9%|8         | 4263/50000 [1:40:27<19:23:07,  1.53s/it]
Outer training loop:   9%|8         | 4264/50000 [1:40:29<19:48:04,  1.56s/it]
Outer training loop:   9%|8         | 4265/50000 [1:40:31<19:34:50,  1.54s/it]
Outer training loop:   9%|8         | 4266/50000 [1:40:32<19:47:17,  1.56s/it]
Outer training loop:   9%|8         | 4267/50000 [1:40:34<20:12:48,  1.59s/it]
Outer training loop:   9%|8         | 4268/50000 [1:40:36<20:21:55,  1.60s/it]
Outer training loop:   9%|8         | 4269/50000 [1:40:37<19:44:49,  1.55s/it]
Outer training loop:   9%|8         | 4270/50000 [1:40:39<19:52:03,  1.56s/it]
Outer training loop:   9%|8         | 4271/50000 [1:40:40<19:33:45,  1.54s/it]
Outer training loop:   9%|8         | 4272/50000 [1:40:42<19:40:07,  1.55s/it]
Outer training loop:   9%|8         | 4273/50000 [1:40:43<19:28:19,  1.53s/it]
Outer training loop:   9%|8         | 4274/50000 [1:40:45<19:45:35,  1.56s/it]
Outer training loop:   9%|8         | 4275/50000 [1:40:46<19:36:47,  1.54s/it]
Outer training loop:   9%|8         | 4276/50000 [1:40:48<19:41:31,  1.55s/it]
Outer training loop:   9%|8         | 4277/50000 [1:40:49<19:26:38,  1.53s/it]
Outer training loop:   9%|8         | 4278/50000 [1:40:51<19:46:00,  1.56s/it]
Outer training loop:   9%|8         | 4279/50000 [1:40:53<20:03:53,  1.58s/it]
Outer training loop:   9%|8         | 4280/50000 [1:40:54<19:52:14,  1.56s/it]
Outer training loop:   9%|8         | 4281/50000 [1:40:56<19:37:29,  1.55s/it]
Outer training loop:   9%|8         | 4282/50000 [1:40:57<19:47:15,  1.56s/it]
Outer training loop:   9%|8         | 4283/50000 [1:40:59<19:31:22,  1.54s/it]
Outer training loop:   9%|8         | 4284/50000 [1:41:00<20:02:51,  1.58s/it]
Outer training loop:   9%|8         | 4285/50000 [1:41:02<20:24:33,  1.61s/it]
Outer training loop:   9%|8         | 4286/50000 [1:41:04<20:08:51,  1.59s/it]
Outer training loop:   9%|8         | 4287/50000 [1:41:05<19:52:45,  1.57s/it]
Outer training loop:   9%|8         | 4288/50000 [1:41:07<19:53:21,  1.57s/it]
Outer training loop:   9%|8         | 4289/50000 [1:41:08<19:30:38,  1.54s/it]
Outer training loop:   9%|8         | 4290/50000 [1:41:10<20:03:04,  1.58s/it]
Outer training loop:   9%|8         | 4291/50000 [1:41:11<19:36:37,  1.54s/it]
Outer training loop:   9%|8         | 4292/50000 [1:41:13<19:35:08,  1.54s/it]
Outer training loop:   9%|8         | 4293/50000 [1:41:14<19:23:49,  1.53s/it]
Outer training loop:   9%|8         | 4294/50000 [1:41:16<19:58:04,  1.57s/it]
Outer training loop:   9%|8         | 4295/50000 [1:41:18<20:13:52,  1.59s/it]
Outer training loop:   9%|8         | 4296/50000 [1:41:19<20:00:31,  1.58s/it]
Outer training loop:   9%|8         | 4297/50000 [1:41:21<19:48:17,  1.56s/it]
Outer training loop:   9%|8         | 4298/50000 [1:41:22<19:49:02,  1.56s/it]
Outer training loop:   9%|8         | 4299/50000 [1:41:24<19:55:57,  1.57s/it]
Outer training loop:   9%|8         | 4300/50000 [1:41:25<19:47:48,  1.56s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.46s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.50s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.47s/it]

Outer training loop:   9%|8         | 4301/50000 [1:41:31<36:40:43,  2.89s/it]
Outer training loop:   9%|8         | 4302/50000 [1:41:33<31:35:53,  2.49s/it]
Outer training loop:   9%|8         | 4303/50000 [1:41:34<28:16:52,  2.23s/it]
Outer training loop:   9%|8         | 4304/50000 [1:41:36<26:13:01,  2.07s/it]
Outer training loop:   9%|8         | 4305/50000 [1:41:38<24:00:01,  1.89s/it]
Outer training loop:   9%|8         | 4306/50000 [1:41:39<22:35:07,  1.78s/it]
Outer training loop:   9%|8         | 4307/50000 [1:41:41<22:12:45,  1.75s/it]
Outer training loop:   9%|8         | 4308/50000 [1:41:42<21:29:53,  1.69s/it]
Outer training loop:   9%|8         | 4309/50000 [1:41:44<20:59:05,  1.65s/it]
Outer training loop:   9%|8         | 4310/50000 [1:41:46<21:03:59,  1.66s/it]
Outer training loop:   9%|8         | 4311/50000 [1:41:47<20:33:44,  1.62s/it]
Outer training loop:   9%|8         | 4312/50000 [1:41:49<20:29:35,  1.61s/it]
Outer training loop:   9%|8         | 4313/50000 [1:41:50<20:05:21,  1.58s/it]
Outer training loop:   9%|8         | 4314/50000 [1:41:52<20:09:46,  1.59s/it]
Outer training loop:   9%|8         | 4315/50000 [1:41:53<19:46:33,  1.56s/it]
Outer training loop:   9%|8         | 4316/50000 [1:41:55<19:45:00,  1.56s/it]
Outer training loop:   9%|8         | 4317/50000 [1:41:57<19:56:26,  1.57s/it]
Outer training loop:   9%|8         | 4318/50000 [1:41:58<20:04:14,  1.58s/it]
Outer training loop:   9%|8         | 4319/50000 [1:42:00<20:02:14,  1.58s/it]
Outer training loop:   9%|8         | 4320/50000 [1:42:01<19:42:03,  1.55s/it]
Outer training loop:   9%|8         | 4321/50000 [1:42:03<19:53:41,  1.57s/it]
Outer training loop:   9%|8         | 4322/50000 [1:42:04<19:55:18,  1.57s/it]
Outer training loop:   9%|8         | 4323/50000 [1:42:06<19:54:49,  1.57s/it]
Outer training loop:   9%|8         | 4324/50000 [1:42:07<19:42:27,  1.55s/it]
Outer training loop:   9%|8         | 4325/50000 [1:42:09<19:40:58,  1.55s/it]
Outer training loop:   9%|8         | 4326/50000 [1:42:11<19:30:08,  1.54s/it]
Outer training loop:   9%|8         | 4327/50000 [1:42:12<19:36:32,  1.55s/it]
Outer training loop:   9%|8         | 4328/50000 [1:42:14<19:21:30,  1.53s/it]
Outer training loop:   9%|8         | 4329/50000 [1:42:15<19:37:45,  1.55s/it]
Outer training loop:   9%|8         | 4330/50000 [1:42:17<20:02:36,  1.58s/it]
Outer training loop:   9%|8         | 4331/50000 [1:42:18<19:38:13,  1.55s/it]
Outer training loop:   9%|8         | 4332/50000 [1:42:20<20:00:58,  1.58s/it]
Outer training loop:   9%|8         | 4333/50000 [1:42:21<19:47:49,  1.56s/it]
Outer training loop:   9%|8         | 4334/50000 [1:42:23<19:45:30,  1.56s/it]
Outer training loop:   9%|8         | 4335/50000 [1:42:25<19:41:06,  1.55s/it]
Outer training loop:   9%|8         | 4336/50000 [1:42:26<19:43:14,  1.55s/it]
Outer training loop:   9%|8         | 4337/50000 [1:42:28<19:22:22,  1.53s/it]
Outer training loop:   9%|8         | 4338/50000 [1:42:29<19:28:25,  1.54s/it]
Outer training loop:   9%|8         | 4339/50000 [1:42:31<19:22:30,  1.53s/it]
Outer training loop:   9%|8         | 4340/50000 [1:42:32<19:11:55,  1.51s/it]
Outer training loop:   9%|8         | 4341/50000 [1:42:34<20:13:17,  1.59s/it]
Outer training loop:   9%|8         | 4342/50000 [1:42:35<20:08:48,  1.59s/it]
Outer training loop:   9%|8         | 4343/50000 [1:42:37<19:56:57,  1.57s/it]
Outer training loop:   9%|8         | 4344/50000 [1:42:39<20:02:40,  1.58s/it]
Outer training loop:   9%|8         | 4345/50000 [1:42:40<19:34:01,  1.54s/it]
Outer training loop:   9%|8         | 4346/50000 [1:42:42<19:46:42,  1.56s/it]
Outer training loop:   9%|8         | 4347/50000 [1:42:43<19:26:12,  1.53s/it]
Outer training loop:   9%|8         | 4348/50000 [1:42:45<19:56:17,  1.57s/it]
Outer training loop:   9%|8         | 4349/50000 [1:42:46<19:52:08,  1.57s/it]
Outer training loop:   9%|8         | 4350/50000 [1:42:48<19:51:59,  1.57s/it]
Outer training loop:   9%|8         | 4351/50000 [1:42:49<19:53:29,  1.57s/it]
Outer training loop:   9%|8         | 4352/50000 [1:42:51<19:57:48,  1.57s/it]
Outer training loop:   9%|8         | 4353/50000 [1:42:53<19:43:28,  1.56s/it]
Outer training loop:   9%|8         | 4354/50000 [1:42:54<19:50:55,  1.57s/it]
Outer training loop:   9%|8         | 4355/50000 [1:42:56<19:51:26,  1.57s/it]
Outer training loop:   9%|8         | 4356/50000 [1:42:57<20:04:03,  1.58s/it]
Outer training loop:   9%|8         | 4357/50000 [1:42:59<19:39:03,  1.55s/it]
Outer training loop:   9%|8         | 4358/50000 [1:43:00<19:49:55,  1.56s/it]
Outer training loop:   9%|8         | 4359/50000 [1:43:02<19:24:14,  1.53s/it]
Outer training loop:   9%|8         | 4360/50000 [1:43:03<19:19:19,  1.52s/it]
Outer training loop:   9%|8         | 4361/50000 [1:43:05<19:46:53,  1.56s/it]
Outer training loop:   9%|8         | 4362/50000 [1:43:07<19:39:52,  1.55s/it]
Outer training loop:   9%|8         | 4363/50000 [1:43:08<20:00:39,  1.58s/it]
Outer training loop:   9%|8         | 4364/50000 [1:43:10<19:50:11,  1.56s/it]
Outer training loop:   9%|8         | 4365/50000 [1:43:11<19:58:20,  1.58s/it]
Outer training loop:   9%|8         | 4366/50000 [1:43:13<19:49:09,  1.56s/it]
Outer training loop:   9%|8         | 4367/50000 [1:43:14<19:59:44,  1.58s/it]
Outer training loop:   9%|8         | 4368/50000 [1:43:16<19:49:56,  1.56s/it]
Outer training loop:   9%|8         | 4369/50000 [1:43:18<19:41:42,  1.55s/it]
Outer training loop:   9%|8         | 4370/50000 [1:43:19<19:31:35,  1.54s/it]
Outer training loop:   9%|8         | 4371/50000 [1:43:21<19:42:53,  1.56s/it]
Outer training loop:   9%|8         | 4372/50000 [1:43:22<19:26:37,  1.53s/it]
Outer training loop:   9%|8         | 4373/50000 [1:43:24<19:32:08,  1.54s/it]
Outer training loop:   9%|8         | 4374/50000 [1:43:25<19:26:55,  1.53s/it]
Outer training loop:   9%|8         | 4375/50000 [1:43:27<19:49:54,  1.56s/it]
Outer training loop:   9%|8         | 4376/50000 [1:43:28<19:31:34,  1.54s/it]
Outer training loop:   9%|8         | 4377/50000 [1:43:30<19:38:12,  1.55s/it]
Outer training loop:   9%|8         | 4378/50000 [1:43:31<19:27:36,  1.54s/it]
Outer training loop:   9%|8         | 4379/50000 [1:43:33<19:29:42,  1.54s/it]
Outer training loop:   9%|8         | 4380/50000 [1:43:35<19:36:10,  1.55s/it]
Outer training loop:   9%|8         | 4381/50000 [1:43:36<19:28:27,  1.54s/it]
Outer training loop:   9%|8         | 4382/50000 [1:43:38<19:50:08,  1.57s/it]
Outer training loop:   9%|8         | 4383/50000 [1:43:39<19:49:28,  1.56s/it]
Outer training loop:   9%|8         | 4384/50000 [1:43:41<19:41:09,  1.55s/it]
Outer training loop:   9%|8         | 4385/50000 [1:43:42<19:33:13,  1.54s/it]
Outer training loop:   9%|8         | 4386/50000 [1:43:44<19:41:47,  1.55s/it]
Outer training loop:   9%|8         | 4387/50000 [1:43:45<19:27:21,  1.54s/it]
Outer training loop:   9%|8         | 4388/50000 [1:43:47<19:39:41,  1.55s/it]
Outer training loop:   9%|8         | 4389/50000 [1:43:48<19:35:02,  1.55s/it]
Outer training loop:   9%|8         | 4390/50000 [1:43:50<19:38:27,  1.55s/it]
Outer training loop:   9%|8         | 4391/50000 [1:43:52<20:05:06,  1.59s/it]
Outer training loop:   9%|8         | 4392/50000 [1:43:53<19:55:13,  1.57s/it]
Outer training loop:   9%|8         | 4393/50000 [1:43:55<19:48:23,  1.56s/it]
Outer training loop:   9%|8         | 4394/50000 [1:43:56<20:16:21,  1.60s/it]
Outer training loop:   9%|8         | 4395/50000 [1:43:58<19:55:07,  1.57s/it]
Outer training loop:   9%|8         | 4396/50000 [1:44:00<20:12:10,  1.59s/it]
Outer training loop:   9%|8         | 4397/50000 [1:44:01<20:09:29,  1.59s/it]
Outer training loop:   9%|8         | 4398/50000 [1:44:03<20:10:46,  1.59s/it]
Outer training loop:   9%|8         | 4399/50000 [1:44:04<20:14:54,  1.60s/it]
Outer training loop:   9%|8         | 4400/50000 [1:44:06<20:00:22,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.49s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.53s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.51s/it]

Outer training loop:   9%|8         | 4401/50000 [1:44:12<37:37:31,  2.97s/it]
Outer training loop:   9%|8         | 4402/50000 [1:44:14<32:44:45,  2.59s/it]
Outer training loop:   9%|8         | 4403/50000 [1:44:15<28:54:43,  2.28s/it]
Outer training loop:   9%|8         | 4404/50000 [1:44:17<26:11:50,  2.07s/it]
Outer training loop:   9%|8         | 4405/50000 [1:44:19<24:12:50,  1.91s/it]
Outer training loop:   9%|8         | 4406/50000 [1:44:20<23:00:47,  1.82s/it]
Outer training loop:   9%|8         | 4407/50000 [1:44:22<21:54:35,  1.73s/it]
Outer training loop:   9%|8         | 4408/50000 [1:44:23<21:13:45,  1.68s/it]
Outer training loop:   9%|8         | 4409/50000 [1:44:25<21:02:11,  1.66s/it]
Outer training loop:   9%|8         | 4410/50000 [1:44:26<20:38:59,  1.63s/it]
Outer training loop:   9%|8         | 4411/50000 [1:44:28<20:17:11,  1.60s/it]
Outer training loop:   9%|8         | 4412/50000 [1:44:30<20:19:17,  1.60s/it]
Outer training loop:   9%|8         | 4413/50000 [1:44:31<20:04:51,  1.59s/it]
Outer training loop:   9%|8         | 4414/50000 [1:44:33<19:44:33,  1.56s/it]
Outer training loop:   9%|8         | 4415/50000 [1:44:34<20:03:22,  1.58s/it]
Outer training loop:   9%|8         | 4416/50000 [1:44:36<20:11:44,  1.59s/it]
Outer training loop:   9%|8         | 4417/50000 [1:44:37<19:38:01,  1.55s/it]
Outer training loop:   9%|8         | 4418/50000 [1:44:39<19:28:57,  1.54s/it]
Outer training loop:   9%|8         | 4419/50000 [1:44:40<19:40:45,  1.55s/it]
Outer training loop:   9%|8         | 4420/50000 [1:44:42<20:15:36,  1.60s/it]
Outer training loop:   9%|8         | 4421/50000 [1:44:44<19:33:08,  1.54s/it]
Outer training loop:   9%|8         | 4422/50000 [1:44:45<19:38:09,  1.55s/it]
Outer training loop:   9%|8         | 4423/50000 [1:44:47<19:42:47,  1.56s/it]
Outer training loop:   9%|8         | 4424/50000 [1:44:48<19:28:06,  1.54s/it]
Outer training loop:   9%|8         | 4425/50000 [1:44:50<19:57:03,  1.58s/it]
Outer training loop:   9%|8         | 4426/50000 [1:44:51<20:01:28,  1.58s/it]
Outer training loop:   9%|8         | 4427/50000 [1:44:53<20:14:55,  1.60s/it]
Outer training loop:   9%|8         | 4428/50000 [1:44:55<19:48:31,  1.56s/it]
Outer training loop:   9%|8         | 4429/50000 [1:44:56<20:09:05,  1.59s/it]
Outer training loop:   9%|8         | 4430/50000 [1:44:58<20:01:35,  1.58s/it]
Outer training loop:   9%|8         | 4431/50000 [1:44:59<19:32:24,  1.54s/it]
Outer training loop:   9%|8         | 4432/50000 [1:45:01<19:20:35,  1.53s/it]
Outer training loop:   9%|8         | 4433/50000 [1:45:02<19:41:26,  1.56s/it]
Outer training loop:   9%|8         | 4434/50000 [1:45:04<19:33:47,  1.55s/it]
Outer training loop:   9%|8         | 4435/50000 [1:45:05<19:53:24,  1.57s/it]
Outer training loop:   9%|8         | 4436/50000 [1:45:07<19:33:25,  1.55s/it]
Outer training loop:   9%|8         | 4437/50000 [1:45:09<19:48:01,  1.56s/it]
Outer training loop:   9%|8         | 4438/50000 [1:45:10<20:18:07,  1.60s/it]
Outer training loop:   9%|8         | 4439/50000 [1:45:12<20:22:00,  1.61s/it]
Outer training loop:   9%|8         | 4440/50000 [1:45:13<20:25:49,  1.61s/it]
Outer training loop:   9%|8         | 4441/50000 [1:45:15<20:04:58,  1.59s/it]
Outer training loop:   9%|8         | 4442/50000 [1:45:17<20:18:04,  1.60s/it]
Outer training loop:   9%|8         | 4443/50000 [1:45:18<19:54:46,  1.57s/it]
Outer training loop:   9%|8         | 4444/50000 [1:45:20<20:16:10,  1.60s/it]
Outer training loop:   9%|8         | 4445/50000 [1:45:21<19:59:13,  1.58s/it]
Outer training loop:   9%|8         | 4446/50000 [1:45:23<20:04:57,  1.59s/it]
Outer training loop:   9%|8         | 4447/50000 [1:45:24<19:45:36,  1.56s/it]
Outer training loop:   9%|8         | 4448/50000 [1:45:26<19:53:21,  1.57s/it]
Outer training loop:   9%|8         | 4449/50000 [1:45:28<19:38:57,  1.55s/it]
Outer training loop:   9%|8         | 4450/50000 [1:45:29<19:20:57,  1.53s/it]
Outer training loop:   9%|8         | 4451/50000 [1:45:31<19:42:00,  1.56s/it]
Outer training loop:   9%|8         | 4452/50000 [1:45:32<19:26:46,  1.54s/it]
Outer training loop:   9%|8         | 4453/50000 [1:45:34<20:18:15,  1.60s/it]
Outer training loop:   9%|8         | 4454/50000 [1:45:35<19:50:45,  1.57s/it]
Outer training loop:   9%|8         | 4455/50000 [1:45:37<19:53:16,  1.57s/it]
Outer training loop:   9%|8         | 4456/50000 [1:45:38<19:32:57,  1.55s/it]
Outer training loop:   9%|8         | 4457/50000 [1:45:40<19:37:36,  1.55s/it]
Outer training loop:   9%|8         | 4458/50000 [1:45:42<19:56:26,  1.58s/it]
Outer training loop:   9%|8         | 4459/50000 [1:45:43<19:45:57,  1.56s/it]
Outer training loop:   9%|8         | 4460/50000 [1:45:45<19:51:55,  1.57s/it]
Outer training loop:   9%|8         | 4461/50000 [1:45:46<19:43:47,  1.56s/it]
Outer training loop:   9%|8         | 4462/50000 [1:45:48<20:04:53,  1.59s/it]
Outer training loop:   9%|8         | 4463/50000 [1:45:49<19:50:18,  1.57s/it]
Outer training loop:   9%|8         | 4464/50000 [1:45:51<19:52:35,  1.57s/it]
Outer training loop:   9%|8         | 4465/50000 [1:45:53<19:35:12,  1.55s/it]
Outer training loop:   9%|8         | 4466/50000 [1:45:54<19:47:32,  1.56s/it]
Outer training loop:   9%|8         | 4467/50000 [1:45:56<19:43:39,  1.56s/it]
Outer training loop:   9%|8         | 4468/50000 [1:45:57<19:34:31,  1.55s/it]
Outer training loop:   9%|8         | 4469/50000 [1:45:59<19:17:59,  1.53s/it]
Outer training loop:   9%|8         | 4470/50000 [1:46:00<19:10:28,  1.52s/it]
Outer training loop:   9%|8         | 4471/50000 [1:46:02<19:39:34,  1.55s/it]
Outer training loop:   9%|8         | 4472/50000 [1:46:03<19:36:10,  1.55s/it]
Outer training loop:   9%|8         | 4473/50000 [1:46:05<19:58:52,  1.58s/it]
Outer training loop:   9%|8         | 4474/50000 [1:46:07<19:38:53,  1.55s/it]
Outer training loop:   9%|8         | 4475/50000 [1:46:08<19:47:58,  1.57s/it]
Outer training loop:   9%|8         | 4476/50000 [1:46:10<20:20:30,  1.61s/it]
Outer training loop:   9%|8         | 4477/50000 [1:46:11<19:29:57,  1.54s/it]
Outer training loop:   9%|8         | 4478/50000 [1:46:13<19:25:52,  1.54s/it]
Outer training loop:   9%|8         | 4479/50000 [1:46:14<20:07:21,  1.59s/it]
Outer training loop:   9%|8         | 4480/50000 [1:46:16<20:17:45,  1.61s/it]
Outer training loop:   9%|8         | 4481/50000 [1:46:18<20:07:21,  1.59s/it]
Outer training loop:   9%|8         | 4482/50000 [1:46:19<20:11:15,  1.60s/it]
Outer training loop:   9%|8         | 4483/50000 [1:46:21<19:53:04,  1.57s/it]
Outer training loop:   9%|8         | 4484/50000 [1:46:22<19:41:30,  1.56s/it]
Outer training loop:   9%|8         | 4485/50000 [1:46:24<19:42:51,  1.56s/it]
Outer training loop:   9%|8         | 4486/50000 [1:46:26<19:58:05,  1.58s/it]
Outer training loop:   9%|8         | 4487/50000 [1:46:27<19:38:42,  1.55s/it]
Outer training loop:   9%|8         | 4488/50000 [1:46:29<20:01:35,  1.58s/it]
Outer training loop:   9%|8         | 4489/50000 [1:46:30<20:13:51,  1.60s/it]
Outer training loop:   9%|8         | 4490/50000 [1:46:32<19:59:11,  1.58s/it]
Outer training loop:   9%|8         | 4491/50000 [1:46:33<20:05:29,  1.59s/it]
Outer training loop:   9%|8         | 4492/50000 [1:46:35<19:48:03,  1.57s/it]
Outer training loop:   9%|8         | 4493/50000 [1:46:37<20:09:10,  1.59s/it]
Outer training loop:   9%|8         | 4494/50000 [1:46:38<19:36:20,  1.55s/it]
Outer training loop:   9%|8         | 4495/50000 [1:46:40<19:49:44,  1.57s/it]
Outer training loop:   9%|8         | 4496/50000 [1:46:41<19:40:32,  1.56s/it]
Outer training loop:   9%|8         | 4497/50000 [1:46:43<19:49:23,  1.57s/it]
Outer training loop:   9%|8         | 4498/50000 [1:46:44<19:33:07,  1.55s/it]
Outer training loop:   9%|8         | 4499/50000 [1:46:46<19:22:47,  1.53s/it]
Outer training loop:   9%|9         | 4500/50000 [1:46:47<19:51:20,  1.57s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.56s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.55s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.53s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<01:56,  1.43s/it]

Evaluating policy:   2%|2         | 2/83 [00:02<01:57,  1.45s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<01:58,  1.48s/it]

Evaluating policy:   5%|4         | 4/83 [00:05<01:57,  1.48s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<01:58,  1.52s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<01:55,  1.50s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:57,  1.55s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:57,  1.56s/it]

Evaluating policy:  11%|#         | 9/83 [00:13<01:55,  1.56s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:53,  1.56s/it]

Evaluating policy:  13%|#3        | 11/83 [00:16<01:51,  1.55s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:51,  1.58s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:48,  1.55s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:45,  1.54s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:47,  1.57s/it]

Evaluating policy:  19%|#9        | 16/83 [00:24<01:43,  1.54s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:40,  1.52s/it]

Evaluating policy:  22%|##1       | 18/83 [00:27<01:40,  1.55s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:39,  1.55s/it]

Evaluating policy:  24%|##4       | 20/83 [00:30<01:38,  1.56s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:39,  1.60s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:35,  1.56s/it]

Evaluating policy:  28%|##7       | 23/83 [00:35<01:34,  1.58s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:31,  1.55s/it]

Evaluating policy:  30%|###       | 25/83 [00:38<01:34,  1.63s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:30,  1.59s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:28,  1.58s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:25,  1.55s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:24,  1.57s/it]

Evaluating policy:  36%|###6      | 30/83 [00:46<01:22,  1.55s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:20,  1.55s/it]

Evaluating policy:  39%|###8      | 32/83 [00:49<01:17,  1.52s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:19,  1.59s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:18,  1.61s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:15,  1.57s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:13,  1.56s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:10,  1.53s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:10,  1.58s/it]

Evaluating policy:  47%|####6     | 39/83 [01:00<01:08,  1.56s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:05,  1.51s/it]

Evaluating policy:  49%|####9     | 41/83 [01:03<01:03,  1.52s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:02,  1.52s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:06<01:03,  1.58s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<01:01,  1.57s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:09<00:57,  1.53s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:11<00:56,  1.52s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:12<00:54,  1.53s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:14<00:53,  1.53s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:53,  1.59s/it]

Evaluating policy:  60%|######    | 50/83 [01:17<00:51,  1.56s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:49,  1.54s/it]

Evaluating policy:  63%|######2   | 52/83 [01:20<00:47,  1.54s/it]

Evaluating policy:  64%|######3   | 53/83 [01:22<00:45,  1.53s/it]

Evaluating policy:  65%|######5   | 54/83 [01:23<00:43,  1.52s/it]

Evaluating policy:  66%|######6   | 55/83 [01:25<00:44,  1.59s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:42,  1.58s/it]

Evaluating policy:  69%|######8   | 57/83 [01:28<00:40,  1.57s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:38,  1.55s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:31<00:36,  1.52s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:33<00:34,  1.52s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:34<00:35,  1.63s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:36<00:32,  1.57s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:37<00:30,  1.54s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:39<00:28,  1.52s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:40<00:27,  1.51s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:42<00:25,  1.51s/it]

Evaluating policy:  81%|########  | 67/83 [01:43<00:24,  1.51s/it]

Evaluating policy:  82%|########1 | 68/83 [01:45<00:24,  1.63s/it]

Evaluating policy:  83%|########3 | 69/83 [01:47<00:22,  1.59s/it]

Evaluating policy:  84%|########4 | 70/83 [01:48<00:20,  1.58s/it]

Evaluating policy:  86%|########5 | 71/83 [01:50<00:18,  1.55s/it]

Evaluating policy:  87%|########6 | 72/83 [01:51<00:17,  1.55s/it]

Evaluating policy:  88%|########7 | 73/83 [01:53<00:15,  1.55s/it]

Evaluating policy:  89%|########9 | 74/83 [01:54<00:13,  1.53s/it]

Evaluating policy:  90%|######### | 75/83 [01:56<00:12,  1.56s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:11,  1.66s/it]

Evaluating policy:  93%|#########2| 77/83 [01:59<00:09,  1.63s/it]

Evaluating policy:  94%|#########3| 78/83 [02:01<00:08,  1.61s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.60s/it]

Evaluating policy:  96%|#########6| 80/83 [02:04<00:04,  1.58s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.56s/it]

Evaluating policy:  99%|#########8| 82/83 [02:07<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.52s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:52,  2.10s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:34,  1.91s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.78s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:17,  1.74s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:08,  1.65s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:08,  1.67s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<02:03,  1.63s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<02:01,  1.63s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:58,  1.60s/it]

Evaluating policy:  12%|#2        | 10/83 [00:16<01:55,  1.58s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:57,  1.63s/it]

Evaluating policy:  14%|#4        | 12/83 [00:19<01:51,  1.58s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:47,  1.53s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:48,  1.58s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:45,  1.56s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:44,  1.55s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:42,  1.56s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:40,  1.54s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:38,  1.53s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:36,  1.53s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:37,  1.58s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:34,  1.55s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:33,  1.56s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:34,  1.59s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:31,  1.57s/it]

Evaluating policy:  31%|###1      | 26/83 [00:41<01:29,  1.57s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:29,  1.59s/it]

Evaluating policy:  34%|###3      | 28/83 [00:44<01:25,  1.56s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:22,  1.53s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:20,  1.52s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:18,  1.51s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:19,  1.57s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:18,  1.57s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:15,  1.55s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:13,  1.53s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:14,  1.58s/it]

Evaluating policy:  45%|####4     | 37/83 [00:58<01:11,  1.55s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:09,  1.55s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:07,  1.54s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:05,  1.53s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:03,  1.52s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:05,  1.61s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:03,  1.59s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:09<01:01,  1.57s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<00:59,  1.56s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<00:59,  1.62s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:13<00:57,  1.60s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:54,  1.56s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:52,  1.54s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:49,  1.51s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:50,  1.56s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:50,  1.63s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:47,  1.59s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:44,  1.55s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:42,  1.52s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:41,  1.53s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:39,  1.52s/it]

Evaluating policy:  70%|######9   | 58/83 [01:31<00:41,  1.64s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:38,  1.59s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:34<00:35,  1.55s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:34,  1.55s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:36<00:31,  1.51s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:30,  1.51s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:30,  1.63s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:28,  1.57s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:26,  1.55s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:24,  1.55s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:23,  1.54s/it]

Evaluating policy:  83%|########3 | 69/83 [01:47<00:21,  1.54s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:19,  1.52s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:19,  1.65s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:17,  1.58s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:15,  1.56s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:13,  1.54s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.53s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:10,  1.52s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.52s/it]

Evaluating policy:  94%|#########3| 78/83 [02:01<00:07,  1.51s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.64s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.60s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.57s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.56s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.52s/it]

Outer training loop:   9%|9         | 4501/50000 [1:51:13<1019:44:16, 80.68s/it]
Outer training loop:   9%|9         | 4502/50000 [1:51:14<719:43:24, 56.95s/it] 
Outer training loop:   9%|9         | 4503/50000 [1:51:16<509:32:46, 40.32s/it]
Outer training loop:   9%|9         | 4504/50000 [1:51:17<362:11:28, 28.66s/it]
Outer training loop:   9%|9         | 4505/50000 [1:51:19<259:10:26, 20.51s/it]
Outer training loop:   9%|9         | 4506/50000 [1:51:21<188:57:03, 14.95s/it]
Outer training loop:   9%|9         | 4507/50000 [1:51:22<137:50:20, 10.91s/it]
Outer training loop:   9%|9         | 4508/50000 [1:51:24<102:39:16,  8.12s/it]
Outer training loop:   9%|9         | 4509/50000 [1:51:25<77:57:22,  6.17s/it] 
Outer training loop:   9%|9         | 4510/50000 [1:51:27<60:38:05,  4.80s/it]
Outer training loop:   9%|9         | 4511/50000 [1:51:29<48:06:35,  3.81s/it]
Outer training loop:   9%|9         | 4512/50000 [1:51:30<39:51:30,  3.15s/it]
Outer training loop:   9%|9         | 4513/50000 [1:51:32<33:37:15,  2.66s/it]
Outer training loop:   9%|9         | 4514/50000 [1:51:33<29:31:52,  2.34s/it]
Outer training loop:   9%|9         | 4515/50000 [1:51:35<26:43:51,  2.12s/it]
Outer training loop:   9%|9         | 4516/50000 [1:51:36<24:36:54,  1.95s/it]
Outer training loop:   9%|9         | 4517/50000 [1:51:38<23:03:56,  1.83s/it]
Outer training loop:   9%|9         | 4518/50000 [1:51:40<22:14:59,  1.76s/it]
Outer training loop:   9%|9         | 4519/50000 [1:51:41<21:47:03,  1.72s/it]
Outer training loop:   9%|9         | 4520/50000 [1:51:43<21:01:20,  1.66s/it]
Outer training loop:   9%|9         | 4521/50000 [1:51:44<20:33:15,  1.63s/it]
Outer training loop:   9%|9         | 4522/50000 [1:51:46<20:03:43,  1.59s/it]
Outer training loop:   9%|9         | 4523/50000 [1:51:47<19:54:49,  1.58s/it]
Outer training loop:   9%|9         | 4524/50000 [1:51:49<19:35:55,  1.55s/it]
Outer training loop:   9%|9         | 4525/50000 [1:51:50<20:02:33,  1.59s/it]
Outer training loop:   9%|9         | 4526/50000 [1:51:52<19:50:56,  1.57s/it]
Outer training loop:   9%|9         | 4527/50000 [1:51:54<20:09:09,  1.60s/it]
Outer training loop:   9%|9         | 4528/50000 [1:51:55<20:02:29,  1.59s/it]
Outer training loop:   9%|9         | 4529/50000 [1:51:57<20:11:19,  1.60s/it]
Outer training loop:   9%|9         | 4530/50000 [1:51:58<20:02:36,  1.59s/it]
Outer training loop:   9%|9         | 4531/50000 [1:52:00<20:18:33,  1.61s/it]
Outer training loop:   9%|9         | 4532/50000 [1:52:02<19:57:57,  1.58s/it]
Outer training loop:   9%|9         | 4533/50000 [1:52:03<20:10:31,  1.60s/it]
Outer training loop:   9%|9         | 4534/50000 [1:52:05<19:54:59,  1.58s/it]
Outer training loop:   9%|9         | 4535/50000 [1:52:06<20:01:23,  1.59s/it]
Outer training loop:   9%|9         | 4536/50000 [1:52:08<19:30:22,  1.54s/it]
Outer training loop:   9%|9         | 4537/50000 [1:52:09<19:58:27,  1.58s/it]
Outer training loop:   9%|9         | 4538/50000 [1:52:11<20:00:32,  1.58s/it]
Outer training loop:   9%|9         | 4539/50000 [1:52:13<19:46:35,  1.57s/it]
Outer training loop:   9%|9         | 4540/50000 [1:52:14<19:48:20,  1.57s/it]
Outer training loop:   9%|9         | 4541/50000 [1:52:16<19:34:03,  1.55s/it]
Outer training loop:   9%|9         | 4542/50000 [1:52:17<19:41:25,  1.56s/it]
Outer training loop:   9%|9         | 4543/50000 [1:52:19<19:57:02,  1.58s/it]
Outer training loop:   9%|9         | 4544/50000 [1:52:21<20:18:09,  1.61s/it]
Outer training loop:   9%|9         | 4545/50000 [1:52:22<19:44:56,  1.56s/it]
Outer training loop:   9%|9         | 4546/50000 [1:52:24<19:45:16,  1.56s/it]
Outer training loop:   9%|9         | 4547/50000 [1:52:25<20:00:27,  1.58s/it]
Outer training loop:   9%|9         | 4548/50000 [1:52:27<19:35:58,  1.55s/it]
Outer training loop:   9%|9         | 4549/50000 [1:52:28<19:33:57,  1.55s/it]
Outer training loop:   9%|9         | 4550/50000 [1:52:30<19:15:16,  1.53s/it]
Outer training loop:   9%|9         | 4551/50000 [1:52:31<19:56:05,  1.58s/it]
Outer training loop:   9%|9         | 4552/50000 [1:52:33<19:41:22,  1.56s/it]
Outer training loop:   9%|9         | 4553/50000 [1:52:35<19:45:15,  1.56s/it]
Outer training loop:   9%|9         | 4554/50000 [1:52:36<19:49:53,  1.57s/it]
Outer training loop:   9%|9         | 4555/50000 [1:52:38<19:39:01,  1.56s/it]
Outer training loop:   9%|9         | 4556/50000 [1:52:39<19:30:11,  1.55s/it]
Outer training loop:   9%|9         | 4557/50000 [1:52:41<19:48:58,  1.57s/it]
Outer training loop:   9%|9         | 4558/50000 [1:52:42<19:28:09,  1.54s/it]
Outer training loop:   9%|9         | 4559/50000 [1:52:44<19:59:09,  1.58s/it]
Outer training loop:   9%|9         | 4560/50000 [1:52:45<19:41:02,  1.56s/it]
Outer training loop:   9%|9         | 4561/50000 [1:52:47<19:51:18,  1.57s/it]
Outer training loop:   9%|9         | 4562/50000 [1:52:49<20:11:17,  1.60s/it]
Outer training loop:   9%|9         | 4563/50000 [1:52:50<20:02:56,  1.59s/it]
Outer training loop:   9%|9         | 4564/50000 [1:52:52<19:57:43,  1.58s/it]
Outer training loop:   9%|9         | 4565/50000 [1:52:53<19:50:24,  1.57s/it]
Outer training loop:   9%|9         | 4566/50000 [1:52:55<19:45:19,  1.57s/it]
Outer training loop:   9%|9         | 4567/50000 [1:52:56<19:23:58,  1.54s/it]
Outer training loop:   9%|9         | 4568/50000 [1:52:58<19:41:09,  1.56s/it]
Outer training loop:   9%|9         | 4569/50000 [1:53:00<20:17:18,  1.61s/it]
Outer training loop:   9%|9         | 4570/50000 [1:53:01<20:06:15,  1.59s/it]
Outer training loop:   9%|9         | 4571/50000 [1:53:03<19:56:49,  1.58s/it]
Outer training loop:   9%|9         | 4572/50000 [1:53:04<19:38:33,  1.56s/it]
Outer training loop:   9%|9         | 4573/50000 [1:53:06<19:29:20,  1.54s/it]
Outer training loop:   9%|9         | 4574/50000 [1:53:07<19:26:58,  1.54s/it]
Outer training loop:   9%|9         | 4575/50000 [1:53:09<19:42:26,  1.56s/it]
Outer training loop:   9%|9         | 4576/50000 [1:53:10<19:23:10,  1.54s/it]
Outer training loop:   9%|9         | 4577/50000 [1:53:12<19:32:08,  1.55s/it]
Outer training loop:   9%|9         | 4578/50000 [1:53:13<19:04:23,  1.51s/it]
Outer training loop:   9%|9         | 4579/50000 [1:53:15<19:19:36,  1.53s/it]
Outer training loop:   9%|9         | 4580/50000 [1:53:17<19:10:04,  1.52s/it]
Outer training loop:   9%|9         | 4581/50000 [1:53:18<20:02:23,  1.59s/it]
Outer training loop:   9%|9         | 4582/50000 [1:53:20<19:51:30,  1.57s/it]
Outer training loop:   9%|9         | 4583/50000 [1:53:21<19:57:02,  1.58s/it]
Outer training loop:   9%|9         | 4584/50000 [1:53:23<19:40:02,  1.56s/it]
Outer training loop:   9%|9         | 4585/50000 [1:53:25<19:46:45,  1.57s/it]
Outer training loop:   9%|9         | 4586/50000 [1:53:26<19:59:56,  1.59s/it]
Outer training loop:   9%|9         | 4587/50000 [1:53:28<19:44:05,  1.56s/it]
Outer training loop:   9%|9         | 4588/50000 [1:53:29<19:37:42,  1.56s/it]
Outer training loop:   9%|9         | 4589/50000 [1:53:31<19:34:57,  1.55s/it]
Outer training loop:   9%|9         | 4590/50000 [1:53:32<19:33:35,  1.55s/it]
Outer training loop:   9%|9         | 4591/50000 [1:53:34<19:56:56,  1.58s/it]
Outer training loop:   9%|9         | 4592/50000 [1:53:36<19:51:42,  1.57s/it]
Outer training loop:   9%|9         | 4593/50000 [1:53:37<19:47:12,  1.57s/it]
Outer training loop:   9%|9         | 4594/50000 [1:53:39<19:43:00,  1.56s/it]
Outer training loop:   9%|9         | 4595/50000 [1:53:40<19:34:37,  1.55s/it]
Outer training loop:   9%|9         | 4596/50000 [1:53:42<19:33:46,  1.55s/it]
Outer training loop:   9%|9         | 4597/50000 [1:53:43<19:41:47,  1.56s/it]
Outer training loop:   9%|9         | 4598/50000 [1:53:45<19:30:02,  1.55s/it]
Outer training loop:   9%|9         | 4599/50000 [1:53:46<19:44:01,  1.56s/it]
Outer training loop:   9%|9         | 4600/50000 [1:53:48<19:30:49,  1.55s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.45s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.49s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.45s/it]

Outer training loop:   9%|9         | 4601/50000 [1:53:54<36:15:15,  2.87s/it]
Outer training loop:   9%|9         | 4602/50000 [1:53:56<31:33:11,  2.50s/it]
Outer training loop:   9%|9         | 4603/50000 [1:53:57<27:45:38,  2.20s/it]
Outer training loop:   9%|9         | 4604/50000 [1:53:59<25:39:11,  2.03s/it]
Outer training loop:   9%|9         | 4605/50000 [1:54:00<23:38:30,  1.87s/it]
Outer training loop:   9%|9         | 4606/50000 [1:54:02<22:34:49,  1.79s/it]
Outer training loop:   9%|9         | 4607/50000 [1:54:03<21:28:49,  1.70s/it]
Outer training loop:   9%|9         | 4608/50000 [1:54:05<21:09:50,  1.68s/it]
Outer training loop:   9%|9         | 4609/50000 [1:54:06<20:25:52,  1.62s/it]
Outer training loop:   9%|9         | 4610/50000 [1:54:08<20:19:17,  1.61s/it]
Outer training loop:   9%|9         | 4611/50000 [1:54:09<19:39:39,  1.56s/it]
Outer training loop:   9%|9         | 4612/50000 [1:54:11<19:45:18,  1.57s/it]
Outer training loop:   9%|9         | 4613/50000 [1:54:12<19:34:38,  1.55s/it]
Outer training loop:   9%|9         | 4614/50000 [1:54:14<19:32:56,  1.55s/it]
Outer training loop:   9%|9         | 4615/50000 [1:54:16<19:34:10,  1.55s/it]
Outer training loop:   9%|9         | 4616/50000 [1:54:17<20:31:14,  1.63s/it]
Outer training loop:   9%|9         | 4617/50000 [1:54:19<20:06:14,  1.59s/it]
Outer training loop:   9%|9         | 4618/50000 [1:54:20<19:50:45,  1.57s/it]
Outer training loop:   9%|9         | 4619/50000 [1:54:22<19:56:27,  1.58s/it]
Outer training loop:   9%|9         | 4620/50000 [1:54:24<19:58:35,  1.58s/it]
Outer training loop:   9%|9         | 4621/50000 [1:54:25<19:48:38,  1.57s/it]
Outer training loop:   9%|9         | 4622/50000 [1:54:27<19:43:01,  1.56s/it]
Outer training loop:   9%|9         | 4623/50000 [1:54:28<20:15:50,  1.61s/it]
Outer training loop:   9%|9         | 4624/50000 [1:54:30<19:52:59,  1.58s/it]
Outer training loop:   9%|9         | 4625/50000 [1:54:32<19:55:05,  1.58s/it]
Outer training loop:   9%|9         | 4626/50000 [1:54:33<19:59:32,  1.59s/it]
Outer training loop:   9%|9         | 4627/50000 [1:54:35<20:12:46,  1.60s/it]
Outer training loop:   9%|9         | 4628/50000 [1:54:36<19:47:49,  1.57s/it]
Outer training loop:   9%|9         | 4629/50000 [1:54:38<20:02:31,  1.59s/it]
Outer training loop:   9%|9         | 4630/50000 [1:54:39<19:42:09,  1.56s/it]
Outer training loop:   9%|9         | 4631/50000 [1:54:41<19:51:06,  1.58s/it]
Outer training loop:   9%|9         | 4632/50000 [1:54:43<19:59:14,  1.59s/it]
Outer training loop:   9%|9         | 4633/50000 [1:54:44<19:40:29,  1.56s/it]
Outer training loop:   9%|9         | 4634/50000 [1:54:46<19:38:13,  1.56s/it]
Outer training loop:   9%|9         | 4635/50000 [1:54:47<19:44:44,  1.57s/it]
Outer training loop:   9%|9         | 4636/50000 [1:54:49<19:48:21,  1.57s/it]
Outer training loop:   9%|9         | 4637/50000 [1:54:50<19:37:45,  1.56s/it]
Outer training loop:   9%|9         | 4638/50000 [1:54:52<19:14:15,  1.53s/it]
Outer training loop:   9%|9         | 4639/50000 [1:54:53<19:41:19,  1.56s/it]
Outer training loop:   9%|9         | 4640/50000 [1:54:55<19:28:18,  1.55s/it]
Outer training loop:   9%|9         | 4641/50000 [1:54:57<19:32:24,  1.55s/it]
Outer training loop:   9%|9         | 4642/50000 [1:54:58<19:47:24,  1.57s/it]
Outer training loop:   9%|9         | 4643/50000 [1:55:00<19:40:10,  1.56s/it]
Outer training loop:   9%|9         | 4644/50000 [1:55:01<19:22:03,  1.54s/it]
Outer training loop:   9%|9         | 4645/50000 [1:55:03<19:43:14,  1.57s/it]
Outer training loop:   9%|9         | 4646/50000 [1:55:04<19:34:24,  1.55s/it]
Outer training loop:   9%|9         | 4647/50000 [1:55:06<19:37:55,  1.56s/it]
Outer training loop:   9%|9         | 4648/50000 [1:55:07<19:38:00,  1.56s/it]
Outer training loop:   9%|9         | 4649/50000 [1:55:09<19:45:39,  1.57s/it]
Outer training loop:   9%|9         | 4650/50000 [1:55:11<19:36:30,  1.56s/it]
Outer training loop:   9%|9         | 4651/50000 [1:55:12<19:58:28,  1.59s/it]
Outer training loop:   9%|9         | 4652/50000 [1:55:14<19:22:49,  1.54s/it]
Outer training loop:   9%|9         | 4653/50000 [1:55:15<19:21:58,  1.54s/it]
Outer training loop:   9%|9         | 4654/50000 [1:55:17<19:12:36,  1.53s/it]
Outer training loop:   9%|9         | 4655/50000 [1:55:18<19:35:17,  1.56s/it]
Outer training loop:   9%|9         | 4656/50000 [1:55:20<19:33:44,  1.55s/it]
Outer training loop:   9%|9         | 4657/50000 [1:55:21<19:54:12,  1.58s/it]
Outer training loop:   9%|9         | 4658/50000 [1:55:23<19:17:51,  1.53s/it]
Outer training loop:   9%|9         | 4659/50000 [1:55:25<19:50:56,  1.58s/it]
Outer training loop:   9%|9         | 4660/50000 [1:55:26<19:33:07,  1.55s/it]
Outer training loop:   9%|9         | 4661/50000 [1:55:28<19:41:13,  1.56s/it]
Outer training loop:   9%|9         | 4662/50000 [1:55:29<19:28:26,  1.55s/it]
Outer training loop:   9%|9         | 4663/50000 [1:55:31<19:47:23,  1.57s/it]
Outer training loop:   9%|9         | 4664/50000 [1:55:32<19:15:09,  1.53s/it]
Outer training loop:   9%|9         | 4665/50000 [1:55:34<19:47:57,  1.57s/it]
Outer training loop:   9%|9         | 4666/50000 [1:55:36<20:05:37,  1.60s/it]
Outer training loop:   9%|9         | 4667/50000 [1:55:37<19:58:18,  1.59s/it]
Outer training loop:   9%|9         | 4668/50000 [1:55:39<19:23:13,  1.54s/it]
Outer training loop:   9%|9         | 4669/50000 [1:55:40<19:22:18,  1.54s/it]
Outer training loop:   9%|9         | 4670/50000 [1:55:42<19:35:26,  1.56s/it]
Outer training loop:   9%|9         | 4671/50000 [1:55:43<19:21:09,  1.54s/it]
Outer training loop:   9%|9         | 4672/50000 [1:55:45<19:22:19,  1.54s/it]
Outer training loop:   9%|9         | 4673/50000 [1:55:46<19:18:22,  1.53s/it]
Outer training loop:   9%|9         | 4674/50000 [1:55:48<19:32:47,  1.55s/it]
Outer training loop:   9%|9         | 4675/50000 [1:55:49<19:40:50,  1.56s/it]
Outer training loop:   9%|9         | 4676/50000 [1:55:51<20:02:18,  1.59s/it]
Outer training loop:   9%|9         | 4677/50000 [1:55:53<20:12:56,  1.61s/it]
Outer training loop:   9%|9         | 4678/50000 [1:55:54<19:53:15,  1.58s/it]
Outer training loop:   9%|9         | 4679/50000 [1:55:56<20:09:15,  1.60s/it]
Outer training loop:   9%|9         | 4680/50000 [1:55:57<19:43:28,  1.57s/it]
Outer training loop:   9%|9         | 4681/50000 [1:55:59<19:42:14,  1.57s/it]
Outer training loop:   9%|9         | 4682/50000 [1:56:01<19:56:37,  1.58s/it]
Outer training loop:   9%|9         | 4683/50000 [1:56:02<19:59:54,  1.59s/it]
Outer training loop:   9%|9         | 4684/50000 [1:56:04<19:47:13,  1.57s/it]
Outer training loop:   9%|9         | 4685/50000 [1:56:05<19:33:44,  1.55s/it]
Outer training loop:   9%|9         | 4686/50000 [1:56:07<19:41:13,  1.56s/it]
Outer training loop:   9%|9         | 4687/50000 [1:56:08<19:34:21,  1.56s/it]
Outer training loop:   9%|9         | 4688/50000 [1:56:10<19:30:00,  1.55s/it]
Outer training loop:   9%|9         | 4689/50000 [1:56:11<19:30:34,  1.55s/it]
Outer training loop:   9%|9         | 4690/50000 [1:56:13<19:33:59,  1.55s/it]
Outer training loop:   9%|9         | 4691/50000 [1:56:15<19:27:54,  1.55s/it]
Outer training loop:   9%|9         | 4692/50000 [1:56:16<19:25:23,  1.54s/it]
Outer training loop:   9%|9         | 4693/50000 [1:56:18<19:25:18,  1.54s/it]
Outer training loop:   9%|9         | 4694/50000 [1:56:19<19:29:46,  1.55s/it]
Outer training loop:   9%|9         | 4695/50000 [1:56:21<19:16:13,  1.53s/it]
Outer training loop:   9%|9         | 4696/50000 [1:56:22<19:29:24,  1.55s/it]
Outer training loop:   9%|9         | 4697/50000 [1:56:24<19:14:26,  1.53s/it]
Outer training loop:   9%|9         | 4698/50000 [1:56:25<19:17:59,  1.53s/it]
Outer training loop:   9%|9         | 4699/50000 [1:56:27<19:13:03,  1.53s/it]
Outer training loop:   9%|9         | 4700/50000 [1:56:28<18:57:45,  1.51s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.46s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.50s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.48s/it]

Outer training loop:   9%|9         | 4701/50000 [1:56:34<36:20:17,  2.89s/it]
Outer training loop:   9%|9         | 4702/50000 [1:56:36<31:30:58,  2.50s/it]
Outer training loop:   9%|9         | 4703/50000 [1:56:38<27:54:38,  2.22s/it]
Outer training loop:   9%|9         | 4704/50000 [1:56:39<25:29:30,  2.03s/it]
Outer training loop:   9%|9         | 4705/50000 [1:56:41<23:31:05,  1.87s/it]
Outer training loop:   9%|9         | 4706/50000 [1:56:42<22:16:40,  1.77s/it]
Outer training loop:   9%|9         | 4707/50000 [1:56:44<21:03:13,  1.67s/it]
Outer training loop:   9%|9         | 4708/50000 [1:56:45<20:42:24,  1.65s/it]
Outer training loop:   9%|9         | 4709/50000 [1:56:47<20:25:34,  1.62s/it]
Outer training loop:   9%|9         | 4710/50000 [1:56:48<20:18:19,  1.61s/it]
Outer training loop:   9%|9         | 4711/50000 [1:56:50<20:14:13,  1.61s/it]
Outer training loop:   9%|9         | 4712/50000 [1:56:51<20:06:06,  1.60s/it]
Outer training loop:   9%|9         | 4713/50000 [1:56:53<20:22:29,  1.62s/it]
Outer training loop:   9%|9         | 4714/50000 [1:56:55<20:09:59,  1.60s/it]
Outer training loop:   9%|9         | 4715/50000 [1:56:56<19:45:35,  1.57s/it]
Outer training loop:   9%|9         | 4716/50000 [1:56:58<19:56:40,  1.59s/it]
Outer training loop:   9%|9         | 4717/50000 [1:56:59<19:37:22,  1.56s/it]
Outer training loop:   9%|9         | 4718/50000 [1:57:01<19:48:23,  1.57s/it]
Outer training loop:   9%|9         | 4719/50000 [1:57:03<19:54:22,  1.58s/it]
Outer training loop:   9%|9         | 4720/50000 [1:57:04<19:52:15,  1.58s/it]
Outer training loop:   9%|9         | 4721/50000 [1:57:06<20:30:23,  1.63s/it]
Outer training loop:   9%|9         | 4722/50000 [1:57:07<20:01:42,  1.59s/it]
Outer training loop:   9%|9         | 4723/50000 [1:57:09<19:28:45,  1.55s/it]
Outer training loop:   9%|9         | 4724/50000 [1:57:11<19:57:15,  1.59s/it]
Outer training loop:   9%|9         | 4725/50000 [1:57:12<19:50:52,  1.58s/it]
Outer training loop:   9%|9         | 4726/50000 [1:57:14<20:11:50,  1.61s/it]
Outer training loop:   9%|9         | 4727/50000 [1:57:15<20:21:15,  1.62s/it]
Outer training loop:   9%|9         | 4728/50000 [1:57:17<20:08:28,  1.60s/it]
Outer training loop:   9%|9         | 4729/50000 [1:57:18<19:41:38,  1.57s/it]
Outer training loop:   9%|9         | 4730/50000 [1:57:20<19:53:43,  1.58s/it]
Outer training loop:   9%|9         | 4731/50000 [1:57:22<19:48:17,  1.57s/it]
Outer training loop:   9%|9         | 4732/50000 [1:57:23<19:52:06,  1.58s/it]
Outer training loop:   9%|9         | 4733/50000 [1:57:25<19:26:04,  1.55s/it]
Outer training loop:   9%|9         | 4734/50000 [1:57:26<19:42:49,  1.57s/it]
Outer training loop:   9%|9         | 4735/50000 [1:57:28<19:22:00,  1.54s/it]
Outer training loop:   9%|9         | 4736/50000 [1:57:29<19:27:18,  1.55s/it]
Outer training loop:   9%|9         | 4737/50000 [1:57:31<19:39:48,  1.56s/it]
Outer training loop:   9%|9         | 4738/50000 [1:57:32<19:29:46,  1.55s/it]
Outer training loop:   9%|9         | 4739/50000 [1:57:34<19:30:52,  1.55s/it]
Outer training loop:   9%|9         | 4740/50000 [1:57:36<19:58:26,  1.59s/it]
Outer training loop:   9%|9         | 4741/50000 [1:57:37<19:34:57,  1.56s/it]
Outer training loop:   9%|9         | 4742/50000 [1:57:39<19:30:36,  1.55s/it]
Outer training loop:   9%|9         | 4743/50000 [1:57:40<19:22:59,  1.54s/it]
Outer training loop:   9%|9         | 4744/50000 [1:57:42<19:23:09,  1.54s/it]
Outer training loop:   9%|9         | 4745/50000 [1:57:43<19:15:44,  1.53s/it]
Outer training loop:   9%|9         | 4746/50000 [1:57:45<19:40:11,  1.56s/it]
Outer training loop:   9%|9         | 4747/50000 [1:57:46<19:22:07,  1.54s/it]
Outer training loop:   9%|9         | 4748/50000 [1:57:48<19:45:27,  1.57s/it]
Outer training loop:   9%|9         | 4749/50000 [1:57:50<19:30:42,  1.55s/it]
Outer training loop:  10%|9         | 4750/50000 [1:57:51<19:39:04,  1.56s/it]
Outer training loop:  10%|9         | 4751/50000 [1:57:53<19:36:26,  1.56s/it]
Outer training loop:  10%|9         | 4752/50000 [1:57:54<19:31:04,  1.55s/it]
Outer training loop:  10%|9         | 4753/50000 [1:57:56<19:29:58,  1.55s/it]
Outer training loop:  10%|9         | 4754/50000 [1:57:57<19:33:49,  1.56s/it]
Outer training loop:  10%|9         | 4755/50000 [1:57:59<19:20:12,  1.54s/it]
Outer training loop:  10%|9         | 4756/50000 [1:58:00<19:35:03,  1.56s/it]
Outer training loop:  10%|9         | 4757/50000 [1:58:02<19:22:57,  1.54s/it]
Outer training loop:  10%|9         | 4758/50000 [1:58:04<19:57:23,  1.59s/it]
Outer training loop:  10%|9         | 4759/50000 [1:58:05<19:21:56,  1.54s/it]
Outer training loop:  10%|9         | 4760/50000 [1:58:07<19:43:57,  1.57s/it]
Outer training loop:  10%|9         | 4761/50000 [1:58:08<19:17:15,  1.53s/it]
Outer training loop:  10%|9         | 4762/50000 [1:58:10<19:46:22,  1.57s/it]
Outer training loop:  10%|9         | 4763/50000 [1:58:12<20:12:09,  1.61s/it]
Outer training loop:  10%|9         | 4764/50000 [1:58:13<20:17:35,  1.61s/it]
Outer training loop:  10%|9         | 4765/50000 [1:58:15<19:57:02,  1.59s/it]
Outer training loop:  10%|9         | 4766/50000 [1:58:16<20:04:31,  1.60s/it]
Outer training loop:  10%|9         | 4767/50000 [1:58:18<19:53:35,  1.58s/it]
Outer training loop:  10%|9         | 4768/50000 [1:58:19<20:06:01,  1.60s/it]
Outer training loop:  10%|9         | 4769/50000 [1:58:21<19:53:15,  1.58s/it]
Outer training loop:  10%|9         | 4770/50000 [1:58:23<19:52:38,  1.58s/it]
Outer training loop:  10%|9         | 4771/50000 [1:58:24<19:35:20,  1.56s/it]
Outer training loop:  10%|9         | 4772/50000 [1:58:26<19:29:31,  1.55s/it]
Outer training loop:  10%|9         | 4773/50000 [1:58:27<19:30:00,  1.55s/it]
Outer training loop:  10%|9         | 4774/50000 [1:58:29<19:33:48,  1.56s/it]
Outer training loop:  10%|9         | 4775/50000 [1:58:30<19:26:48,  1.55s/it]
Outer training loop:  10%|9         | 4776/50000 [1:58:32<19:26:24,  1.55s/it]
Outer training loop:  10%|9         | 4777/50000 [1:58:34<19:57:57,  1.59s/it]
Outer training loop:  10%|9         | 4778/50000 [1:58:35<19:42:17,  1.57s/it]
Outer training loop:  10%|9         | 4779/50000 [1:58:37<19:43:18,  1.57s/it]
Outer training loop:  10%|9         | 4780/50000 [1:58:38<19:17:12,  1.54s/it]
Outer training loop:  10%|9         | 4781/50000 [1:58:40<19:48:01,  1.58s/it]
Outer training loop:  10%|9         | 4782/50000 [1:58:41<19:32:35,  1.56s/it]
Outer training loop:  10%|9         | 4783/50000 [1:58:43<19:52:03,  1.58s/it]
Outer training loop:  10%|9         | 4784/50000 [1:58:44<19:35:23,  1.56s/it]
Outer training loop:  10%|9         | 4785/50000 [1:58:46<19:39:46,  1.57s/it]
Outer training loop:  10%|9         | 4786/50000 [1:58:48<19:30:36,  1.55s/it]
Outer training loop:  10%|9         | 4787/50000 [1:58:49<19:40:48,  1.57s/it]
Outer training loop:  10%|9         | 4788/50000 [1:58:51<19:26:22,  1.55s/it]
Outer training loop:  10%|9         | 4789/50000 [1:58:52<19:26:08,  1.55s/it]
Outer training loop:  10%|9         | 4790/50000 [1:58:54<19:13:06,  1.53s/it]
Outer training loop:  10%|9         | 4791/50000 [1:58:55<19:16:37,  1.54s/it]
Outer training loop:  10%|9         | 4792/50000 [1:58:57<19:02:31,  1.52s/it]
Outer training loop:  10%|9         | 4793/50000 [1:58:58<19:26:19,  1.55s/it]
Outer training loop:  10%|9         | 4794/50000 [1:59:00<19:04:43,  1.52s/it]
Outer training loop:  10%|9         | 4795/50000 [1:59:01<19:16:16,  1.53s/it]
Outer training loop:  10%|9         | 4796/50000 [1:59:03<19:17:01,  1.54s/it]
Outer training loop:  10%|9         | 4797/50000 [1:59:04<19:36:08,  1.56s/it]
Outer training loop:  10%|9         | 4798/50000 [1:59:06<19:18:50,  1.54s/it]
Outer training loop:  10%|9         | 4799/50000 [1:59:07<19:03:17,  1.52s/it]
Outer training loop:  10%|9         | 4800/50000 [1:59:09<19:17:56,  1.54s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.59s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.57s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.56s/it]

Outer training loop:  10%|9         | 4801/50000 [1:59:15<36:54:53,  2.94s/it]
Outer training loop:  10%|9         | 4802/50000 [1:59:17<31:24:21,  2.50s/it]
Outer training loop:  10%|9         | 4803/50000 [1:59:18<28:16:45,  2.25s/it]
Outer training loop:  10%|9         | 4804/50000 [1:59:20<25:25:22,  2.03s/it]
Outer training loop:  10%|9         | 4805/50000 [1:59:21<23:44:25,  1.89s/it]
Outer training loop:  10%|9         | 4806/50000 [1:59:23<22:19:09,  1.78s/it]
Outer training loop:  10%|9         | 4807/50000 [1:59:25<21:46:03,  1.73s/it]
Outer training loop:  10%|9         | 4808/50000 [1:59:26<21:02:40,  1.68s/it]
Outer training loop:  10%|9         | 4809/50000 [1:59:28<20:51:39,  1.66s/it]
Outer training loop:  10%|9         | 4810/50000 [1:59:29<20:57:08,  1.67s/it]
Outer training loop:  10%|9         | 4811/50000 [1:59:31<20:41:38,  1.65s/it]
Outer training loop:  10%|9         | 4812/50000 [1:59:33<20:44:30,  1.65s/it]
Outer training loop:  10%|9         | 4813/50000 [1:59:34<20:51:51,  1.66s/it]
Outer training loop:  10%|9         | 4814/50000 [1:59:36<20:10:18,  1.61s/it]
Outer training loop:  10%|9         | 4815/50000 [1:59:38<20:30:32,  1.63s/it]
Outer training loop:  10%|9         | 4816/50000 [1:59:39<20:32:00,  1.64s/it]
Outer training loop:  10%|9         | 4817/50000 [1:59:41<19:55:37,  1.59s/it]
Outer training loop:  10%|9         | 4818/50000 [1:59:42<19:50:41,  1.58s/it]
Outer training loop:  10%|9         | 4819/50000 [1:59:44<20:26:29,  1.63s/it]
Outer training loop:  10%|9         | 4820/50000 [1:59:46<20:28:57,  1.63s/it]
Outer training loop:  10%|9         | 4821/50000 [1:59:47<20:38:12,  1.64s/it]
Outer training loop:  10%|9         | 4822/50000 [1:59:49<20:32:44,  1.64s/it]
Outer training loop:  10%|9         | 4823/50000 [1:59:51<20:31:12,  1.64s/it]
Outer training loop:  10%|9         | 4824/50000 [1:59:52<20:25:12,  1.63s/it]
Outer training loop:  10%|9         | 4825/50000 [1:59:54<20:30:13,  1.63s/it]
Outer training loop:  10%|9         | 4826/50000 [1:59:55<20:11:25,  1.61s/it]
Outer training loop:  10%|9         | 4827/50000 [1:59:57<20:26:03,  1.63s/it]
Outer training loop:  10%|9         | 4828/50000 [1:59:59<20:24:48,  1.63s/it]
Outer training loop:  10%|9         | 4829/50000 [2:00:00<20:25:54,  1.63s/it]
Outer training loop:  10%|9         | 4830/50000 [2:00:02<20:14:26,  1.61s/it]
Outer training loop:  10%|9         | 4831/50000 [2:00:04<20:28:38,  1.63s/it]
Outer training loop:  10%|9         | 4832/50000 [2:00:05<20:17:35,  1.62s/it]
Outer training loop:  10%|9         | 4833/50000 [2:00:07<20:13:39,  1.61s/it]
Outer training loop:  10%|9         | 4834/50000 [2:00:08<20:10:57,  1.61s/it]
Outer training loop:  10%|9         | 4835/50000 [2:00:10<20:07:27,  1.60s/it]
Outer training loop:  10%|9         | 4836/50000 [2:00:11<19:59:08,  1.59s/it]
Outer training loop:  10%|9         | 4837/50000 [2:00:13<20:14:42,  1.61s/it]
Outer training loop:  10%|9         | 4838/50000 [2:00:15<19:46:17,  1.58s/it]
Outer training loop:  10%|9         | 4839/50000 [2:00:16<19:35:09,  1.56s/it]
Outer training loop:  10%|9         | 4840/50000 [2:00:18<20:10:55,  1.61s/it]
Outer training loop:  10%|9         | 4841/50000 [2:00:20<20:14:37,  1.61s/it]
Outer training loop:  10%|9         | 4842/50000 [2:00:21<20:35:01,  1.64s/it]
Outer training loop:  10%|9         | 4843/50000 [2:00:23<20:30:05,  1.63s/it]
Outer training loop:  10%|9         | 4844/50000 [2:00:24<19:58:33,  1.59s/it]
Outer training loop:  10%|9         | 4845/50000 [2:00:26<19:48:16,  1.58s/it]
Outer training loop:  10%|9         | 4846/50000 [2:00:28<20:00:47,  1.60s/it]
Outer training loop:  10%|9         | 4847/50000 [2:00:29<19:49:54,  1.58s/it]
Outer training loop:  10%|9         | 4848/50000 [2:00:31<20:05:47,  1.60s/it]
Outer training loop:  10%|9         | 4849/50000 [2:00:32<19:51:09,  1.58s/it]
Outer training loop:  10%|9         | 4850/50000 [2:00:34<20:03:22,  1.60s/it]
Outer training loop:  10%|9         | 4851/50000 [2:00:35<19:49:31,  1.58s/it]
Outer training loop:  10%|9         | 4852/50000 [2:00:37<19:39:53,  1.57s/it]
Outer training loop:  10%|9         | 4853/50000 [2:00:38<19:26:52,  1.55s/it]
Outer training loop:  10%|9         | 4854/50000 [2:00:40<19:51:35,  1.58s/it]
Outer training loop:  10%|9         | 4855/50000 [2:00:42<19:29:42,  1.55s/it]
Outer training loop:  10%|9         | 4856/50000 [2:00:43<19:53:10,  1.59s/it]
Outer training loop:  10%|9         | 4857/50000 [2:00:45<19:40:42,  1.57s/it]
Outer training loop:  10%|9         | 4858/50000 [2:00:46<19:53:39,  1.59s/it]
Outer training loop:  10%|9         | 4859/50000 [2:00:48<19:38:49,  1.57s/it]
Outer training loop:  10%|9         | 4860/50000 [2:00:50<20:34:41,  1.64s/it]
Outer training loop:  10%|9         | 4861/50000 [2:00:51<20:01:35,  1.60s/it]
Outer training loop:  10%|9         | 4862/50000 [2:00:53<19:51:59,  1.58s/it]
Outer training loop:  10%|9         | 4863/50000 [2:00:54<19:51:58,  1.58s/it]
Outer training loop:  10%|9         | 4864/50000 [2:00:56<19:27:03,  1.55s/it]
Outer training loop:  10%|9         | 4865/50000 [2:00:58<19:56:56,  1.59s/it]
Outer training loop:  10%|9         | 4866/50000 [2:00:59<19:49:06,  1.58s/it]
Outer training loop:  10%|9         | 4867/50000 [2:01:01<19:44:48,  1.58s/it]
Outer training loop:  10%|9         | 4868/50000 [2:01:02<19:53:01,  1.59s/it]
Outer training loop:  10%|9         | 4869/50000 [2:01:04<20:10:00,  1.61s/it]
Outer training loop:  10%|9         | 4870/50000 [2:01:05<19:50:08,  1.58s/it]
Outer training loop:  10%|9         | 4871/50000 [2:01:07<19:31:34,  1.56s/it]
Outer training loop:  10%|9         | 4872/50000 [2:01:09<19:52:21,  1.59s/it]
Outer training loop:  10%|9         | 4873/50000 [2:01:10<19:30:04,  1.56s/it]
Outer training loop:  10%|9         | 4874/50000 [2:01:12<19:37:04,  1.57s/it]
Outer training loop:  10%|9         | 4875/50000 [2:01:13<19:04:11,  1.52s/it]
Outer training loop:  10%|9         | 4876/50000 [2:01:15<19:11:00,  1.53s/it]
Outer training loop:  10%|9         | 4877/50000 [2:01:16<19:18:49,  1.54s/it]
Outer training loop:  10%|9         | 4878/50000 [2:01:18<19:21:11,  1.54s/it]
Outer training loop:  10%|9         | 4879/50000 [2:01:20<20:06:25,  1.60s/it]
Outer training loop:  10%|9         | 4880/50000 [2:01:21<20:01:25,  1.60s/it]
Outer training loop:  10%|9         | 4881/50000 [2:01:23<19:26:37,  1.55s/it]
Outer training loop:  10%|9         | 4882/50000 [2:01:24<19:43:00,  1.57s/it]
Outer training loop:  10%|9         | 4883/50000 [2:01:26<19:50:12,  1.58s/it]
Outer training loop:  10%|9         | 4884/50000 [2:01:27<19:58:38,  1.59s/it]
Outer training loop:  10%|9         | 4885/50000 [2:01:29<19:46:25,  1.58s/it]
Outer training loop:  10%|9         | 4886/50000 [2:01:31<19:51:20,  1.58s/it]
Outer training loop:  10%|9         | 4887/50000 [2:01:32<19:52:27,  1.59s/it]
Outer training loop:  10%|9         | 4888/50000 [2:01:34<19:56:29,  1.59s/it]
Outer training loop:  10%|9         | 4889/50000 [2:01:35<19:42:23,  1.57s/it]
Outer training loop:  10%|9         | 4890/50000 [2:01:37<19:47:11,  1.58s/it]
Outer training loop:  10%|9         | 4891/50000 [2:01:38<19:42:53,  1.57s/it]
Outer training loop:  10%|9         | 4892/50000 [2:01:40<19:48:55,  1.58s/it]
Outer training loop:  10%|9         | 4893/50000 [2:01:42<19:36:27,  1.56s/it]
Outer training loop:  10%|9         | 4894/50000 [2:01:43<19:44:34,  1.58s/it]
Outer training loop:  10%|9         | 4895/50000 [2:01:45<19:40:15,  1.57s/it]
Outer training loop:  10%|9         | 4896/50000 [2:01:46<19:07:41,  1.53s/it]
Outer training loop:  10%|9         | 4897/50000 [2:01:48<19:07:23,  1.53s/it]
Outer training loop:  10%|9         | 4898/50000 [2:01:49<19:20:09,  1.54s/it]
Outer training loop:  10%|9         | 4899/50000 [2:01:51<19:28:14,  1.55s/it]
Outer training loop:  10%|9         | 4900/50000 [2:01:52<19:36:51,  1.57s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.51s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.51s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.51s/it]

Outer training loop:  10%|9         | 4901/50000 [2:01:59<36:55:23,  2.95s/it]
Outer training loop:  10%|9         | 4902/50000 [2:02:00<31:23:38,  2.51s/it]
Outer training loop:  10%|9         | 4903/50000 [2:02:02<27:58:05,  2.23s/it]
Outer training loop:  10%|9         | 4904/50000 [2:02:03<25:10:43,  2.01s/it]
Outer training loop:  10%|9         | 4905/50000 [2:02:05<23:58:52,  1.91s/it]
Outer training loop:  10%|9         | 4906/50000 [2:02:06<22:23:27,  1.79s/it]
Outer training loop:  10%|9         | 4907/50000 [2:02:08<22:19:44,  1.78s/it]
Outer training loop:  10%|9         | 4908/50000 [2:02:10<21:24:42,  1.71s/it]
Outer training loop:  10%|9         | 4909/50000 [2:02:11<20:54:23,  1.67s/it]
Outer training loop:  10%|9         | 4910/50000 [2:02:13<20:26:45,  1.63s/it]
Outer training loop:  10%|9         | 4911/50000 [2:02:14<20:11:09,  1.61s/it]
Outer training loop:  10%|9         | 4912/50000 [2:02:16<19:39:27,  1.57s/it]
Outer training loop:  10%|9         | 4913/50000 [2:02:17<19:32:21,  1.56s/it]
Outer training loop:  10%|9         | 4914/50000 [2:02:19<19:34:30,  1.56s/it]
Outer training loop:  10%|9         | 4915/50000 [2:02:21<19:46:04,  1.58s/it]
Outer training loop:  10%|9         | 4916/50000 [2:02:22<19:15:36,  1.54s/it]
Outer training loop:  10%|9         | 4917/50000 [2:02:23<19:12:11,  1.53s/it]
Outer training loop:  10%|9         | 4918/50000 [2:02:25<19:03:38,  1.52s/it]
Outer training loop:  10%|9         | 4919/50000 [2:02:27<19:14:09,  1.54s/it]
Outer training loop:  10%|9         | 4920/50000 [2:02:28<19:10:41,  1.53s/it]
Outer training loop:  10%|9         | 4921/50000 [2:02:30<19:22:10,  1.55s/it]
Outer training loop:  10%|9         | 4922/50000 [2:02:31<18:58:45,  1.52s/it]
Outer training loop:  10%|9         | 4923/50000 [2:02:33<19:27:22,  1.55s/it]
Outer training loop:  10%|9         | 4924/50000 [2:02:34<19:33:10,  1.56s/it]
Outer training loop:  10%|9         | 4925/50000 [2:02:36<19:29:13,  1.56s/it]
Outer training loop:  10%|9         | 4926/50000 [2:02:37<19:06:24,  1.53s/it]
Outer training loop:  10%|9         | 4927/50000 [2:02:39<19:07:27,  1.53s/it]
Outer training loop:  10%|9         | 4928/50000 [2:02:40<19:12:37,  1.53s/it]
Outer training loop:  10%|9         | 4929/50000 [2:02:42<19:16:28,  1.54s/it]
Outer training loop:  10%|9         | 4930/50000 [2:02:43<19:18:02,  1.54s/it]
Outer training loop:  10%|9         | 4931/50000 [2:02:45<19:37:04,  1.57s/it]
Outer training loop:  10%|9         | 4932/50000 [2:02:47<19:33:12,  1.56s/it]
Outer training loop:  10%|9         | 4933/50000 [2:02:48<19:24:35,  1.55s/it]
Outer training loop:  10%|9         | 4934/50000 [2:02:50<19:06:28,  1.53s/it]
Outer training loop:  10%|9         | 4935/50000 [2:02:51<19:14:04,  1.54s/it]
Outer training loop:  10%|9         | 4936/50000 [2:02:53<19:16:20,  1.54s/it]
Outer training loop:  10%|9         | 4937/50000 [2:02:54<19:32:55,  1.56s/it]
Outer training loop:  10%|9         | 4938/50000 [2:02:56<19:33:59,  1.56s/it]
Outer training loop:  10%|9         | 4939/50000 [2:02:57<19:28:30,  1.56s/it]
Outer training loop:  10%|9         | 4940/50000 [2:02:59<19:20:48,  1.55s/it]
Outer training loop:  10%|9         | 4941/50000 [2:03:01<19:16:58,  1.54s/it]
Outer training loop:  10%|9         | 4942/50000 [2:03:02<19:16:50,  1.54s/it]
Outer training loop:  10%|9         | 4943/50000 [2:03:04<19:29:08,  1.56s/it]
Outer training loop:  10%|9         | 4944/50000 [2:03:05<19:24:21,  1.55s/it]
Outer training loop:  10%|9         | 4945/50000 [2:03:07<19:27:37,  1.55s/it]
Outer training loop:  10%|9         | 4946/50000 [2:03:08<19:23:16,  1.55s/it]
Outer training loop:  10%|9         | 4947/50000 [2:03:10<19:37:28,  1.57s/it]
Outer training loop:  10%|9         | 4948/50000 [2:03:11<19:28:56,  1.56s/it]
Outer training loop:  10%|9         | 4949/50000 [2:03:13<19:25:06,  1.55s/it]
Outer training loop:  10%|9         | 4950/50000 [2:03:14<19:07:06,  1.53s/it]
Outer training loop:  10%|9         | 4951/50000 [2:03:16<19:11:26,  1.53s/it]
Outer training loop:  10%|9         | 4952/50000 [2:03:18<19:11:24,  1.53s/it]
Outer training loop:  10%|9         | 4953/50000 [2:03:19<19:29:15,  1.56s/it]
Outer training loop:  10%|9         | 4954/50000 [2:03:21<19:12:20,  1.53s/it]
Outer training loop:  10%|9         | 4955/50000 [2:03:22<19:13:07,  1.54s/it]
Outer training loop:  10%|9         | 4956/50000 [2:03:24<19:09:33,  1.53s/it]
Outer training loop:  10%|9         | 4957/50000 [2:03:25<19:48:33,  1.58s/it]
Outer training loop:  10%|9         | 4958/50000 [2:03:27<19:33:01,  1.56s/it]
Outer training loop:  10%|9         | 4959/50000 [2:03:29<19:44:03,  1.58s/it]
Outer training loop:  10%|9         | 4960/50000 [2:03:30<19:28:14,  1.56s/it]
Outer training loop:  10%|9         | 4961/50000 [2:03:32<19:14:21,  1.54s/it]
Outer training loop:  10%|9         | 4962/50000 [2:03:33<19:40:14,  1.57s/it]
Outer training loop:  10%|9         | 4963/50000 [2:03:35<20:07:10,  1.61s/it]
Outer training loop:  10%|9         | 4964/50000 [2:03:36<19:25:15,  1.55s/it]
Outer training loop:  10%|9         | 4965/50000 [2:03:38<19:45:22,  1.58s/it]
Outer training loop:  10%|9         | 4966/50000 [2:03:39<19:38:25,  1.57s/it]
Outer training loop:  10%|9         | 4967/50000 [2:03:41<19:55:41,  1.59s/it]
Outer training loop:  10%|9         | 4968/50000 [2:03:43<19:37:03,  1.57s/it]
Outer training loop:  10%|9         | 4969/50000 [2:03:44<19:44:18,  1.58s/it]
Outer training loop:  10%|9         | 4970/50000 [2:03:46<19:51:25,  1.59s/it]
Outer training loop:  10%|9         | 4971/50000 [2:03:47<19:51:42,  1.59s/it]
Outer training loop:  10%|9         | 4972/50000 [2:03:49<19:42:01,  1.58s/it]
Outer training loop:  10%|9         | 4973/50000 [2:03:51<19:49:52,  1.59s/it]
Outer training loop:  10%|9         | 4974/50000 [2:03:52<19:34:08,  1.56s/it]
Outer training loop:  10%|9         | 4975/50000 [2:03:54<19:44:45,  1.58s/it]
Outer training loop:  10%|9         | 4976/50000 [2:03:55<19:29:31,  1.56s/it]
Outer training loop:  10%|9         | 4977/50000 [2:03:57<19:29:25,  1.56s/it]
Outer training loop:  10%|9         | 4978/50000 [2:03:58<19:09:45,  1.53s/it]
Outer training loop:  10%|9         | 4979/50000 [2:04:00<19:11:36,  1.53s/it]
Outer training loop:  10%|9         | 4980/50000 [2:04:01<19:15:06,  1.54s/it]
Outer training loop:  10%|9         | 4981/50000 [2:04:03<19:28:49,  1.56s/it]
Outer training loop:  10%|9         | 4982/50000 [2:04:05<19:25:47,  1.55s/it]
Outer training loop:  10%|9         | 4983/50000 [2:04:06<19:29:00,  1.56s/it]
Outer training loop:  10%|9         | 4984/50000 [2:04:08<19:07:35,  1.53s/it]
Outer training loop:  10%|9         | 4985/50000 [2:04:09<19:03:15,  1.52s/it]
Outer training loop:  10%|9         | 4986/50000 [2:04:11<19:39:22,  1.57s/it]
Outer training loop:  10%|9         | 4987/50000 [2:04:12<19:52:58,  1.59s/it]
Outer training loop:  10%|9         | 4988/50000 [2:04:14<19:26:20,  1.55s/it]
Outer training loop:  10%|9         | 4989/50000 [2:04:15<19:10:13,  1.53s/it]
Outer training loop:  10%|9         | 4990/50000 [2:04:17<19:35:05,  1.57s/it]
Outer training loop:  10%|9         | 4991/50000 [2:04:19<19:38:40,  1.57s/it]
Outer training loop:  10%|9         | 4992/50000 [2:04:20<19:34:31,  1.57s/it]
Outer training loop:  10%|9         | 4993/50000 [2:04:22<19:11:17,  1.53s/it]
Outer training loop:  10%|9         | 4994/50000 [2:04:23<19:34:57,  1.57s/it]
Outer training loop:  10%|9         | 4995/50000 [2:04:25<19:38:27,  1.57s/it]
Outer training loop:  10%|9         | 4996/50000 [2:04:26<19:24:59,  1.55s/it]
Outer training loop:  10%|9         | 4997/50000 [2:04:28<19:14:05,  1.54s/it]
Outer training loop:  10%|9         | 4998/50000 [2:04:29<19:43:04,  1.58s/it]
Outer training loop:  10%|9         | 4999/50000 [2:04:31<19:38:53,  1.57s/it]
Outer training loop:  10%|#         | 5000/50000 [2:04:33<19:38:42,  1.57s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.56s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.53s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:01,  1.48s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:03,  1.52s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:05,  1.57s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:05,  1.59s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:03,  1.58s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:00,  1.57s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<02:00,  1.59s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:58,  1.57s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:59,  1.61s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:56,  1.59s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:53,  1.57s/it]

Evaluating policy:  14%|#4        | 12/83 [00:19<01:52,  1.59s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:49,  1.57s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:46,  1.54s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:46,  1.57s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:43,  1.55s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:40,  1.52s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:40,  1.54s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:36,  1.51s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:35,  1.52s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:36,  1.56s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:33,  1.54s/it]

Evaluating policy:  28%|##7       | 23/83 [00:35<01:32,  1.55s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:30,  1.54s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:33,  1.60s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:30,  1.59s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:28,  1.58s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:25,  1.55s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:25,  1.58s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:23,  1.57s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:20,  1.55s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:18,  1.55s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:21,  1.62s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:17,  1.58s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:14,  1.56s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:13,  1.56s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:10,  1.53s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:11,  1.58s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:08,  1.57s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:05,  1.52s/it]

Evaluating policy:  49%|####9     | 41/83 [01:03<01:02,  1.49s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:05,  1.60s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:02,  1.55s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<01:00,  1.54s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<00:58,  1.53s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:11<00:55,  1.51s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:13<00:54,  1.50s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:55,  1.59s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:52,  1.55s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:50,  1.54s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:49,  1.55s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:47,  1.54s/it]

Evaluating policy:  64%|######3   | 53/83 [01:22<00:47,  1.58s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:47,  1.65s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:44,  1.61s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:42,  1.56s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:40,  1.55s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:39,  1.56s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:37,  1.56s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:34<00:38,  1.66s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:35,  1.60s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:37<00:32,  1.56s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:31,  1.56s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:29,  1.55s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:27,  1.53s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:26,  1.54s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:26,  1.63s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:24,  1.62s/it]

Evaluating policy:  83%|########3 | 69/83 [01:48<00:21,  1.57s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:20,  1.57s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:18,  1.54s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:16,  1.53s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:15,  1.53s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:13,  1.53s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:13,  1.66s/it]

Evaluating policy:  92%|#########1| 76/83 [01:59<00:11,  1.63s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.61s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.58s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.59s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.57s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.53s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.66s/it]

Evaluating policy: 100%|##########| 83/83 [02:10<00:00,  1.60s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<01:56,  1.42s/it]

Evaluating policy:   2%|2         | 2/83 [00:02<01:56,  1.44s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<01:56,  1.45s/it]

Evaluating policy:   5%|4         | 4/83 [00:05<01:55,  1.46s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<01:53,  1.46s/it]

Evaluating policy:   7%|7         | 6/83 [00:08<01:52,  1.46s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:52,  1.49s/it]

Evaluating policy:  10%|9         | 8/83 [00:11<01:51,  1.49s/it]

Evaluating policy:  11%|#         | 9/83 [00:13<02:01,  1.64s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:57,  1.61s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:58,  1.64s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:54,  1.61s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:50,  1.58s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:50,  1.61s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:45,  1.55s/it]

Evaluating policy:  19%|#9        | 16/83 [00:24<01:43,  1.54s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:43,  1.56s/it]

Evaluating policy:  22%|##1       | 18/83 [00:27<01:41,  1.56s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:39,  1.55s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:43,  1.64s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:40,  1.62s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:35,  1.57s/it]

Evaluating policy:  28%|##7       | 23/83 [00:35<01:33,  1.57s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:34,  1.59s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:31,  1.58s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:29,  1.57s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:26,  1.55s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:29,  1.63s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:26,  1.60s/it]

Evaluating policy:  36%|###6      | 30/83 [00:46<01:22,  1.57s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:19,  1.53s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:21,  1.59s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:18,  1.57s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:15,  1.53s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:12,  1.51s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:10,  1.50s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:12,  1.57s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:10,  1.57s/it]

Evaluating policy:  47%|####6     | 39/83 [01:00<01:08,  1.56s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:06,  1.54s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:07,  1.61s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:04,  1.58s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:02,  1.55s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<01:00,  1.54s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<00:59,  1.57s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<01:00,  1.64s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:13<00:57,  1.60s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:54,  1.56s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:52,  1.54s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:50,  1.53s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:48,  1.51s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:49,  1.61s/it]

Evaluating policy:  64%|######3   | 53/83 [01:22<00:47,  1.60s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:45,  1.56s/it]

Evaluating policy:  66%|######6   | 55/83 [01:25<00:43,  1.57s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:42,  1.56s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:40,  1.54s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:41,  1.66s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:38,  1.60s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:33<00:36,  1.57s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:34,  1.57s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:36<00:32,  1.55s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:30,  1.53s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:39<00:28,  1.51s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:29,  1.62s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:26,  1.58s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:25,  1.57s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:23,  1.54s/it]

Evaluating policy:  83%|########3 | 69/83 [01:47<00:21,  1.56s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:20,  1.58s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:18,  1.57s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:18,  1.67s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:16,  1.62s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:14,  1.57s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.55s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:10,  1.54s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.55s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.53s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.54s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.65s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.60s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.58s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.54s/it]

Outer training loop:  10%|#         | 5001/50000 [2:08:59<1013:02:29, 81.05s/it]
Outer training loop:  10%|#         | 5002/50000 [2:09:01<714:36:59, 57.17s/it] 
Outer training loop:  10%|#         | 5003/50000 [2:09:02<506:00:19, 40.48s/it]
Outer training loop:  10%|#         | 5004/50000 [2:09:04<359:47:13, 28.79s/it]
Outer training loop:  10%|#         | 5005/50000 [2:09:05<257:48:55, 20.63s/it]
Outer training loop:  10%|#         | 5006/50000 [2:09:07<187:53:35, 15.03s/it]
Outer training loop:  10%|#         | 5007/50000 [2:09:09<137:04:00, 10.97s/it]
Outer training loop:  10%|#         | 5008/50000 [2:09:10<101:26:35,  8.12s/it]
Outer training loop:  10%|#         | 5009/50000 [2:09:12<77:06:00,  6.17s/it] 
Outer training loop:  10%|#         | 5010/50000 [2:09:13<59:53:21,  4.79s/it]
Outer training loop:  10%|#         | 5011/50000 [2:09:15<48:00:12,  3.84s/it]
Outer training loop:  10%|#         | 5012/50000 [2:09:16<39:17:37,  3.14s/it]
Outer training loop:  10%|#         | 5013/50000 [2:09:18<33:27:57,  2.68s/it]
Outer training loop:  10%|#         | 5014/50000 [2:09:20<29:01:08,  2.32s/it]
Outer training loop:  10%|#         | 5015/50000 [2:09:21<26:24:45,  2.11s/it]
Outer training loop:  10%|#         | 5016/50000 [2:09:23<24:09:40,  1.93s/it]
Outer training loop:  10%|#         | 5017/50000 [2:09:24<22:55:59,  1.84s/it]
Outer training loop:  10%|#         | 5018/50000 [2:09:26<22:03:39,  1.77s/it]
Outer training loop:  10%|#         | 5019/50000 [2:09:27<21:06:28,  1.69s/it]
Outer training loop:  10%|#         | 5020/50000 [2:09:29<20:16:20,  1.62s/it]
Outer training loop:  10%|#         | 5021/50000 [2:09:30<19:53:11,  1.59s/it]
Outer training loop:  10%|#         | 5022/50000 [2:09:32<19:34:00,  1.57s/it]
Outer training loop:  10%|#         | 5023/50000 [2:09:33<19:33:35,  1.57s/it]
Outer training loop:  10%|#         | 5024/50000 [2:09:35<20:02:38,  1.60s/it]
Outer training loop:  10%|#         | 5025/50000 [2:09:37<19:46:00,  1.58s/it]
Outer training loop:  10%|#         | 5026/50000 [2:09:38<19:53:17,  1.59s/it]
Outer training loop:  10%|#         | 5027/50000 [2:09:40<19:57:39,  1.60s/it]
Outer training loop:  10%|#         | 5028/50000 [2:09:41<19:57:09,  1.60s/it]
Outer training loop:  10%|#         | 5029/50000 [2:09:43<19:15:57,  1.54s/it]
Outer training loop:  10%|#         | 5030/50000 [2:09:45<19:27:26,  1.56s/it]
Outer training loop:  10%|#         | 5031/50000 [2:09:46<19:09:19,  1.53s/it]
Outer training loop:  10%|#         | 5032/50000 [2:09:47<18:49:21,  1.51s/it]
Outer training loop:  10%|#         | 5033/50000 [2:09:49<19:15:47,  1.54s/it]
Outer training loop:  10%|#         | 5034/50000 [2:09:51<19:21:30,  1.55s/it]
Outer training loop:  10%|#         | 5035/50000 [2:09:52<19:02:41,  1.52s/it]
Outer training loop:  10%|#         | 5036/50000 [2:09:54<19:06:21,  1.53s/it]
Outer training loop:  10%|#         | 5037/50000 [2:09:55<19:19:34,  1.55s/it]
Outer training loop:  10%|#         | 5038/50000 [2:09:57<20:37:21,  1.65s/it]
Outer training loop:  10%|#         | 5039/50000 [2:09:59<20:06:10,  1.61s/it]
Outer training loop:  10%|#         | 5040/50000 [2:10:00<20:16:39,  1.62s/it]
Outer training loop:  10%|#         | 5041/50000 [2:10:02<19:58:09,  1.60s/it]
Outer training loop:  10%|#         | 5042/50000 [2:10:03<19:35:28,  1.57s/it]
Outer training loop:  10%|#         | 5043/50000 [2:10:05<20:00:44,  1.60s/it]
Outer training loop:  10%|#         | 5044/50000 [2:10:07<19:44:26,  1.58s/it]
Outer training loop:  10%|#         | 5045/50000 [2:10:08<19:35:27,  1.57s/it]
Outer training loop:  10%|#         | 5046/50000 [2:10:10<19:39:53,  1.57s/it]
Outer training loop:  10%|#         | 5047/50000 [2:10:11<19:45:14,  1.58s/it]
Outer training loop:  10%|#         | 5048/50000 [2:10:13<19:18:57,  1.55s/it]
Outer training loop:  10%|#         | 5049/50000 [2:10:14<19:41:06,  1.58s/it]
Outer training loop:  10%|#         | 5050/50000 [2:10:16<19:27:27,  1.56s/it]
Outer training loop:  10%|#         | 5051/50000 [2:10:18<19:51:01,  1.59s/it]
Outer training loop:  10%|#         | 5052/50000 [2:10:19<19:35:06,  1.57s/it]
Outer training loop:  10%|#         | 5053/50000 [2:10:21<19:37:49,  1.57s/it]
Outer training loop:  10%|#         | 5054/50000 [2:10:22<19:19:28,  1.55s/it]
Outer training loop:  10%|#         | 5055/50000 [2:10:24<19:23:51,  1.55s/it]
Outer training loop:  10%|#         | 5056/50000 [2:10:25<19:27:51,  1.56s/it]
Outer training loop:  10%|#         | 5057/50000 [2:10:27<19:32:12,  1.56s/it]
Outer training loop:  10%|#         | 5058/50000 [2:10:28<19:10:24,  1.54s/it]
Outer training loop:  10%|#         | 5059/50000 [2:10:30<19:09:08,  1.53s/it]
Outer training loop:  10%|#         | 5060/50000 [2:10:31<19:04:13,  1.53s/it]
Outer training loop:  10%|#         | 5061/50000 [2:10:33<19:37:47,  1.57s/it]
Outer training loop:  10%|#         | 5062/50000 [2:10:35<19:12:50,  1.54s/it]
Outer training loop:  10%|#         | 5063/50000 [2:10:36<19:27:29,  1.56s/it]
Outer training loop:  10%|#         | 5064/50000 [2:10:38<19:00:21,  1.52s/it]
Outer training loop:  10%|#         | 5065/50000 [2:10:39<19:21:53,  1.55s/it]
Outer training loop:  10%|#         | 5066/50000 [2:10:41<19:16:09,  1.54s/it]
Outer training loop:  10%|#         | 5067/50000 [2:10:42<19:31:13,  1.56s/it]
Outer training loop:  10%|#         | 5068/50000 [2:10:44<19:40:23,  1.58s/it]
Outer training loop:  10%|#         | 5069/50000 [2:10:45<19:24:47,  1.56s/it]
Outer training loop:  10%|#         | 5070/50000 [2:10:47<19:35:20,  1.57s/it]
Outer training loop:  10%|#         | 5071/50000 [2:10:49<19:34:24,  1.57s/it]
Outer training loop:  10%|#         | 5072/50000 [2:10:50<19:48:53,  1.59s/it]
Outer training loop:  10%|#         | 5073/50000 [2:10:52<19:24:12,  1.55s/it]
Outer training loop:  10%|#         | 5074/50000 [2:10:53<19:32:37,  1.57s/it]
Outer training loop:  10%|#         | 5075/50000 [2:10:55<19:23:28,  1.55s/it]
Outer training loop:  10%|#         | 5076/50000 [2:10:56<19:18:39,  1.55s/it]
Outer training loop:  10%|#         | 5077/50000 [2:10:58<19:10:44,  1.54s/it]
Outer training loop:  10%|#         | 5078/50000 [2:11:00<19:38:49,  1.57s/it]
Outer training loop:  10%|#         | 5079/50000 [2:11:01<19:20:20,  1.55s/it]
Outer training loop:  10%|#         | 5080/50000 [2:11:02<19:05:13,  1.53s/it]
Outer training loop:  10%|#         | 5081/50000 [2:11:04<19:26:06,  1.56s/it]
Outer training loop:  10%|#         | 5082/50000 [2:11:06<19:13:50,  1.54s/it]
Outer training loop:  10%|#         | 5083/50000 [2:11:07<19:30:20,  1.56s/it]
Outer training loop:  10%|#         | 5084/50000 [2:11:09<19:22:06,  1.55s/it]
Outer training loop:  10%|#         | 5085/50000 [2:11:10<19:41:28,  1.58s/it]
Outer training loop:  10%|#         | 5086/50000 [2:11:12<19:26:39,  1.56s/it]
Outer training loop:  10%|#         | 5087/50000 [2:11:13<19:32:45,  1.57s/it]
Outer training loop:  10%|#         | 5088/50000 [2:11:15<20:09:02,  1.62s/it]
Outer training loop:  10%|#         | 5089/50000 [2:11:17<19:29:10,  1.56s/it]
Outer training loop:  10%|#         | 5090/50000 [2:11:18<19:33:27,  1.57s/it]
Outer training loop:  10%|#         | 5091/50000 [2:11:20<19:59:27,  1.60s/it]
Outer training loop:  10%|#         | 5092/50000 [2:11:21<19:45:10,  1.58s/it]
Outer training loop:  10%|#         | 5093/50000 [2:11:23<19:41:35,  1.58s/it]
Outer training loop:  10%|#         | 5094/50000 [2:11:25<20:15:56,  1.62s/it]
Outer training loop:  10%|#         | 5095/50000 [2:11:26<20:05:04,  1.61s/it]
Outer training loop:  10%|#         | 5096/50000 [2:11:28<20:18:04,  1.63s/it]
Outer training loop:  10%|#         | 5097/50000 [2:11:30<19:56:49,  1.60s/it]
Outer training loop:  10%|#         | 5098/50000 [2:11:31<19:48:01,  1.59s/it]
Outer training loop:  10%|#         | 5099/50000 [2:11:33<19:08:13,  1.53s/it]
Outer training loop:  10%|#         | 5100/50000 [2:11:34<19:45:11,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.52s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.52s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.52s/it]

Outer training loop:  10%|#         | 5101/50000 [2:11:40<36:41:47,  2.94s/it]
Outer training loop:  10%|#         | 5102/50000 [2:11:42<31:30:49,  2.53s/it]
Outer training loop:  10%|#         | 5103/50000 [2:11:43<27:18:57,  2.19s/it]
Outer training loop:  10%|#         | 5104/50000 [2:11:45<25:05:16,  2.01s/it]
Outer training loop:  10%|#         | 5105/50000 [2:11:46<23:00:29,  1.84s/it]
Outer training loop:  10%|#         | 5106/50000 [2:11:48<22:02:02,  1.77s/it]
Outer training loop:  10%|#         | 5107/50000 [2:11:49<21:17:55,  1.71s/it]
Outer training loop:  10%|#         | 5108/50000 [2:11:51<21:22:32,  1.71s/it]
Outer training loop:  10%|#         | 5109/50000 [2:11:53<20:35:31,  1.65s/it]
Outer training loop:  10%|#         | 5110/50000 [2:11:54<20:37:12,  1.65s/it]
Outer training loop:  10%|#         | 5111/50000 [2:11:56<20:19:52,  1.63s/it]
Outer training loop:  10%|#         | 5112/50000 [2:11:58<20:12:49,  1.62s/it]
Outer training loop:  10%|#         | 5113/50000 [2:11:59<19:44:34,  1.58s/it]
Outer training loop:  10%|#         | 5114/50000 [2:12:01<19:59:22,  1.60s/it]
Outer training loop:  10%|#         | 5115/50000 [2:12:02<19:46:11,  1.59s/it]
Outer training loop:  10%|#         | 5116/50000 [2:12:04<19:42:24,  1.58s/it]
Outer training loop:  10%|#         | 5117/50000 [2:12:05<19:23:10,  1.55s/it]
Outer training loop:  10%|#         | 5118/50000 [2:12:07<19:34:54,  1.57s/it]
Outer training loop:  10%|#         | 5119/50000 [2:12:09<19:41:01,  1.58s/it]
Outer training loop:  10%|#         | 5120/50000 [2:12:10<19:58:08,  1.60s/it]
Outer training loop:  10%|#         | 5121/50000 [2:12:12<20:03:14,  1.61s/it]
Outer training loop:  10%|#         | 5122/50000 [2:12:13<20:06:27,  1.61s/it]
Outer training loop:  10%|#         | 5123/50000 [2:12:15<19:48:29,  1.59s/it]
Outer training loop:  10%|#         | 5124/50000 [2:12:16<19:22:36,  1.55s/it]
Outer training loop:  10%|#         | 5125/50000 [2:12:18<19:43:34,  1.58s/it]
Outer training loop:  10%|#         | 5126/50000 [2:12:20<19:30:15,  1.56s/it]
Outer training loop:  10%|#         | 5127/50000 [2:12:21<19:56:13,  1.60s/it]
Outer training loop:  10%|#         | 5128/50000 [2:12:23<20:15:05,  1.62s/it]
Outer training loop:  10%|#         | 5129/50000 [2:12:24<19:52:19,  1.59s/it]
Outer training loop:  10%|#         | 5130/50000 [2:12:26<19:21:46,  1.55s/it]
Outer training loop:  10%|#         | 5131/50000 [2:12:28<19:30:54,  1.57s/it]
Outer training loop:  10%|#         | 5132/50000 [2:12:29<19:20:52,  1.55s/it]
Outer training loop:  10%|#         | 5133/50000 [2:12:31<19:42:37,  1.58s/it]
Outer training loop:  10%|#         | 5134/50000 [2:12:32<19:28:21,  1.56s/it]
Outer training loop:  10%|#         | 5135/50000 [2:12:34<20:38:48,  1.66s/it]
Outer training loop:  10%|#         | 5136/50000 [2:12:36<20:13:55,  1.62s/it]
Outer training loop:  10%|#         | 5137/50000 [2:12:37<20:03:01,  1.61s/it]
Outer training loop:  10%|#         | 5138/50000 [2:12:39<20:13:52,  1.62s/it]
Outer training loop:  10%|#         | 5139/50000 [2:12:40<19:54:27,  1.60s/it]
Outer training loop:  10%|#         | 5140/50000 [2:12:42<19:44:32,  1.58s/it]
Outer training loop:  10%|#         | 5141/50000 [2:12:44<19:58:38,  1.60s/it]
Outer training loop:  10%|#         | 5142/50000 [2:12:45<19:30:27,  1.57s/it]
Outer training loop:  10%|#         | 5143/50000 [2:12:47<19:40:42,  1.58s/it]
Outer training loop:  10%|#         | 5144/50000 [2:12:48<19:38:30,  1.58s/it]
Outer training loop:  10%|#         | 5145/50000 [2:12:50<19:47:10,  1.59s/it]
Outer training loop:  10%|#         | 5146/50000 [2:12:51<19:08:54,  1.54s/it]
Outer training loop:  10%|#         | 5147/50000 [2:12:53<19:37:17,  1.57s/it]
Outer training loop:  10%|#         | 5148/50000 [2:12:55<19:53:41,  1.60s/it]
Outer training loop:  10%|#         | 5149/50000 [2:12:56<19:40:26,  1.58s/it]
Outer training loop:  10%|#         | 5150/50000 [2:12:58<19:48:16,  1.59s/it]
Outer training loop:  10%|#         | 5151/50000 [2:12:59<19:33:50,  1.57s/it]
Outer training loop:  10%|#         | 5152/50000 [2:13:01<19:40:34,  1.58s/it]
Outer training loop:  10%|#         | 5153/50000 [2:13:02<19:23:01,  1.56s/it]
Outer training loop:  10%|#         | 5154/50000 [2:13:04<19:11:47,  1.54s/it]
Outer training loop:  10%|#         | 5155/50000 [2:13:05<19:19:08,  1.55s/it]
Outer training loop:  10%|#         | 5156/50000 [2:13:07<19:14:29,  1.54s/it]
Outer training loop:  10%|#         | 5157/50000 [2:13:09<19:25:41,  1.56s/it]
Outer training loop:  10%|#         | 5158/50000 [2:13:10<19:06:37,  1.53s/it]
Outer training loop:  10%|#         | 5159/50000 [2:13:12<19:19:42,  1.55s/it]
Outer training loop:  10%|#         | 5160/50000 [2:13:13<18:50:26,  1.51s/it]
Outer training loop:  10%|#         | 5161/50000 [2:13:15<19:18:28,  1.55s/it]
Outer training loop:  10%|#         | 5162/50000 [2:13:16<19:19:07,  1.55s/it]
Outer training loop:  10%|#         | 5163/50000 [2:13:18<19:23:39,  1.56s/it]
Outer training loop:  10%|#         | 5164/50000 [2:13:19<19:14:12,  1.54s/it]
Outer training loop:  10%|#         | 5165/50000 [2:13:21<19:07:23,  1.54s/it]
Outer training loop:  10%|#         | 5166/50000 [2:13:22<19:06:04,  1.53s/it]
Outer training loop:  10%|#         | 5167/50000 [2:13:24<19:25:38,  1.56s/it]
Outer training loop:  10%|#         | 5168/50000 [2:13:26<19:48:34,  1.59s/it]
Outer training loop:  10%|#         | 5169/50000 [2:13:27<20:02:01,  1.61s/it]
Outer training loop:  10%|#         | 5170/50000 [2:13:29<19:35:54,  1.57s/it]
Outer training loop:  10%|#         | 5171/50000 [2:13:30<19:33:24,  1.57s/it]
Outer training loop:  10%|#         | 5172/50000 [2:13:32<19:26:44,  1.56s/it]
Outer training loop:  10%|#         | 5173/50000 [2:13:34<19:43:46,  1.58s/it]
Outer training loop:  10%|#         | 5174/50000 [2:13:35<19:43:24,  1.58s/it]
Outer training loop:  10%|#         | 5175/50000 [2:13:37<19:39:43,  1.58s/it]
Outer training loop:  10%|#         | 5176/50000 [2:13:38<19:26:13,  1.56s/it]
Outer training loop:  10%|#         | 5177/50000 [2:13:40<19:41:20,  1.58s/it]
Outer training loop:  10%|#         | 5178/50000 [2:13:41<19:15:40,  1.55s/it]
Outer training loop:  10%|#         | 5179/50000 [2:13:43<19:35:31,  1.57s/it]
Outer training loop:  10%|#         | 5180/50000 [2:13:45<19:37:28,  1.58s/it]
Outer training loop:  10%|#         | 5181/50000 [2:13:46<19:35:53,  1.57s/it]
Outer training loop:  10%|#         | 5182/50000 [2:13:48<19:38:31,  1.58s/it]
Outer training loop:  10%|#         | 5183/50000 [2:13:49<19:18:08,  1.55s/it]
Outer training loop:  10%|#         | 5184/50000 [2:13:51<19:39:43,  1.58s/it]
Outer training loop:  10%|#         | 5185/50000 [2:13:53<19:54:07,  1.60s/it]
Outer training loop:  10%|#         | 5186/50000 [2:13:54<19:33:54,  1.57s/it]
Outer training loop:  10%|#         | 5187/50000 [2:13:56<19:37:53,  1.58s/it]
Outer training loop:  10%|#         | 5188/50000 [2:13:57<19:20:48,  1.55s/it]
Outer training loop:  10%|#         | 5189/50000 [2:13:59<19:22:03,  1.56s/it]
Outer training loop:  10%|#         | 5190/50000 [2:14:00<19:14:01,  1.55s/it]
Outer training loop:  10%|#         | 5191/50000 [2:14:02<19:24:08,  1.56s/it]
Outer training loop:  10%|#         | 5192/50000 [2:14:03<19:41:49,  1.58s/it]
Outer training loop:  10%|#         | 5193/50000 [2:14:05<19:53:17,  1.60s/it]
Outer training loop:  10%|#         | 5194/50000 [2:14:07<19:51:08,  1.60s/it]
Outer training loop:  10%|#         | 5195/50000 [2:14:08<19:38:46,  1.58s/it]
Outer training loop:  10%|#         | 5196/50000 [2:14:10<19:59:54,  1.61s/it]
Outer training loop:  10%|#         | 5197/50000 [2:14:11<19:20:38,  1.55s/it]
Outer training loop:  10%|#         | 5198/50000 [2:14:13<19:27:11,  1.56s/it]
Outer training loop:  10%|#         | 5199/50000 [2:14:14<19:38:57,  1.58s/it]
Outer training loop:  10%|#         | 5200/50000 [2:14:16<19:56:59,  1.60s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.56s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.56s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.52s/it]

Outer training loop:  10%|#         | 5201/50000 [2:14:22<36:47:47,  2.96s/it]
Outer training loop:  10%|#         | 5202/50000 [2:14:24<31:46:41,  2.55s/it]
Outer training loop:  10%|#         | 5203/50000 [2:14:25<28:13:21,  2.27s/it]
Outer training loop:  10%|#         | 5204/50000 [2:14:27<25:03:51,  2.01s/it]
Outer training loop:  10%|#         | 5205/50000 [2:14:28<23:11:36,  1.86s/it]
Outer training loop:  10%|#         | 5206/50000 [2:14:30<22:15:49,  1.79s/it]
Outer training loop:  10%|#         | 5207/50000 [2:14:32<21:09:32,  1.70s/it]
Outer training loop:  10%|#         | 5208/50000 [2:14:33<20:57:37,  1.68s/it]
Outer training loop:  10%|#         | 5209/50000 [2:14:35<20:52:04,  1.68s/it]
Outer training loop:  10%|#         | 5210/50000 [2:14:36<20:25:50,  1.64s/it]
Outer training loop:  10%|#         | 5211/50000 [2:14:38<20:04:33,  1.61s/it]
Outer training loop:  10%|#         | 5212/50000 [2:14:40<20:04:05,  1.61s/it]
Outer training loop:  10%|#         | 5213/50000 [2:14:41<19:44:42,  1.59s/it]
Outer training loop:  10%|#         | 5214/50000 [2:14:43<20:01:17,  1.61s/it]
Outer training loop:  10%|#         | 5215/50000 [2:14:44<19:31:27,  1.57s/it]
Outer training loop:  10%|#         | 5216/50000 [2:14:46<19:31:43,  1.57s/it]
Outer training loop:  10%|#         | 5217/50000 [2:14:47<19:21:31,  1.56s/it]
Outer training loop:  10%|#         | 5218/50000 [2:14:49<19:21:28,  1.56s/it]
Outer training loop:  10%|#         | 5219/50000 [2:14:50<19:29:37,  1.57s/it]
Outer training loop:  10%|#         | 5220/50000 [2:14:52<19:34:17,  1.57s/it]
Outer training loop:  10%|#         | 5221/50000 [2:14:54<19:15:55,  1.55s/it]
Outer training loop:  10%|#         | 5222/50000 [2:14:55<19:29:52,  1.57s/it]
Outer training loop:  10%|#         | 5223/50000 [2:14:57<19:31:23,  1.57s/it]
Outer training loop:  10%|#         | 5224/50000 [2:14:58<19:47:54,  1.59s/it]
Outer training loop:  10%|#         | 5225/50000 [2:15:00<19:14:40,  1.55s/it]
Outer training loop:  10%|#         | 5226/50000 [2:15:01<19:18:36,  1.55s/it]
Outer training loop:  10%|#         | 5227/50000 [2:15:03<19:48:23,  1.59s/it]
Outer training loop:  10%|#         | 5228/50000 [2:15:05<20:00:21,  1.61s/it]
Outer training loop:  10%|#         | 5229/50000 [2:15:06<19:30:37,  1.57s/it]
Outer training loop:  10%|#         | 5230/50000 [2:15:08<19:24:56,  1.56s/it]
Outer training loop:  10%|#         | 5231/50000 [2:15:09<18:55:31,  1.52s/it]
Outer training loop:  10%|#         | 5232/50000 [2:15:11<20:01:17,  1.61s/it]
Outer training loop:  10%|#         | 5233/50000 [2:15:12<19:29:04,  1.57s/it]
Outer training loop:  10%|#         | 5234/50000 [2:15:14<19:31:14,  1.57s/it]
Outer training loop:  10%|#         | 5235/50000 [2:15:15<19:13:10,  1.55s/it]
Outer training loop:  10%|#         | 5236/50000 [2:15:17<19:31:40,  1.57s/it]
Outer training loop:  10%|#         | 5237/50000 [2:15:19<19:04:26,  1.53s/it]
Outer training loop:  10%|#         | 5238/50000 [2:15:20<19:31:19,  1.57s/it]
Outer training loop:  10%|#         | 5239/50000 [2:15:22<19:20:45,  1.56s/it]
Outer training loop:  10%|#         | 5240/50000 [2:15:23<19:31:11,  1.57s/it]
Outer training loop:  10%|#         | 5241/50000 [2:15:25<19:41:57,  1.58s/it]
Outer training loop:  10%|#         | 5242/50000 [2:15:26<19:25:20,  1.56s/it]
Outer training loop:  10%|#         | 5243/50000 [2:15:28<19:34:55,  1.58s/it]
Outer training loop:  10%|#         | 5244/50000 [2:15:30<19:12:59,  1.55s/it]
Outer training loop:  10%|#         | 5245/50000 [2:15:31<19:27:03,  1.56s/it]
Outer training loop:  10%|#         | 5246/50000 [2:15:33<19:16:19,  1.55s/it]
Outer training loop:  10%|#         | 5247/50000 [2:15:34<19:46:32,  1.59s/it]
Outer training loop:  10%|#         | 5248/50000 [2:15:36<19:37:47,  1.58s/it]
Outer training loop:  10%|#         | 5249/50000 [2:15:38<19:55:21,  1.60s/it]
Outer training loop:  10%|#         | 5250/50000 [2:15:39<19:39:23,  1.58s/it]
Outer training loop:  11%|#         | 5251/50000 [2:15:41<19:18:36,  1.55s/it]
Outer training loop:  11%|#         | 5252/50000 [2:15:42<19:18:30,  1.55s/it]
Outer training loop:  11%|#         | 5253/50000 [2:15:44<18:57:34,  1.53s/it]
Outer training loop:  11%|#         | 5254/50000 [2:15:45<18:58:19,  1.53s/it]
Outer training loop:  11%|#         | 5255/50000 [2:15:47<19:14:50,  1.55s/it]
Outer training loop:  11%|#         | 5256/50000 [2:15:48<19:07:42,  1.54s/it]
Outer training loop:  11%|#         | 5257/50000 [2:15:50<19:39:50,  1.58s/it]
Outer training loop:  11%|#         | 5258/50000 [2:15:51<19:25:03,  1.56s/it]
Outer training loop:  11%|#         | 5259/50000 [2:15:53<19:08:37,  1.54s/it]
Outer training loop:  11%|#         | 5260/50000 [2:15:55<19:28:14,  1.57s/it]
Outer training loop:  11%|#         | 5261/50000 [2:15:56<19:40:07,  1.58s/it]
Outer training loop:  11%|#         | 5262/50000 [2:15:58<19:30:39,  1.57s/it]
Outer training loop:  11%|#         | 5263/50000 [2:15:59<19:13:01,  1.55s/it]
Outer training loop:  11%|#         | 5264/50000 [2:16:01<18:57:17,  1.53s/it]
Outer training loop:  11%|#         | 5265/50000 [2:16:02<19:20:30,  1.56s/it]
Outer training loop:  11%|#         | 5266/50000 [2:16:04<19:42:29,  1.59s/it]
Outer training loop:  11%|#         | 5267/50000 [2:16:06<20:00:50,  1.61s/it]
Outer training loop:  11%|#         | 5268/50000 [2:16:07<19:21:16,  1.56s/it]
Outer training loop:  11%|#         | 5269/50000 [2:16:09<19:26:30,  1.56s/it]
Outer training loop:  11%|#         | 5270/50000 [2:16:10<19:30:56,  1.57s/it]
Outer training loop:  11%|#         | 5271/50000 [2:16:12<19:33:52,  1.57s/it]
Outer training loop:  11%|#         | 5272/50000 [2:16:13<19:17:24,  1.55s/it]
Outer training loop:  11%|#         | 5273/50000 [2:16:15<19:25:05,  1.56s/it]
Outer training loop:  11%|#         | 5274/50000 [2:16:16<18:57:55,  1.53s/it]
Outer training loop:  11%|#         | 5275/50000 [2:16:18<19:10:29,  1.54s/it]
Outer training loop:  11%|#         | 5276/50000 [2:16:20<19:20:56,  1.56s/it]
Outer training loop:  11%|#         | 5277/50000 [2:16:21<19:07:51,  1.54s/it]
Outer training loop:  11%|#         | 5278/50000 [2:16:23<18:58:18,  1.53s/it]
Outer training loop:  11%|#         | 5279/50000 [2:16:24<19:10:46,  1.54s/it]
Outer training loop:  11%|#         | 5280/50000 [2:16:26<19:24:16,  1.56s/it]
Outer training loop:  11%|#         | 5281/50000 [2:16:27<19:28:16,  1.57s/it]
Outer training loop:  11%|#         | 5282/50000 [2:16:29<19:44:39,  1.59s/it]
Outer training loop:  11%|#         | 5283/50000 [2:16:31<19:54:51,  1.60s/it]
Outer training loop:  11%|#         | 5284/50000 [2:16:32<19:33:12,  1.57s/it]
Outer training loop:  11%|#         | 5285/50000 [2:16:34<19:51:17,  1.60s/it]
Outer training loop:  11%|#         | 5286/50000 [2:16:35<19:19:16,  1.56s/it]
Outer training loop:  11%|#         | 5287/50000 [2:16:37<19:38:36,  1.58s/it]
Outer training loop:  11%|#         | 5288/50000 [2:16:38<19:34:27,  1.58s/it]
Outer training loop:  11%|#         | 5289/50000 [2:16:40<19:50:58,  1.60s/it]
Outer training loop:  11%|#         | 5290/50000 [2:16:42<19:22:32,  1.56s/it]
Outer training loop:  11%|#         | 5291/50000 [2:16:43<19:33:06,  1.57s/it]
Outer training loop:  11%|#         | 5292/50000 [2:16:45<19:16:04,  1.55s/it]
Outer training loop:  11%|#         | 5293/50000 [2:16:46<19:22:19,  1.56s/it]
Outer training loop:  11%|#         | 5294/50000 [2:16:48<19:42:02,  1.59s/it]
Outer training loop:  11%|#         | 5295/50000 [2:16:49<19:18:52,  1.56s/it]
Outer training loop:  11%|#         | 5296/50000 [2:16:51<19:11:16,  1.55s/it]
Outer training loop:  11%|#         | 5297/50000 [2:16:53<19:34:21,  1.58s/it]
Outer training loop:  11%|#         | 5298/50000 [2:16:54<19:29:08,  1.57s/it]
Outer training loop:  11%|#         | 5299/50000 [2:16:56<19:31:22,  1.57s/it]
Outer training loop:  11%|#         | 5300/50000 [2:16:57<18:59:58,  1.53s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.50s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.50s/it]

Outer training loop:  11%|#         | 5301/50000 [2:17:03<36:19:34,  2.93s/it]
Outer training loop:  11%|#         | 5302/50000 [2:17:05<31:46:54,  2.56s/it]
Outer training loop:  11%|#         | 5303/50000 [2:17:06<27:49:09,  2.24s/it]
Outer training loop:  11%|#         | 5304/50000 [2:17:08<25:10:03,  2.03s/it]
Outer training loop:  11%|#         | 5305/50000 [2:17:10<23:25:59,  1.89s/it]
Outer training loop:  11%|#         | 5306/50000 [2:17:11<22:29:05,  1.81s/it]
Outer training loop:  11%|#         | 5307/50000 [2:17:13<21:10:50,  1.71s/it]
Outer training loop:  11%|#         | 5308/50000 [2:17:14<20:54:50,  1.68s/it]
Outer training loop:  11%|#         | 5309/50000 [2:17:16<20:13:58,  1.63s/it]
Outer training loop:  11%|#         | 5310/50000 [2:17:17<20:02:10,  1.61s/it]
Outer training loop:  11%|#         | 5311/50000 [2:17:19<19:39:54,  1.58s/it]
Outer training loop:  11%|#         | 5312/50000 [2:17:20<19:42:24,  1.59s/it]
Outer training loop:  11%|#         | 5313/50000 [2:17:22<19:34:02,  1.58s/it]
Outer training loop:  11%|#         | 5314/50000 [2:17:24<19:49:14,  1.60s/it]
Outer training loop:  11%|#         | 5315/50000 [2:17:25<19:31:24,  1.57s/it]
Outer training loop:  11%|#         | 5316/50000 [2:17:27<19:07:31,  1.54s/it]
Outer training loop:  11%|#         | 5317/50000 [2:17:28<19:13:15,  1.55s/it]
Outer training loop:  11%|#         | 5318/50000 [2:17:30<19:17:14,  1.55s/it]
Outer training loop:  11%|#         | 5319/50000 [2:17:31<19:19:21,  1.56s/it]
Outer training loop:  11%|#         | 5320/50000 [2:17:33<19:19:25,  1.56s/it]
Outer training loop:  11%|#         | 5321/50000 [2:17:35<19:46:51,  1.59s/it]
Outer training loop:  11%|#         | 5322/50000 [2:17:36<19:54:00,  1.60s/it]
Outer training loop:  11%|#         | 5323/50000 [2:17:38<19:27:04,  1.57s/it]
Outer training loop:  11%|#         | 5324/50000 [2:17:39<19:17:45,  1.55s/it]
Outer training loop:  11%|#         | 5325/50000 [2:17:41<19:32:18,  1.57s/it]
Outer training loop:  11%|#         | 5326/50000 [2:17:42<19:19:04,  1.56s/it]
Outer training loop:  11%|#         | 5327/50000 [2:17:44<19:24:02,  1.56s/it]
Outer training loop:  11%|#         | 5328/50000 [2:17:45<19:06:14,  1.54s/it]
Outer training loop:  11%|#         | 5329/50000 [2:17:47<20:19:02,  1.64s/it]
Outer training loop:  11%|#         | 5330/50000 [2:17:49<19:50:12,  1.60s/it]
Outer training loop:  11%|#         | 5331/50000 [2:17:50<19:42:40,  1.59s/it]
Outer training loop:  11%|#         | 5332/50000 [2:17:52<19:27:02,  1.57s/it]
Outer training loop:  11%|#         | 5333/50000 [2:17:54<19:41:46,  1.59s/it]
Outer training loop:  11%|#         | 5334/50000 [2:17:55<20:01:02,  1.61s/it]
Outer training loop:  11%|#         | 5335/50000 [2:17:57<19:26:07,  1.57s/it]
Outer training loop:  11%|#         | 5336/50000 [2:17:58<19:23:15,  1.56s/it]
Outer training loop:  11%|#         | 5337/50000 [2:18:00<19:31:17,  1.57s/it]
Outer training loop:  11%|#         | 5338/50000 [2:18:01<19:31:05,  1.57s/it]
Outer training loop:  11%|#         | 5339/50000 [2:18:03<20:08:21,  1.62s/it]
Outer training loop:  11%|#         | 5340/50000 [2:18:05<20:29:30,  1.65s/it]
Outer training loop:  11%|#         | 5341/50000 [2:18:06<19:52:36,  1.60s/it]
Outer training loop:  11%|#         | 5342/50000 [2:18:08<19:29:37,  1.57s/it]
Outer training loop:  11%|#         | 5343/50000 [2:18:09<19:35:28,  1.58s/it]
Outer training loop:  11%|#         | 5344/50000 [2:18:11<19:29:58,  1.57s/it]
Outer training loop:  11%|#         | 5345/50000 [2:18:13<19:28:52,  1.57s/it]
Outer training loop:  11%|#         | 5346/50000 [2:18:14<19:11:43,  1.55s/it]
Outer training loop:  11%|#         | 5347/50000 [2:18:16<19:24:32,  1.56s/it]
Outer training loop:  11%|#         | 5348/50000 [2:18:17<19:20:27,  1.56s/it]
Outer training loop:  11%|#         | 5349/50000 [2:18:19<19:43:37,  1.59s/it]
Outer training loop:  11%|#         | 5350/50000 [2:18:20<19:31:00,  1.57s/it]
Outer training loop:  11%|#         | 5351/50000 [2:18:22<19:55:16,  1.61s/it]
Outer training loop:  11%|#         | 5352/50000 [2:18:24<19:30:23,  1.57s/it]
Outer training loop:  11%|#         | 5353/50000 [2:18:25<19:39:30,  1.59s/it]
Outer training loop:  11%|#         | 5354/50000 [2:18:27<19:19:42,  1.56s/it]
Outer training loop:  11%|#         | 5355/50000 [2:18:28<19:18:29,  1.56s/it]
Outer training loop:  11%|#         | 5356/50000 [2:18:30<19:12:44,  1.55s/it]
Outer training loop:  11%|#         | 5357/50000 [2:18:31<19:37:30,  1.58s/it]
Outer training loop:  11%|#         | 5358/50000 [2:18:33<19:23:08,  1.56s/it]
Outer training loop:  11%|#         | 5359/50000 [2:18:35<19:40:02,  1.59s/it]
Outer training loop:  11%|#         | 5360/50000 [2:18:36<19:23:04,  1.56s/it]
Outer training loop:  11%|#         | 5361/50000 [2:18:38<19:39:44,  1.59s/it]
Outer training loop:  11%|#         | 5362/50000 [2:18:39<18:57:13,  1.53s/it]
Outer training loop:  11%|#         | 5363/50000 [2:18:41<19:04:42,  1.54s/it]
Outer training loop:  11%|#         | 5364/50000 [2:18:42<18:55:21,  1.53s/it]
Outer training loop:  11%|#         | 5365/50000 [2:18:44<19:02:46,  1.54s/it]
Outer training loop:  11%|#         | 5366/50000 [2:18:45<18:56:14,  1.53s/it]
Outer training loop:  11%|#         | 5367/50000 [2:18:47<19:21:47,  1.56s/it]
Outer training loop:  11%|#         | 5368/50000 [2:18:48<19:21:24,  1.56s/it]
Outer training loop:  11%|#         | 5369/50000 [2:18:50<19:31:28,  1.57s/it]
Outer training loop:  11%|#         | 5370/50000 [2:18:52<19:25:46,  1.57s/it]
Outer training loop:  11%|#         | 5371/50000 [2:18:53<19:44:34,  1.59s/it]
Outer training loop:  11%|#         | 5372/50000 [2:18:55<19:18:42,  1.56s/it]
Outer training loop:  11%|#         | 5373/50000 [2:18:56<19:28:38,  1.57s/it]
Outer training loop:  11%|#         | 5374/50000 [2:18:58<19:25:55,  1.57s/it]
Outer training loop:  11%|#         | 5375/50000 [2:19:00<19:41:47,  1.59s/it]
Outer training loop:  11%|#         | 5376/50000 [2:19:01<19:14:17,  1.55s/it]
Outer training loop:  11%|#         | 5377/50000 [2:19:03<19:39:09,  1.59s/it]
Outer training loop:  11%|#         | 5378/50000 [2:19:04<19:28:49,  1.57s/it]
Outer training loop:  11%|#         | 5379/50000 [2:19:06<20:14:28,  1.63s/it]
Outer training loop:  11%|#         | 5380/50000 [2:19:07<19:36:39,  1.58s/it]
Outer training loop:  11%|#         | 5381/50000 [2:19:09<19:54:16,  1.61s/it]
Outer training loop:  11%|#         | 5382/50000 [2:19:11<19:31:10,  1.57s/it]
Outer training loop:  11%|#         | 5383/50000 [2:19:12<19:20:56,  1.56s/it]
Outer training loop:  11%|#         | 5384/50000 [2:19:14<19:46:39,  1.60s/it]
Outer training loop:  11%|#         | 5385/50000 [2:19:15<19:43:36,  1.59s/it]
Outer training loop:  11%|#         | 5386/50000 [2:19:17<19:28:46,  1.57s/it]
Outer training loop:  11%|#         | 5387/50000 [2:19:19<19:49:23,  1.60s/it]
Outer training loop:  11%|#         | 5388/50000 [2:19:20<19:37:23,  1.58s/it]
Outer training loop:  11%|#         | 5389/50000 [2:19:22<19:47:58,  1.60s/it]
Outer training loop:  11%|#         | 5390/50000 [2:19:23<19:52:46,  1.60s/it]
Outer training loop:  11%|#         | 5391/50000 [2:19:25<20:12:31,  1.63s/it]
Outer training loop:  11%|#         | 5392/50000 [2:19:27<19:57:29,  1.61s/it]
Outer training loop:  11%|#         | 5393/50000 [2:19:28<19:59:10,  1.61s/it]
Outer training loop:  11%|#         | 5394/50000 [2:19:30<19:39:45,  1.59s/it]
Outer training loop:  11%|#         | 5395/50000 [2:19:31<19:34:14,  1.58s/it]
Outer training loop:  11%|#         | 5396/50000 [2:19:33<19:15:03,  1.55s/it]
Outer training loop:  11%|#         | 5397/50000 [2:19:34<19:38:18,  1.59s/it]
Outer training loop:  11%|#         | 5398/50000 [2:19:36<19:15:39,  1.55s/it]
Outer training loop:  11%|#         | 5399/50000 [2:19:38<19:24:49,  1.57s/it]
Outer training loop:  11%|#         | 5400/50000 [2:19:39<19:03:24,  1.54s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.49s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.52s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.48s/it]

Outer training loop:  11%|#         | 5401/50000 [2:19:45<35:47:48,  2.89s/it]
Outer training loop:  11%|#         | 5402/50000 [2:19:47<30:44:52,  2.48s/it]
Outer training loop:  11%|#         | 5403/50000 [2:19:48<27:15:07,  2.20s/it]
Outer training loop:  11%|#         | 5404/50000 [2:19:50<25:15:45,  2.04s/it]
Outer training loop:  11%|#         | 5405/50000 [2:19:51<23:11:49,  1.87s/it]
Outer training loop:  11%|#         | 5406/50000 [2:19:53<22:12:57,  1.79s/it]
Outer training loop:  11%|#         | 5407/50000 [2:19:54<20:58:21,  1.69s/it]
Outer training loop:  11%|#         | 5408/50000 [2:19:56<20:44:48,  1.67s/it]
Outer training loop:  11%|#         | 5409/50000 [2:19:57<20:05:04,  1.62s/it]
Outer training loop:  11%|#         | 5410/50000 [2:19:59<20:15:04,  1.63s/it]
Outer training loop:  11%|#         | 5411/50000 [2:20:01<19:51:06,  1.60s/it]
Outer training loop:  11%|#         | 5412/50000 [2:20:02<19:58:50,  1.61s/it]
Outer training loop:  11%|#         | 5413/50000 [2:20:04<19:31:11,  1.58s/it]
Outer training loop:  11%|#         | 5414/50000 [2:20:05<19:13:53,  1.55s/it]
Outer training loop:  11%|#         | 5415/50000 [2:20:07<19:17:09,  1.56s/it]
Outer training loop:  11%|#         | 5416/50000 [2:20:09<19:37:51,  1.59s/it]
Outer training loop:  11%|#         | 5417/50000 [2:20:10<19:07:01,  1.54s/it]
Outer training loop:  11%|#         | 5418/50000 [2:20:11<18:48:54,  1.52s/it]
Outer training loop:  11%|#         | 5419/50000 [2:20:13<19:07:48,  1.54s/it]
Outer training loop:  11%|#         | 5420/50000 [2:20:15<19:14:39,  1.55s/it]
Outer training loop:  11%|#         | 5421/50000 [2:20:16<19:04:09,  1.54s/it]
Outer training loop:  11%|#         | 5422/50000 [2:20:18<19:17:47,  1.56s/it]
Outer training loop:  11%|#         | 5423/50000 [2:20:19<18:42:35,  1.51s/it]
Outer training loop:  11%|#         | 5424/50000 [2:20:21<18:33:15,  1.50s/it]
Outer training loop:  11%|#         | 5425/50000 [2:20:22<18:44:15,  1.51s/it]
Outer training loop:  11%|#         | 5426/50000 [2:20:24<19:20:18,  1.56s/it]
Outer training loop:  11%|#         | 5427/50000 [2:20:25<19:42:29,  1.59s/it]
Outer training loop:  11%|#         | 5428/50000 [2:20:27<19:16:35,  1.56s/it]
Outer training loop:  11%|#         | 5429/50000 [2:20:29<19:15:48,  1.56s/it]
Outer training loop:  11%|#         | 5430/50000 [2:20:30<19:13:24,  1.55s/it]
Outer training loop:  11%|#         | 5431/50000 [2:20:32<19:20:23,  1.56s/it]
Outer training loop:  11%|#         | 5432/50000 [2:20:33<18:58:58,  1.53s/it]
Outer training loop:  11%|#         | 5433/50000 [2:20:35<19:15:28,  1.56s/it]
Outer training loop:  11%|#         | 5434/50000 [2:20:36<19:23:28,  1.57s/it]
Outer training loop:  11%|#         | 5435/50000 [2:20:38<19:35:54,  1.58s/it]
Outer training loop:  11%|#         | 5436/50000 [2:20:40<19:35:23,  1.58s/it]
Outer training loop:  11%|#         | 5437/50000 [2:20:41<19:42:57,  1.59s/it]
Outer training loop:  11%|#         | 5438/50000 [2:20:43<19:13:42,  1.55s/it]
Outer training loop:  11%|#         | 5439/50000 [2:20:44<19:15:28,  1.56s/it]
Outer training loop:  11%|#         | 5440/50000 [2:20:46<18:52:20,  1.52s/it]
Outer training loop:  11%|#         | 5441/50000 [2:20:47<19:10:56,  1.55s/it]
Outer training loop:  11%|#         | 5442/50000 [2:20:49<19:11:00,  1.55s/it]
Outer training loop:  11%|#         | 5443/50000 [2:20:50<19:25:25,  1.57s/it]
Outer training loop:  11%|#         | 5444/50000 [2:20:52<19:08:24,  1.55s/it]
Outer training loop:  11%|#         | 5445/50000 [2:20:53<19:03:25,  1.54s/it]
Outer training loop:  11%|#         | 5446/50000 [2:20:55<19:06:50,  1.54s/it]
Outer training loop:  11%|#         | 5447/50000 [2:20:56<19:08:33,  1.55s/it]
Outer training loop:  11%|#         | 5448/50000 [2:20:58<18:57:50,  1.53s/it]
Outer training loop:  11%|#         | 5449/50000 [2:21:00<19:19:16,  1.56s/it]
Outer training loop:  11%|#         | 5450/50000 [2:21:01<19:33:51,  1.58s/it]
Outer training loop:  11%|#         | 5451/50000 [2:21:03<20:01:05,  1.62s/it]
Outer training loop:  11%|#         | 5452/50000 [2:21:04<19:34:13,  1.58s/it]
Outer training loop:  11%|#         | 5453/50000 [2:21:06<19:41:08,  1.59s/it]
Outer training loop:  11%|#         | 5454/50000 [2:21:08<19:13:48,  1.55s/it]
Outer training loop:  11%|#         | 5455/50000 [2:21:09<19:00:47,  1.54s/it]
Outer training loop:  11%|#         | 5456/50000 [2:21:11<19:09:59,  1.55s/it]
Outer training loop:  11%|#         | 5457/50000 [2:21:12<19:04:15,  1.54s/it]
Outer training loop:  11%|#         | 5458/50000 [2:21:14<19:15:33,  1.56s/it]
Outer training loop:  11%|#         | 5459/50000 [2:21:15<19:21:57,  1.57s/it]
Outer training loop:  11%|#         | 5460/50000 [2:21:17<19:05:05,  1.54s/it]
Outer training loop:  11%|#         | 5461/50000 [2:21:18<19:27:02,  1.57s/it]
Outer training loop:  11%|#         | 5462/50000 [2:21:20<19:32:38,  1.58s/it]
Outer training loop:  11%|#         | 5463/50000 [2:21:21<18:58:14,  1.53s/it]
Outer training loop:  11%|#         | 5464/50000 [2:21:23<19:16:13,  1.56s/it]
Outer training loop:  11%|#         | 5465/50000 [2:21:25<19:05:52,  1.54s/it]
Outer training loop:  11%|#         | 5466/50000 [2:21:26<19:24:00,  1.57s/it]
Outer training loop:  11%|#         | 5467/50000 [2:21:28<19:21:08,  1.56s/it]
Outer training loop:  11%|#         | 5468/50000 [2:21:29<19:20:25,  1.56s/it]
Outer training loop:  11%|#         | 5469/50000 [2:21:31<18:51:54,  1.53s/it]
Outer training loop:  11%|#         | 5470/50000 [2:21:32<19:01:40,  1.54s/it]
Outer training loop:  11%|#         | 5471/50000 [2:21:34<19:05:25,  1.54s/it]
Outer training loop:  11%|#         | 5472/50000 [2:21:36<19:27:59,  1.57s/it]
Outer training loop:  11%|#         | 5473/50000 [2:21:37<19:20:52,  1.56s/it]
Outer training loop:  11%|#         | 5474/50000 [2:21:39<18:55:16,  1.53s/it]
Outer training loop:  11%|#         | 5475/50000 [2:21:40<18:48:56,  1.52s/it]
Outer training loop:  11%|#         | 5476/50000 [2:21:42<19:49:00,  1.60s/it]
Outer training loop:  11%|#         | 5477/50000 [2:21:43<19:24:00,  1.57s/it]
Outer training loop:  11%|#         | 5478/50000 [2:21:45<19:18:47,  1.56s/it]
Outer training loop:  11%|#         | 5479/50000 [2:21:46<19:16:46,  1.56s/it]
Outer training loop:  11%|#         | 5480/50000 [2:21:48<19:37:41,  1.59s/it]
Outer training loop:  11%|#         | 5481/50000 [2:21:50<19:56:23,  1.61s/it]
Outer training loop:  11%|#         | 5482/50000 [2:21:51<19:31:47,  1.58s/it]
Outer training loop:  11%|#         | 5483/50000 [2:21:53<19:20:00,  1.56s/it]
Outer training loop:  11%|#         | 5484/50000 [2:21:54<19:26:41,  1.57s/it]
Outer training loop:  11%|#         | 5485/50000 [2:21:56<19:13:52,  1.56s/it]
Outer training loop:  11%|#         | 5486/50000 [2:21:58<19:34:02,  1.58s/it]
Outer training loop:  11%|#         | 5487/50000 [2:21:59<19:41:19,  1.59s/it]
Outer training loop:  11%|#         | 5488/50000 [2:22:01<19:35:01,  1.58s/it]
Outer training loop:  11%|#         | 5489/50000 [2:22:02<19:32:55,  1.58s/it]
Outer training loop:  11%|#         | 5490/50000 [2:22:04<19:21:09,  1.57s/it]
Outer training loop:  11%|#         | 5491/50000 [2:22:05<19:25:17,  1.57s/it]
Outer training loop:  11%|#         | 5492/50000 [2:22:07<19:00:11,  1.54s/it]
Outer training loop:  11%|#         | 5493/50000 [2:22:08<19:21:38,  1.57s/it]
Outer training loop:  11%|#         | 5494/50000 [2:22:10<19:11:53,  1.55s/it]
Outer training loop:  11%|#         | 5495/50000 [2:22:12<19:20:47,  1.56s/it]
Outer training loop:  11%|#         | 5496/50000 [2:22:13<18:54:27,  1.53s/it]
Outer training loop:  11%|#         | 5497/50000 [2:22:15<19:12:00,  1.55s/it]
Outer training loop:  11%|#         | 5498/50000 [2:22:16<19:02:34,  1.54s/it]
Outer training loop:  11%|#         | 5499/50000 [2:22:18<19:26:29,  1.57s/it]
Outer training loop:  11%|#1        | 5500/50000 [2:22:19<19:10:14,  1.55s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.50s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.53s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.51s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:12,  1.62s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:09,  1.60s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:07,  1.60s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:03,  1.56s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:02,  1.57s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:01,  1.57s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:58,  1.56s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:58,  1.58s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:56,  1.58s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:55,  1.58s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:52,  1.57s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:51,  1.56s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:51,  1.59s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:47,  1.55s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:44,  1.53s/it]

Evaluating policy:  19%|#9        | 16/83 [00:24<01:43,  1.54s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:40,  1.52s/it]

Evaluating policy:  22%|##1       | 18/83 [00:27<01:37,  1.49s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:39,  1.55s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:38,  1.57s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:35,  1.55s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:40,  1.64s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:36,  1.60s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:33,  1.58s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:31,  1.58s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:31,  1.61s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:29,  1.60s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:26,  1.57s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:23,  1.54s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:24,  1.60s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:22,  1.60s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:19,  1.56s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:18,  1.56s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:19,  1.62s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:15,  1.58s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:13,  1.56s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:10,  1.53s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:08,  1.53s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:10,  1.61s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:06,  1.56s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:04,  1.54s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:02,  1.53s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:01,  1.53s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<00:59,  1.54s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<01:00,  1.59s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<00:57,  1.56s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:13<00:54,  1.52s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:14<00:53,  1.52s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:51,  1.51s/it]

Evaluating policy:  60%|######    | 50/83 [01:17<00:49,  1.50s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:50,  1.59s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:48,  1.57s/it]

Evaluating policy:  64%|######3   | 53/83 [01:22<00:46,  1.54s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:44,  1.54s/it]

Evaluating policy:  66%|######6   | 55/83 [01:25<00:42,  1.52s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:41,  1.53s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:43,  1.66s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:40,  1.64s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:38,  1.59s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:33<00:35,  1.56s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:34,  1.55s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:36<00:32,  1.53s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:30,  1.54s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:31,  1.64s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:28,  1.60s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:26,  1.57s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:24,  1.56s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:23,  1.56s/it]

Evaluating policy:  83%|########3 | 69/83 [01:47<00:21,  1.55s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:21,  1.65s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:19,  1.60s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:17,  1.63s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:15,  1.58s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:13,  1.55s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.55s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:10,  1.54s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.55s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:08,  1.68s/it]

Evaluating policy:  95%|#########5| 79/83 [02:04<00:06,  1.66s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.57s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.55s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.58s/it]

Evaluating policy: 100%|##########| 83/83 [02:10<00:00,  1.54s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:01,  1.48s/it]

Evaluating policy:   2%|2         | 2/83 [00:02<01:58,  1.47s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:10,  1.63s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:07,  1.61s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:05,  1.60s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:01,  1.57s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<01:59,  1.57s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:58,  1.59s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:56,  1.58s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:55,  1.58s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:51,  1.55s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:48,  1.53s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:49,  1.57s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:46,  1.54s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:44,  1.53s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:46,  1.59s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:44,  1.58s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:40,  1.55s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:40,  1.57s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:37,  1.55s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:34,  1.53s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:36,  1.58s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:34,  1.57s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:31,  1.55s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:29,  1.54s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:31,  1.60s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:27,  1.57s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:25,  1.56s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:26,  1.60s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:23,  1.57s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:20,  1.55s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:19,  1.56s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:16,  1.52s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:17,  1.58s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:14,  1.55s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:12,  1.54s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:11,  1.55s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:09,  1.55s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:13,  1.67s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:09,  1.61s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:06,  1.59s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:04,  1.58s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:02,  1.56s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:09<01:04,  1.65s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<01:01,  1.61s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<00:58,  1.59s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:14<00:57,  1.59s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:54,  1.55s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:17<00:55,  1.63s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:52,  1.59s/it]

Evaluating policy:  61%|######1   | 51/83 [01:20<00:50,  1.58s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:48,  1.55s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:46,  1.54s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:43,  1.51s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:45,  1.61s/it]

Evaluating policy:  67%|######7   | 56/83 [01:28<00:42,  1.58s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:40,  1.58s/it]

Evaluating policy:  70%|######9   | 58/83 [01:31<00:38,  1.55s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:36,  1.53s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:34<00:35,  1.52s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:33,  1.53s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:37<00:34,  1.63s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:39<00:31,  1.58s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:29,  1.54s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:42<00:27,  1.53s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:26,  1.55s/it]

Evaluating policy:  81%|########  | 67/83 [01:45<00:24,  1.52s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:22,  1.50s/it]

Evaluating policy:  83%|########3 | 69/83 [01:48<00:22,  1.62s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:20,  1.58s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:18,  1.56s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:16,  1.53s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:15,  1.52s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:13,  1.51s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.51s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:10,  1.50s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.63s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.57s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.57s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.55s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.51s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.50s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.51s/it]

Outer training loop:  11%|#1        | 5501/50000 [2:26:45<1000:27:24, 80.94s/it]
Outer training loop:  11%|#1        | 5502/50000 [2:26:47<705:53:38, 57.11s/it] 
Outer training loop:  11%|#1        | 5503/50000 [2:26:49<499:45:10, 40.43s/it]
Outer training loop:  11%|#1        | 5504/50000 [2:26:51<357:14:15, 28.90s/it]
Outer training loop:  11%|#1        | 5505/50000 [2:26:52<255:39:58, 20.69s/it]
Outer training loop:  11%|#1        | 5506/50000 [2:26:54<184:51:22, 14.96s/it]
Outer training loop:  11%|#1        | 5507/50000 [2:26:55<135:37:50, 10.97s/it]
Outer training loop:  11%|#1        | 5508/50000 [2:26:57<100:56:33,  8.17s/it]
Outer training loop:  11%|#1        | 5509/50000 [2:26:58<76:06:45,  6.16s/it] 
Outer training loop:  11%|#1        | 5510/50000 [2:27:00<59:27:26,  4.81s/it]
Outer training loop:  11%|#1        | 5511/50000 [2:27:01<46:58:33,  3.80s/it]
Outer training loop:  11%|#1        | 5512/50000 [2:27:03<38:53:45,  3.15s/it]
Outer training loop:  11%|#1        | 5513/50000 [2:27:05<33:17:42,  2.69s/it]
Outer training loop:  11%|#1        | 5514/50000 [2:27:06<29:22:39,  2.38s/it]
Outer training loop:  11%|#1        | 5515/50000 [2:27:08<26:08:22,  2.12s/it]
Outer training loop:  11%|#1        | 5516/50000 [2:27:09<24:15:13,  1.96s/it]
Outer training loop:  11%|#1        | 5517/50000 [2:27:11<22:38:22,  1.83s/it]
Outer training loop:  11%|#1        | 5518/50000 [2:27:13<21:49:55,  1.77s/it]
Outer training loop:  11%|#1        | 5519/50000 [2:27:14<20:57:04,  1.70s/it]
Outer training loop:  11%|#1        | 5520/50000 [2:27:16<20:33:19,  1.66s/it]
Outer training loop:  11%|#1        | 5521/50000 [2:27:17<20:15:41,  1.64s/it]
Outer training loop:  11%|#1        | 5522/50000 [2:27:19<20:09:07,  1.63s/it]
Outer training loop:  11%|#1        | 5523/50000 [2:27:20<19:47:48,  1.60s/it]
Outer training loop:  11%|#1        | 5524/50000 [2:27:22<19:48:23,  1.60s/it]
Outer training loop:  11%|#1        | 5525/50000 [2:27:24<19:17:07,  1.56s/it]
Outer training loop:  11%|#1        | 5526/50000 [2:27:25<19:30:33,  1.58s/it]
Outer training loop:  11%|#1        | 5527/50000 [2:27:27<19:17:47,  1.56s/it]
Outer training loop:  11%|#1        | 5528/50000 [2:27:28<19:10:15,  1.55s/it]
Outer training loop:  11%|#1        | 5529/50000 [2:27:30<19:21:43,  1.57s/it]
Outer training loop:  11%|#1        | 5530/50000 [2:27:31<19:33:19,  1.58s/it]
Outer training loop:  11%|#1        | 5531/50000 [2:27:33<19:37:28,  1.59s/it]
Outer training loop:  11%|#1        | 5532/50000 [2:27:35<19:59:26,  1.62s/it]
Outer training loop:  11%|#1        | 5533/50000 [2:27:36<19:44:49,  1.60s/it]
Outer training loop:  11%|#1        | 5534/50000 [2:27:38<19:35:41,  1.59s/it]
Outer training loop:  11%|#1        | 5535/50000 [2:27:39<19:33:24,  1.58s/it]
Outer training loop:  11%|#1        | 5536/50000 [2:27:41<19:48:31,  1.60s/it]
Outer training loop:  11%|#1        | 5537/50000 [2:27:42<19:02:09,  1.54s/it]
Outer training loop:  11%|#1        | 5538/50000 [2:27:44<19:31:40,  1.58s/it]
Outer training loop:  11%|#1        | 5539/50000 [2:27:46<19:17:56,  1.56s/it]
Outer training loop:  11%|#1        | 5540/50000 [2:27:47<19:18:37,  1.56s/it]
Outer training loop:  11%|#1        | 5541/50000 [2:27:49<19:07:16,  1.55s/it]
Outer training loop:  11%|#1        | 5542/50000 [2:27:50<19:28:35,  1.58s/it]
Outer training loop:  11%|#1        | 5543/50000 [2:27:52<19:11:50,  1.55s/it]
Outer training loop:  11%|#1        | 5544/50000 [2:27:53<19:02:05,  1.54s/it]
Outer training loop:  11%|#1        | 5545/50000 [2:27:55<19:30:56,  1.58s/it]
Outer training loop:  11%|#1        | 5546/50000 [2:27:57<19:33:05,  1.58s/it]
Outer training loop:  11%|#1        | 5547/50000 [2:27:58<19:09:22,  1.55s/it]
Outer training loop:  11%|#1        | 5548/50000 [2:28:00<18:50:58,  1.53s/it]
Outer training loop:  11%|#1        | 5549/50000 [2:28:01<19:08:21,  1.55s/it]
Outer training loop:  11%|#1        | 5550/50000 [2:28:03<19:02:59,  1.54s/it]
Outer training loop:  11%|#1        | 5551/50000 [2:28:04<19:34:16,  1.59s/it]
Outer training loop:  11%|#1        | 5552/50000 [2:28:06<19:26:29,  1.57s/it]
Outer training loop:  11%|#1        | 5553/50000 [2:28:08<19:37:13,  1.59s/it]
Outer training loop:  11%|#1        | 5554/50000 [2:28:09<19:18:06,  1.56s/it]
Outer training loop:  11%|#1        | 5555/50000 [2:28:11<19:19:38,  1.57s/it]
Outer training loop:  11%|#1        | 5556/50000 [2:28:12<19:06:16,  1.55s/it]
Outer training loop:  11%|#1        | 5557/50000 [2:28:14<19:55:31,  1.61s/it]
Outer training loop:  11%|#1        | 5558/50000 [2:28:16<19:45:06,  1.60s/it]
Outer training loop:  11%|#1        | 5559/50000 [2:28:17<19:42:21,  1.60s/it]
Outer training loop:  11%|#1        | 5560/50000 [2:28:19<19:22:38,  1.57s/it]
Outer training loop:  11%|#1        | 5561/50000 [2:28:20<19:17:59,  1.56s/it]
Outer training loop:  11%|#1        | 5562/50000 [2:28:22<19:09:22,  1.55s/it]
Outer training loop:  11%|#1        | 5563/50000 [2:28:23<19:35:20,  1.59s/it]
Outer training loop:  11%|#1        | 5564/50000 [2:28:25<19:35:03,  1.59s/it]
Outer training loop:  11%|#1        | 5565/50000 [2:28:27<19:42:34,  1.60s/it]
Outer training loop:  11%|#1        | 5566/50000 [2:28:28<19:00:14,  1.54s/it]
Outer training loop:  11%|#1        | 5567/50000 [2:28:30<19:23:19,  1.57s/it]
Outer training loop:  11%|#1        | 5568/50000 [2:28:31<19:09:55,  1.55s/it]
Outer training loop:  11%|#1        | 5569/50000 [2:28:33<19:33:32,  1.58s/it]
Outer training loop:  11%|#1        | 5570/50000 [2:28:34<19:26:25,  1.58s/it]
Outer training loop:  11%|#1        | 5571/50000 [2:28:36<19:36:48,  1.59s/it]
Outer training loop:  11%|#1        | 5572/50000 [2:28:38<19:30:33,  1.58s/it]
Outer training loop:  11%|#1        | 5573/50000 [2:28:39<19:39:12,  1.59s/it]
Outer training loop:  11%|#1        | 5574/50000 [2:28:41<19:28:52,  1.58s/it]
Outer training loop:  11%|#1        | 5575/50000 [2:28:42<19:39:15,  1.59s/it]
Outer training loop:  11%|#1        | 5576/50000 [2:28:44<19:03:12,  1.54s/it]
Outer training loop:  11%|#1        | 5577/50000 [2:28:45<19:23:45,  1.57s/it]
Outer training loop:  11%|#1        | 5578/50000 [2:28:47<19:13:00,  1.56s/it]
Outer training loop:  11%|#1        | 5579/50000 [2:28:48<19:20:02,  1.57s/it]
Outer training loop:  11%|#1        | 5580/50000 [2:28:50<19:19:01,  1.57s/it]
Outer training loop:  11%|#1        | 5581/50000 [2:28:52<19:17:41,  1.56s/it]
Outer training loop:  11%|#1        | 5582/50000 [2:28:53<18:48:14,  1.52s/it]
Outer training loop:  11%|#1        | 5583/50000 [2:28:55<19:05:02,  1.55s/it]
Outer training loop:  11%|#1        | 5584/50000 [2:28:56<18:48:18,  1.52s/it]
Outer training loop:  11%|#1        | 5585/50000 [2:28:58<19:16:30,  1.56s/it]
Outer training loop:  11%|#1        | 5586/50000 [2:28:59<18:50:53,  1.53s/it]
Outer training loop:  11%|#1        | 5587/50000 [2:29:01<18:58:08,  1.54s/it]
Outer training loop:  11%|#1        | 5588/50000 [2:29:02<19:01:50,  1.54s/it]
Outer training loop:  11%|#1        | 5589/50000 [2:29:04<18:52:58,  1.53s/it]
Outer training loop:  11%|#1        | 5590/50000 [2:29:05<19:06:26,  1.55s/it]
Outer training loop:  11%|#1        | 5591/50000 [2:29:07<18:47:03,  1.52s/it]
Outer training loop:  11%|#1        | 5592/50000 [2:29:08<18:47:17,  1.52s/it]
Outer training loop:  11%|#1        | 5593/50000 [2:29:10<19:09:26,  1.55s/it]
Outer training loop:  11%|#1        | 5594/50000 [2:29:12<19:00:37,  1.54s/it]
Outer training loop:  11%|#1        | 5595/50000 [2:29:13<18:49:52,  1.53s/it]
Outer training loop:  11%|#1        | 5596/50000 [2:29:15<18:46:57,  1.52s/it]
Outer training loop:  11%|#1        | 5597/50000 [2:29:16<18:39:50,  1.51s/it]
Outer training loop:  11%|#1        | 5598/50000 [2:29:18<18:42:25,  1.52s/it]
Outer training loop:  11%|#1        | 5599/50000 [2:29:19<18:52:14,  1.53s/it]
Outer training loop:  11%|#1        | 5600/50000 [2:29:21<18:48:17,  1.52s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.47s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.53s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.50s/it]

Outer training loop:  11%|#1        | 5601/50000 [2:29:27<35:59:15,  2.92s/it]
Outer training loop:  11%|#1        | 5602/50000 [2:29:28<31:16:33,  2.54s/it]
Outer training loop:  11%|#1        | 5603/50000 [2:29:30<27:27:59,  2.23s/it]
Outer training loop:  11%|#1        | 5604/50000 [2:29:32<25:42:32,  2.08s/it]
Outer training loop:  11%|#1        | 5605/50000 [2:29:33<23:52:36,  1.94s/it]
Outer training loop:  11%|#1        | 5606/50000 [2:29:35<22:18:46,  1.81s/it]
Outer training loop:  11%|#1        | 5607/50000 [2:29:36<21:22:28,  1.73s/it]
Outer training loop:  11%|#1        | 5608/50000 [2:29:38<20:25:19,  1.66s/it]
Outer training loop:  11%|#1        | 5609/50000 [2:29:39<20:17:15,  1.65s/it]
Outer training loop:  11%|#1        | 5610/50000 [2:29:41<20:11:26,  1.64s/it]
Outer training loop:  11%|#1        | 5611/50000 [2:29:43<20:00:57,  1.62s/it]
Outer training loop:  11%|#1        | 5612/50000 [2:29:44<19:36:47,  1.59s/it]
Outer training loop:  11%|#1        | 5613/50000 [2:29:46<19:50:34,  1.61s/it]
Outer training loop:  11%|#1        | 5614/50000 [2:29:47<19:53:14,  1.61s/it]
Outer training loop:  11%|#1        | 5615/50000 [2:29:49<19:54:07,  1.61s/it]
Outer training loop:  11%|#1        | 5616/50000 [2:29:51<19:28:49,  1.58s/it]
Outer training loop:  11%|#1        | 5617/50000 [2:29:52<19:30:27,  1.58s/it]
Outer training loop:  11%|#1        | 5618/50000 [2:29:54<19:20:49,  1.57s/it]
Outer training loop:  11%|#1        | 5619/50000 [2:29:55<19:03:17,  1.55s/it]
Outer training loop:  11%|#1        | 5620/50000 [2:29:57<19:25:49,  1.58s/it]
Outer training loop:  11%|#1        | 5621/50000 [2:29:58<19:06:46,  1.55s/it]
Outer training loop:  11%|#1        | 5622/50000 [2:30:00<19:19:17,  1.57s/it]
Outer training loop:  11%|#1        | 5623/50000 [2:30:01<19:00:40,  1.54s/it]
Outer training loop:  11%|#1        | 5624/50000 [2:30:03<18:47:52,  1.52s/it]
Outer training loop:  11%|#1        | 5625/50000 [2:30:04<19:00:17,  1.54s/it]
Outer training loop:  11%|#1        | 5626/50000 [2:30:06<19:42:44,  1.60s/it]
Outer training loop:  11%|#1        | 5627/50000 [2:30:08<19:10:15,  1.56s/it]
Outer training loop:  11%|#1        | 5628/50000 [2:30:09<19:02:34,  1.54s/it]
Outer training loop:  11%|#1        | 5629/50000 [2:30:11<19:16:53,  1.56s/it]
Outer training loop:  11%|#1        | 5630/50000 [2:30:12<19:08:19,  1.55s/it]
Outer training loop:  11%|#1        | 5631/50000 [2:30:14<19:15:52,  1.56s/it]
Outer training loop:  11%|#1        | 5632/50000 [2:30:16<19:22:38,  1.57s/it]
Outer training loop:  11%|#1        | 5633/50000 [2:30:17<19:20:48,  1.57s/it]
Outer training loop:  11%|#1        | 5634/50000 [2:30:19<19:36:46,  1.59s/it]
Outer training loop:  11%|#1        | 5635/50000 [2:30:20<19:15:42,  1.56s/it]
Outer training loop:  11%|#1        | 5636/50000 [2:30:22<18:46:53,  1.52s/it]
Outer training loop:  11%|#1        | 5637/50000 [2:30:23<19:02:18,  1.54s/it]
Outer training loop:  11%|#1        | 5638/50000 [2:30:25<19:17:43,  1.57s/it]
Outer training loop:  11%|#1        | 5639/50000 [2:30:26<19:02:11,  1.54s/it]
Outer training loop:  11%|#1        | 5640/50000 [2:30:28<18:47:16,  1.52s/it]
Outer training loop:  11%|#1        | 5641/50000 [2:30:29<18:52:02,  1.53s/it]
Outer training loop:  11%|#1        | 5642/50000 [2:30:31<18:31:31,  1.50s/it]
Outer training loop:  11%|#1        | 5643/50000 [2:30:32<18:57:42,  1.54s/it]
Outer training loop:  11%|#1        | 5644/50000 [2:30:34<18:56:25,  1.54s/it]
Outer training loop:  11%|#1        | 5645/50000 [2:30:36<19:21:29,  1.57s/it]
Outer training loop:  11%|#1        | 5646/50000 [2:30:37<19:00:27,  1.54s/it]
Outer training loop:  11%|#1        | 5647/50000 [2:30:39<18:53:58,  1.53s/it]
Outer training loop:  11%|#1        | 5648/50000 [2:30:40<19:06:14,  1.55s/it]
Outer training loop:  11%|#1        | 5649/50000 [2:30:42<19:23:58,  1.57s/it]
Outer training loop:  11%|#1        | 5650/50000 [2:30:43<18:56:19,  1.54s/it]
Outer training loop:  11%|#1        | 5651/50000 [2:30:45<19:18:21,  1.57s/it]
Outer training loop:  11%|#1        | 5652/50000 [2:30:46<19:03:40,  1.55s/it]
Outer training loop:  11%|#1        | 5653/50000 [2:30:48<19:10:32,  1.56s/it]
Outer training loop:  11%|#1        | 5654/50000 [2:30:50<19:39:27,  1.60s/it]
Outer training loop:  11%|#1        | 5655/50000 [2:30:51<19:40:31,  1.60s/it]
Outer training loop:  11%|#1        | 5656/50000 [2:30:53<19:26:05,  1.58s/it]
Outer training loop:  11%|#1        | 5657/50000 [2:30:54<19:40:09,  1.60s/it]
Outer training loop:  11%|#1        | 5658/50000 [2:30:56<19:14:11,  1.56s/it]
Outer training loop:  11%|#1        | 5659/50000 [2:30:57<19:06:57,  1.55s/it]
Outer training loop:  11%|#1        | 5660/50000 [2:30:59<19:01:29,  1.54s/it]
Outer training loop:  11%|#1        | 5661/50000 [2:31:01<19:01:30,  1.54s/it]
Outer training loop:  11%|#1        | 5662/50000 [2:31:02<19:00:35,  1.54s/it]
Outer training loop:  11%|#1        | 5663/50000 [2:31:04<19:10:52,  1.56s/it]
Outer training loop:  11%|#1        | 5664/50000 [2:31:05<18:53:08,  1.53s/it]
Outer training loop:  11%|#1        | 5665/50000 [2:31:07<19:13:13,  1.56s/it]
Outer training loop:  11%|#1        | 5666/50000 [2:31:08<18:55:12,  1.54s/it]
Outer training loop:  11%|#1        | 5667/50000 [2:31:10<19:10:25,  1.56s/it]
Outer training loop:  11%|#1        | 5668/50000 [2:31:11<19:23:26,  1.57s/it]
Outer training loop:  11%|#1        | 5669/50000 [2:31:13<19:12:01,  1.56s/it]
Outer training loop:  11%|#1        | 5670/50000 [2:31:14<18:55:44,  1.54s/it]
Outer training loop:  11%|#1        | 5671/50000 [2:31:16<18:31:28,  1.50s/it]
Outer training loop:  11%|#1        | 5672/50000 [2:31:17<18:46:21,  1.52s/it]
Outer training loop:  11%|#1        | 5673/50000 [2:31:19<18:31:21,  1.50s/it]
Outer training loop:  11%|#1        | 5674/50000 [2:31:21<19:03:13,  1.55s/it]
Outer training loop:  11%|#1        | 5675/50000 [2:31:22<19:01:06,  1.54s/it]
Outer training loop:  11%|#1        | 5676/50000 [2:31:24<19:14:02,  1.56s/it]
Outer training loop:  11%|#1        | 5677/50000 [2:31:25<18:48:58,  1.53s/it]
Outer training loop:  11%|#1        | 5678/50000 [2:31:27<19:45:25,  1.60s/it]
Outer training loop:  11%|#1        | 5679/50000 [2:31:29<19:30:54,  1.59s/it]
Outer training loop:  11%|#1        | 5680/50000 [2:31:30<19:38:58,  1.60s/it]
Outer training loop:  11%|#1        | 5681/50000 [2:31:32<19:24:41,  1.58s/it]
Outer training loop:  11%|#1        | 5682/50000 [2:31:33<19:38:15,  1.60s/it]
Outer training loop:  11%|#1        | 5683/50000 [2:31:35<19:36:55,  1.59s/it]
Outer training loop:  11%|#1        | 5684/50000 [2:31:36<19:39:07,  1.60s/it]
Outer training loop:  11%|#1        | 5685/50000 [2:31:38<19:18:44,  1.57s/it]
Outer training loop:  11%|#1        | 5686/50000 [2:31:40<19:29:31,  1.58s/it]
Outer training loop:  11%|#1        | 5687/50000 [2:31:41<19:06:32,  1.55s/it]
Outer training loop:  11%|#1        | 5688/50000 [2:31:43<19:09:10,  1.56s/it]
Outer training loop:  11%|#1        | 5689/50000 [2:31:44<19:12:14,  1.56s/it]
Outer training loop:  11%|#1        | 5690/50000 [2:31:46<19:28:22,  1.58s/it]
Outer training loop:  11%|#1        | 5691/50000 [2:31:47<19:30:19,  1.58s/it]
Outer training loop:  11%|#1        | 5692/50000 [2:31:49<19:31:42,  1.59s/it]
Outer training loop:  11%|#1        | 5693/50000 [2:31:51<19:22:07,  1.57s/it]
Outer training loop:  11%|#1        | 5694/50000 [2:31:52<19:19:44,  1.57s/it]
Outer training loop:  11%|#1        | 5695/50000 [2:31:54<19:20:28,  1.57s/it]
Outer training loop:  11%|#1        | 5696/50000 [2:31:55<19:35:59,  1.59s/it]
Outer training loop:  11%|#1        | 5697/50000 [2:31:57<19:27:54,  1.58s/it]
Outer training loop:  11%|#1        | 5698/50000 [2:31:58<19:26:45,  1.58s/it]
Outer training loop:  11%|#1        | 5699/50000 [2:32:00<19:12:50,  1.56s/it]
Outer training loop:  11%|#1        | 5700/50000 [2:32:02<19:19:52,  1.57s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.55s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.52s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.58s/it]

Outer training loop:  11%|#1        | 5701/50000 [2:32:08<36:46:55,  2.99s/it]
Outer training loop:  11%|#1        | 5702/50000 [2:32:09<31:30:33,  2.56s/it]
Outer training loop:  11%|#1        | 5703/50000 [2:32:11<27:55:13,  2.27s/it]
Outer training loop:  11%|#1        | 5704/50000 [2:32:13<25:00:00,  2.03s/it]
Outer training loop:  11%|#1        | 5705/50000 [2:32:14<23:23:31,  1.90s/it]
Outer training loop:  11%|#1        | 5706/50000 [2:32:16<21:46:24,  1.77s/it]
Outer training loop:  11%|#1        | 5707/50000 [2:32:17<20:56:50,  1.70s/it]
Outer training loop:  11%|#1        | 5708/50000 [2:32:19<20:44:28,  1.69s/it]
Outer training loop:  11%|#1        | 5709/50000 [2:32:20<20:16:37,  1.65s/it]
Outer training loop:  11%|#1        | 5710/50000 [2:32:22<20:05:29,  1.63s/it]
Outer training loop:  11%|#1        | 5711/50000 [2:32:23<19:43:41,  1.60s/it]
Outer training loop:  11%|#1        | 5712/50000 [2:32:25<19:30:55,  1.59s/it]
Outer training loop:  11%|#1        | 5713/50000 [2:32:27<19:49:01,  1.61s/it]
Outer training loop:  11%|#1        | 5714/50000 [2:32:28<19:32:19,  1.59s/it]
Outer training loop:  11%|#1        | 5715/50000 [2:32:30<19:24:07,  1.58s/it]
Outer training loop:  11%|#1        | 5716/50000 [2:32:31<19:21:04,  1.57s/it]
Outer training loop:  11%|#1        | 5717/50000 [2:32:33<19:10:33,  1.56s/it]
Outer training loop:  11%|#1        | 5718/50000 [2:32:34<19:23:47,  1.58s/it]
Outer training loop:  11%|#1        | 5719/50000 [2:32:36<19:58:28,  1.62s/it]
Outer training loop:  11%|#1        | 5720/50000 [2:32:38<19:37:30,  1.60s/it]
Outer training loop:  11%|#1        | 5721/50000 [2:32:39<19:11:04,  1.56s/it]
Outer training loop:  11%|#1        | 5722/50000 [2:32:41<19:28:55,  1.58s/it]
Outer training loop:  11%|#1        | 5723/50000 [2:32:43<19:43:08,  1.60s/it]
Outer training loop:  11%|#1        | 5724/50000 [2:32:44<19:49:11,  1.61s/it]
Outer training loop:  11%|#1        | 5725/50000 [2:32:46<19:36:28,  1.59s/it]
Outer training loop:  11%|#1        | 5726/50000 [2:32:47<19:50:50,  1.61s/it]
Outer training loop:  11%|#1        | 5727/50000 [2:32:49<19:14:46,  1.56s/it]
Outer training loop:  11%|#1        | 5728/50000 [2:32:50<19:07:53,  1.56s/it]
Outer training loop:  11%|#1        | 5729/50000 [2:32:52<18:51:04,  1.53s/it]
Outer training loop:  11%|#1        | 5730/50000 [2:32:53<19:05:54,  1.55s/it]
Outer training loop:  11%|#1        | 5731/50000 [2:32:55<19:40:35,  1.60s/it]
Outer training loop:  11%|#1        | 5732/50000 [2:32:57<19:25:28,  1.58s/it]
Outer training loop:  11%|#1        | 5733/50000 [2:32:58<19:46:29,  1.61s/it]
Outer training loop:  11%|#1        | 5734/50000 [2:33:00<19:26:35,  1.58s/it]
Outer training loop:  11%|#1        | 5735/50000 [2:33:01<19:00:41,  1.55s/it]
Outer training loop:  11%|#1        | 5736/50000 [2:33:03<19:13:08,  1.56s/it]
Outer training loop:  11%|#1        | 5737/50000 [2:33:04<18:51:04,  1.53s/it]
Outer training loop:  11%|#1        | 5738/50000 [2:33:06<19:18:41,  1.57s/it]
Outer training loop:  11%|#1        | 5739/50000 [2:33:08<19:29:29,  1.59s/it]
Outer training loop:  11%|#1        | 5740/50000 [2:33:09<19:29:20,  1.59s/it]
Outer training loop:  11%|#1        | 5741/50000 [2:33:11<19:11:47,  1.56s/it]
Outer training loop:  11%|#1        | 5742/50000 [2:33:12<19:07:39,  1.56s/it]
Outer training loop:  11%|#1        | 5743/50000 [2:33:14<19:05:48,  1.55s/it]
Outer training loop:  11%|#1        | 5744/50000 [2:33:15<19:20:39,  1.57s/it]
Outer training loop:  11%|#1        | 5745/50000 [2:33:17<19:18:10,  1.57s/it]
Outer training loop:  11%|#1        | 5746/50000 [2:33:19<19:34:02,  1.59s/it]
Outer training loop:  11%|#1        | 5747/50000 [2:33:20<19:30:22,  1.59s/it]
Outer training loop:  11%|#1        | 5748/50000 [2:33:22<18:55:23,  1.54s/it]
Outer training loop:  11%|#1        | 5749/50000 [2:33:23<19:04:04,  1.55s/it]
Outer training loop:  12%|#1        | 5750/50000 [2:33:25<19:13:29,  1.56s/it]
Outer training loop:  12%|#1        | 5751/50000 [2:33:27<20:05:11,  1.63s/it]
Outer training loop:  12%|#1        | 5752/50000 [2:33:28<19:32:53,  1.59s/it]
Outer training loop:  12%|#1        | 5753/50000 [2:33:30<19:24:06,  1.58s/it]
Outer training loop:  12%|#1        | 5754/50000 [2:33:31<19:39:54,  1.60s/it]
Outer training loop:  12%|#1        | 5755/50000 [2:33:33<19:59:17,  1.63s/it]
Outer training loop:  12%|#1        | 5756/50000 [2:33:35<19:42:59,  1.60s/it]
Outer training loop:  12%|#1        | 5757/50000 [2:33:36<19:29:27,  1.59s/it]
Outer training loop:  12%|#1        | 5758/50000 [2:33:38<19:53:03,  1.62s/it]
Outer training loop:  12%|#1        | 5759/50000 [2:33:39<19:55:08,  1.62s/it]
Outer training loop:  12%|#1        | 5760/50000 [2:33:41<19:32:08,  1.59s/it]
Outer training loop:  12%|#1        | 5761/50000 [2:33:43<19:46:48,  1.61s/it]
Outer training loop:  12%|#1        | 5762/50000 [2:33:44<19:19:12,  1.57s/it]
Outer training loop:  12%|#1        | 5763/50000 [2:33:46<19:33:07,  1.59s/it]
Outer training loop:  12%|#1        | 5764/50000 [2:33:47<19:17:16,  1.57s/it]
Outer training loop:  12%|#1        | 5765/50000 [2:33:49<19:48:30,  1.61s/it]
Outer training loop:  12%|#1        | 5766/50000 [2:33:51<19:38:34,  1.60s/it]
Outer training loop:  12%|#1        | 5767/50000 [2:33:52<19:34:48,  1.59s/it]
Outer training loop:  12%|#1        | 5768/50000 [2:33:54<19:18:52,  1.57s/it]
Outer training loop:  12%|#1        | 5769/50000 [2:33:55<19:06:32,  1.56s/it]
Outer training loop:  12%|#1        | 5770/50000 [2:33:57<19:09:03,  1.56s/it]
Outer training loop:  12%|#1        | 5771/50000 [2:33:58<19:31:18,  1.59s/it]
Outer training loop:  12%|#1        | 5772/50000 [2:34:00<19:12:13,  1.56s/it]
Outer training loop:  12%|#1        | 5773/50000 [2:34:01<18:54:12,  1.54s/it]
Outer training loop:  12%|#1        | 5774/50000 [2:34:03<19:15:50,  1.57s/it]
Outer training loop:  12%|#1        | 5775/50000 [2:34:05<19:13:03,  1.56s/it]
Outer training loop:  12%|#1        | 5776/50000 [2:34:06<19:25:27,  1.58s/it]
Outer training loop:  12%|#1        | 5777/50000 [2:34:08<19:03:04,  1.55s/it]
Outer training loop:  12%|#1        | 5778/50000 [2:34:09<19:00:27,  1.55s/it]
Outer training loop:  12%|#1        | 5779/50000 [2:34:11<18:54:46,  1.54s/it]
Outer training loop:  12%|#1        | 5780/50000 [2:34:12<19:08:11,  1.56s/it]
Outer training loop:  12%|#1        | 5781/50000 [2:34:14<19:15:41,  1.57s/it]
Outer training loop:  12%|#1        | 5782/50000 [2:34:15<19:11:57,  1.56s/it]
Outer training loop:  12%|#1        | 5783/50000 [2:34:17<18:56:21,  1.54s/it]
Outer training loop:  12%|#1        | 5784/50000 [2:34:19<19:32:27,  1.59s/it]
Outer training loop:  12%|#1        | 5785/50000 [2:34:20<19:26:59,  1.58s/it]
Outer training loop:  12%|#1        | 5786/50000 [2:34:22<19:39:29,  1.60s/it]
Outer training loop:  12%|#1        | 5787/50000 [2:34:23<19:24:22,  1.58s/it]
Outer training loop:  12%|#1        | 5788/50000 [2:34:25<19:41:16,  1.60s/it]
Outer training loop:  12%|#1        | 5789/50000 [2:34:27<19:10:52,  1.56s/it]
Outer training loop:  12%|#1        | 5790/50000 [2:34:28<19:07:12,  1.56s/it]
Outer training loop:  12%|#1        | 5791/50000 [2:34:30<19:19:43,  1.57s/it]
Outer training loop:  12%|#1        | 5792/50000 [2:34:31<19:33:05,  1.59s/it]
Outer training loop:  12%|#1        | 5793/50000 [2:34:33<19:22:16,  1.58s/it]
Outer training loop:  12%|#1        | 5794/50000 [2:34:34<19:36:14,  1.60s/it]
Outer training loop:  12%|#1        | 5795/50000 [2:34:36<19:10:34,  1.56s/it]
Outer training loop:  12%|#1        | 5796/50000 [2:34:38<19:02:27,  1.55s/it]
Outer training loop:  12%|#1        | 5797/50000 [2:34:39<18:55:49,  1.54s/it]
Outer training loop:  12%|#1        | 5798/50000 [2:34:41<18:57:37,  1.54s/it]
Outer training loop:  12%|#1        | 5799/50000 [2:34:42<18:58:33,  1.55s/it]
Outer training loop:  12%|#1        | 5800/50000 [2:34:44<19:00:53,  1.55s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.45s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.53s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.50s/it]

Outer training loop:  12%|#1        | 5801/50000 [2:34:50<36:38:27,  2.98s/it]
Outer training loop:  12%|#1        | 5802/50000 [2:34:52<31:15:02,  2.55s/it]
Outer training loop:  12%|#1        | 5803/50000 [2:34:53<27:30:06,  2.24s/it]
Outer training loop:  12%|#1        | 5804/50000 [2:34:55<24:54:41,  2.03s/it]
Outer training loop:  12%|#1        | 5805/50000 [2:34:56<23:21:45,  1.90s/it]
Outer training loop:  12%|#1        | 5806/50000 [2:34:58<21:55:31,  1.79s/it]
Outer training loop:  12%|#1        | 5807/50000 [2:34:59<21:02:26,  1.71s/it]
Outer training loop:  12%|#1        | 5808/50000 [2:35:01<20:14:39,  1.65s/it]
Outer training loop:  12%|#1        | 5809/50000 [2:35:02<20:05:03,  1.64s/it]
Outer training loop:  12%|#1        | 5810/50000 [2:35:04<19:48:01,  1.61s/it]
Outer training loop:  12%|#1        | 5811/50000 [2:35:06<19:50:19,  1.62s/it]
Outer training loop:  12%|#1        | 5812/50000 [2:35:07<19:10:33,  1.56s/it]
Outer training loop:  12%|#1        | 5813/50000 [2:35:09<19:27:13,  1.58s/it]
Outer training loop:  12%|#1        | 5814/50000 [2:35:10<19:54:58,  1.62s/it]
Outer training loop:  12%|#1        | 5815/50000 [2:35:12<19:23:14,  1.58s/it]
Outer training loop:  12%|#1        | 5816/50000 [2:35:13<19:26:15,  1.58s/it]
Outer training loop:  12%|#1        | 5817/50000 [2:35:15<18:53:20,  1.54s/it]
Outer training loop:  12%|#1        | 5818/50000 [2:35:16<18:37:11,  1.52s/it]
Outer training loop:  12%|#1        | 5819/50000 [2:35:18<19:07:44,  1.56s/it]
Outer training loop:  12%|#1        | 5820/50000 [2:35:20<19:26:48,  1.58s/it]
Outer training loop:  12%|#1        | 5821/50000 [2:35:21<19:55:57,  1.62s/it]
Outer training loop:  12%|#1        | 5822/50000 [2:35:23<19:26:44,  1.58s/it]
Outer training loop:  12%|#1        | 5823/50000 [2:35:24<19:46:34,  1.61s/it]
Outer training loop:  12%|#1        | 5824/50000 [2:35:26<19:30:04,  1.59s/it]
Outer training loop:  12%|#1        | 5825/50000 [2:35:28<19:29:44,  1.59s/it]
Outer training loop:  12%|#1        | 5826/50000 [2:35:29<19:04:39,  1.55s/it]
Outer training loop:  12%|#1        | 5827/50000 [2:35:31<18:47:42,  1.53s/it]
Outer training loop:  12%|#1        | 5828/50000 [2:35:32<18:33:38,  1.51s/it]
Outer training loop:  12%|#1        | 5829/50000 [2:35:34<18:51:51,  1.54s/it]
Outer training loop:  12%|#1        | 5830/50000 [2:35:35<18:46:46,  1.53s/it]
Outer training loop:  12%|#1        | 5831/50000 [2:35:37<19:08:41,  1.56s/it]
Outer training loop:  12%|#1        | 5832/50000 [2:35:38<18:56:20,  1.54s/it]
Outer training loop:  12%|#1        | 5833/50000 [2:35:40<19:10:39,  1.56s/it]
Outer training loop:  12%|#1        | 5834/50000 [2:35:41<19:16:42,  1.57s/it]
Outer training loop:  12%|#1        | 5835/50000 [2:35:43<19:13:55,  1.57s/it]
Outer training loop:  12%|#1        | 5836/50000 [2:35:45<18:52:10,  1.54s/it]
Outer training loop:  12%|#1        | 5837/50000 [2:35:46<18:56:58,  1.54s/it]
Outer training loop:  12%|#1        | 5838/50000 [2:35:48<18:50:57,  1.54s/it]
Outer training loop:  12%|#1        | 5839/50000 [2:35:49<19:09:56,  1.56s/it]
Outer training loop:  12%|#1        | 5840/50000 [2:35:51<19:16:42,  1.57s/it]
Outer training loop:  12%|#1        | 5841/50000 [2:35:52<19:28:32,  1.59s/it]
Outer training loop:  12%|#1        | 5842/50000 [2:35:54<19:11:04,  1.56s/it]
Outer training loop:  12%|#1        | 5843/50000 [2:35:56<19:31:45,  1.59s/it]
Outer training loop:  12%|#1        | 5844/50000 [2:35:57<19:03:09,  1.55s/it]
Outer training loop:  12%|#1        | 5845/50000 [2:35:59<19:27:52,  1.59s/it]
Outer training loop:  12%|#1        | 5846/50000 [2:36:00<19:04:57,  1.56s/it]
Outer training loop:  12%|#1        | 5847/50000 [2:36:02<19:11:08,  1.56s/it]
Outer training loop:  12%|#1        | 5848/50000 [2:36:03<19:37:00,  1.60s/it]
Outer training loop:  12%|#1        | 5849/50000 [2:36:05<19:42:02,  1.61s/it]
Outer training loop:  12%|#1        | 5850/50000 [2:36:07<19:36:01,  1.60s/it]
Outer training loop:  12%|#1        | 5851/50000 [2:36:08<19:52:56,  1.62s/it]
Outer training loop:  12%|#1        | 5852/50000 [2:36:10<19:38:06,  1.60s/it]
Outer training loop:  12%|#1        | 5853/50000 [2:36:11<19:31:37,  1.59s/it]
Outer training loop:  12%|#1        | 5854/50000 [2:36:13<19:14:40,  1.57s/it]
Outer training loop:  12%|#1        | 5855/50000 [2:36:15<19:31:08,  1.59s/it]
Outer training loop:  12%|#1        | 5856/50000 [2:36:16<19:04:28,  1.56s/it]
Outer training loop:  12%|#1        | 5857/50000 [2:36:18<19:19:42,  1.58s/it]
Outer training loop:  12%|#1        | 5858/50000 [2:36:19<19:07:16,  1.56s/it]
Outer training loop:  12%|#1        | 5859/50000 [2:36:21<19:24:57,  1.58s/it]
Outer training loop:  12%|#1        | 5860/50000 [2:36:22<19:19:04,  1.58s/it]
Outer training loop:  12%|#1        | 5861/50000 [2:36:24<19:13:25,  1.57s/it]
Outer training loop:  12%|#1        | 5862/50000 [2:36:25<18:56:35,  1.55s/it]
Outer training loop:  12%|#1        | 5863/50000 [2:36:27<19:08:01,  1.56s/it]
Outer training loop:  12%|#1        | 5864/50000 [2:36:29<18:39:59,  1.52s/it]
Outer training loop:  12%|#1        | 5865/50000 [2:36:30<19:21:31,  1.58s/it]
Outer training loop:  12%|#1        | 5866/50000 [2:36:32<19:00:50,  1.55s/it]
Outer training loop:  12%|#1        | 5867/50000 [2:36:33<19:10:33,  1.56s/it]
Outer training loop:  12%|#1        | 5868/50000 [2:36:35<19:03:55,  1.56s/it]
Outer training loop:  12%|#1        | 5869/50000 [2:36:36<19:20:18,  1.58s/it]
Outer training loop:  12%|#1        | 5870/50000 [2:36:38<19:06:31,  1.56s/it]
Outer training loop:  12%|#1        | 5871/50000 [2:36:40<19:28:20,  1.59s/it]
Outer training loop:  12%|#1        | 5872/50000 [2:36:41<19:10:19,  1.56s/it]
Outer training loop:  12%|#1        | 5873/50000 [2:36:43<19:13:57,  1.57s/it]
Outer training loop:  12%|#1        | 5874/50000 [2:36:44<18:53:01,  1.54s/it]
Outer training loop:  12%|#1        | 5875/50000 [2:36:46<19:16:42,  1.57s/it]
Outer training loop:  12%|#1        | 5876/50000 [2:36:47<19:12:41,  1.57s/it]
Outer training loop:  12%|#1        | 5877/50000 [2:36:49<19:38:52,  1.60s/it]
Outer training loop:  12%|#1        | 5878/50000 [2:36:51<19:34:00,  1.60s/it]
Outer training loop:  12%|#1        | 5879/50000 [2:36:52<19:38:17,  1.60s/it]
Outer training loop:  12%|#1        | 5880/50000 [2:36:54<19:11:14,  1.57s/it]
Outer training loop:  12%|#1        | 5881/50000 [2:36:55<19:06:43,  1.56s/it]
Outer training loop:  12%|#1        | 5882/50000 [2:36:57<18:56:46,  1.55s/it]
Outer training loop:  12%|#1        | 5883/50000 [2:36:58<18:51:44,  1.54s/it]
Outer training loop:  12%|#1        | 5884/50000 [2:37:00<19:05:44,  1.56s/it]
Outer training loop:  12%|#1        | 5885/50000 [2:37:01<18:39:29,  1.52s/it]
Outer training loop:  12%|#1        | 5886/50000 [2:37:03<18:47:21,  1.53s/it]
Outer training loop:  12%|#1        | 5887/50000 [2:37:04<18:37:38,  1.52s/it]
Outer training loop:  12%|#1        | 5888/50000 [2:37:06<19:06:15,  1.56s/it]
Outer training loop:  12%|#1        | 5889/50000 [2:37:08<19:00:40,  1.55s/it]
Outer training loop:  12%|#1        | 5890/50000 [2:37:09<19:12:28,  1.57s/it]
Outer training loop:  12%|#1        | 5891/50000 [2:37:11<19:22:29,  1.58s/it]
Outer training loop:  12%|#1        | 5892/50000 [2:37:12<19:08:37,  1.56s/it]
Outer training loop:  12%|#1        | 5893/50000 [2:37:14<19:02:26,  1.55s/it]
Outer training loop:  12%|#1        | 5894/50000 [2:37:15<19:01:36,  1.55s/it]
Outer training loop:  12%|#1        | 5895/50000 [2:37:17<18:48:27,  1.54s/it]
Outer training loop:  12%|#1        | 5896/50000 [2:37:19<18:53:57,  1.54s/it]
Outer training loop:  12%|#1        | 5897/50000 [2:37:20<18:50:33,  1.54s/it]
Outer training loop:  12%|#1        | 5898/50000 [2:37:22<19:32:22,  1.59s/it]
Outer training loop:  12%|#1        | 5899/50000 [2:37:23<19:11:12,  1.57s/it]
Outer training loop:  12%|#1        | 5900/50000 [2:37:25<19:29:08,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.57s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.56s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.55s/it]

Outer training loop:  12%|#1        | 5901/50000 [2:37:31<36:11:39,  2.95s/it]
Outer training loop:  12%|#1        | 5902/50000 [2:37:33<31:20:07,  2.56s/it]
Outer training loop:  12%|#1        | 5903/50000 [2:37:34<27:51:37,  2.27s/it]
Outer training loop:  12%|#1        | 5904/50000 [2:37:36<24:49:48,  2.03s/it]
Outer training loop:  12%|#1        | 5905/50000 [2:37:37<23:19:54,  1.90s/it]
Outer training loop:  12%|#1        | 5906/50000 [2:37:39<21:48:06,  1.78s/it]
Outer training loop:  12%|#1        | 5907/50000 [2:37:40<21:17:39,  1.74s/it]
Outer training loop:  12%|#1        | 5908/50000 [2:37:42<20:20:32,  1.66s/it]
Outer training loop:  12%|#1        | 5909/50000 [2:37:44<19:55:59,  1.63s/it]
Outer training loop:  12%|#1        | 5910/50000 [2:37:45<19:42:47,  1.61s/it]
Outer training loop:  12%|#1        | 5911/50000 [2:37:47<19:42:48,  1.61s/it]
Outer training loop:  12%|#1        | 5912/50000 [2:37:48<19:15:35,  1.57s/it]
Outer training loop:  12%|#1        | 5913/50000 [2:37:50<19:37:53,  1.60s/it]
Outer training loop:  12%|#1        | 5914/50000 [2:37:51<19:23:13,  1.58s/it]
Outer training loop:  12%|#1        | 5915/50000 [2:37:53<19:44:46,  1.61s/it]
Outer training loop:  12%|#1        | 5916/50000 [2:37:55<19:28:00,  1.59s/it]
Outer training loop:  12%|#1        | 5917/50000 [2:37:56<19:30:51,  1.59s/it]
Outer training loop:  12%|#1        | 5918/50000 [2:37:58<19:35:24,  1.60s/it]
Outer training loop:  12%|#1        | 5919/50000 [2:37:59<19:14:11,  1.57s/it]
Outer training loop:  12%|#1        | 5920/50000 [2:38:01<19:27:49,  1.59s/it]
Outer training loop:  12%|#1        | 5921/50000 [2:38:02<19:04:20,  1.56s/it]
Outer training loop:  12%|#1        | 5922/50000 [2:38:04<18:59:01,  1.55s/it]
Outer training loop:  12%|#1        | 5923/50000 [2:38:06<19:05:16,  1.56s/it]
Outer training loop:  12%|#1        | 5924/50000 [2:38:07<18:59:02,  1.55s/it]
Outer training loop:  12%|#1        | 5925/50000 [2:38:09<19:04:58,  1.56s/it]
Outer training loop:  12%|#1        | 5926/50000 [2:38:10<19:21:54,  1.58s/it]
Outer training loop:  12%|#1        | 5927/50000 [2:38:12<19:00:25,  1.55s/it]
Outer training loop:  12%|#1        | 5928/50000 [2:38:13<18:51:22,  1.54s/it]
Outer training loop:  12%|#1        | 5929/50000 [2:38:15<19:01:39,  1.55s/it]
Outer training loop:  12%|#1        | 5930/50000 [2:38:16<18:29:59,  1.51s/it]
Outer training loop:  12%|#1        | 5931/50000 [2:38:18<18:59:37,  1.55s/it]
Outer training loop:  12%|#1        | 5932/50000 [2:38:19<18:56:57,  1.55s/it]
Outer training loop:  12%|#1        | 5933/50000 [2:38:21<18:57:36,  1.55s/it]
Outer training loop:  12%|#1        | 5934/50000 [2:38:23<18:48:06,  1.54s/it]
Outer training loop:  12%|#1        | 5935/50000 [2:38:24<19:14:09,  1.57s/it]
Outer training loop:  12%|#1        | 5936/50000 [2:38:26<19:03:09,  1.56s/it]
Outer training loop:  12%|#1        | 5937/50000 [2:38:27<19:33:02,  1.60s/it]
Outer training loop:  12%|#1        | 5938/50000 [2:38:29<19:24:09,  1.59s/it]
Outer training loop:  12%|#1        | 5939/50000 [2:38:31<19:30:52,  1.59s/it]
Outer training loop:  12%|#1        | 5940/50000 [2:38:32<19:04:29,  1.56s/it]
Outer training loop:  12%|#1        | 5941/50000 [2:38:34<19:21:50,  1.58s/it]
Outer training loop:  12%|#1        | 5942/50000 [2:38:35<19:12:25,  1.57s/it]
Outer training loop:  12%|#1        | 5943/50000 [2:38:37<19:40:46,  1.61s/it]
Outer training loop:  12%|#1        | 5944/50000 [2:38:39<19:56:19,  1.63s/it]
Outer training loop:  12%|#1        | 5945/50000 [2:38:40<20:15:09,  1.65s/it]
Outer training loop:  12%|#1        | 5946/50000 [2:38:42<20:07:31,  1.64s/it]
Outer training loop:  12%|#1        | 5947/50000 [2:38:43<19:25:41,  1.59s/it]
Outer training loop:  12%|#1        | 5948/50000 [2:38:45<19:24:13,  1.59s/it]
Outer training loop:  12%|#1        | 5949/50000 [2:38:47<19:23:56,  1.59s/it]
Outer training loop:  12%|#1        | 5950/50000 [2:38:48<19:30:55,  1.59s/it]
Outer training loop:  12%|#1        | 5951/50000 [2:38:50<19:07:49,  1.56s/it]
Outer training loop:  12%|#1        | 5952/50000 [2:38:51<19:05:10,  1.56s/it]
Outer training loop:  12%|#1        | 5953/50000 [2:38:53<19:18:20,  1.58s/it]
Outer training loop:  12%|#1        | 5954/50000 [2:38:54<18:56:02,  1.55s/it]
Outer training loop:  12%|#1        | 5955/50000 [2:38:56<19:07:24,  1.56s/it]
Outer training loop:  12%|#1        | 5956/50000 [2:38:58<19:18:07,  1.58s/it]
Outer training loop:  12%|#1        | 5957/50000 [2:38:59<18:45:34,  1.53s/it]
Outer training loop:  12%|#1        | 5958/50000 [2:39:00<18:30:03,  1.51s/it]
Outer training loop:  12%|#1        | 5959/50000 [2:39:02<18:27:47,  1.51s/it]
Outer training loop:  12%|#1        | 5960/50000 [2:39:04<18:44:45,  1.53s/it]
Outer training loop:  12%|#1        | 5961/50000 [2:39:05<18:47:19,  1.54s/it]
Outer training loop:  12%|#1        | 5962/50000 [2:39:07<19:02:35,  1.56s/it]
Outer training loop:  12%|#1        | 5963/50000 [2:39:08<18:40:10,  1.53s/it]
Outer training loop:  12%|#1        | 5964/50000 [2:39:10<18:47:49,  1.54s/it]
Outer training loop:  12%|#1        | 5965/50000 [2:39:11<18:31:12,  1.51s/it]
Outer training loop:  12%|#1        | 5966/50000 [2:39:13<18:42:24,  1.53s/it]
Outer training loop:  12%|#1        | 5967/50000 [2:39:14<18:50:41,  1.54s/it]
Outer training loop:  12%|#1        | 5968/50000 [2:39:16<18:41:42,  1.53s/it]
Outer training loop:  12%|#1        | 5969/50000 [2:39:17<19:15:12,  1.57s/it]
Outer training loop:  12%|#1        | 5970/50000 [2:39:19<18:55:56,  1.55s/it]
Outer training loop:  12%|#1        | 5971/50000 [2:39:21<19:20:06,  1.58s/it]
Outer training loop:  12%|#1        | 5972/50000 [2:39:22<19:11:08,  1.57s/it]
Outer training loop:  12%|#1        | 5973/50000 [2:39:24<19:08:59,  1.57s/it]
Outer training loop:  12%|#1        | 5974/50000 [2:39:25<19:15:43,  1.58s/it]
Outer training loop:  12%|#1        | 5975/50000 [2:39:27<19:00:38,  1.55s/it]
Outer training loop:  12%|#1        | 5976/50000 [2:39:28<18:51:59,  1.54s/it]
Outer training loop:  12%|#1        | 5977/50000 [2:39:30<19:00:31,  1.55s/it]
Outer training loop:  12%|#1        | 5978/50000 [2:39:32<19:12:58,  1.57s/it]
Outer training loop:  12%|#1        | 5979/50000 [2:39:33<19:23:14,  1.59s/it]
Outer training loop:  12%|#1        | 5980/50000 [2:39:35<19:06:49,  1.56s/it]
Outer training loop:  12%|#1        | 5981/50000 [2:39:36<19:08:28,  1.57s/it]
Outer training loop:  12%|#1        | 5982/50000 [2:39:38<19:02:43,  1.56s/it]
Outer training loop:  12%|#1        | 5983/50000 [2:39:39<19:12:12,  1.57s/it]
Outer training loop:  12%|#1        | 5984/50000 [2:39:41<19:07:46,  1.56s/it]
Outer training loop:  12%|#1        | 5985/50000 [2:39:42<19:00:49,  1.56s/it]
Outer training loop:  12%|#1        | 5986/50000 [2:39:44<18:50:22,  1.54s/it]
Outer training loop:  12%|#1        | 5987/50000 [2:39:46<18:58:01,  1.55s/it]
Outer training loop:  12%|#1        | 5988/50000 [2:39:47<18:52:46,  1.54s/it]
Outer training loop:  12%|#1        | 5989/50000 [2:39:49<19:12:02,  1.57s/it]
Outer training loop:  12%|#1        | 5990/50000 [2:39:50<18:55:19,  1.55s/it]
Outer training loop:  12%|#1        | 5991/50000 [2:39:52<19:07:22,  1.56s/it]
Outer training loop:  12%|#1        | 5992/50000 [2:39:53<18:41:49,  1.53s/it]
Outer training loop:  12%|#1        | 5993/50000 [2:39:55<18:59:36,  1.55s/it]
Outer training loop:  12%|#1        | 5994/50000 [2:39:56<19:05:45,  1.56s/it]
Outer training loop:  12%|#1        | 5995/50000 [2:39:58<19:35:54,  1.60s/it]
Outer training loop:  12%|#1        | 5996/50000 [2:40:00<19:18:16,  1.58s/it]
Outer training loop:  12%|#1        | 5997/50000 [2:40:01<19:03:27,  1.56s/it]
Outer training loop:  12%|#1        | 5998/50000 [2:40:03<18:53:55,  1.55s/it]
Outer training loop:  12%|#1        | 5999/50000 [2:40:04<18:56:37,  1.55s/it]
Outer training loop:  12%|#2        | 6000/50000 [2:40:06<18:45:20,  1.53s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.47s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.53s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.49s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:00,  1.47s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:00,  1.49s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:00,  1.50s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:01,  1.54s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<01:59,  1.53s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<01:59,  1.55s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:57,  1.55s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:55,  1.53s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:58,  1.61s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:53,  1.56s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:54,  1.59s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:53,  1.59s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:49,  1.56s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:50,  1.60s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:47,  1.58s/it]

Evaluating policy:  19%|#9        | 16/83 [00:24<01:43,  1.55s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:43,  1.56s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:39,  1.54s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:38,  1.54s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:36,  1.54s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:38,  1.58s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:34,  1.56s/it]

Evaluating policy:  28%|##7       | 23/83 [00:35<01:32,  1.55s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:34,  1.59s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:30,  1.57s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:29,  1.57s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:26,  1.55s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:27,  1.58s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:22,  1.53s/it]

Evaluating policy:  36%|###6      | 30/83 [00:46<01:20,  1.52s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:19,  1.52s/it]

Evaluating policy:  39%|###8      | 32/83 [00:49<01:21,  1.59s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:19,  1.59s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:16,  1.57s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:13,  1.52s/it]

Evaluating policy:  43%|####3     | 36/83 [00:55<01:10,  1.51s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:12,  1.57s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:08,  1.53s/it]

Evaluating policy:  47%|####6     | 39/83 [01:00<01:07,  1.52s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:04,  1.49s/it]

Evaluating policy:  49%|####9     | 41/83 [01:03<01:04,  1.53s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:05,  1.60s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:06<01:02,  1.57s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<01:00,  1.55s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:09<00:57,  1.51s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:11<00:55,  1.50s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:12<00:54,  1.51s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:14<00:56,  1.61s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:53,  1.56s/it]

Evaluating policy:  60%|######    | 50/83 [01:17<00:50,  1.54s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:49,  1.54s/it]

Evaluating policy:  63%|######2   | 52/83 [01:20<00:47,  1.53s/it]

Evaluating policy:  64%|######3   | 53/83 [01:22<00:44,  1.50s/it]

Evaluating policy:  65%|######5   | 54/83 [01:23<00:46,  1.60s/it]

Evaluating policy:  66%|######6   | 55/83 [01:25<00:44,  1.57s/it]

Evaluating policy:  67%|######7   | 56/83 [01:26<00:42,  1.56s/it]

Evaluating policy:  69%|######8   | 57/83 [01:28<00:40,  1.56s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:38,  1.55s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:31<00:36,  1.53s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:33<00:35,  1.54s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:34<00:35,  1.63s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:36<00:32,  1.57s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:37<00:31,  1.56s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:39<00:29,  1.54s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:40<00:27,  1.53s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:42<00:25,  1.52s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:25,  1.61s/it]

Evaluating policy:  82%|########1 | 68/83 [01:45<00:23,  1.59s/it]

Evaluating policy:  83%|########3 | 69/83 [01:47<00:21,  1.54s/it]

Evaluating policy:  84%|########4 | 70/83 [01:48<00:19,  1.53s/it]

Evaluating policy:  86%|########5 | 71/83 [01:50<00:18,  1.53s/it]

Evaluating policy:  87%|########6 | 72/83 [01:51<00:16,  1.54s/it]

Evaluating policy:  88%|########7 | 73/83 [01:53<00:15,  1.52s/it]

Evaluating policy:  89%|########9 | 74/83 [01:54<00:13,  1.52s/it]

Evaluating policy:  90%|######### | 75/83 [01:56<00:13,  1.66s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:11,  1.62s/it]

Evaluating policy:  93%|#########2| 77/83 [01:59<00:09,  1.61s/it]

Evaluating policy:  94%|#########3| 78/83 [02:01<00:07,  1.57s/it]

Evaluating policy:  95%|#########5| 79/83 [02:02<00:06,  1.57s/it]

Evaluating policy:  96%|#########6| 80/83 [02:04<00:04,  1.56s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.55s/it]

Evaluating policy:  99%|#########8| 82/83 [02:07<00:01,  1.68s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.59s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:03,  1.50s/it]

Evaluating policy:   2%|2         | 2/83 [00:02<02:00,  1.49s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<01:58,  1.49s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:00,  1.52s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<01:58,  1.51s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<01:56,  1.52s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:54,  1.50s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:56,  1.56s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<02:05,  1.70s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:59,  1.63s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:58,  1.64s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:54,  1.61s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:50,  1.58s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:50,  1.60s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:47,  1.58s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:45,  1.58s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:43,  1.56s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:44,  1.61s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:41,  1.59s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:38,  1.57s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:40,  1.62s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:35,  1.57s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:32,  1.55s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:33,  1.58s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:30,  1.56s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:28,  1.56s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:25,  1.53s/it]

Evaluating policy:  34%|###3      | 28/83 [00:44<01:26,  1.58s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:23,  1.54s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:20,  1.52s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:18,  1.52s/it]

Evaluating policy:  39%|###8      | 32/83 [00:49<01:16,  1.50s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:18,  1.57s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:15,  1.54s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:13,  1.53s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:11,  1.53s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:13,  1.60s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:10,  1.57s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:08,  1.56s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:07,  1.56s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:05,  1.55s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:07,  1.65s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:04,  1.61s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<01:01,  1.58s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<00:59,  1.58s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<00:58,  1.57s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:14<00:59,  1.67s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:56,  1.63s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:53,  1.58s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:50,  1.54s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:49,  1.53s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:46,  1.52s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:48,  1.63s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:45,  1.58s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:44,  1.58s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:42,  1.56s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:40,  1.55s/it]

Evaluating policy:  70%|######9   | 58/83 [01:31<00:40,  1.60s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:40,  1.67s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:34<00:36,  1.59s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:34,  1.58s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:37<00:32,  1.57s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:30,  1.54s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:28,  1.52s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:27,  1.51s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:27,  1.61s/it]

Evaluating policy:  81%|########  | 67/83 [01:45<00:25,  1.58s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:23,  1.56s/it]

Evaluating policy:  83%|########3 | 69/83 [01:48<00:22,  1.59s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:20,  1.54s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:18,  1.54s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:16,  1.51s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:16,  1.64s/it]

Evaluating policy:  89%|########9 | 74/83 [01:56<00:14,  1.59s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.56s/it]

Evaluating policy:  92%|#########1| 76/83 [01:59<00:10,  1.54s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.53s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.52s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.53s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.52s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.51s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.65s/it]

Evaluating policy: 100%|##########| 83/83 [02:10<00:00,  1.61s/it]

Outer training loop:  12%|#2        | 6001/50000 [2:44:32<988:26:30, 80.87s/it]
Outer training loop:  12%|#2        | 6002/50000 [2:44:33<697:35:44, 57.08s/it]
Outer training loop:  12%|#2        | 6003/50000 [2:44:35<493:51:25, 40.41s/it]
Outer training loop:  12%|#2        | 6004/50000 [2:44:36<351:23:44, 28.75s/it]
Outer training loop:  12%|#2        | 6005/50000 [2:44:38<251:30:48, 20.58s/it]
Outer training loop:  12%|#2        | 6006/50000 [2:44:39<181:23:51, 14.84s/it]
Outer training loop:  12%|#2        | 6007/50000 [2:44:41<132:12:04, 10.82s/it]
Outer training loop:  12%|#2        | 6008/50000 [2:44:42<98:16:54,  8.04s/it] 
Outer training loop:  12%|#2        | 6009/50000 [2:44:44<76:13:22,  6.24s/it]
Outer training loop:  12%|#2        | 6010/50000 [2:44:46<58:55:45,  4.82s/it]
Outer training loop:  12%|#2        | 6011/50000 [2:44:47<46:54:37,  3.84s/it]
Outer training loop:  12%|#2        | 6012/50000 [2:44:49<38:29:25,  3.15s/it]
Outer training loop:  12%|#2        | 6013/50000 [2:44:51<32:41:52,  2.68s/it]
Outer training loop:  12%|#2        | 6014/50000 [2:44:52<28:25:45,  2.33s/it]
Outer training loop:  12%|#2        | 6015/50000 [2:44:54<25:38:00,  2.10s/it]
Outer training loop:  12%|#2        | 6016/50000 [2:44:55<23:29:03,  1.92s/it]
Outer training loop:  12%|#2        | 6017/50000 [2:44:57<22:05:57,  1.81s/it]
Outer training loop:  12%|#2        | 6018/50000 [2:44:58<21:17:58,  1.74s/it]
Outer training loop:  12%|#2        | 6019/50000 [2:45:00<20:05:12,  1.64s/it]
Outer training loop:  12%|#2        | 6020/50000 [2:45:01<19:41:14,  1.61s/it]
Outer training loop:  12%|#2        | 6021/50000 [2:45:03<20:02:10,  1.64s/it]
Outer training loop:  12%|#2        | 6022/50000 [2:45:05<20:02:09,  1.64s/it]
Outer training loop:  12%|#2        | 6023/50000 [2:45:06<19:43:39,  1.61s/it]
Outer training loop:  12%|#2        | 6024/50000 [2:45:08<19:41:02,  1.61s/it]
Outer training loop:  12%|#2        | 6025/50000 [2:45:09<19:18:16,  1.58s/it]
Outer training loop:  12%|#2        | 6026/50000 [2:45:11<19:32:54,  1.60s/it]
Outer training loop:  12%|#2        | 6027/50000 [2:45:12<19:32:00,  1.60s/it]
Outer training loop:  12%|#2        | 6028/50000 [2:45:14<19:37:11,  1.61s/it]
Outer training loop:  12%|#2        | 6029/50000 [2:45:16<19:19:54,  1.58s/it]
Outer training loop:  12%|#2        | 6030/50000 [2:45:17<18:56:17,  1.55s/it]
Outer training loop:  12%|#2        | 6031/50000 [2:45:19<19:30:26,  1.60s/it]
Outer training loop:  12%|#2        | 6032/50000 [2:45:20<19:20:37,  1.58s/it]
Outer training loop:  12%|#2        | 6033/50000 [2:45:22<19:01:24,  1.56s/it]
Outer training loop:  12%|#2        | 6034/50000 [2:45:23<19:08:49,  1.57s/it]
Outer training loop:  12%|#2        | 6035/50000 [2:45:25<19:00:14,  1.56s/it]
Outer training loop:  12%|#2        | 6036/50000 [2:45:26<18:54:35,  1.55s/it]
Outer training loop:  12%|#2        | 6037/50000 [2:45:28<19:02:57,  1.56s/it]
Outer training loop:  12%|#2        | 6038/50000 [2:45:30<18:52:21,  1.55s/it]
Outer training loop:  12%|#2        | 6039/50000 [2:45:31<18:48:48,  1.54s/it]
Outer training loop:  12%|#2        | 6040/50000 [2:45:33<19:20:09,  1.58s/it]
Outer training loop:  12%|#2        | 6041/50000 [2:45:34<19:16:58,  1.58s/it]
Outer training loop:  12%|#2        | 6042/50000 [2:45:36<19:11:28,  1.57s/it]
Outer training loop:  12%|#2        | 6043/50000 [2:45:38<19:14:45,  1.58s/it]
Outer training loop:  12%|#2        | 6044/50000 [2:45:39<18:58:30,  1.55s/it]
Outer training loop:  12%|#2        | 6045/50000 [2:45:41<18:50:33,  1.54s/it]
Outer training loop:  12%|#2        | 6046/50000 [2:45:42<18:47:54,  1.54s/it]
Outer training loop:  12%|#2        | 6047/50000 [2:45:44<18:59:43,  1.56s/it]
Outer training loop:  12%|#2        | 6048/50000 [2:45:45<18:52:37,  1.55s/it]
Outer training loop:  12%|#2        | 6049/50000 [2:45:47<19:02:21,  1.56s/it]
Outer training loop:  12%|#2        | 6050/50000 [2:45:48<19:28:52,  1.60s/it]
Outer training loop:  12%|#2        | 6051/50000 [2:45:50<19:28:41,  1.60s/it]
Outer training loop:  12%|#2        | 6052/50000 [2:45:51<18:47:24,  1.54s/it]
Outer training loop:  12%|#2        | 6053/50000 [2:45:53<19:01:30,  1.56s/it]
Outer training loop:  12%|#2        | 6054/50000 [2:45:54<18:33:07,  1.52s/it]
Outer training loop:  12%|#2        | 6055/50000 [2:45:56<18:55:59,  1.55s/it]
Outer training loop:  12%|#2        | 6056/50000 [2:45:58<18:34:49,  1.52s/it]
Outer training loop:  12%|#2        | 6057/50000 [2:45:59<18:52:33,  1.55s/it]
Outer training loop:  12%|#2        | 6058/50000 [2:46:01<19:13:36,  1.58s/it]
Outer training loop:  12%|#2        | 6059/50000 [2:46:02<18:58:00,  1.55s/it]
Outer training loop:  12%|#2        | 6060/50000 [2:46:04<18:42:36,  1.53s/it]
Outer training loop:  12%|#2        | 6061/50000 [2:46:05<18:54:26,  1.55s/it]
Outer training loop:  12%|#2        | 6062/50000 [2:46:07<18:57:03,  1.55s/it]
Outer training loop:  12%|#2        | 6063/50000 [2:46:09<19:14:45,  1.58s/it]
Outer training loop:  12%|#2        | 6064/50000 [2:46:10<19:03:53,  1.56s/it]
Outer training loop:  12%|#2        | 6065/50000 [2:46:12<19:22:24,  1.59s/it]
Outer training loop:  12%|#2        | 6066/50000 [2:46:13<19:30:50,  1.60s/it]
Outer training loop:  12%|#2        | 6067/50000 [2:46:15<19:10:10,  1.57s/it]
Outer training loop:  12%|#2        | 6068/50000 [2:46:16<18:56:44,  1.55s/it]
Outer training loop:  12%|#2        | 6069/50000 [2:46:18<19:11:52,  1.57s/it]
Outer training loop:  12%|#2        | 6070/50000 [2:46:20<19:27:55,  1.60s/it]
Outer training loop:  12%|#2        | 6071/50000 [2:46:21<19:08:05,  1.57s/it]
Outer training loop:  12%|#2        | 6072/50000 [2:46:23<19:27:02,  1.59s/it]
Outer training loop:  12%|#2        | 6073/50000 [2:46:24<18:57:43,  1.55s/it]
Outer training loop:  12%|#2        | 6074/50000 [2:46:26<19:09:39,  1.57s/it]
Outer training loop:  12%|#2        | 6075/50000 [2:46:27<18:51:40,  1.55s/it]
Outer training loop:  12%|#2        | 6076/50000 [2:46:29<19:12:16,  1.57s/it]
Outer training loop:  12%|#2        | 6077/50000 [2:46:31<19:22:02,  1.59s/it]
Outer training loop:  12%|#2        | 6078/50000 [2:46:32<19:12:39,  1.57s/it]
Outer training loop:  12%|#2        | 6079/50000 [2:46:34<19:40:18,  1.61s/it]
Outer training loop:  12%|#2        | 6080/50000 [2:46:35<19:36:51,  1.61s/it]
Outer training loop:  12%|#2        | 6081/50000 [2:46:37<19:54:30,  1.63s/it]
Outer training loop:  12%|#2        | 6082/50000 [2:46:39<19:59:50,  1.64s/it]
Outer training loop:  12%|#2        | 6083/50000 [2:46:40<19:36:59,  1.61s/it]
Outer training loop:  12%|#2        | 6084/50000 [2:46:42<19:11:31,  1.57s/it]
Outer training loop:  12%|#2        | 6085/50000 [2:46:43<19:19:04,  1.58s/it]
Outer training loop:  12%|#2        | 6086/50000 [2:46:45<18:46:54,  1.54s/it]
Outer training loop:  12%|#2        | 6087/50000 [2:46:46<18:53:57,  1.55s/it]
Outer training loop:  12%|#2        | 6088/50000 [2:46:48<18:41:55,  1.53s/it]
Outer training loop:  12%|#2        | 6089/50000 [2:46:50<18:55:29,  1.55s/it]
Outer training loop:  12%|#2        | 6090/50000 [2:46:51<18:53:49,  1.55s/it]
Outer training loop:  12%|#2        | 6091/50000 [2:46:53<19:23:54,  1.59s/it]
Outer training loop:  12%|#2        | 6092/50000 [2:46:54<18:49:39,  1.54s/it]
Outer training loop:  12%|#2        | 6093/50000 [2:46:56<19:21:13,  1.59s/it]
Outer training loop:  12%|#2        | 6094/50000 [2:46:57<18:57:07,  1.55s/it]
Outer training loop:  12%|#2        | 6095/50000 [2:46:59<19:05:33,  1.57s/it]
Outer training loop:  12%|#2        | 6096/50000 [2:47:00<18:50:14,  1.54s/it]
Outer training loop:  12%|#2        | 6097/50000 [2:47:02<18:58:08,  1.56s/it]
Outer training loop:  12%|#2        | 6098/50000 [2:47:04<18:51:24,  1.55s/it]
Outer training loop:  12%|#2        | 6099/50000 [2:47:05<19:22:15,  1.59s/it]
Outer training loop:  12%|#2        | 6100/50000 [2:47:07<19:02:43,  1.56s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.47s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.51s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.48s/it]

Outer training loop:  12%|#2        | 6101/50000 [2:47:13<35:44:21,  2.93s/it]
Outer training loop:  12%|#2        | 6102/50000 [2:47:15<31:00:18,  2.54s/it]
Outer training loop:  12%|#2        | 6103/50000 [2:47:16<27:23:16,  2.25s/it]
Outer training loop:  12%|#2        | 6104/50000 [2:47:18<25:20:54,  2.08s/it]
Outer training loop:  12%|#2        | 6105/50000 [2:47:19<23:11:41,  1.90s/it]
Outer training loop:  12%|#2        | 6106/50000 [2:47:21<22:08:23,  1.82s/it]
Outer training loop:  12%|#2        | 6107/50000 [2:47:22<20:56:37,  1.72s/it]
Outer training loop:  12%|#2        | 6108/50000 [2:47:24<20:23:38,  1.67s/it]
Outer training loop:  12%|#2        | 6109/50000 [2:47:25<19:39:39,  1.61s/it]
Outer training loop:  12%|#2        | 6110/50000 [2:47:27<19:38:11,  1.61s/it]
Outer training loop:  12%|#2        | 6111/50000 [2:47:29<19:16:53,  1.58s/it]
Outer training loop:  12%|#2        | 6112/50000 [2:47:30<19:32:19,  1.60s/it]
Outer training loop:  12%|#2        | 6113/50000 [2:47:32<19:02:07,  1.56s/it]
Outer training loop:  12%|#2        | 6114/50000 [2:47:33<19:19:57,  1.59s/it]
Outer training loop:  12%|#2        | 6115/50000 [2:47:35<19:12:02,  1.58s/it]
Outer training loop:  12%|#2        | 6116/50000 [2:47:37<19:37:12,  1.61s/it]
Outer training loop:  12%|#2        | 6117/50000 [2:47:38<19:15:11,  1.58s/it]
Outer training loop:  12%|#2        | 6118/50000 [2:47:40<19:39:03,  1.61s/it]
Outer training loop:  12%|#2        | 6119/50000 [2:47:41<19:05:19,  1.57s/it]
Outer training loop:  12%|#2        | 6120/50000 [2:47:43<18:58:54,  1.56s/it]
Outer training loop:  12%|#2        | 6121/50000 [2:47:44<19:03:24,  1.56s/it]
Outer training loop:  12%|#2        | 6122/50000 [2:47:46<19:03:30,  1.56s/it]
Outer training loop:  12%|#2        | 6123/50000 [2:47:48<19:22:43,  1.59s/it]
Outer training loop:  12%|#2        | 6124/50000 [2:47:49<19:23:07,  1.59s/it]
Outer training loop:  12%|#2        | 6125/50000 [2:47:51<18:59:01,  1.56s/it]
Outer training loop:  12%|#2        | 6126/50000 [2:47:52<19:11:21,  1.57s/it]
Outer training loop:  12%|#2        | 6127/50000 [2:47:54<18:39:53,  1.53s/it]
Outer training loop:  12%|#2        | 6128/50000 [2:47:55<18:46:32,  1.54s/it]
Outer training loop:  12%|#2        | 6129/50000 [2:47:57<18:48:18,  1.54s/it]
Outer training loop:  12%|#2        | 6130/50000 [2:47:58<19:07:52,  1.57s/it]
Outer training loop:  12%|#2        | 6131/50000 [2:48:00<18:49:37,  1.55s/it]
Outer training loop:  12%|#2        | 6132/50000 [2:48:01<18:38:04,  1.53s/it]
Outer training loop:  12%|#2        | 6133/50000 [2:48:03<18:56:13,  1.55s/it]
Outer training loop:  12%|#2        | 6134/50000 [2:48:05<19:13:18,  1.58s/it]
Outer training loop:  12%|#2        | 6135/50000 [2:48:06<19:19:52,  1.59s/it]
Outer training loop:  12%|#2        | 6136/50000 [2:48:08<19:21:44,  1.59s/it]
Outer training loop:  12%|#2        | 6137/50000 [2:48:09<19:11:06,  1.57s/it]
Outer training loop:  12%|#2        | 6138/50000 [2:48:11<19:04:41,  1.57s/it]
Outer training loop:  12%|#2        | 6139/50000 [2:48:12<18:54:18,  1.55s/it]
Outer training loop:  12%|#2        | 6140/50000 [2:48:14<18:56:20,  1.55s/it]
Outer training loop:  12%|#2        | 6141/50000 [2:48:16<19:23:17,  1.59s/it]
Outer training loop:  12%|#2        | 6142/50000 [2:48:17<18:54:10,  1.55s/it]
Outer training loop:  12%|#2        | 6143/50000 [2:48:19<19:18:10,  1.58s/it]
Outer training loop:  12%|#2        | 6144/50000 [2:48:20<19:16:44,  1.58s/it]
Outer training loop:  12%|#2        | 6145/50000 [2:48:22<19:16:38,  1.58s/it]
Outer training loop:  12%|#2        | 6146/50000 [2:48:23<18:55:07,  1.55s/it]
Outer training loop:  12%|#2        | 6147/50000 [2:48:25<19:25:14,  1.59s/it]
Outer training loop:  12%|#2        | 6148/50000 [2:48:27<18:47:58,  1.54s/it]
Outer training loop:  12%|#2        | 6149/50000 [2:48:28<18:47:13,  1.54s/it]
Outer training loop:  12%|#2        | 6150/50000 [2:48:30<19:06:11,  1.57s/it]
Outer training loop:  12%|#2        | 6151/50000 [2:48:31<18:48:16,  1.54s/it]
Outer training loop:  12%|#2        | 6152/50000 [2:48:33<18:28:16,  1.52s/it]
Outer training loop:  12%|#2        | 6153/50000 [2:48:34<18:52:30,  1.55s/it]
Outer training loop:  12%|#2        | 6154/50000 [2:48:36<18:40:58,  1.53s/it]
Outer training loop:  12%|#2        | 6155/50000 [2:48:37<18:46:28,  1.54s/it]
Outer training loop:  12%|#2        | 6156/50000 [2:48:39<18:45:37,  1.54s/it]
Outer training loop:  12%|#2        | 6157/50000 [2:48:40<18:29:14,  1.52s/it]
Outer training loop:  12%|#2        | 6158/50000 [2:48:42<18:16:30,  1.50s/it]
Outer training loop:  12%|#2        | 6159/50000 [2:48:43<18:27:37,  1.52s/it]
Outer training loop:  12%|#2        | 6160/50000 [2:48:45<18:39:57,  1.53s/it]
Outer training loop:  12%|#2        | 6161/50000 [2:48:47<19:02:03,  1.56s/it]
Outer training loop:  12%|#2        | 6162/50000 [2:48:48<19:12:49,  1.58s/it]
Outer training loop:  12%|#2        | 6163/50000 [2:48:50<19:00:23,  1.56s/it]
Outer training loop:  12%|#2        | 6164/50000 [2:48:51<18:44:13,  1.54s/it]
Outer training loop:  12%|#2        | 6165/50000 [2:48:53<19:01:44,  1.56s/it]
Outer training loop:  12%|#2        | 6166/50000 [2:48:54<19:10:23,  1.57s/it]
Outer training loop:  12%|#2        | 6167/50000 [2:48:56<19:19:53,  1.59s/it]
Outer training loop:  12%|#2        | 6168/50000 [2:48:58<19:06:44,  1.57s/it]
Outer training loop:  12%|#2        | 6169/50000 [2:48:59<18:57:28,  1.56s/it]
Outer training loop:  12%|#2        | 6170/50000 [2:49:00<18:26:48,  1.52s/it]
Outer training loop:  12%|#2        | 6171/50000 [2:49:02<18:43:51,  1.54s/it]
Outer training loop:  12%|#2        | 6172/50000 [2:49:04<19:02:22,  1.56s/it]
Outer training loop:  12%|#2        | 6173/50000 [2:49:05<19:52:02,  1.63s/it]
Outer training loop:  12%|#2        | 6174/50000 [2:49:07<19:33:01,  1.61s/it]
Outer training loop:  12%|#2        | 6175/50000 [2:49:09<19:17:04,  1.58s/it]
Outer training loop:  12%|#2        | 6176/50000 [2:49:10<19:25:02,  1.60s/it]
Outer training loop:  12%|#2        | 6177/50000 [2:49:12<19:05:00,  1.57s/it]
Outer training loop:  12%|#2        | 6178/50000 [2:49:13<19:08:46,  1.57s/it]
Outer training loop:  12%|#2        | 6179/50000 [2:49:15<18:44:47,  1.54s/it]
Outer training loop:  12%|#2        | 6180/50000 [2:49:16<18:55:38,  1.55s/it]
Outer training loop:  12%|#2        | 6181/50000 [2:49:18<18:47:53,  1.54s/it]
Outer training loop:  12%|#2        | 6182/50000 [2:49:20<19:18:47,  1.59s/it]
Outer training loop:  12%|#2        | 6183/50000 [2:49:21<18:55:31,  1.55s/it]
Outer training loop:  12%|#2        | 6184/50000 [2:49:22<18:42:43,  1.54s/it]
Outer training loop:  12%|#2        | 6185/50000 [2:49:24<18:59:18,  1.56s/it]
Outer training loop:  12%|#2        | 6186/50000 [2:49:26<19:10:15,  1.58s/it]
Outer training loop:  12%|#2        | 6187/50000 [2:49:27<19:14:05,  1.58s/it]
Outer training loop:  12%|#2        | 6188/50000 [2:49:29<18:48:50,  1.55s/it]
Outer training loop:  12%|#2        | 6189/50000 [2:49:30<18:59:10,  1.56s/it]
Outer training loop:  12%|#2        | 6190/50000 [2:49:32<18:45:04,  1.54s/it]
Outer training loop:  12%|#2        | 6191/50000 [2:49:34<19:11:17,  1.58s/it]
Outer training loop:  12%|#2        | 6192/50000 [2:49:35<19:00:33,  1.56s/it]
Outer training loop:  12%|#2        | 6193/50000 [2:49:37<19:02:24,  1.56s/it]
Outer training loop:  12%|#2        | 6194/50000 [2:49:38<19:03:23,  1.57s/it]
Outer training loop:  12%|#2        | 6195/50000 [2:49:40<19:03:52,  1.57s/it]
Outer training loop:  12%|#2        | 6196/50000 [2:49:41<18:41:29,  1.54s/it]
Outer training loop:  12%|#2        | 6197/50000 [2:49:43<18:51:28,  1.55s/it]
Outer training loop:  12%|#2        | 6198/50000 [2:49:44<18:33:40,  1.53s/it]
Outer training loop:  12%|#2        | 6199/50000 [2:49:46<18:48:24,  1.55s/it]
Outer training loop:  12%|#2        | 6200/50000 [2:49:47<18:30:18,  1.52s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.50s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.50s/it]

Outer training loop:  12%|#2        | 6201/50000 [2:49:54<35:36:54,  2.93s/it]
Outer training loop:  12%|#2        | 6202/50000 [2:49:55<30:24:54,  2.50s/it]
Outer training loop:  12%|#2        | 6203/50000 [2:49:57<26:46:45,  2.20s/it]
Outer training loop:  12%|#2        | 6204/50000 [2:49:58<24:21:42,  2.00s/it]
Outer training loop:  12%|#2        | 6205/50000 [2:50:00<22:34:30,  1.86s/it]
Outer training loop:  12%|#2        | 6206/50000 [2:50:01<21:30:56,  1.77s/it]
Outer training loop:  12%|#2        | 6207/50000 [2:50:03<20:37:23,  1.70s/it]
Outer training loop:  12%|#2        | 6208/50000 [2:50:04<19:55:27,  1.64s/it]
Outer training loop:  12%|#2        | 6209/50000 [2:50:06<19:33:03,  1.61s/it]
Outer training loop:  12%|#2        | 6210/50000 [2:50:07<19:15:09,  1.58s/it]
Outer training loop:  12%|#2        | 6211/50000 [2:50:09<19:18:10,  1.59s/it]
Outer training loop:  12%|#2        | 6212/50000 [2:50:10<18:53:12,  1.55s/it]
Outer training loop:  12%|#2        | 6213/50000 [2:50:12<18:51:08,  1.55s/it]
Outer training loop:  12%|#2        | 6214/50000 [2:50:13<18:29:20,  1.52s/it]
Outer training loop:  12%|#2        | 6215/50000 [2:50:15<18:45:48,  1.54s/it]
Outer training loop:  12%|#2        | 6216/50000 [2:50:16<18:51:57,  1.55s/it]
Outer training loop:  12%|#2        | 6217/50000 [2:50:18<18:56:14,  1.56s/it]
Outer training loop:  12%|#2        | 6218/50000 [2:50:20<19:08:15,  1.57s/it]
Outer training loop:  12%|#2        | 6219/50000 [2:50:21<19:22:03,  1.59s/it]
Outer training loop:  12%|#2        | 6220/50000 [2:50:23<19:37:39,  1.61s/it]
Outer training loop:  12%|#2        | 6221/50000 [2:50:25<19:45:04,  1.62s/it]
Outer training loop:  12%|#2        | 6222/50000 [2:50:26<19:31:35,  1.61s/it]
Outer training loop:  12%|#2        | 6223/50000 [2:50:28<19:30:52,  1.60s/it]
Outer training loop:  12%|#2        | 6224/50000 [2:50:29<18:54:00,  1.55s/it]
Outer training loop:  12%|#2        | 6225/50000 [2:50:31<18:53:37,  1.55s/it]
Outer training loop:  12%|#2        | 6226/50000 [2:50:32<18:40:15,  1.54s/it]
Outer training loop:  12%|#2        | 6227/50000 [2:50:34<19:31:18,  1.61s/it]
Outer training loop:  12%|#2        | 6228/50000 [2:50:36<19:34:17,  1.61s/it]
Outer training loop:  12%|#2        | 6229/50000 [2:50:37<19:38:57,  1.62s/it]
Outer training loop:  12%|#2        | 6230/50000 [2:50:39<19:01:40,  1.57s/it]
Outer training loop:  12%|#2        | 6231/50000 [2:50:40<19:02:31,  1.57s/it]
Outer training loop:  12%|#2        | 6232/50000 [2:50:42<19:20:47,  1.59s/it]
Outer training loop:  12%|#2        | 6233/50000 [2:50:44<19:35:34,  1.61s/it]
Outer training loop:  12%|#2        | 6234/50000 [2:50:45<19:12:01,  1.58s/it]
Outer training loop:  12%|#2        | 6235/50000 [2:50:47<18:49:44,  1.55s/it]
Outer training loop:  12%|#2        | 6236/50000 [2:50:48<19:12:59,  1.58s/it]
Outer training loop:  12%|#2        | 6237/50000 [2:50:50<19:00:11,  1.56s/it]
Outer training loop:  12%|#2        | 6238/50000 [2:50:51<19:23:29,  1.60s/it]
Outer training loop:  12%|#2        | 6239/50000 [2:50:53<19:15:16,  1.58s/it]
Outer training loop:  12%|#2        | 6240/50000 [2:50:55<19:24:48,  1.60s/it]
Outer training loop:  12%|#2        | 6241/50000 [2:50:56<19:07:53,  1.57s/it]
Outer training loop:  12%|#2        | 6242/50000 [2:50:58<19:00:26,  1.56s/it]
Outer training loop:  12%|#2        | 6243/50000 [2:50:59<18:43:43,  1.54s/it]
Outer training loop:  12%|#2        | 6244/50000 [2:51:01<18:49:58,  1.55s/it]
Outer training loop:  12%|#2        | 6245/50000 [2:51:02<18:50:35,  1.55s/it]
Outer training loop:  12%|#2        | 6246/50000 [2:51:04<19:07:50,  1.57s/it]
Outer training loop:  12%|#2        | 6247/50000 [2:51:05<19:01:27,  1.57s/it]
Outer training loop:  12%|#2        | 6248/50000 [2:51:07<18:54:40,  1.56s/it]
Outer training loop:  12%|#2        | 6249/50000 [2:51:08<18:30:05,  1.52s/it]
Outer training loop:  12%|#2        | 6250/50000 [2:51:10<18:35:29,  1.53s/it]
Outer training loop:  13%|#2        | 6251/50000 [2:51:12<18:31:47,  1.52s/it]
Outer training loop:  13%|#2        | 6252/50000 [2:51:13<18:53:57,  1.56s/it]
Outer training loop:  13%|#2        | 6253/50000 [2:51:15<18:56:51,  1.56s/it]
Outer training loop:  13%|#2        | 6254/50000 [2:51:16<18:45:51,  1.54s/it]
Outer training loop:  13%|#2        | 6255/50000 [2:51:18<19:00:06,  1.56s/it]
Outer training loop:  13%|#2        | 6256/50000 [2:51:19<18:59:00,  1.56s/it]
Outer training loop:  13%|#2        | 6257/50000 [2:51:21<18:46:21,  1.54s/it]
Outer training loop:  13%|#2        | 6258/50000 [2:51:22<18:34:55,  1.53s/it]
Outer training loop:  13%|#2        | 6259/50000 [2:51:24<18:45:51,  1.54s/it]
Outer training loop:  13%|#2        | 6260/50000 [2:51:25<18:31:37,  1.52s/it]
Outer training loop:  13%|#2        | 6261/50000 [2:51:27<18:51:41,  1.55s/it]
Outer training loop:  13%|#2        | 6262/50000 [2:51:29<18:35:18,  1.53s/it]
Outer training loop:  13%|#2        | 6263/50000 [2:51:30<18:46:21,  1.55s/it]
Outer training loop:  13%|#2        | 6264/50000 [2:51:32<18:22:50,  1.51s/it]
Outer training loop:  13%|#2        | 6265/50000 [2:51:33<18:20:54,  1.51s/it]
Outer training loop:  13%|#2        | 6266/50000 [2:51:35<18:36:21,  1.53s/it]
Outer training loop:  13%|#2        | 6267/50000 [2:51:36<18:32:07,  1.53s/it]
Outer training loop:  13%|#2        | 6268/50000 [2:51:38<18:42:13,  1.54s/it]
Outer training loop:  13%|#2        | 6269/50000 [2:51:39<18:26:06,  1.52s/it]
Outer training loop:  13%|#2        | 6270/50000 [2:51:41<19:23:20,  1.60s/it]
Outer training loop:  13%|#2        | 6271/50000 [2:51:43<19:11:55,  1.58s/it]
Outer training loop:  13%|#2        | 6272/50000 [2:51:44<19:03:42,  1.57s/it]
Outer training loop:  13%|#2        | 6273/50000 [2:51:46<18:44:52,  1.54s/it]
Outer training loop:  13%|#2        | 6274/50000 [2:51:47<18:56:06,  1.56s/it]
Outer training loop:  13%|#2        | 6275/50000 [2:51:49<18:59:06,  1.56s/it]
Outer training loop:  13%|#2        | 6276/50000 [2:51:50<18:59:34,  1.56s/it]
Outer training loop:  13%|#2        | 6277/50000 [2:51:52<18:50:40,  1.55s/it]
Outer training loop:  13%|#2        | 6278/50000 [2:51:53<18:48:28,  1.55s/it]
Outer training loop:  13%|#2        | 6279/50000 [2:51:55<19:01:39,  1.57s/it]
Outer training loop:  13%|#2        | 6280/50000 [2:51:57<19:21:06,  1.59s/it]
Outer training loop:  13%|#2        | 6281/50000 [2:51:58<18:57:11,  1.56s/it]
Outer training loop:  13%|#2        | 6282/50000 [2:52:00<19:10:42,  1.58s/it]
Outer training loop:  13%|#2        | 6283/50000 [2:52:01<18:57:15,  1.56s/it]
Outer training loop:  13%|#2        | 6284/50000 [2:52:03<19:15:46,  1.59s/it]
Outer training loop:  13%|#2        | 6285/50000 [2:52:04<19:05:46,  1.57s/it]
Outer training loop:  13%|#2        | 6286/50000 [2:52:06<19:04:25,  1.57s/it]
Outer training loop:  13%|#2        | 6287/50000 [2:52:08<19:05:41,  1.57s/it]
Outer training loop:  13%|#2        | 6288/50000 [2:52:09<19:14:53,  1.59s/it]
Outer training loop:  13%|#2        | 6289/50000 [2:52:11<18:53:12,  1.56s/it]
Outer training loop:  13%|#2        | 6290/50000 [2:52:12<19:30:23,  1.61s/it]
Outer training loop:  13%|#2        | 6291/50000 [2:52:14<19:08:30,  1.58s/it]
Outer training loop:  13%|#2        | 6292/50000 [2:52:15<19:10:43,  1.58s/it]
Outer training loop:  13%|#2        | 6293/50000 [2:52:17<18:38:58,  1.54s/it]
Outer training loop:  13%|#2        | 6294/50000 [2:52:18<18:29:49,  1.52s/it]
Outer training loop:  13%|#2        | 6295/50000 [2:52:20<18:56:26,  1.56s/it]
Outer training loop:  13%|#2        | 6296/50000 [2:52:21<18:30:14,  1.52s/it]
Outer training loop:  13%|#2        | 6297/50000 [2:52:23<18:38:58,  1.54s/it]
Outer training loop:  13%|#2        | 6298/50000 [2:52:24<18:15:09,  1.50s/it]
Outer training loop:  13%|#2        | 6299/50000 [2:52:26<18:31:52,  1.53s/it]
Outer training loop:  13%|#2        | 6300/50000 [2:52:28<18:17:16,  1.51s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.49s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.50s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.49s/it]

Outer training loop:  13%|#2        | 6301/50000 [2:52:34<34:58:31,  2.88s/it]
Outer training loop:  13%|#2        | 6302/50000 [2:52:35<30:20:14,  2.50s/it]
Outer training loop:  13%|#2        | 6303/50000 [2:52:37<26:49:34,  2.21s/it]
Outer training loop:  13%|#2        | 6304/50000 [2:52:38<24:58:00,  2.06s/it]
Outer training loop:  13%|#2        | 6305/50000 [2:52:40<23:00:06,  1.90s/it]
Outer training loop:  13%|#2        | 6306/50000 [2:52:42<21:45:52,  1.79s/it]
Outer training loop:  13%|#2        | 6307/50000 [2:52:43<20:50:47,  1.72s/it]
Outer training loop:  13%|#2        | 6308/50000 [2:52:45<20:16:28,  1.67s/it]
Outer training loop:  13%|#2        | 6309/50000 [2:52:46<19:52:05,  1.64s/it]
Outer training loop:  13%|#2        | 6310/50000 [2:52:48<19:37:39,  1.62s/it]
Outer training loop:  13%|#2        | 6311/50000 [2:52:49<19:47:39,  1.63s/it]
Outer training loop:  13%|#2        | 6312/50000 [2:52:51<19:39:02,  1.62s/it]
Outer training loop:  13%|#2        | 6313/50000 [2:52:53<19:13:38,  1.58s/it]
Outer training loop:  13%|#2        | 6314/50000 [2:52:54<19:16:02,  1.59s/it]
Outer training loop:  13%|#2        | 6315/50000 [2:52:56<18:53:42,  1.56s/it]
Outer training loop:  13%|#2        | 6316/50000 [2:52:57<19:07:49,  1.58s/it]
Outer training loop:  13%|#2        | 6317/50000 [2:52:59<19:24:12,  1.60s/it]
Outer training loop:  13%|#2        | 6318/50000 [2:53:00<18:39:26,  1.54s/it]
Outer training loop:  13%|#2        | 6319/50000 [2:53:02<19:07:58,  1.58s/it]
Outer training loop:  13%|#2        | 6320/50000 [2:53:03<18:59:00,  1.56s/it]
Outer training loop:  13%|#2        | 6321/50000 [2:53:05<18:56:13,  1.56s/it]
Outer training loop:  13%|#2        | 6322/50000 [2:53:07<19:09:00,  1.58s/it]
Outer training loop:  13%|#2        | 6323/50000 [2:53:08<19:26:22,  1.60s/it]
Outer training loop:  13%|#2        | 6324/50000 [2:53:10<19:10:57,  1.58s/it]
Outer training loop:  13%|#2        | 6325/50000 [2:53:11<19:22:48,  1.60s/it]
Outer training loop:  13%|#2        | 6326/50000 [2:53:13<19:03:32,  1.57s/it]
Outer training loop:  13%|#2        | 6327/50000 [2:53:15<19:10:14,  1.58s/it]
Outer training loop:  13%|#2        | 6328/50000 [2:53:16<18:57:19,  1.56s/it]
Outer training loop:  13%|#2        | 6329/50000 [2:53:18<19:12:38,  1.58s/it]
Outer training loop:  13%|#2        | 6330/50000 [2:53:19<19:14:50,  1.59s/it]
Outer training loop:  13%|#2        | 6331/50000 [2:53:21<19:26:37,  1.60s/it]
Outer training loop:  13%|#2        | 6332/50000 [2:53:23<19:18:48,  1.59s/it]
Outer training loop:  13%|#2        | 6333/50000 [2:53:24<19:15:01,  1.59s/it]
Outer training loop:  13%|#2        | 6334/50000 [2:53:26<19:04:31,  1.57s/it]
Outer training loop:  13%|#2        | 6335/50000 [2:53:27<19:04:42,  1.57s/it]
Outer training loop:  13%|#2        | 6336/50000 [2:53:29<19:04:50,  1.57s/it]
Outer training loop:  13%|#2        | 6337/50000 [2:53:30<18:56:10,  1.56s/it]
Outer training loop:  13%|#2        | 6338/50000 [2:53:32<18:28:20,  1.52s/it]
Outer training loop:  13%|#2        | 6339/50000 [2:53:33<18:58:02,  1.56s/it]
Outer training loop:  13%|#2        | 6340/50000 [2:53:35<18:55:02,  1.56s/it]
Outer training loop:  13%|#2        | 6341/50000 [2:53:37<19:04:23,  1.57s/it]
Outer training loop:  13%|#2        | 6342/50000 [2:53:38<18:27:04,  1.52s/it]
Outer training loop:  13%|#2        | 6343/50000 [2:53:40<18:41:04,  1.54s/it]
Outer training loop:  13%|#2        | 6344/50000 [2:53:41<18:34:27,  1.53s/it]
Outer training loop:  13%|#2        | 6345/50000 [2:53:43<18:41:01,  1.54s/it]
Outer training loop:  13%|#2        | 6346/50000 [2:53:44<18:13:31,  1.50s/it]
Outer training loop:  13%|#2        | 6347/50000 [2:53:46<18:25:56,  1.52s/it]
Outer training loop:  13%|#2        | 6348/50000 [2:53:47<18:16:46,  1.51s/it]
Outer training loop:  13%|#2        | 6349/50000 [2:53:49<18:22:42,  1.52s/it]
Outer training loop:  13%|#2        | 6350/50000 [2:53:50<18:37:46,  1.54s/it]
Outer training loop:  13%|#2        | 6351/50000 [2:53:52<18:59:30,  1.57s/it]
Outer training loop:  13%|#2        | 6352/50000 [2:53:53<18:37:52,  1.54s/it]
Outer training loop:  13%|#2        | 6353/50000 [2:53:55<18:50:09,  1.55s/it]
Outer training loop:  13%|#2        | 6354/50000 [2:53:56<18:36:49,  1.54s/it]
Outer training loop:  13%|#2        | 6355/50000 [2:53:58<18:45:27,  1.55s/it]
Outer training loop:  13%|#2        | 6356/50000 [2:53:59<18:32:21,  1.53s/it]
Outer training loop:  13%|#2        | 6357/50000 [2:54:01<18:53:40,  1.56s/it]
Outer training loop:  13%|#2        | 6358/50000 [2:54:03<18:32:08,  1.53s/it]
Outer training loop:  13%|#2        | 6359/50000 [2:54:04<18:35:12,  1.53s/it]
Outer training loop:  13%|#2        | 6360/50000 [2:54:06<18:28:53,  1.52s/it]
Outer training loop:  13%|#2        | 6361/50000 [2:54:07<18:40:01,  1.54s/it]
Outer training loop:  13%|#2        | 6362/50000 [2:54:09<18:45:15,  1.55s/it]
Outer training loop:  13%|#2        | 6363/50000 [2:54:10<19:19:31,  1.59s/it]
Outer training loop:  13%|#2        | 6364/50000 [2:54:12<18:56:33,  1.56s/it]
Outer training loop:  13%|#2        | 6365/50000 [2:54:13<18:52:02,  1.56s/it]
Outer training loop:  13%|#2        | 6366/50000 [2:54:15<18:39:22,  1.54s/it]
Outer training loop:  13%|#2        | 6367/50000 [2:54:17<19:33:52,  1.61s/it]
Outer training loop:  13%|#2        | 6368/50000 [2:54:18<19:06:05,  1.58s/it]
Outer training loop:  13%|#2        | 6369/50000 [2:54:20<19:03:44,  1.57s/it]
Outer training loop:  13%|#2        | 6370/50000 [2:54:21<18:55:14,  1.56s/it]
Outer training loop:  13%|#2        | 6371/50000 [2:54:23<19:23:51,  1.60s/it]
Outer training loop:  13%|#2        | 6372/50000 [2:54:25<19:17:03,  1.59s/it]
Outer training loop:  13%|#2        | 6373/50000 [2:54:26<19:29:54,  1.61s/it]
Outer training loop:  13%|#2        | 6374/50000 [2:54:28<19:12:23,  1.58s/it]
Outer training loop:  13%|#2        | 6375/50000 [2:54:29<19:02:33,  1.57s/it]
Outer training loop:  13%|#2        | 6376/50000 [2:54:31<18:39:37,  1.54s/it]
Outer training loop:  13%|#2        | 6377/50000 [2:54:32<19:11:44,  1.58s/it]
Outer training loop:  13%|#2        | 6378/50000 [2:54:34<19:09:24,  1.58s/it]
Outer training loop:  13%|#2        | 6379/50000 [2:54:36<19:10:42,  1.58s/it]
Outer training loop:  13%|#2        | 6380/50000 [2:54:37<18:54:40,  1.56s/it]
Outer training loop:  13%|#2        | 6381/50000 [2:54:39<19:17:35,  1.59s/it]
Outer training loop:  13%|#2        | 6382/50000 [2:54:40<19:10:24,  1.58s/it]
Outer training loop:  13%|#2        | 6383/50000 [2:54:42<19:13:52,  1.59s/it]
Outer training loop:  13%|#2        | 6384/50000 [2:54:44<19:09:42,  1.58s/it]
Outer training loop:  13%|#2        | 6385/50000 [2:54:45<18:58:04,  1.57s/it]
Outer training loop:  13%|#2        | 6386/50000 [2:54:47<18:39:23,  1.54s/it]
Outer training loop:  13%|#2        | 6387/50000 [2:54:48<18:35:39,  1.53s/it]
Outer training loop:  13%|#2        | 6388/50000 [2:54:50<18:40:03,  1.54s/it]
Outer training loop:  13%|#2        | 6389/50000 [2:54:51<18:50:33,  1.56s/it]
Outer training loop:  13%|#2        | 6390/50000 [2:54:53<18:35:49,  1.54s/it]
Outer training loop:  13%|#2        | 6391/50000 [2:54:54<18:54:34,  1.56s/it]
Outer training loop:  13%|#2        | 6392/50000 [2:54:56<18:53:01,  1.56s/it]
Outer training loop:  13%|#2        | 6393/50000 [2:54:57<18:47:52,  1.55s/it]
Outer training loop:  13%|#2        | 6394/50000 [2:54:59<18:34:29,  1.53s/it]
Outer training loop:  13%|#2        | 6395/50000 [2:55:00<18:34:39,  1.53s/it]
Outer training loop:  13%|#2        | 6396/50000 [2:55:02<18:35:32,  1.53s/it]
Outer training loop:  13%|#2        | 6397/50000 [2:55:04<18:58:56,  1.57s/it]
Outer training loop:  13%|#2        | 6398/50000 [2:55:05<18:55:25,  1.56s/it]
Outer training loop:  13%|#2        | 6399/50000 [2:55:07<18:39:11,  1.54s/it]
Outer training loop:  13%|#2        | 6400/50000 [2:55:08<19:01:36,  1.57s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.49s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.49s/it]

Outer training loop:  13%|#2        | 6401/50000 [2:55:14<35:19:12,  2.92s/it]
Outer training loop:  13%|#2        | 6402/50000 [2:55:16<30:10:57,  2.49s/it]
Outer training loop:  13%|#2        | 6403/50000 [2:55:17<26:44:31,  2.21s/it]
Outer training loop:  13%|#2        | 6404/50000 [2:55:19<24:15:23,  2.00s/it]
Outer training loop:  13%|#2        | 6405/50000 [2:55:21<22:55:33,  1.89s/it]
Outer training loop:  13%|#2        | 6406/50000 [2:55:22<21:27:16,  1.77s/it]
Outer training loop:  13%|#2        | 6407/50000 [2:55:24<20:45:19,  1.71s/it]
Outer training loop:  13%|#2        | 6408/50000 [2:55:25<20:13:23,  1.67s/it]
Outer training loop:  13%|#2        | 6409/50000 [2:55:27<19:56:36,  1.65s/it]
Outer training loop:  13%|#2        | 6410/50000 [2:55:28<19:31:36,  1.61s/it]
Outer training loop:  13%|#2        | 6411/50000 [2:55:30<19:43:30,  1.63s/it]
Outer training loop:  13%|#2        | 6412/50000 [2:55:31<19:13:47,  1.59s/it]
Outer training loop:  13%|#2        | 6413/50000 [2:55:33<19:39:37,  1.62s/it]
Outer training loop:  13%|#2        | 6414/50000 [2:55:35<20:02:17,  1.66s/it]
Outer training loop:  13%|#2        | 6415/50000 [2:55:36<19:35:53,  1.62s/it]
Outer training loop:  13%|#2        | 6416/50000 [2:55:38<19:22:57,  1.60s/it]
Outer training loop:  13%|#2        | 6417/50000 [2:55:40<19:25:42,  1.60s/it]
Outer training loop:  13%|#2        | 6418/50000 [2:55:41<19:14:30,  1.59s/it]
Outer training loop:  13%|#2        | 6419/50000 [2:55:43<19:00:38,  1.57s/it]
Outer training loop:  13%|#2        | 6420/50000 [2:55:44<18:46:02,  1.55s/it]
Outer training loop:  13%|#2        | 6421/50000 [2:55:46<19:04:18,  1.58s/it]
Outer training loop:  13%|#2        | 6422/50000 [2:55:47<18:58:10,  1.57s/it]
Outer training loop:  13%|#2        | 6423/50000 [2:55:49<19:07:31,  1.58s/it]
Outer training loop:  13%|#2        | 6424/50000 [2:55:51<18:51:09,  1.56s/it]
Outer training loop:  13%|#2        | 6425/50000 [2:55:52<18:55:32,  1.56s/it]
Outer training loop:  13%|#2        | 6426/50000 [2:55:54<18:40:25,  1.54s/it]
Outer training loop:  13%|#2        | 6427/50000 [2:55:55<19:08:51,  1.58s/it]
Outer training loop:  13%|#2        | 6428/50000 [2:55:57<18:46:53,  1.55s/it]
Outer training loop:  13%|#2        | 6429/50000 [2:55:58<18:56:26,  1.56s/it]
Outer training loop:  13%|#2        | 6430/50000 [2:56:00<18:52:56,  1.56s/it]
Outer training loop:  13%|#2        | 6431/50000 [2:56:01<18:56:14,  1.56s/it]
Outer training loop:  13%|#2        | 6432/50000 [2:56:03<18:57:34,  1.57s/it]
Outer training loop:  13%|#2        | 6433/50000 [2:56:05<19:04:25,  1.58s/it]
Outer training loop:  13%|#2        | 6434/50000 [2:56:06<18:44:43,  1.55s/it]
Outer training loop:  13%|#2        | 6435/50000 [2:56:08<19:00:26,  1.57s/it]
Outer training loop:  13%|#2        | 6436/50000 [2:56:09<18:48:29,  1.55s/it]
Outer training loop:  13%|#2        | 6437/50000 [2:56:11<18:52:35,  1.56s/it]
Outer training loop:  13%|#2        | 6438/50000 [2:56:12<18:38:27,  1.54s/it]
Outer training loop:  13%|#2        | 6439/50000 [2:56:14<18:45:37,  1.55s/it]
Outer training loop:  13%|#2        | 6440/50000 [2:56:15<18:33:59,  1.53s/it]
Outer training loop:  13%|#2        | 6441/50000 [2:56:17<18:46:36,  1.55s/it]
Outer training loop:  13%|#2        | 6442/50000 [2:56:18<18:38:00,  1.54s/it]
Outer training loop:  13%|#2        | 6443/50000 [2:56:20<18:50:42,  1.56s/it]
Outer training loop:  13%|#2        | 6444/50000 [2:56:22<18:43:15,  1.55s/it]
Outer training loop:  13%|#2        | 6445/50000 [2:56:23<19:01:52,  1.57s/it]
Outer training loop:  13%|#2        | 6446/50000 [2:56:25<19:02:14,  1.57s/it]
Outer training loop:  13%|#2        | 6447/50000 [2:56:26<19:05:52,  1.58s/it]
Outer training loop:  13%|#2        | 6448/50000 [2:56:28<18:41:29,  1.55s/it]
Outer training loop:  13%|#2        | 6449/50000 [2:56:29<18:52:08,  1.56s/it]
Outer training loop:  13%|#2        | 6450/50000 [2:56:31<18:39:29,  1.54s/it]
Outer training loop:  13%|#2        | 6451/50000 [2:56:33<19:02:10,  1.57s/it]
Outer training loop:  13%|#2        | 6452/50000 [2:56:34<18:52:29,  1.56s/it]
Outer training loop:  13%|#2        | 6453/50000 [2:56:36<18:47:07,  1.55s/it]
Outer training loop:  13%|#2        | 6454/50000 [2:56:37<18:45:14,  1.55s/it]
Outer training loop:  13%|#2        | 6455/50000 [2:56:39<18:33:14,  1.53s/it]
Outer training loop:  13%|#2        | 6456/50000 [2:56:40<18:33:08,  1.53s/it]
Outer training loop:  13%|#2        | 6457/50000 [2:56:42<18:50:35,  1.56s/it]
Outer training loop:  13%|#2        | 6458/50000 [2:56:43<18:44:14,  1.55s/it]
Outer training loop:  13%|#2        | 6459/50000 [2:56:45<18:48:36,  1.56s/it]
Outer training loop:  13%|#2        | 6460/50000 [2:56:47<19:00:49,  1.57s/it]
Outer training loop:  13%|#2        | 6461/50000 [2:56:48<19:22:05,  1.60s/it]
Outer training loop:  13%|#2        | 6462/50000 [2:56:50<19:18:06,  1.60s/it]
Outer training loop:  13%|#2        | 6463/50000 [2:56:51<19:18:44,  1.60s/it]
Outer training loop:  13%|#2        | 6464/50000 [2:56:53<19:47:26,  1.64s/it]
Outer training loop:  13%|#2        | 6465/50000 [2:56:55<19:16:02,  1.59s/it]
Outer training loop:  13%|#2        | 6466/50000 [2:56:56<18:52:17,  1.56s/it]
Outer training loop:  13%|#2        | 6467/50000 [2:56:58<18:58:26,  1.57s/it]
Outer training loop:  13%|#2        | 6468/50000 [2:56:59<18:29:34,  1.53s/it]
Outer training loop:  13%|#2        | 6469/50000 [2:57:01<18:23:12,  1.52s/it]
Outer training loop:  13%|#2        | 6470/50000 [2:57:02<18:35:38,  1.54s/it]
Outer training loop:  13%|#2        | 6471/50000 [2:57:04<19:00:03,  1.57s/it]
Outer training loop:  13%|#2        | 6472/50000 [2:57:05<18:50:39,  1.56s/it]
Outer training loop:  13%|#2        | 6473/50000 [2:57:07<18:47:54,  1.55s/it]
Outer training loop:  13%|#2        | 6474/50000 [2:57:08<18:42:29,  1.55s/it]
Outer training loop:  13%|#2        | 6475/50000 [2:57:10<18:54:19,  1.56s/it]
Outer training loop:  13%|#2        | 6476/50000 [2:57:12<18:38:58,  1.54s/it]
Outer training loop:  13%|#2        | 6477/50000 [2:57:13<19:14:38,  1.59s/it]
Outer training loop:  13%|#2        | 6478/50000 [2:57:15<19:04:24,  1.58s/it]
Outer training loop:  13%|#2        | 6479/50000 [2:57:16<19:18:17,  1.60s/it]
Outer training loop:  13%|#2        | 6480/50000 [2:57:18<18:46:54,  1.55s/it]
Outer training loop:  13%|#2        | 6481/50000 [2:57:20<18:59:17,  1.57s/it]
Outer training loop:  13%|#2        | 6482/50000 [2:57:21<18:42:37,  1.55s/it]
Outer training loop:  13%|#2        | 6483/50000 [2:57:23<18:56:00,  1.57s/it]
Outer training loop:  13%|#2        | 6484/50000 [2:57:24<18:28:56,  1.53s/it]
Outer training loop:  13%|#2        | 6485/50000 [2:57:26<18:20:57,  1.52s/it]
Outer training loop:  13%|#2        | 6486/50000 [2:57:27<18:18:41,  1.51s/it]
Outer training loop:  13%|#2        | 6487/50000 [2:57:29<18:43:09,  1.55s/it]
Outer training loop:  13%|#2        | 6488/50000 [2:57:30<18:23:39,  1.52s/it]
Outer training loop:  13%|#2        | 6489/50000 [2:57:32<18:47:51,  1.56s/it]
Outer training loop:  13%|#2        | 6490/50000 [2:57:33<18:42:46,  1.55s/it]
Outer training loop:  13%|#2        | 6491/50000 [2:57:35<19:14:41,  1.59s/it]
Outer training loop:  13%|#2        | 6492/50000 [2:57:37<19:17:24,  1.60s/it]
Outer training loop:  13%|#2        | 6493/50000 [2:57:38<18:41:18,  1.55s/it]
Outer training loop:  13%|#2        | 6494/50000 [2:57:40<19:01:04,  1.57s/it]
Outer training loop:  13%|#2        | 6495/50000 [2:57:41<19:05:58,  1.58s/it]
Outer training loop:  13%|#2        | 6496/50000 [2:57:43<18:51:10,  1.56s/it]
Outer training loop:  13%|#2        | 6497/50000 [2:57:44<18:30:03,  1.53s/it]
Outer training loop:  13%|#2        | 6498/50000 [2:57:46<18:43:05,  1.55s/it]
Outer training loop:  13%|#2        | 6499/50000 [2:57:48<19:03:07,  1.58s/it]
Outer training loop:  13%|#3        | 6500/50000 [2:57:49<19:00:48,  1.57s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.51s/it]

Evaluating policy:  67%|######6   | 2/3 [00:02<00:01,  1.50s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.50s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:02,  1.49s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:03,  1.52s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:00,  1.51s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:01,  1.54s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<01:59,  1.53s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<01:57,  1.53s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:59,  1.57s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:56,  1.55s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:57,  1.59s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:56,  1.60s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:57,  1.63s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:53,  1.60s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:49,  1.56s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:45,  1.53s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:46,  1.57s/it]

Evaluating policy:  19%|#9        | 16/83 [00:24<01:42,  1.53s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:40,  1.52s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:39,  1.54s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:39,  1.55s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:38,  1.57s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:38,  1.58s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:35,  1.56s/it]

Evaluating policy:  28%|##7       | 23/83 [00:35<01:33,  1.55s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:32,  1.57s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:34,  1.63s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:31,  1.61s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:28,  1.58s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:25,  1.55s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:25,  1.57s/it]

Evaluating policy:  36%|###6      | 30/83 [00:46<01:22,  1.56s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:20,  1.54s/it]

Evaluating policy:  39%|###8      | 32/83 [00:49<01:18,  1.54s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:20,  1.60s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:16,  1.56s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:13,  1.54s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:10,  1.51s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:12,  1.57s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:09,  1.54s/it]

Evaluating policy:  47%|####6     | 39/83 [01:00<01:08,  1.55s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:05,  1.52s/it]

Evaluating policy:  49%|####9     | 41/83 [01:03<01:02,  1.50s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:00,  1.49s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:02,  1.57s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<01:00,  1.56s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:09<00:57,  1.52s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:11<00:55,  1.50s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:12<00:53,  1.49s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:14<00:52,  1.50s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:54,  1.59s/it]

Evaluating policy:  60%|######    | 50/83 [01:17<00:52,  1.58s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:49,  1.55s/it]

Evaluating policy:  63%|######2   | 52/83 [01:20<00:47,  1.54s/it]

Evaluating policy:  64%|######3   | 53/83 [01:22<00:45,  1.51s/it]

Evaluating policy:  65%|######5   | 54/83 [01:23<00:43,  1.50s/it]

Evaluating policy:  66%|######6   | 55/83 [01:25<00:43,  1.57s/it]

Evaluating policy:  67%|######7   | 56/83 [01:26<00:42,  1.57s/it]

Evaluating policy:  69%|######8   | 57/83 [01:28<00:40,  1.56s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:39,  1.57s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:31<00:36,  1.53s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:33<00:36,  1.58s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:36,  1.66s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:36<00:33,  1.61s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:31,  1.58s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:39<00:29,  1.58s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:27,  1.55s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:42<00:26,  1.55s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:24,  1.55s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:24,  1.66s/it]

Evaluating policy:  83%|########3 | 69/83 [01:47<00:22,  1.61s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:20,  1.59s/it]

Evaluating policy:  86%|########5 | 71/83 [01:50<00:18,  1.55s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:16,  1.52s/it]

Evaluating policy:  88%|########7 | 73/83 [01:53<00:15,  1.53s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:13,  1.53s/it]

Evaluating policy:  90%|######### | 75/83 [01:56<00:12,  1.52s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:11,  1.64s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.62s/it]

Evaluating policy:  94%|#########3| 78/83 [02:01<00:07,  1.59s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.59s/it]

Evaluating policy:  96%|#########6| 80/83 [02:04<00:04,  1.57s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.55s/it]

Evaluating policy:  99%|#########8| 82/83 [02:07<00:01,  1.55s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.54s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:34,  1.88s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.77s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:17,  1.72s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:10,  1.65s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:06,  1.62s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:02,  1.59s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<02:02,  1.62s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:59,  1.59s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:57,  1.59s/it]

Evaluating policy:  12%|#2        | 10/83 [00:16<01:58,  1.63s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:55,  1.60s/it]

Evaluating policy:  14%|#4        | 12/83 [00:19<01:51,  1.57s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:51,  1.59s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:48,  1.57s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:44,  1.53s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:46,  1.59s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:44,  1.58s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:42,  1.57s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:43,  1.62s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:40,  1.59s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:36,  1.55s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:37,  1.59s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:35,  1.58s/it]

Evaluating policy:  29%|##8       | 24/83 [00:38<01:32,  1.56s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:33,  1.61s/it]

Evaluating policy:  31%|###1      | 26/83 [00:41<01:30,  1.59s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:30,  1.61s/it]

Evaluating policy:  34%|###3      | 28/83 [00:44<01:27,  1.59s/it]

Evaluating policy:  35%|###4      | 29/83 [00:46<01:26,  1.60s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:24,  1.59s/it]

Evaluating policy:  37%|###7      | 31/83 [00:49<01:20,  1.55s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:19,  1.55s/it]

Evaluating policy:  40%|###9      | 33/83 [00:52<01:16,  1.53s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:18,  1.60s/it]

Evaluating policy:  42%|####2     | 35/83 [00:55<01:14,  1.56s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:12,  1.53s/it]

Evaluating policy:  45%|####4     | 37/83 [00:58<01:10,  1.53s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:08,  1.52s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:09,  1.58s/it]

Evaluating policy:  48%|####8     | 40/83 [01:03<01:07,  1.58s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:05,  1.56s/it]

Evaluating policy:  51%|#####     | 42/83 [01:06<01:03,  1.55s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:08<01:04,  1.62s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:09<01:02,  1.59s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:11<00:59,  1.57s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<00:57,  1.55s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:14<00:55,  1.53s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:56,  1.60s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:17<00:53,  1.57s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:50,  1.53s/it]

Evaluating policy:  61%|######1   | 51/83 [01:20<00:49,  1.54s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:47,  1.52s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:45,  1.52s/it]

Evaluating policy:  65%|######5   | 54/83 [01:25<00:46,  1.59s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:43,  1.56s/it]

Evaluating policy:  67%|######7   | 56/83 [01:28<00:41,  1.55s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:40,  1.56s/it]

Evaluating policy:  70%|######9   | 58/83 [01:31<00:39,  1.57s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:36,  1.53s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:34<00:37,  1.62s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:36<00:35,  1.61s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:37<00:33,  1.59s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:39<00:31,  1.55s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:29,  1.54s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:42<00:27,  1.51s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:25,  1.51s/it]

Evaluating policy:  81%|########  | 67/83 [01:45<00:26,  1.64s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:23,  1.59s/it]

Evaluating policy:  83%|########3 | 69/83 [01:48<00:21,  1.57s/it]

Evaluating policy:  84%|########4 | 70/83 [01:50<00:20,  1.55s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:18,  1.53s/it]

Evaluating policy:  87%|########6 | 72/83 [01:53<00:16,  1.52s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:15,  1.51s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:13,  1.51s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:13,  1.63s/it]

Evaluating policy:  92%|#########1| 76/83 [01:59<00:11,  1.59s/it]

Evaluating policy:  93%|#########2| 77/83 [02:01<00:09,  1.61s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.56s/it]

Evaluating policy:  95%|#########5| 79/83 [02:04<00:06,  1.55s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.55s/it]

Evaluating policy:  98%|#########7| 81/83 [02:07<00:03,  1.53s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.52s/it]

Evaluating policy: 100%|##########| 83/83 [02:10<00:00,  1.49s/it]

Outer training loop:  13%|#3        | 6501/50000 [3:02:15<976:15:55, 80.80s/it]
Outer training loop:  13%|#3        | 6502/50000 [3:02:17<690:28:11, 57.14s/it]
Outer training loop:  13%|#3        | 6503/50000 [3:02:18<488:59:09, 40.47s/it]
Outer training loop:  13%|#3        | 6504/50000 [3:02:20<348:39:14, 28.86s/it]
Outer training loop:  13%|#3        | 6505/50000 [3:02:22<249:45:39, 20.67s/it]
Outer training loop:  13%|#3        | 6506/50000 [3:02:23<180:21:27, 14.93s/it]
Outer training loop:  13%|#3        | 6507/50000 [3:02:25<131:42:28, 10.90s/it]
Outer training loop:  13%|#3        | 6508/50000 [3:02:26<97:52:28,  8.10s/it] 
Outer training loop:  13%|#3        | 6509/50000 [3:02:28<74:06:42,  6.13s/it]
Outer training loop:  13%|#3        | 6510/50000 [3:02:29<57:31:09,  4.76s/it]
Outer training loop:  13%|#3        | 6511/50000 [3:02:31<45:52:37,  3.80s/it]
Outer training loop:  13%|#3        | 6512/50000 [3:02:32<37:35:52,  3.11s/it]
Outer training loop:  13%|#3        | 6513/50000 [3:02:34<31:50:48,  2.64s/it]
Outer training loop:  13%|#3        | 6514/50000 [3:02:36<28:12:56,  2.34s/it]
Outer training loop:  13%|#3        | 6515/50000 [3:02:37<25:18:21,  2.10s/it]
Outer training loop:  13%|#3        | 6516/50000 [3:02:39<23:22:03,  1.93s/it]
Outer training loop:  13%|#3        | 6517/50000 [3:02:40<21:50:36,  1.81s/it]
Outer training loop:  13%|#3        | 6518/50000 [3:02:42<21:21:49,  1.77s/it]
Outer training loop:  13%|#3        | 6519/50000 [3:02:43<20:41:24,  1.71s/it]
Outer training loop:  13%|#3        | 6520/50000 [3:02:45<20:19:35,  1.68s/it]
Outer training loop:  13%|#3        | 6521/50000 [3:02:47<19:45:22,  1.64s/it]
Outer training loop:  13%|#3        | 6522/50000 [3:02:48<19:43:24,  1.63s/it]
Outer training loop:  13%|#3        | 6523/50000 [3:02:50<19:33:00,  1.62s/it]
Outer training loop:  13%|#3        | 6524/50000 [3:02:51<19:40:36,  1.63s/it]
Outer training loop:  13%|#3        | 6525/50000 [3:02:53<19:20:37,  1.60s/it]
Outer training loop:  13%|#3        | 6526/50000 [3:02:54<19:08:56,  1.59s/it]
Outer training loop:  13%|#3        | 6527/50000 [3:02:56<19:24:18,  1.61s/it]
Outer training loop:  13%|#3        | 6528/50000 [3:02:58<19:02:14,  1.58s/it]
Outer training loop:  13%|#3        | 6529/50000 [3:02:59<18:53:33,  1.56s/it]
Outer training loop:  13%|#3        | 6530/50000 [3:03:01<18:48:30,  1.56s/it]
Outer training loop:  13%|#3        | 6531/50000 [3:03:02<19:04:04,  1.58s/it]
Outer training loop:  13%|#3        | 6532/50000 [3:03:04<18:41:08,  1.55s/it]
Outer training loop:  13%|#3        | 6533/50000 [3:03:05<19:03:53,  1.58s/it]
Outer training loop:  13%|#3        | 6534/50000 [3:03:07<18:56:24,  1.57s/it]
Outer training loop:  13%|#3        | 6535/50000 [3:03:09<19:07:10,  1.58s/it]
Outer training loop:  13%|#3        | 6536/50000 [3:03:10<19:20:20,  1.60s/it]
Outer training loop:  13%|#3        | 6537/50000 [3:03:12<19:19:10,  1.60s/it]
Outer training loop:  13%|#3        | 6538/50000 [3:03:13<19:16:45,  1.60s/it]
Outer training loop:  13%|#3        | 6539/50000 [3:03:15<19:13:25,  1.59s/it]
Outer training loop:  13%|#3        | 6540/50000 [3:03:17<18:59:12,  1.57s/it]
Outer training loop:  13%|#3        | 6541/50000 [3:03:18<19:02:03,  1.58s/it]
Outer training loop:  13%|#3        | 6542/50000 [3:03:20<18:51:05,  1.56s/it]
Outer training loop:  13%|#3        | 6543/50000 [3:03:21<19:03:58,  1.58s/it]
Outer training loop:  13%|#3        | 6544/50000 [3:03:23<19:15:20,  1.60s/it]
Outer training loop:  13%|#3        | 6545/50000 [3:03:25<19:40:43,  1.63s/it]
Outer training loop:  13%|#3        | 6546/50000 [3:03:26<19:16:43,  1.60s/it]
Outer training loop:  13%|#3        | 6547/50000 [3:03:28<19:18:58,  1.60s/it]
Outer training loop:  13%|#3        | 6548/50000 [3:03:29<19:14:32,  1.59s/it]
Outer training loop:  13%|#3        | 6549/50000 [3:03:31<19:12:11,  1.59s/it]
Outer training loop:  13%|#3        | 6550/50000 [3:03:32<19:04:15,  1.58s/it]
Outer training loop:  13%|#3        | 6551/50000 [3:03:34<19:16:47,  1.60s/it]
Outer training loop:  13%|#3        | 6552/50000 [3:03:36<19:24:14,  1.61s/it]
Outer training loop:  13%|#3        | 6553/50000 [3:03:37<18:52:56,  1.56s/it]
Outer training loop:  13%|#3        | 6554/50000 [3:03:39<19:19:27,  1.60s/it]
Outer training loop:  13%|#3        | 6555/50000 [3:03:40<19:11:58,  1.59s/it]
Outer training loop:  13%|#3        | 6556/50000 [3:03:42<19:04:00,  1.58s/it]
Outer training loop:  13%|#3        | 6557/50000 [3:03:44<19:05:04,  1.58s/it]
Outer training loop:  13%|#3        | 6558/50000 [3:03:45<19:20:36,  1.60s/it]
Outer training loop:  13%|#3        | 6559/50000 [3:03:47<19:07:51,  1.59s/it]
Outer training loop:  13%|#3        | 6560/50000 [3:03:48<19:04:22,  1.58s/it]
Outer training loop:  13%|#3        | 6561/50000 [3:03:50<19:20:55,  1.60s/it]
Outer training loop:  13%|#3        | 6562/50000 [3:03:52<19:20:39,  1.60s/it]
Outer training loop:  13%|#3        | 6563/50000 [3:03:53<19:12:29,  1.59s/it]
Outer training loop:  13%|#3        | 6564/50000 [3:03:55<19:04:16,  1.58s/it]
Outer training loop:  13%|#3        | 6565/50000 [3:03:56<18:40:20,  1.55s/it]
Outer training loop:  13%|#3        | 6566/50000 [3:03:58<18:52:46,  1.56s/it]
Outer training loop:  13%|#3        | 6567/50000 [3:03:59<18:38:00,  1.54s/it]
Outer training loop:  13%|#3        | 6568/50000 [3:04:01<18:57:47,  1.57s/it]
Outer training loop:  13%|#3        | 6569/50000 [3:04:02<18:48:53,  1.56s/it]
Outer training loop:  13%|#3        | 6570/50000 [3:04:04<19:14:29,  1.59s/it]
Outer training loop:  13%|#3        | 6571/50000 [3:04:06<19:01:42,  1.58s/it]
Outer training loop:  13%|#3        | 6572/50000 [3:04:07<19:00:03,  1.58s/it]
Outer training loop:  13%|#3        | 6573/50000 [3:04:09<18:43:04,  1.55s/it]
Outer training loop:  13%|#3        | 6574/50000 [3:04:10<19:05:32,  1.58s/it]
Outer training loop:  13%|#3        | 6575/50000 [3:04:12<18:34:16,  1.54s/it]
Outer training loop:  13%|#3        | 6576/50000 [3:04:13<18:31:44,  1.54s/it]
Outer training loop:  13%|#3        | 6577/50000 [3:04:15<18:21:59,  1.52s/it]
Outer training loop:  13%|#3        | 6578/50000 [3:04:17<18:46:01,  1.56s/it]
Outer training loop:  13%|#3        | 6579/50000 [3:04:18<18:40:28,  1.55s/it]
Outer training loop:  13%|#3        | 6580/50000 [3:04:20<18:59:35,  1.57s/it]
Outer training loop:  13%|#3        | 6581/50000 [3:04:21<18:52:14,  1.56s/it]
Outer training loop:  13%|#3        | 6582/50000 [3:04:23<19:08:21,  1.59s/it]
Outer training loop:  13%|#3        | 6583/50000 [3:04:24<19:02:16,  1.58s/it]
Outer training loop:  13%|#3        | 6584/50000 [3:04:26<19:10:31,  1.59s/it]
Outer training loop:  13%|#3        | 6585/50000 [3:04:28<19:06:55,  1.59s/it]
Outer training loop:  13%|#3        | 6586/50000 [3:04:29<19:10:16,  1.59s/it]
Outer training loop:  13%|#3        | 6587/50000 [3:04:31<18:39:16,  1.55s/it]
Outer training loop:  13%|#3        | 6588/50000 [3:04:32<18:11:30,  1.51s/it]
Outer training loop:  13%|#3        | 6589/50000 [3:04:34<18:55:17,  1.57s/it]
Outer training loop:  13%|#3        | 6590/50000 [3:04:35<18:45:15,  1.56s/it]
Outer training loop:  13%|#3        | 6591/50000 [3:04:37<18:49:57,  1.56s/it]
Outer training loop:  13%|#3        | 6592/50000 [3:04:38<18:31:10,  1.54s/it]
Outer training loop:  13%|#3        | 6593/50000 [3:04:40<18:48:25,  1.56s/it]
Outer training loop:  13%|#3        | 6594/50000 [3:04:42<19:01:42,  1.58s/it]
Outer training loop:  13%|#3        | 6595/50000 [3:04:43<19:55:43,  1.65s/it]
Outer training loop:  13%|#3        | 6596/50000 [3:04:45<19:23:50,  1.61s/it]
Outer training loop:  13%|#3        | 6597/50000 [3:04:47<19:12:45,  1.59s/it]
Outer training loop:  13%|#3        | 6598/50000 [3:04:48<18:49:59,  1.56s/it]
Outer training loop:  13%|#3        | 6599/50000 [3:04:50<18:49:16,  1.56s/it]
Outer training loop:  13%|#3        | 6600/50000 [3:04:51<18:38:42,  1.55s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.46s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.51s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.48s/it]

Outer training loop:  13%|#3        | 6601/50000 [3:04:57<35:08:28,  2.92s/it]
Outer training loop:  13%|#3        | 6602/50000 [3:04:59<30:18:10,  2.51s/it]
Outer training loop:  13%|#3        | 6603/50000 [3:05:00<26:40:57,  2.21s/it]
Outer training loop:  13%|#3        | 6604/50000 [3:05:02<24:21:32,  2.02s/it]
Outer training loop:  13%|#3        | 6605/50000 [3:05:03<22:31:52,  1.87s/it]
Outer training loop:  13%|#3        | 6606/50000 [3:05:05<21:42:25,  1.80s/it]
Outer training loop:  13%|#3        | 6607/50000 [3:05:07<20:47:38,  1.73s/it]
Outer training loop:  13%|#3        | 6608/50000 [3:05:08<19:56:44,  1.65s/it]
Outer training loop:  13%|#3        | 6609/50000 [3:05:10<19:32:11,  1.62s/it]
Outer training loop:  13%|#3        | 6610/50000 [3:05:11<19:29:50,  1.62s/it]
Outer training loop:  13%|#3        | 6611/50000 [3:05:13<19:24:00,  1.61s/it]
Outer training loop:  13%|#3        | 6612/50000 [3:05:14<19:30:01,  1.62s/it]
Outer training loop:  13%|#3        | 6613/50000 [3:05:16<18:57:29,  1.57s/it]
Outer training loop:  13%|#3        | 6614/50000 [3:05:18<19:23:38,  1.61s/it]
Outer training loop:  13%|#3        | 6615/50000 [3:05:19<19:19:33,  1.60s/it]
Outer training loop:  13%|#3        | 6616/50000 [3:05:21<18:47:25,  1.56s/it]
Outer training loop:  13%|#3        | 6617/50000 [3:05:22<19:02:43,  1.58s/it]
Outer training loop:  13%|#3        | 6618/50000 [3:05:24<18:41:15,  1.55s/it]
Outer training loop:  13%|#3        | 6619/50000 [3:05:25<19:08:09,  1.59s/it]
Outer training loop:  13%|#3        | 6620/50000 [3:05:27<19:03:15,  1.58s/it]
Outer training loop:  13%|#3        | 6621/50000 [3:05:29<19:21:03,  1.61s/it]
Outer training loop:  13%|#3        | 6622/50000 [3:05:30<18:55:59,  1.57s/it]
Outer training loop:  13%|#3        | 6623/50000 [3:05:32<18:39:51,  1.55s/it]
Outer training loop:  13%|#3        | 6624/50000 [3:05:33<18:32:26,  1.54s/it]
Outer training loop:  13%|#3        | 6625/50000 [3:05:35<19:10:14,  1.59s/it]
Outer training loop:  13%|#3        | 6626/50000 [3:05:36<18:49:10,  1.56s/it]
Outer training loop:  13%|#3        | 6627/50000 [3:05:38<19:15:03,  1.60s/it]
Outer training loop:  13%|#3        | 6628/50000 [3:05:40<19:08:27,  1.59s/it]
Outer training loop:  13%|#3        | 6629/50000 [3:05:41<18:53:19,  1.57s/it]
Outer training loop:  13%|#3        | 6630/50000 [3:05:43<18:49:24,  1.56s/it]
Outer training loop:  13%|#3        | 6631/50000 [3:05:44<18:59:32,  1.58s/it]
Outer training loop:  13%|#3        | 6632/50000 [3:05:46<19:16:51,  1.60s/it]
Outer training loop:  13%|#3        | 6633/50000 [3:05:48<19:13:56,  1.60s/it]
Outer training loop:  13%|#3        | 6634/50000 [3:05:49<18:57:47,  1.57s/it]
Outer training loop:  13%|#3        | 6635/50000 [3:05:51<19:08:47,  1.59s/it]
Outer training loop:  13%|#3        | 6636/50000 [3:05:52<18:25:19,  1.53s/it]
Outer training loop:  13%|#3        | 6637/50000 [3:05:54<18:25:59,  1.53s/it]
Outer training loop:  13%|#3        | 6638/50000 [3:05:55<18:11:33,  1.51s/it]
Outer training loop:  13%|#3        | 6639/50000 [3:05:57<18:43:43,  1.55s/it]
Outer training loop:  13%|#3        | 6640/50000 [3:05:58<18:16:00,  1.52s/it]
Outer training loop:  13%|#3        | 6641/50000 [3:06:00<18:31:26,  1.54s/it]
Outer training loop:  13%|#3        | 6642/50000 [3:06:01<18:38:05,  1.55s/it]
Outer training loop:  13%|#3        | 6643/50000 [3:06:03<18:52:48,  1.57s/it]
Outer training loop:  13%|#3        | 6644/50000 [3:06:04<18:39:29,  1.55s/it]
Outer training loop:  13%|#3        | 6645/50000 [3:06:06<19:05:04,  1.58s/it]
Outer training loop:  13%|#3        | 6646/50000 [3:06:08<19:07:22,  1.59s/it]
Outer training loop:  13%|#3        | 6647/50000 [3:06:09<19:23:16,  1.61s/it]
Outer training loop:  13%|#3        | 6648/50000 [3:06:11<19:06:39,  1.59s/it]
Outer training loop:  13%|#3        | 6649/50000 [3:06:13<19:17:20,  1.60s/it]
Outer training loop:  13%|#3        | 6650/50000 [3:06:14<18:56:07,  1.57s/it]
Outer training loop:  13%|#3        | 6651/50000 [3:06:16<19:08:46,  1.59s/it]
Outer training loop:  13%|#3        | 6652/50000 [3:06:17<18:39:40,  1.55s/it]
Outer training loop:  13%|#3        | 6653/50000 [3:06:19<18:58:27,  1.58s/it]
Outer training loop:  13%|#3        | 6654/50000 [3:06:20<18:51:16,  1.57s/it]
Outer training loop:  13%|#3        | 6655/50000 [3:06:22<19:15:48,  1.60s/it]
Outer training loop:  13%|#3        | 6656/50000 [3:06:24<19:07:34,  1.59s/it]
Outer training loop:  13%|#3        | 6657/50000 [3:06:25<19:09:52,  1.59s/it]
Outer training loop:  13%|#3        | 6658/50000 [3:06:27<18:39:11,  1.55s/it]
Outer training loop:  13%|#3        | 6659/50000 [3:06:28<18:47:11,  1.56s/it]
Outer training loop:  13%|#3        | 6660/50000 [3:06:30<18:59:28,  1.58s/it]
Outer training loop:  13%|#3        | 6661/50000 [3:06:31<19:15:24,  1.60s/it]
Outer training loop:  13%|#3        | 6662/50000 [3:06:33<19:04:11,  1.58s/it]
Outer training loop:  13%|#3        | 6663/50000 [3:06:34<18:46:47,  1.56s/it]
Outer training loop:  13%|#3        | 6664/50000 [3:06:36<19:09:18,  1.59s/it]
Outer training loop:  13%|#3        | 6665/50000 [3:06:38<19:21:24,  1.61s/it]
Outer training loop:  13%|#3        | 6666/50000 [3:06:39<19:01:37,  1.58s/it]
Outer training loop:  13%|#3        | 6667/50000 [3:06:41<19:18:20,  1.60s/it]
Outer training loop:  13%|#3        | 6668/50000 [3:06:42<18:49:57,  1.56s/it]
Outer training loop:  13%|#3        | 6669/50000 [3:06:44<18:24:29,  1.53s/it]
Outer training loop:  13%|#3        | 6670/50000 [3:06:45<18:33:28,  1.54s/it]
Outer training loop:  13%|#3        | 6671/50000 [3:06:47<18:43:26,  1.56s/it]
Outer training loop:  13%|#3        | 6672/50000 [3:06:49<18:30:11,  1.54s/it]
Outer training loop:  13%|#3        | 6673/50000 [3:06:50<18:43:53,  1.56s/it]
Outer training loop:  13%|#3        | 6674/50000 [3:06:52<18:28:08,  1.53s/it]
Outer training loop:  13%|#3        | 6675/50000 [3:06:53<18:09:46,  1.51s/it]
Outer training loop:  13%|#3        | 6676/50000 [3:06:55<18:44:55,  1.56s/it]
Outer training loop:  13%|#3        | 6677/50000 [3:06:56<18:50:56,  1.57s/it]
Outer training loop:  13%|#3        | 6678/50000 [3:06:58<18:48:32,  1.56s/it]
Outer training loop:  13%|#3        | 6679/50000 [3:06:59<18:37:37,  1.55s/it]
Outer training loop:  13%|#3        | 6680/50000 [3:07:01<18:39:53,  1.55s/it]
Outer training loop:  13%|#3        | 6681/50000 [3:07:02<18:30:36,  1.54s/it]
Outer training loop:  13%|#3        | 6682/50000 [3:07:04<18:42:17,  1.55s/it]
Outer training loop:  13%|#3        | 6683/50000 [3:07:06<18:46:04,  1.56s/it]
Outer training loop:  13%|#3        | 6684/50000 [3:07:07<19:02:09,  1.58s/it]
Outer training loop:  13%|#3        | 6685/50000 [3:07:09<18:37:45,  1.55s/it]
Outer training loop:  13%|#3        | 6686/50000 [3:07:10<18:57:12,  1.58s/it]
Outer training loop:  13%|#3        | 6687/50000 [3:07:12<18:25:45,  1.53s/it]
Outer training loop:  13%|#3        | 6688/50000 [3:07:13<18:41:14,  1.55s/it]
Outer training loop:  13%|#3        | 6689/50000 [3:07:15<18:38:49,  1.55s/it]
Outer training loop:  13%|#3        | 6690/50000 [3:07:17<18:57:13,  1.58s/it]
Outer training loop:  13%|#3        | 6691/50000 [3:07:18<18:43:31,  1.56s/it]
Outer training loop:  13%|#3        | 6692/50000 [3:07:20<19:25:59,  1.62s/it]
Outer training loop:  13%|#3        | 6693/50000 [3:07:21<19:31:42,  1.62s/it]
Outer training loop:  13%|#3        | 6694/50000 [3:07:23<19:19:22,  1.61s/it]
Outer training loop:  13%|#3        | 6695/50000 [3:07:25<19:09:43,  1.59s/it]
Outer training loop:  13%|#3        | 6696/50000 [3:07:26<18:59:21,  1.58s/it]
Outer training loop:  13%|#3        | 6697/50000 [3:07:28<18:55:39,  1.57s/it]
Outer training loop:  13%|#3        | 6698/50000 [3:07:29<19:11:57,  1.60s/it]
Outer training loop:  13%|#3        | 6699/50000 [3:07:31<19:15:52,  1.60s/it]
Outer training loop:  13%|#3        | 6700/50000 [3:07:33<19:04:32,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.49s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.50s/it]

Outer training loop:  13%|#3        | 6701/50000 [3:07:39<35:31:33,  2.95s/it]
Outer training loop:  13%|#3        | 6702/50000 [3:07:40<30:49:30,  2.56s/it]
Outer training loop:  13%|#3        | 6703/50000 [3:07:42<26:57:15,  2.24s/it]
Outer training loop:  13%|#3        | 6704/50000 [3:07:43<24:12:34,  2.01s/it]
Outer training loop:  13%|#3        | 6705/50000 [3:07:45<22:38:26,  1.88s/it]
Outer training loop:  13%|#3        | 6706/50000 [3:07:46<21:18:01,  1.77s/it]
Outer training loop:  13%|#3        | 6707/50000 [3:07:48<20:56:32,  1.74s/it]
Outer training loop:  13%|#3        | 6708/50000 [3:07:50<20:26:28,  1.70s/it]
Outer training loop:  13%|#3        | 6709/50000 [3:07:51<20:03:06,  1.67s/it]
Outer training loop:  13%|#3        | 6710/50000 [3:07:53<19:51:49,  1.65s/it]
Outer training loop:  13%|#3        | 6711/50000 [3:07:54<19:37:24,  1.63s/it]
Outer training loop:  13%|#3        | 6712/50000 [3:07:56<18:59:14,  1.58s/it]
Outer training loop:  13%|#3        | 6713/50000 [3:07:57<18:51:46,  1.57s/it]
Outer training loop:  13%|#3        | 6714/50000 [3:07:59<18:33:02,  1.54s/it]
Outer training loop:  13%|#3        | 6715/50000 [3:08:01<19:00:56,  1.58s/it]
Outer training loop:  13%|#3        | 6716/50000 [3:08:02<18:47:28,  1.56s/it]
Outer training loop:  13%|#3        | 6717/50000 [3:08:04<19:05:11,  1.59s/it]
Outer training loop:  13%|#3        | 6718/50000 [3:08:05<18:50:40,  1.57s/it]
Outer training loop:  13%|#3        | 6719/50000 [3:08:07<18:39:16,  1.55s/it]
Outer training loop:  13%|#3        | 6720/50000 [3:08:08<18:24:02,  1.53s/it]
Outer training loop:  13%|#3        | 6721/50000 [3:08:10<18:42:21,  1.56s/it]
Outer training loop:  13%|#3        | 6722/50000 [3:08:11<18:32:48,  1.54s/it]
Outer training loop:  13%|#3        | 6723/50000 [3:08:13<18:32:53,  1.54s/it]
Outer training loop:  13%|#3        | 6724/50000 [3:08:15<18:44:48,  1.56s/it]
Outer training loop:  13%|#3        | 6725/50000 [3:08:16<18:24:23,  1.53s/it]
Outer training loop:  13%|#3        | 6726/50000 [3:08:18<18:34:39,  1.55s/it]
Outer training loop:  13%|#3        | 6727/50000 [3:08:19<18:38:58,  1.55s/it]
Outer training loop:  13%|#3        | 6728/50000 [3:08:21<19:08:25,  1.59s/it]
Outer training loop:  13%|#3        | 6729/50000 [3:08:22<18:52:38,  1.57s/it]
Outer training loop:  13%|#3        | 6730/50000 [3:08:24<18:48:26,  1.56s/it]
Outer training loop:  13%|#3        | 6731/50000 [3:08:26<19:01:51,  1.58s/it]
Outer training loop:  13%|#3        | 6732/50000 [3:08:27<18:38:13,  1.55s/it]
Outer training loop:  13%|#3        | 6733/50000 [3:08:29<18:43:24,  1.56s/it]
Outer training loop:  13%|#3        | 6734/50000 [3:08:30<18:35:41,  1.55s/it]
Outer training loop:  13%|#3        | 6735/50000 [3:08:32<18:39:32,  1.55s/it]
Outer training loop:  13%|#3        | 6736/50000 [3:08:33<18:39:11,  1.55s/it]
Outer training loop:  13%|#3        | 6737/50000 [3:08:35<18:52:11,  1.57s/it]
Outer training loop:  13%|#3        | 6738/50000 [3:08:36<18:36:13,  1.55s/it]
Outer training loop:  13%|#3        | 6739/50000 [3:08:38<19:23:26,  1.61s/it]
Outer training loop:  13%|#3        | 6740/50000 [3:08:40<18:53:19,  1.57s/it]
Outer training loop:  13%|#3        | 6741/50000 [3:08:41<19:04:33,  1.59s/it]
Outer training loop:  13%|#3        | 6742/50000 [3:08:43<19:05:30,  1.59s/it]
Outer training loop:  13%|#3        | 6743/50000 [3:08:44<18:55:36,  1.58s/it]
Outer training loop:  13%|#3        | 6744/50000 [3:08:46<18:46:06,  1.56s/it]
Outer training loop:  13%|#3        | 6745/50000 [3:08:47<18:42:02,  1.56s/it]
Outer training loop:  13%|#3        | 6746/50000 [3:08:49<18:52:56,  1.57s/it]
Outer training loop:  13%|#3        | 6747/50000 [3:08:51<18:46:30,  1.56s/it]
Outer training loop:  13%|#3        | 6748/50000 [3:08:52<18:51:21,  1.57s/it]
Outer training loop:  13%|#3        | 6749/50000 [3:08:54<18:43:40,  1.56s/it]
Outer training loop:  14%|#3        | 6750/50000 [3:08:55<18:26:30,  1.54s/it]
Outer training loop:  14%|#3        | 6751/50000 [3:08:57<18:41:56,  1.56s/it]
Outer training loop:  14%|#3        | 6752/50000 [3:08:58<18:29:40,  1.54s/it]
Outer training loop:  14%|#3        | 6753/50000 [3:09:00<18:40:02,  1.55s/it]
Outer training loop:  14%|#3        | 6754/50000 [3:09:01<18:32:45,  1.54s/it]
Outer training loop:  14%|#3        | 6755/50000 [3:09:03<18:48:43,  1.57s/it]
Outer training loop:  14%|#3        | 6756/50000 [3:09:05<18:54:45,  1.57s/it]
Outer training loop:  14%|#3        | 6757/50000 [3:09:06<19:04:39,  1.59s/it]
Outer training loop:  14%|#3        | 6758/50000 [3:09:08<18:42:58,  1.56s/it]
Outer training loop:  14%|#3        | 6759/50000 [3:09:09<18:43:18,  1.56s/it]
Outer training loop:  14%|#3        | 6760/50000 [3:09:11<18:26:12,  1.53s/it]
Outer training loop:  14%|#3        | 6761/50000 [3:09:12<18:43:30,  1.56s/it]
Outer training loop:  14%|#3        | 6762/50000 [3:09:14<18:31:53,  1.54s/it]
Outer training loop:  14%|#3        | 6763/50000 [3:09:15<18:46:19,  1.56s/it]
Outer training loop:  14%|#3        | 6764/50000 [3:09:17<18:41:53,  1.56s/it]
Outer training loop:  14%|#3        | 6765/50000 [3:09:19<19:12:08,  1.60s/it]
Outer training loop:  14%|#3        | 6766/50000 [3:09:20<19:02:18,  1.59s/it]
Outer training loop:  14%|#3        | 6767/50000 [3:09:22<19:12:32,  1.60s/it]
Outer training loop:  14%|#3        | 6768/50000 [3:09:23<18:53:21,  1.57s/it]
Outer training loop:  14%|#3        | 6769/50000 [3:09:25<18:54:27,  1.57s/it]
Outer training loop:  14%|#3        | 6770/50000 [3:09:26<18:23:34,  1.53s/it]
Outer training loop:  14%|#3        | 6771/50000 [3:09:28<18:32:44,  1.54s/it]
Outer training loop:  14%|#3        | 6772/50000 [3:09:30<18:24:15,  1.53s/it]
Outer training loop:  14%|#3        | 6773/50000 [3:09:31<18:34:44,  1.55s/it]
Outer training loop:  14%|#3        | 6774/50000 [3:09:33<18:20:50,  1.53s/it]
Outer training loop:  14%|#3        | 6775/50000 [3:09:34<18:44:44,  1.56s/it]
Outer training loop:  14%|#3        | 6776/50000 [3:09:36<18:36:32,  1.55s/it]
Outer training loop:  14%|#3        | 6777/50000 [3:09:37<18:43:59,  1.56s/it]
Outer training loop:  14%|#3        | 6778/50000 [3:09:39<18:30:45,  1.54s/it]
Outer training loop:  14%|#3        | 6779/50000 [3:09:40<18:55:01,  1.58s/it]
Outer training loop:  14%|#3        | 6780/50000 [3:09:42<18:54:07,  1.57s/it]
Outer training loop:  14%|#3        | 6781/50000 [3:09:44<19:06:49,  1.59s/it]
Outer training loop:  14%|#3        | 6782/50000 [3:09:45<18:40:34,  1.56s/it]
Outer training loop:  14%|#3        | 6783/50000 [3:09:47<18:50:03,  1.57s/it]
Outer training loop:  14%|#3        | 6784/50000 [3:09:48<18:27:55,  1.54s/it]
Outer training loop:  14%|#3        | 6785/50000 [3:09:50<18:38:28,  1.55s/it]
Outer training loop:  14%|#3        | 6786/50000 [3:09:51<18:21:04,  1.53s/it]
Outer training loop:  14%|#3        | 6787/50000 [3:09:53<18:34:16,  1.55s/it]
Outer training loop:  14%|#3        | 6788/50000 [3:09:54<18:28:27,  1.54s/it]
Outer training loop:  14%|#3        | 6789/50000 [3:09:56<19:08:29,  1.59s/it]
Outer training loop:  14%|#3        | 6790/50000 [3:09:58<18:46:52,  1.56s/it]
Outer training loop:  14%|#3        | 6791/50000 [3:09:59<18:52:53,  1.57s/it]
Outer training loop:  14%|#3        | 6792/50000 [3:10:01<18:37:29,  1.55s/it]
Outer training loop:  14%|#3        | 6793/50000 [3:10:02<18:33:53,  1.55s/it]
Outer training loop:  14%|#3        | 6794/50000 [3:10:04<18:37:15,  1.55s/it]
Outer training loop:  14%|#3        | 6795/50000 [3:10:05<18:49:12,  1.57s/it]
Outer training loop:  14%|#3        | 6796/50000 [3:10:07<18:29:16,  1.54s/it]
Outer training loop:  14%|#3        | 6797/50000 [3:10:08<18:43:03,  1.56s/it]
Outer training loop:  14%|#3        | 6798/50000 [3:10:10<18:41:16,  1.56s/it]
Outer training loop:  14%|#3        | 6799/50000 [3:10:12<18:45:09,  1.56s/it]
Outer training loop:  14%|#3        | 6800/50000 [3:10:13<18:45:28,  1.56s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.54s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.57s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.53s/it]

Outer training loop:  14%|#3        | 6801/50000 [3:10:19<35:48:47,  2.98s/it]
Outer training loop:  14%|#3        | 6802/50000 [3:10:21<31:06:00,  2.59s/it]
Outer training loop:  14%|#3        | 6803/50000 [3:10:23<27:19:20,  2.28s/it]
Outer training loop:  14%|#3        | 6804/50000 [3:10:24<25:07:44,  2.09s/it]
Outer training loop:  14%|#3        | 6805/50000 [3:10:26<23:16:53,  1.94s/it]
Outer training loop:  14%|#3        | 6806/50000 [3:10:28<21:55:38,  1.83s/it]
Outer training loop:  14%|#3        | 6807/50000 [3:10:29<20:36:56,  1.72s/it]
Outer training loop:  14%|#3        | 6808/50000 [3:10:31<20:13:39,  1.69s/it]
Outer training loop:  14%|#3        | 6809/50000 [3:10:32<19:42:07,  1.64s/it]
Outer training loop:  14%|#3        | 6810/50000 [3:10:34<19:53:33,  1.66s/it]
Outer training loop:  14%|#3        | 6811/50000 [3:10:35<19:27:30,  1.62s/it]
Outer training loop:  14%|#3        | 6812/50000 [3:10:37<19:16:13,  1.61s/it]
Outer training loop:  14%|#3        | 6813/50000 [3:10:38<19:10:12,  1.60s/it]
Outer training loop:  14%|#3        | 6814/50000 [3:10:40<19:21:33,  1.61s/it]
Outer training loop:  14%|#3        | 6815/50000 [3:10:42<19:04:29,  1.59s/it]
Outer training loop:  14%|#3        | 6816/50000 [3:10:43<19:10:48,  1.60s/it]
Outer training loop:  14%|#3        | 6817/50000 [3:10:45<18:47:10,  1.57s/it]
Outer training loop:  14%|#3        | 6818/50000 [3:10:46<19:17:03,  1.61s/it]
Outer training loop:  14%|#3        | 6819/50000 [3:10:48<18:51:08,  1.57s/it]
Outer training loop:  14%|#3        | 6820/50000 [3:10:50<19:19:33,  1.61s/it]
Outer training loop:  14%|#3        | 6821/50000 [3:10:51<19:07:15,  1.59s/it]
Outer training loop:  14%|#3        | 6822/50000 [3:10:53<19:12:59,  1.60s/it]
Outer training loop:  14%|#3        | 6823/50000 [3:10:54<18:53:45,  1.58s/it]
Outer training loop:  14%|#3        | 6824/50000 [3:10:56<18:31:58,  1.55s/it]
Outer training loop:  14%|#3        | 6825/50000 [3:10:57<18:36:28,  1.55s/it]
Outer training loop:  14%|#3        | 6826/50000 [3:10:59<18:40:30,  1.56s/it]
Outer training loop:  14%|#3        | 6827/50000 [3:11:01<18:30:38,  1.54s/it]
Outer training loop:  14%|#3        | 6828/50000 [3:11:02<18:39:13,  1.56s/it]
Outer training loop:  14%|#3        | 6829/50000 [3:11:04<18:37:40,  1.55s/it]
Outer training loop:  14%|#3        | 6830/50000 [3:11:05<19:06:52,  1.59s/it]
Outer training loop:  14%|#3        | 6831/50000 [3:11:07<18:48:40,  1.57s/it]
Outer training loop:  14%|#3        | 6832/50000 [3:11:08<18:58:23,  1.58s/it]
Outer training loop:  14%|#3        | 6833/50000 [3:11:10<18:39:26,  1.56s/it]
Outer training loop:  14%|#3        | 6834/50000 [3:11:12<18:46:39,  1.57s/it]
Outer training loop:  14%|#3        | 6835/50000 [3:11:13<18:34:16,  1.55s/it]
Outer training loop:  14%|#3        | 6836/50000 [3:11:15<19:29:09,  1.63s/it]
Outer training loop:  14%|#3        | 6837/50000 [3:11:16<19:02:53,  1.59s/it]
Outer training loop:  14%|#3        | 6838/50000 [3:11:18<19:16:30,  1.61s/it]
Outer training loop:  14%|#3        | 6839/50000 [3:11:20<19:03:27,  1.59s/it]
Outer training loop:  14%|#3        | 6840/50000 [3:11:21<19:06:04,  1.59s/it]
Outer training loop:  14%|#3        | 6841/50000 [3:11:23<18:55:35,  1.58s/it]
Outer training loop:  14%|#3        | 6842/50000 [3:11:24<19:11:33,  1.60s/it]
Outer training loop:  14%|#3        | 6843/50000 [3:11:26<19:00:53,  1.59s/it]
Outer training loop:  14%|#3        | 6844/50000 [3:11:27<19:01:45,  1.59s/it]
Outer training loop:  14%|#3        | 6845/50000 [3:11:29<19:07:20,  1.60s/it]
Outer training loop:  14%|#3        | 6846/50000 [3:11:31<19:22:24,  1.62s/it]
Outer training loop:  14%|#3        | 6847/50000 [3:11:32<18:45:43,  1.57s/it]
Outer training loop:  14%|#3        | 6848/50000 [3:11:34<19:10:04,  1.60s/it]
Outer training loop:  14%|#3        | 6849/50000 [3:11:35<18:54:43,  1.58s/it]
Outer training loop:  14%|#3        | 6850/50000 [3:11:37<19:07:22,  1.60s/it]
Outer training loop:  14%|#3        | 6851/50000 [3:11:39<18:53:31,  1.58s/it]
Outer training loop:  14%|#3        | 6852/50000 [3:11:40<18:57:31,  1.58s/it]
Outer training loop:  14%|#3        | 6853/50000 [3:11:42<19:00:40,  1.59s/it]
Outer training loop:  14%|#3        | 6854/50000 [3:11:43<19:24:49,  1.62s/it]
Outer training loop:  14%|#3        | 6855/50000 [3:11:45<18:59:18,  1.58s/it]
Outer training loop:  14%|#3        | 6856/50000 [3:11:47<19:15:02,  1.61s/it]
Outer training loop:  14%|#3        | 6857/50000 [3:11:48<19:15:08,  1.61s/it]
Outer training loop:  14%|#3        | 6858/50000 [3:11:50<19:27:56,  1.62s/it]
Outer training loop:  14%|#3        | 6859/50000 [3:11:51<18:53:34,  1.58s/it]
Outer training loop:  14%|#3        | 6860/50000 [3:11:53<19:10:40,  1.60s/it]
Outer training loop:  14%|#3        | 6861/50000 [3:11:55<19:03:51,  1.59s/it]
Outer training loop:  14%|#3        | 6862/50000 [3:11:56<19:21:16,  1.62s/it]
Outer training loop:  14%|#3        | 6863/50000 [3:11:58<18:56:50,  1.58s/it]
Outer training loop:  14%|#3        | 6864/50000 [3:11:59<18:38:11,  1.56s/it]
Outer training loop:  14%|#3        | 6865/50000 [3:12:01<18:48:29,  1.57s/it]
Outer training loop:  14%|#3        | 6866/50000 [3:12:02<18:31:19,  1.55s/it]
Outer training loop:  14%|#3        | 6867/50000 [3:12:04<18:46:16,  1.57s/it]
Outer training loop:  14%|#3        | 6868/50000 [3:12:05<18:32:52,  1.55s/it]
Outer training loop:  14%|#3        | 6869/50000 [3:12:07<18:56:17,  1.58s/it]
Outer training loop:  14%|#3        | 6870/50000 [3:12:09<18:27:54,  1.54s/it]
Outer training loop:  14%|#3        | 6871/50000 [3:12:10<18:56:23,  1.58s/it]
Outer training loop:  14%|#3        | 6872/50000 [3:12:12<18:41:08,  1.56s/it]
Outer training loop:  14%|#3        | 6873/50000 [3:12:13<18:51:15,  1.57s/it]
Outer training loop:  14%|#3        | 6874/50000 [3:12:15<18:28:59,  1.54s/it]
Outer training loop:  14%|#3        | 6875/50000 [3:12:17<19:02:38,  1.59s/it]
Outer training loop:  14%|#3        | 6876/50000 [3:12:18<19:06:09,  1.59s/it]
Outer training loop:  14%|#3        | 6877/50000 [3:12:20<19:33:44,  1.63s/it]
Outer training loop:  14%|#3        | 6878/50000 [3:12:21<19:06:47,  1.60s/it]
Outer training loop:  14%|#3        | 6879/50000 [3:12:23<19:10:57,  1.60s/it]
Outer training loop:  14%|#3        | 6880/50000 [3:12:25<18:54:03,  1.58s/it]
Outer training loop:  14%|#3        | 6881/50000 [3:12:26<19:09:52,  1.60s/it]
Outer training loop:  14%|#3        | 6882/50000 [3:12:28<18:51:20,  1.57s/it]
Outer training loop:  14%|#3        | 6883/50000 [3:12:29<19:02:47,  1.59s/it]
Outer training loop:  14%|#3        | 6884/50000 [3:12:31<18:43:01,  1.56s/it]
Outer training loop:  14%|#3        | 6885/50000 [3:12:32<19:01:21,  1.59s/it]
Outer training loop:  14%|#3        | 6886/50000 [3:12:34<19:27:57,  1.63s/it]
Outer training loop:  14%|#3        | 6887/50000 [3:12:36<19:43:19,  1.65s/it]
Outer training loop:  14%|#3        | 6888/50000 [3:12:37<19:26:42,  1.62s/it]
Outer training loop:  14%|#3        | 6889/50000 [3:12:39<19:14:27,  1.61s/it]
Outer training loop:  14%|#3        | 6890/50000 [3:12:41<19:05:52,  1.59s/it]
Outer training loop:  14%|#3        | 6891/50000 [3:12:42<19:16:36,  1.61s/it]
Outer training loop:  14%|#3        | 6892/50000 [3:12:44<19:02:41,  1.59s/it]
Outer training loop:  14%|#3        | 6893/50000 [3:12:45<18:51:09,  1.57s/it]
Outer training loop:  14%|#3        | 6894/50000 [3:12:47<18:59:52,  1.59s/it]
Outer training loop:  14%|#3        | 6895/50000 [3:12:48<18:46:28,  1.57s/it]
Outer training loop:  14%|#3        | 6896/50000 [3:12:50<19:16:20,  1.61s/it]
Outer training loop:  14%|#3        | 6897/50000 [3:12:52<19:02:30,  1.59s/it]
Outer training loop:  14%|#3        | 6898/50000 [3:12:53<19:17:53,  1.61s/it]
Outer training loop:  14%|#3        | 6899/50000 [3:12:55<19:11:35,  1.60s/it]
Outer training loop:  14%|#3        | 6900/50000 [3:12:57<19:04:15,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.56s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.55s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.54s/it]

Outer training loop:  14%|#3        | 6901/50000 [3:13:03<35:14:35,  2.94s/it]
Outer training loop:  14%|#3        | 6902/50000 [3:13:04<30:35:52,  2.56s/it]
Outer training loop:  14%|#3        | 6903/50000 [3:13:06<27:19:53,  2.28s/it]
Outer training loop:  14%|#3        | 6904/50000 [3:13:07<24:40:36,  2.06s/it]
Outer training loop:  14%|#3        | 6905/50000 [3:13:09<23:07:51,  1.93s/it]
Outer training loop:  14%|#3        | 6906/50000 [3:13:11<21:46:58,  1.82s/it]
Outer training loop:  14%|#3        | 6907/50000 [3:13:12<21:05:09,  1.76s/it]
Outer training loop:  14%|#3        | 6908/50000 [3:13:14<20:17:12,  1.69s/it]
Outer training loop:  14%|#3        | 6909/50000 [3:13:15<19:51:49,  1.66s/it]
Outer training loop:  14%|#3        | 6910/50000 [3:13:17<19:17:55,  1.61s/it]
Outer training loop:  14%|#3        | 6911/50000 [3:13:18<19:04:40,  1.59s/it]
Outer training loop:  14%|#3        | 6912/50000 [3:13:20<18:56:12,  1.58s/it]
Outer training loop:  14%|#3        | 6913/50000 [3:13:22<18:45:47,  1.57s/it]
Outer training loop:  14%|#3        | 6914/50000 [3:13:23<18:26:16,  1.54s/it]
Outer training loop:  14%|#3        | 6915/50000 [3:13:25<18:50:25,  1.57s/it]
Outer training loop:  14%|#3        | 6916/50000 [3:13:26<18:48:05,  1.57s/it]
Outer training loop:  14%|#3        | 6917/50000 [3:13:28<19:16:35,  1.61s/it]
Outer training loop:  14%|#3        | 6918/50000 [3:13:29<18:58:06,  1.59s/it]
Outer training loop:  14%|#3        | 6919/50000 [3:13:31<19:00:22,  1.59s/it]
Outer training loop:  14%|#3        | 6920/50000 [3:13:33<18:59:22,  1.59s/it]
Outer training loop:  14%|#3        | 6921/50000 [3:13:34<19:15:35,  1.61s/it]
Outer training loop:  14%|#3        | 6922/50000 [3:13:36<19:44:50,  1.65s/it]
Outer training loop:  14%|#3        | 6923/50000 [3:13:38<19:38:16,  1.64s/it]
Outer training loop:  14%|#3        | 6924/50000 [3:13:39<19:18:32,  1.61s/it]
Outer training loop:  14%|#3        | 6925/50000 [3:13:41<19:23:23,  1.62s/it]
Outer training loop:  14%|#3        | 6926/50000 [3:13:42<19:00:35,  1.59s/it]
Outer training loop:  14%|#3        | 6927/50000 [3:13:44<18:39:25,  1.56s/it]
Outer training loop:  14%|#3        | 6928/50000 [3:13:45<18:46:29,  1.57s/it]
Outer training loop:  14%|#3        | 6929/50000 [3:13:47<18:19:25,  1.53s/it]
Outer training loop:  14%|#3        | 6930/50000 [3:13:48<18:25:01,  1.54s/it]
Outer training loop:  14%|#3        | 6931/50000 [3:13:50<18:40:13,  1.56s/it]
Outer training loop:  14%|#3        | 6932/50000 [3:13:52<18:52:10,  1.58s/it]
Outer training loop:  14%|#3        | 6933/50000 [3:13:53<19:12:09,  1.61s/it]
Outer training loop:  14%|#3        | 6934/50000 [3:13:55<19:23:34,  1.62s/it]
Outer training loop:  14%|#3        | 6935/50000 [3:13:57<19:03:33,  1.59s/it]
Outer training loop:  14%|#3        | 6936/50000 [3:13:58<19:10:45,  1.60s/it]
Outer training loop:  14%|#3        | 6937/50000 [3:14:00<19:02:37,  1.59s/it]
Outer training loop:  14%|#3        | 6938/50000 [3:14:01<19:23:06,  1.62s/it]
Outer training loop:  14%|#3        | 6939/50000 [3:14:03<19:03:04,  1.59s/it]
Outer training loop:  14%|#3        | 6940/50000 [3:14:04<18:50:08,  1.57s/it]
Outer training loop:  14%|#3        | 6941/50000 [3:14:06<18:46:48,  1.57s/it]
Outer training loop:  14%|#3        | 6942/50000 [3:14:08<19:00:34,  1.59s/it]
Outer training loop:  14%|#3        | 6943/50000 [3:14:09<18:59:15,  1.59s/it]
Outer training loop:  14%|#3        | 6944/50000 [3:14:11<19:19:38,  1.62s/it]
Outer training loop:  14%|#3        | 6945/50000 [3:14:13<19:24:57,  1.62s/it]
Outer training loop:  14%|#3        | 6946/50000 [3:14:14<19:21:42,  1.62s/it]
Outer training loop:  14%|#3        | 6947/50000 [3:14:16<19:12:29,  1.61s/it]
Outer training loop:  14%|#3        | 6948/50000 [3:14:17<19:03:51,  1.59s/it]
Outer training loop:  14%|#3        | 6949/50000 [3:14:19<19:20:09,  1.62s/it]
Outer training loop:  14%|#3        | 6950/50000 [3:14:21<19:09:01,  1.60s/it]
Outer training loop:  14%|#3        | 6951/50000 [3:14:22<19:30:02,  1.63s/it]
Outer training loop:  14%|#3        | 6952/50000 [3:14:24<18:55:01,  1.58s/it]
Outer training loop:  14%|#3        | 6953/50000 [3:14:25<19:07:27,  1.60s/it]
Outer training loop:  14%|#3        | 6954/50000 [3:14:27<18:49:52,  1.57s/it]
Outer training loop:  14%|#3        | 6955/50000 [3:14:28<18:56:04,  1.58s/it]
Outer training loop:  14%|#3        | 6956/50000 [3:14:30<18:39:04,  1.56s/it]
Outer training loop:  14%|#3        | 6957/50000 [3:14:32<19:04:03,  1.59s/it]
Outer training loop:  14%|#3        | 6958/50000 [3:14:33<18:52:29,  1.58s/it]
Outer training loop:  14%|#3        | 6959/50000 [3:14:35<19:08:35,  1.60s/it]
Outer training loop:  14%|#3        | 6960/50000 [3:14:36<19:08:15,  1.60s/it]
Outer training loop:  14%|#3        | 6961/50000 [3:14:38<19:16:13,  1.61s/it]
Outer training loop:  14%|#3        | 6962/50000 [3:14:40<19:01:10,  1.59s/it]
Outer training loop:  14%|#3        | 6963/50000 [3:14:41<18:40:30,  1.56s/it]
Outer training loop:  14%|#3        | 6964/50000 [3:14:43<19:03:37,  1.59s/it]
Outer training loop:  14%|#3        | 6965/50000 [3:14:44<19:15:44,  1.61s/it]
Outer training loop:  14%|#3        | 6966/50000 [3:14:46<19:09:30,  1.60s/it]
Outer training loop:  14%|#3        | 6967/50000 [3:14:48<19:25:57,  1.63s/it]
Outer training loop:  14%|#3        | 6968/50000 [3:14:49<19:01:44,  1.59s/it]
Outer training loop:  14%|#3        | 6969/50000 [3:14:51<19:30:25,  1.63s/it]
Outer training loop:  14%|#3        | 6970/50000 [3:14:53<19:20:11,  1.62s/it]
Outer training loop:  14%|#3        | 6971/50000 [3:14:54<19:30:32,  1.63s/it]
Outer training loop:  14%|#3        | 6972/50000 [3:14:56<19:06:30,  1.60s/it]
Outer training loop:  14%|#3        | 6973/50000 [3:14:57<19:17:16,  1.61s/it]
Outer training loop:  14%|#3        | 6974/50000 [3:14:59<18:51:24,  1.58s/it]
Outer training loop:  14%|#3        | 6975/50000 [3:15:01<19:06:39,  1.60s/it]
Outer training loop:  14%|#3        | 6976/50000 [3:15:02<18:39:42,  1.56s/it]
Outer training loop:  14%|#3        | 6977/50000 [3:15:04<19:01:31,  1.59s/it]
Outer training loop:  14%|#3        | 6978/50000 [3:15:05<18:45:46,  1.57s/it]
Outer training loop:  14%|#3        | 6979/50000 [3:15:07<18:59:11,  1.59s/it]
Outer training loop:  14%|#3        | 6980/50000 [3:15:08<18:43:51,  1.57s/it]
Outer training loop:  14%|#3        | 6981/50000 [3:15:10<18:58:29,  1.59s/it]
Outer training loop:  14%|#3        | 6982/50000 [3:15:11<18:38:21,  1.56s/it]
Outer training loop:  14%|#3        | 6983/50000 [3:15:13<19:32:51,  1.64s/it]
Outer training loop:  14%|#3        | 6984/50000 [3:15:15<18:57:49,  1.59s/it]
Outer training loop:  14%|#3        | 6985/50000 [3:15:16<18:44:48,  1.57s/it]
Outer training loop:  14%|#3        | 6986/50000 [3:15:18<18:51:55,  1.58s/it]
Outer training loop:  14%|#3        | 6987/50000 [3:15:19<18:47:01,  1.57s/it]
Outer training loop:  14%|#3        | 6988/50000 [3:15:21<18:55:44,  1.58s/it]
Outer training loop:  14%|#3        | 6989/50000 [3:15:22<18:32:10,  1.55s/it]
Outer training loop:  14%|#3        | 6990/50000 [3:15:24<18:38:48,  1.56s/it]
Outer training loop:  14%|#3        | 6991/50000 [3:15:26<18:59:17,  1.59s/it]
Outer training loop:  14%|#3        | 6992/50000 [3:15:27<19:01:51,  1.59s/it]
Outer training loop:  14%|#3        | 6993/50000 [3:15:29<18:44:14,  1.57s/it]
Outer training loop:  14%|#3        | 6994/50000 [3:15:30<18:51:50,  1.58s/it]
Outer training loop:  14%|#3        | 6995/50000 [3:15:32<18:41:43,  1.57s/it]
Outer training loop:  14%|#3        | 6996/50000 [3:15:34<19:21:05,  1.62s/it]
Outer training loop:  14%|#3        | 6997/50000 [3:15:35<19:03:40,  1.60s/it]
Outer training loop:  14%|#3        | 6998/50000 [3:15:37<18:56:48,  1.59s/it]
Outer training loop:  14%|#3        | 6999/50000 [3:15:38<18:34:55,  1.56s/it]
Outer training loop:  14%|#4        | 7000/50000 [3:15:40<18:56:50,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.46s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.50s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.48s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:14,  1.63s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:11,  1.62s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:08,  1.60s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:03,  1.57s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:03,  1.58s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<01:59,  1.55s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<01:58,  1.56s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:55,  1.55s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:55,  1.56s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:53,  1.55s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:50,  1.54s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:52,  1.58s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:48,  1.55s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:46,  1.55s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:45,  1.56s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:46,  1.60s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:43,  1.58s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:39,  1.54s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:41,  1.58s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:40,  1.60s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:37,  1.57s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:34,  1.56s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:36,  1.61s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:33,  1.59s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:32,  1.60s/it]

Evaluating policy:  31%|###1      | 26/83 [00:41<01:33,  1.64s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:30,  1.62s/it]

Evaluating policy:  34%|###3      | 28/83 [00:44<01:26,  1.58s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:26,  1.60s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:24,  1.60s/it]

Evaluating policy:  37%|###7      | 31/83 [00:49<01:27,  1.68s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:22,  1.61s/it]

Evaluating policy:  40%|###9      | 33/83 [00:52<01:19,  1.59s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:16,  1.56s/it]

Evaluating policy:  42%|####2     | 35/83 [00:55<01:17,  1.62s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:14,  1.58s/it]

Evaluating policy:  45%|####4     | 37/83 [00:58<01:12,  1.57s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:09,  1.54s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:09,  1.57s/it]

Evaluating policy:  48%|####8     | 40/83 [01:03<01:09,  1.62s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:07,  1.60s/it]

Evaluating policy:  51%|#####     | 42/83 [01:06<01:04,  1.58s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:01,  1.55s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:09<01:00,  1.54s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<00:57,  1.51s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<00:58,  1.57s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:14<00:55,  1.54s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:53,  1.53s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:51,  1.50s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:50,  1.52s/it]

Evaluating policy:  61%|######1   | 51/83 [01:20<00:48,  1.51s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:49,  1.60s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:46,  1.57s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:44,  1.54s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:42,  1.52s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:40,  1.50s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:39,  1.52s/it]

Evaluating policy:  70%|######9   | 58/83 [01:31<00:40,  1.62s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:38,  1.59s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:34<00:36,  1.59s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:34,  1.56s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:37<00:32,  1.53s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:30,  1.52s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:28,  1.52s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:42<00:28,  1.61s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:26,  1.57s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:24,  1.54s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:23,  1.55s/it]

Evaluating policy:  83%|########3 | 69/83 [01:48<00:21,  1.53s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:20,  1.55s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:18,  1.52s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:17,  1.62s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:15,  1.59s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:14,  1.56s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.56s/it]

Evaluating policy:  92%|#########1| 76/83 [01:59<00:10,  1.56s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.55s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.54s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.63s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:05,  1.74s/it]

Evaluating policy:  98%|#########7| 81/83 [02:07<00:03,  1.66s/it]

Evaluating policy:  99%|#########8| 82/83 [02:09<00:01,  1.64s/it]

Evaluating policy: 100%|##########| 83/83 [02:10<00:00,  1.58s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<01:59,  1.46s/it]

Evaluating policy:   2%|2         | 2/83 [00:02<01:59,  1.47s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<01:57,  1.47s/it]

Evaluating policy:   5%|4         | 4/83 [00:05<01:56,  1.47s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<01:55,  1.48s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:05,  1.64s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<02:03,  1.63s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:59,  1.59s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:59,  1.61s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:54,  1.57s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:55,  1.61s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:51,  1.57s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:48,  1.55s/it]

Evaluating policy:  17%|#6        | 14/83 [00:22<01:49,  1.59s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:48,  1.60s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:45,  1.57s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:44,  1.59s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:40,  1.54s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:38,  1.54s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:36,  1.52s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:36,  1.55s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:34,  1.55s/it]

Evaluating policy:  28%|##7       | 23/83 [00:35<01:33,  1.56s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:35,  1.62s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:32,  1.60s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:30,  1.58s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:26,  1.55s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:27,  1.59s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:23,  1.55s/it]

Evaluating policy:  36%|###6      | 30/83 [00:46<01:20,  1.53s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:19,  1.52s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:20,  1.57s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:17,  1.55s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:15,  1.54s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:12,  1.52s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:11,  1.52s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:13,  1.60s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:09,  1.55s/it]

Evaluating policy:  47%|####6     | 39/83 [01:00<01:07,  1.54s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:05,  1.52s/it]

Evaluating policy:  49%|####9     | 41/83 [01:03<01:04,  1.53s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:05,  1.60s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:03,  1.59s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<01:01,  1.57s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<00:59,  1.55s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:11<00:58,  1.58s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:13<00:59,  1.65s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:55,  1.59s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:16<00:53,  1.56s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:50,  1.54s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:49,  1.55s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:51,  1.66s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:49,  1.64s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:46,  1.60s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:44,  1.58s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:42,  1.56s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:40,  1.55s/it]

Evaluating policy:  70%|######9   | 58/83 [01:31<00:41,  1.65s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:38,  1.62s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:34<00:36,  1.58s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:34,  1.57s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:37<00:32,  1.57s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:31,  1.56s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:29,  1.53s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:42<00:29,  1.63s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:26,  1.59s/it]

Evaluating policy:  81%|########  | 67/83 [01:45<00:25,  1.56s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:23,  1.54s/it]

Evaluating policy:  83%|########3 | 69/83 [01:48<00:21,  1.53s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:19,  1.51s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:18,  1.53s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:16,  1.54s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:16,  1.65s/it]

Evaluating policy:  89%|########9 | 74/83 [01:56<00:14,  1.61s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.59s/it]

Evaluating policy:  92%|#########1| 76/83 [01:59<00:11,  1.58s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.56s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.55s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.56s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.57s/it]

Evaluating policy:  98%|#########7| 81/83 [02:07<00:03,  1.70s/it]

Evaluating policy:  99%|#########8| 82/83 [02:09<00:01,  1.64s/it]

Evaluating policy: 100%|##########| 83/83 [02:10<00:00,  1.62s/it]

Outer training loop:  14%|#4        | 7001/50000 [3:20:07<970:49:21, 81.28s/it]
Outer training loop:  14%|#4        | 7002/50000 [3:20:09<685:01:14, 57.35s/it]
Outer training loop:  14%|#4        | 7003/50000 [3:20:10<485:11:34, 40.62s/it]
Outer training loop:  14%|#4        | 7004/50000 [3:20:12<344:59:08, 28.89s/it]
Outer training loop:  14%|#4        | 7005/50000 [3:20:13<247:18:24, 20.71s/it]
Outer training loop:  14%|#4        | 7006/50000 [3:20:15<178:44:08, 14.97s/it]
Outer training loop:  14%|#4        | 7007/50000 [3:20:16<130:16:54, 10.91s/it]
Outer training loop:  14%|#4        | 7008/50000 [3:20:18<98:19:48,  8.23s/it] 
Outer training loop:  14%|#4        | 7009/50000 [3:20:20<74:16:23,  6.22s/it]
Outer training loop:  14%|#4        | 7010/50000 [3:20:22<57:47:21,  4.84s/it]
Outer training loop:  14%|#4        | 7011/50000 [3:20:23<45:48:28,  3.84s/it]
Outer training loop:  14%|#4        | 7012/50000 [3:20:25<37:35:48,  3.15s/it]
Outer training loop:  14%|#4        | 7013/50000 [3:20:26<31:44:06,  2.66s/it]
Outer training loop:  14%|#4        | 7014/50000 [3:20:28<28:13:11,  2.36s/it]
Outer training loop:  14%|#4        | 7015/50000 [3:20:29<25:11:50,  2.11s/it]
Outer training loop:  14%|#4        | 7016/50000 [3:20:31<23:20:40,  1.96s/it]
Outer training loop:  14%|#4        | 7017/50000 [3:20:32<21:41:17,  1.82s/it]
Outer training loop:  14%|#4        | 7018/50000 [3:20:34<21:15:48,  1.78s/it]
Outer training loop:  14%|#4        | 7019/50000 [3:20:36<20:27:50,  1.71s/it]
Outer training loop:  14%|#4        | 7020/50000 [3:20:37<19:50:06,  1.66s/it]
Outer training loop:  14%|#4        | 7021/50000 [3:20:39<19:13:14,  1.61s/it]
Outer training loop:  14%|#4        | 7022/50000 [3:20:40<19:13:41,  1.61s/it]
Outer training loop:  14%|#4        | 7023/50000 [3:20:42<19:15:02,  1.61s/it]
Outer training loop:  14%|#4        | 7024/50000 [3:20:44<19:23:36,  1.62s/it]
Outer training loop:  14%|#4        | 7025/50000 [3:20:45<18:49:01,  1.58s/it]
Outer training loop:  14%|#4        | 7026/50000 [3:20:47<19:10:57,  1.61s/it]
Outer training loop:  14%|#4        | 7027/50000 [3:20:48<19:02:33,  1.60s/it]
Outer training loop:  14%|#4        | 7028/50000 [3:20:50<19:17:39,  1.62s/it]
Outer training loop:  14%|#4        | 7029/50000 [3:20:51<18:57:39,  1.59s/it]
Outer training loop:  14%|#4        | 7030/50000 [3:20:53<18:53:41,  1.58s/it]
Outer training loop:  14%|#4        | 7031/50000 [3:20:55<18:43:01,  1.57s/it]
Outer training loop:  14%|#4        | 7032/50000 [3:20:56<18:48:19,  1.58s/it]
Outer training loop:  14%|#4        | 7033/50000 [3:20:58<18:29:21,  1.55s/it]
Outer training loop:  14%|#4        | 7034/50000 [3:20:59<18:46:28,  1.57s/it]
Outer training loop:  14%|#4        | 7035/50000 [3:21:01<18:26:02,  1.54s/it]
Outer training loop:  14%|#4        | 7036/50000 [3:21:02<18:24:12,  1.54s/it]
Outer training loop:  14%|#4        | 7037/50000 [3:21:04<18:39:00,  1.56s/it]
Outer training loop:  14%|#4        | 7038/50000 [3:21:06<18:51:52,  1.58s/it]
Outer training loop:  14%|#4        | 7039/50000 [3:21:07<18:43:36,  1.57s/it]
Outer training loop:  14%|#4        | 7040/50000 [3:21:09<18:24:38,  1.54s/it]
Outer training loop:  14%|#4        | 7041/50000 [3:21:10<18:39:42,  1.56s/it]
Outer training loop:  14%|#4        | 7042/50000 [3:21:12<18:32:51,  1.55s/it]
Outer training loop:  14%|#4        | 7043/50000 [3:21:13<18:36:59,  1.56s/it]
Outer training loop:  14%|#4        | 7044/50000 [3:21:15<18:33:39,  1.56s/it]
Outer training loop:  14%|#4        | 7045/50000 [3:21:17<19:00:24,  1.59s/it]
Outer training loop:  14%|#4        | 7046/50000 [3:21:18<18:41:12,  1.57s/it]
Outer training loop:  14%|#4        | 7047/50000 [3:21:20<19:07:31,  1.60s/it]
Outer training loop:  14%|#4        | 7048/50000 [3:21:21<19:14:18,  1.61s/it]
Outer training loop:  14%|#4        | 7049/50000 [3:21:23<18:58:21,  1.59s/it]
Outer training loop:  14%|#4        | 7050/50000 [3:21:25<19:10:10,  1.61s/it]
Outer training loop:  14%|#4        | 7051/50000 [3:21:26<18:52:26,  1.58s/it]
Outer training loop:  14%|#4        | 7052/50000 [3:21:27<18:16:50,  1.53s/it]
Outer training loop:  14%|#4        | 7053/50000 [3:21:29<18:34:15,  1.56s/it]
Outer training loop:  14%|#4        | 7054/50000 [3:21:31<18:48:04,  1.58s/it]
Outer training loop:  14%|#4        | 7055/50000 [3:21:32<18:50:04,  1.58s/it]
Outer training loop:  14%|#4        | 7056/50000 [3:21:34<18:53:54,  1.58s/it]
Outer training loop:  14%|#4        | 7057/50000 [3:21:36<19:18:24,  1.62s/it]
Outer training loop:  14%|#4        | 7058/50000 [3:21:37<18:49:26,  1.58s/it]
Outer training loop:  14%|#4        | 7059/50000 [3:21:39<18:43:06,  1.57s/it]
Outer training loop:  14%|#4        | 7060/50000 [3:21:40<18:31:51,  1.55s/it]
Outer training loop:  14%|#4        | 7061/50000 [3:21:42<18:31:43,  1.55s/it]
Outer training loop:  14%|#4        | 7062/50000 [3:21:43<18:28:41,  1.55s/it]
Outer training loop:  14%|#4        | 7063/50000 [3:21:45<18:02:47,  1.51s/it]
Outer training loop:  14%|#4        | 7064/50000 [3:21:46<18:34:28,  1.56s/it]
Outer training loop:  14%|#4        | 7065/50000 [3:21:48<18:45:47,  1.57s/it]
Outer training loop:  14%|#4        | 7066/50000 [3:21:49<18:37:09,  1.56s/it]
Outer training loop:  14%|#4        | 7067/50000 [3:21:51<18:46:42,  1.57s/it]
Outer training loop:  14%|#4        | 7068/50000 [3:21:53<18:22:16,  1.54s/it]
Outer training loop:  14%|#4        | 7069/50000 [3:21:54<18:51:19,  1.58s/it]
Outer training loop:  14%|#4        | 7070/50000 [3:21:56<18:35:32,  1.56s/it]
Outer training loop:  14%|#4        | 7071/50000 [3:21:57<18:41:15,  1.57s/it]
Outer training loop:  14%|#4        | 7072/50000 [3:21:59<18:16:46,  1.53s/it]
Outer training loop:  14%|#4        | 7073/50000 [3:22:00<18:42:21,  1.57s/it]
Outer training loop:  14%|#4        | 7074/50000 [3:22:02<18:35:05,  1.56s/it]
Outer training loop:  14%|#4        | 7075/50000 [3:22:04<18:47:22,  1.58s/it]
Outer training loop:  14%|#4        | 7076/50000 [3:22:05<18:36:29,  1.56s/it]
Outer training loop:  14%|#4        | 7077/50000 [3:22:07<18:45:17,  1.57s/it]
Outer training loop:  14%|#4        | 7078/50000 [3:22:08<18:18:41,  1.54s/it]
Outer training loop:  14%|#4        | 7079/50000 [3:22:10<18:23:38,  1.54s/it]
Outer training loop:  14%|#4        | 7080/50000 [3:22:11<18:00:49,  1.51s/it]
Outer training loop:  14%|#4        | 7081/50000 [3:22:13<18:15:58,  1.53s/it]
Outer training loop:  14%|#4        | 7082/50000 [3:22:14<17:57:21,  1.51s/it]
Outer training loop:  14%|#4        | 7083/50000 [3:22:16<18:12:51,  1.53s/it]
Outer training loop:  14%|#4        | 7084/50000 [3:22:17<18:19:12,  1.54s/it]
Outer training loop:  14%|#4        | 7085/50000 [3:22:19<18:23:08,  1.54s/it]
Outer training loop:  14%|#4        | 7086/50000 [3:22:20<18:26:37,  1.55s/it]
Outer training loop:  14%|#4        | 7087/50000 [3:22:22<18:46:26,  1.57s/it]
Outer training loop:  14%|#4        | 7088/50000 [3:22:24<18:54:16,  1.59s/it]
Outer training loop:  14%|#4        | 7089/50000 [3:22:25<18:59:49,  1.59s/it]
Outer training loop:  14%|#4        | 7090/50000 [3:22:27<18:19:17,  1.54s/it]
Outer training loop:  14%|#4        | 7091/50000 [3:22:28<18:16:27,  1.53s/it]
Outer training loop:  14%|#4        | 7092/50000 [3:22:30<18:52:43,  1.58s/it]
Outer training loop:  14%|#4        | 7093/50000 [3:22:32<19:01:23,  1.60s/it]
Outer training loop:  14%|#4        | 7094/50000 [3:22:33<18:56:06,  1.59s/it]
Outer training loop:  14%|#4        | 7095/50000 [3:22:35<18:36:14,  1.56s/it]
Outer training loop:  14%|#4        | 7096/50000 [3:22:36<18:53:40,  1.59s/it]
Outer training loop:  14%|#4        | 7097/50000 [3:22:38<18:31:36,  1.55s/it]
Outer training loop:  14%|#4        | 7098/50000 [3:22:39<18:45:37,  1.57s/it]
Outer training loop:  14%|#4        | 7099/50000 [3:22:41<18:36:23,  1.56s/it]
Outer training loop:  14%|#4        | 7100/50000 [3:22:43<18:56:27,  1.59s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.42s/it]

Evaluating policy:  67%|######6   | 2/3 [00:02<00:01,  1.44s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.45s/it]

Outer training loop:  14%|#4        | 7101/50000 [3:22:48<34:36:20,  2.90s/it]
Outer training loop:  14%|#4        | 7102/50000 [3:22:50<29:37:15,  2.49s/it]
Outer training loop:  14%|#4        | 7103/50000 [3:22:52<26:27:59,  2.22s/it]
Outer training loop:  14%|#4        | 7104/50000 [3:22:53<24:02:57,  2.02s/it]
Outer training loop:  14%|#4        | 7105/50000 [3:22:55<22:42:42,  1.91s/it]
Outer training loop:  14%|#4        | 7106/50000 [3:22:56<21:36:34,  1.81s/it]
Outer training loop:  14%|#4        | 7107/50000 [3:22:58<20:55:14,  1.76s/it]
Outer training loop:  14%|#4        | 7108/50000 [3:23:00<20:04:10,  1.68s/it]
Outer training loop:  14%|#4        | 7109/50000 [3:23:01<19:55:51,  1.67s/it]
Outer training loop:  14%|#4        | 7110/50000 [3:23:03<19:10:28,  1.61s/it]
Outer training loop:  14%|#4        | 7111/50000 [3:23:04<19:53:36,  1.67s/it]
Outer training loop:  14%|#4        | 7112/50000 [3:23:06<19:30:57,  1.64s/it]
Outer training loop:  14%|#4        | 7113/50000 [3:23:07<18:58:22,  1.59s/it]
Outer training loop:  14%|#4        | 7114/50000 [3:23:09<19:06:45,  1.60s/it]
Outer training loop:  14%|#4        | 7115/50000 [3:23:11<18:35:48,  1.56s/it]
Outer training loop:  14%|#4        | 7116/50000 [3:23:12<18:36:04,  1.56s/it]
Outer training loop:  14%|#4        | 7117/50000 [3:23:14<18:22:16,  1.54s/it]
Outer training loop:  14%|#4        | 7118/50000 [3:23:15<18:42:32,  1.57s/it]
Outer training loop:  14%|#4        | 7119/50000 [3:23:17<18:35:29,  1.56s/it]
Outer training loop:  14%|#4        | 7120/50000 [3:23:18<18:26:46,  1.55s/it]
Outer training loop:  14%|#4        | 7121/50000 [3:23:20<18:18:55,  1.54s/it]
Outer training loop:  14%|#4        | 7122/50000 [3:23:21<18:31:41,  1.56s/it]
Outer training loop:  14%|#4        | 7123/50000 [3:23:23<18:23:52,  1.54s/it]
Outer training loop:  14%|#4        | 7124/50000 [3:23:25<18:53:13,  1.59s/it]
Outer training loop:  14%|#4        | 7125/50000 [3:23:26<18:36:31,  1.56s/it]
Outer training loop:  14%|#4        | 7126/50000 [3:23:28<18:40:01,  1.57s/it]
Outer training loop:  14%|#4        | 7127/50000 [3:23:29<18:28:04,  1.55s/it]
Outer training loop:  14%|#4        | 7128/50000 [3:23:31<18:47:56,  1.58s/it]
Outer training loop:  14%|#4        | 7129/50000 [3:23:32<18:35:57,  1.56s/it]
Outer training loop:  14%|#4        | 7130/50000 [3:23:34<19:06:21,  1.60s/it]
Outer training loop:  14%|#4        | 7131/50000 [3:23:36<19:00:30,  1.60s/it]
Outer training loop:  14%|#4        | 7132/50000 [3:23:37<19:02:40,  1.60s/it]
Outer training loop:  14%|#4        | 7133/50000 [3:23:39<18:39:11,  1.57s/it]
Outer training loop:  14%|#4        | 7134/50000 [3:23:40<18:29:59,  1.55s/it]
Outer training loop:  14%|#4        | 7135/50000 [3:23:42<18:26:28,  1.55s/it]
Outer training loop:  14%|#4        | 7136/50000 [3:23:43<18:38:43,  1.57s/it]
Outer training loop:  14%|#4        | 7137/50000 [3:23:45<18:22:58,  1.54s/it]
Outer training loop:  14%|#4        | 7138/50000 [3:23:47<18:32:13,  1.56s/it]
Outer training loop:  14%|#4        | 7139/50000 [3:23:48<18:09:42,  1.53s/it]
Outer training loop:  14%|#4        | 7140/50000 [3:23:50<18:46:45,  1.58s/it]
Outer training loop:  14%|#4        | 7141/50000 [3:23:51<19:09:27,  1.61s/it]
Outer training loop:  14%|#4        | 7142/50000 [3:23:53<18:56:54,  1.59s/it]
Outer training loop:  14%|#4        | 7143/50000 [3:23:54<18:51:27,  1.58s/it]
Outer training loop:  14%|#4        | 7144/50000 [3:23:56<19:03:04,  1.60s/it]
Outer training loop:  14%|#4        | 7145/50000 [3:23:58<19:07:34,  1.61s/it]
Outer training loop:  14%|#4        | 7146/50000 [3:23:59<19:10:06,  1.61s/it]
Outer training loop:  14%|#4        | 7147/50000 [3:24:01<19:01:21,  1.60s/it]
Outer training loop:  14%|#4        | 7148/50000 [3:24:02<18:31:39,  1.56s/it]
Outer training loop:  14%|#4        | 7149/50000 [3:24:04<18:30:27,  1.55s/it]
Outer training loop:  14%|#4        | 7150/50000 [3:24:06<18:58:48,  1.59s/it]
Outer training loop:  14%|#4        | 7151/50000 [3:24:07<18:43:40,  1.57s/it]
Outer training loop:  14%|#4        | 7152/50000 [3:24:09<18:52:59,  1.59s/it]
Outer training loop:  14%|#4        | 7153/50000 [3:24:10<18:55:30,  1.59s/it]
Outer training loop:  14%|#4        | 7154/50000 [3:24:12<18:44:19,  1.57s/it]
Outer training loop:  14%|#4        | 7155/50000 [3:24:14<19:08:42,  1.61s/it]
Outer training loop:  14%|#4        | 7156/50000 [3:24:15<18:43:13,  1.57s/it]
Outer training loop:  14%|#4        | 7157/50000 [3:24:17<18:34:24,  1.56s/it]
Outer training loop:  14%|#4        | 7158/50000 [3:24:18<18:56:57,  1.59s/it]
Outer training loop:  14%|#4        | 7159/50000 [3:24:20<18:54:06,  1.59s/it]
Outer training loop:  14%|#4        | 7160/50000 [3:24:21<18:56:21,  1.59s/it]
Outer training loop:  14%|#4        | 7161/50000 [3:24:23<19:23:21,  1.63s/it]
Outer training loop:  14%|#4        | 7162/50000 [3:24:25<19:21:47,  1.63s/it]
Outer training loop:  14%|#4        | 7163/50000 [3:24:26<18:37:47,  1.57s/it]
Outer training loop:  14%|#4        | 7164/50000 [3:24:28<19:04:45,  1.60s/it]
Outer training loop:  14%|#4        | 7165/50000 [3:24:29<18:47:25,  1.58s/it]
Outer training loop:  14%|#4        | 7166/50000 [3:24:31<18:55:21,  1.59s/it]
Outer training loop:  14%|#4        | 7167/50000 [3:24:33<18:51:24,  1.58s/it]
Outer training loop:  14%|#4        | 7168/50000 [3:24:34<19:02:59,  1.60s/it]
Outer training loop:  14%|#4        | 7169/50000 [3:24:36<18:29:59,  1.55s/it]
Outer training loop:  14%|#4        | 7170/50000 [3:24:37<18:43:25,  1.57s/it]
Outer training loop:  14%|#4        | 7171/50000 [3:24:39<18:57:27,  1.59s/it]
Outer training loop:  14%|#4        | 7172/50000 [3:24:41<19:04:11,  1.60s/it]
Outer training loop:  14%|#4        | 7173/50000 [3:24:42<18:42:52,  1.57s/it]
Outer training loop:  14%|#4        | 7174/50000 [3:24:44<18:39:50,  1.57s/it]
Outer training loop:  14%|#4        | 7175/50000 [3:24:45<18:32:37,  1.56s/it]
Outer training loop:  14%|#4        | 7176/50000 [3:24:47<18:43:58,  1.57s/it]
Outer training loop:  14%|#4        | 7177/50000 [3:24:48<18:29:03,  1.55s/it]
Outer training loop:  14%|#4        | 7178/50000 [3:24:50<18:51:50,  1.59s/it]
Outer training loop:  14%|#4        | 7179/50000 [3:24:51<18:37:06,  1.57s/it]
Outer training loop:  14%|#4        | 7180/50000 [3:24:53<19:01:30,  1.60s/it]
Outer training loop:  14%|#4        | 7181/50000 [3:24:55<18:39:48,  1.57s/it]
Outer training loop:  14%|#4        | 7182/50000 [3:24:56<18:48:10,  1.58s/it]
Outer training loop:  14%|#4        | 7183/50000 [3:24:58<18:28:55,  1.55s/it]
Outer training loop:  14%|#4        | 7184/50000 [3:24:59<18:51:44,  1.59s/it]
Outer training loop:  14%|#4        | 7185/50000 [3:25:01<18:31:34,  1.56s/it]
Outer training loop:  14%|#4        | 7186/50000 [3:25:03<18:36:36,  1.56s/it]
Outer training loop:  14%|#4        | 7187/50000 [3:25:04<18:30:55,  1.56s/it]
Outer training loop:  14%|#4        | 7188/50000 [3:25:06<18:51:38,  1.59s/it]
Outer training loop:  14%|#4        | 7189/50000 [3:25:07<19:00:58,  1.60s/it]
Outer training loop:  14%|#4        | 7190/50000 [3:25:09<18:38:35,  1.57s/it]
Outer training loop:  14%|#4        | 7191/50000 [3:25:10<18:25:14,  1.55s/it]
Outer training loop:  14%|#4        | 7192/50000 [3:25:12<18:36:09,  1.56s/it]
Outer training loop:  14%|#4        | 7193/50000 [3:25:13<18:24:31,  1.55s/it]
Outer training loop:  14%|#4        | 7194/50000 [3:25:15<18:54:38,  1.59s/it]
Outer training loop:  14%|#4        | 7195/50000 [3:25:17<18:51:29,  1.59s/it]
Outer training loop:  14%|#4        | 7196/50000 [3:25:18<18:57:25,  1.59s/it]
Outer training loop:  14%|#4        | 7197/50000 [3:25:20<18:51:44,  1.59s/it]
Outer training loop:  14%|#4        | 7198/50000 [3:25:21<18:35:45,  1.56s/it]
Outer training loop:  14%|#4        | 7199/50000 [3:25:23<18:21:36,  1.54s/it]
Outer training loop:  14%|#4        | 7200/50000 [3:25:25<18:48:30,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.57s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.56s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.52s/it]

Outer training loop:  14%|#4        | 7201/50000 [3:25:31<34:47:30,  2.93s/it]
Outer training loop:  14%|#4        | 7202/50000 [3:25:32<29:49:48,  2.51s/it]
Outer training loop:  14%|#4        | 7203/50000 [3:25:34<26:48:28,  2.26s/it]
Outer training loop:  14%|#4        | 7204/50000 [3:25:35<24:15:41,  2.04s/it]
Outer training loop:  14%|#4        | 7205/50000 [3:25:37<22:40:47,  1.91s/it]
Outer training loop:  14%|#4        | 7206/50000 [3:25:39<21:21:31,  1.80s/it]
Outer training loop:  14%|#4        | 7207/50000 [3:25:40<20:28:57,  1.72s/it]
Outer training loop:  14%|#4        | 7208/50000 [3:25:42<20:12:22,  1.70s/it]
Outer training loop:  14%|#4        | 7209/50000 [3:25:43<20:07:08,  1.69s/it]
Outer training loop:  14%|#4        | 7210/50000 [3:25:45<19:20:15,  1.63s/it]
Outer training loop:  14%|#4        | 7211/50000 [3:25:47<19:34:10,  1.65s/it]
Outer training loop:  14%|#4        | 7212/50000 [3:25:48<18:58:34,  1.60s/it]
Outer training loop:  14%|#4        | 7213/50000 [3:25:50<19:20:40,  1.63s/it]
Outer training loop:  14%|#4        | 7214/50000 [3:25:51<18:42:46,  1.57s/it]
Outer training loop:  14%|#4        | 7215/50000 [3:25:53<18:36:17,  1.57s/it]
Outer training loop:  14%|#4        | 7216/50000 [3:25:54<18:59:36,  1.60s/it]
Outer training loop:  14%|#4        | 7217/50000 [3:25:56<18:34:48,  1.56s/it]
Outer training loop:  14%|#4        | 7218/50000 [3:25:57<18:36:18,  1.57s/it]
Outer training loop:  14%|#4        | 7219/50000 [3:25:59<18:03:47,  1.52s/it]
Outer training loop:  14%|#4        | 7220/50000 [3:26:00<18:19:26,  1.54s/it]
Outer training loop:  14%|#4        | 7221/50000 [3:26:02<18:13:11,  1.53s/it]
Outer training loop:  14%|#4        | 7222/50000 [3:26:04<18:28:20,  1.55s/it]
Outer training loop:  14%|#4        | 7223/50000 [3:26:05<18:45:27,  1.58s/it]
Outer training loop:  14%|#4        | 7224/50000 [3:26:07<18:41:09,  1.57s/it]
Outer training loop:  14%|#4        | 7225/50000 [3:26:08<18:23:40,  1.55s/it]
Outer training loop:  14%|#4        | 7226/50000 [3:26:10<18:31:46,  1.56s/it]
Outer training loop:  14%|#4        | 7227/50000 [3:26:11<18:32:50,  1.56s/it]
Outer training loop:  14%|#4        | 7228/50000 [3:26:13<18:37:13,  1.57s/it]
Outer training loop:  14%|#4        | 7229/50000 [3:26:15<18:30:00,  1.56s/it]
Outer training loop:  14%|#4        | 7230/50000 [3:26:16<18:32:42,  1.56s/it]
Outer training loop:  14%|#4        | 7231/50000 [3:26:18<18:31:52,  1.56s/it]
Outer training loop:  14%|#4        | 7232/50000 [3:26:19<18:37:12,  1.57s/it]
Outer training loop:  14%|#4        | 7233/50000 [3:26:21<18:11:13,  1.53s/it]
Outer training loop:  14%|#4        | 7234/50000 [3:26:22<18:33:13,  1.56s/it]
Outer training loop:  14%|#4        | 7235/50000 [3:26:24<18:24:18,  1.55s/it]
Outer training loop:  14%|#4        | 7236/50000 [3:26:25<18:11:05,  1.53s/it]
Outer training loop:  14%|#4        | 7237/50000 [3:26:27<18:22:25,  1.55s/it]
Outer training loop:  14%|#4        | 7238/50000 [3:26:28<18:02:59,  1.52s/it]
Outer training loop:  14%|#4        | 7239/50000 [3:26:30<18:41:05,  1.57s/it]
Outer training loop:  14%|#4        | 7240/50000 [3:26:32<18:23:44,  1.55s/it]
Outer training loop:  14%|#4        | 7241/50000 [3:26:33<18:58:16,  1.60s/it]
Outer training loop:  14%|#4        | 7242/50000 [3:26:35<18:42:06,  1.57s/it]
Outer training loop:  14%|#4        | 7243/50000 [3:26:36<18:58:50,  1.60s/it]
Outer training loop:  14%|#4        | 7244/50000 [3:26:38<18:43:09,  1.58s/it]
Outer training loop:  14%|#4        | 7245/50000 [3:26:40<18:50:16,  1.59s/it]
Outer training loop:  14%|#4        | 7246/50000 [3:26:41<18:43:01,  1.58s/it]
Outer training loop:  14%|#4        | 7247/50000 [3:26:43<18:45:38,  1.58s/it]
Outer training loop:  14%|#4        | 7248/50000 [3:26:44<18:42:37,  1.58s/it]
Outer training loop:  14%|#4        | 7249/50000 [3:26:46<18:47:09,  1.58s/it]
Outer training loop:  14%|#4        | 7250/50000 [3:26:47<18:31:00,  1.56s/it]
Outer training loop:  15%|#4        | 7251/50000 [3:26:49<18:47:31,  1.58s/it]
Outer training loop:  15%|#4        | 7252/50000 [3:26:51<18:34:59,  1.56s/it]
Outer training loop:  15%|#4        | 7253/50000 [3:26:52<18:44:43,  1.58s/it]
Outer training loop:  15%|#4        | 7254/50000 [3:26:54<18:45:08,  1.58s/it]
Outer training loop:  15%|#4        | 7255/50000 [3:26:55<18:05:16,  1.52s/it]
Outer training loop:  15%|#4        | 7256/50000 [3:26:57<17:53:18,  1.51s/it]
Outer training loop:  15%|#4        | 7257/50000 [3:26:58<17:58:34,  1.51s/it]
Outer training loop:  15%|#4        | 7258/50000 [3:27:00<18:31:55,  1.56s/it]
Outer training loop:  15%|#4        | 7259/50000 [3:27:01<18:47:03,  1.58s/it]
Outer training loop:  15%|#4        | 7260/50000 [3:27:03<18:46:30,  1.58s/it]
Outer training loop:  15%|#4        | 7261/50000 [3:27:05<18:39:09,  1.57s/it]
Outer training loop:  15%|#4        | 7262/50000 [3:27:06<18:06:52,  1.53s/it]
Outer training loop:  15%|#4        | 7263/50000 [3:27:08<18:27:00,  1.55s/it]
Outer training loop:  15%|#4        | 7264/50000 [3:27:09<18:19:10,  1.54s/it]
Outer training loop:  15%|#4        | 7265/50000 [3:27:11<18:42:19,  1.58s/it]
Outer training loop:  15%|#4        | 7266/50000 [3:27:12<18:19:28,  1.54s/it]
Outer training loop:  15%|#4        | 7267/50000 [3:27:14<18:12:23,  1.53s/it]
Outer training loop:  15%|#4        | 7268/50000 [3:27:15<18:34:30,  1.56s/it]
Outer training loop:  15%|#4        | 7269/50000 [3:27:17<18:51:05,  1.59s/it]
Outer training loop:  15%|#4        | 7270/50000 [3:27:19<18:57:12,  1.60s/it]
Outer training loop:  15%|#4        | 7271/50000 [3:27:20<18:32:01,  1.56s/it]
Outer training loop:  15%|#4        | 7272/50000 [3:27:22<18:48:49,  1.59s/it]
Outer training loop:  15%|#4        | 7273/50000 [3:27:23<18:34:10,  1.56s/it]
Outer training loop:  15%|#4        | 7274/50000 [3:27:25<18:27:59,  1.56s/it]
Outer training loop:  15%|#4        | 7275/50000 [3:27:26<18:12:21,  1.53s/it]
Outer training loop:  15%|#4        | 7276/50000 [3:27:28<18:21:13,  1.55s/it]
Outer training loop:  15%|#4        | 7277/50000 [3:27:29<18:06:41,  1.53s/it]
Outer training loop:  15%|#4        | 7278/50000 [3:27:31<18:16:58,  1.54s/it]
Outer training loop:  15%|#4        | 7279/50000 [3:27:32<18:08:38,  1.53s/it]
Outer training loop:  15%|#4        | 7280/50000 [3:27:34<18:00:34,  1.52s/it]
Outer training loop:  15%|#4        | 7281/50000 [3:27:36<18:26:35,  1.55s/it]
Outer training loop:  15%|#4        | 7282/50000 [3:27:37<18:38:54,  1.57s/it]
Outer training loop:  15%|#4        | 7283/50000 [3:27:39<18:25:58,  1.55s/it]
Outer training loop:  15%|#4        | 7284/50000 [3:27:40<19:02:37,  1.60s/it]
Outer training loop:  15%|#4        | 7285/50000 [3:27:42<18:42:52,  1.58s/it]
Outer training loop:  15%|#4        | 7286/50000 [3:27:43<18:33:39,  1.56s/it]
Outer training loop:  15%|#4        | 7287/50000 [3:27:45<18:41:29,  1.58s/it]
Outer training loop:  15%|#4        | 7288/50000 [3:27:47<18:34:03,  1.56s/it]
Outer training loop:  15%|#4        | 7289/50000 [3:27:48<18:35:49,  1.57s/it]
Outer training loop:  15%|#4        | 7290/50000 [3:27:50<18:46:17,  1.58s/it]
Outer training loop:  15%|#4        | 7291/50000 [3:27:51<18:48:12,  1.58s/it]
Outer training loop:  15%|#4        | 7292/50000 [3:27:53<18:24:24,  1.55s/it]
Outer training loop:  15%|#4        | 7293/50000 [3:27:54<18:39:16,  1.57s/it]
Outer training loop:  15%|#4        | 7294/50000 [3:27:56<18:24:54,  1.55s/it]
Outer training loop:  15%|#4        | 7295/50000 [3:27:58<18:34:05,  1.57s/it]
Outer training loop:  15%|#4        | 7296/50000 [3:27:59<18:29:41,  1.56s/it]
Outer training loop:  15%|#4        | 7297/50000 [3:28:01<18:23:18,  1.55s/it]
Outer training loop:  15%|#4        | 7298/50000 [3:28:02<18:16:51,  1.54s/it]
Outer training loop:  15%|#4        | 7299/50000 [3:28:04<18:52:55,  1.59s/it]
Outer training loop:  15%|#4        | 7300/50000 [3:28:05<18:41:42,  1.58s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.45s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.51s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.47s/it]

Outer training loop:  15%|#4        | 7301/50000 [3:28:12<34:49:51,  2.94s/it]
Outer training loop:  15%|#4        | 7302/50000 [3:28:13<30:00:52,  2.53s/it]
Outer training loop:  15%|#4        | 7303/50000 [3:28:15<26:09:45,  2.21s/it]
Outer training loop:  15%|#4        | 7304/50000 [3:28:16<23:51:38,  2.01s/it]
Outer training loop:  15%|#4        | 7305/50000 [3:28:18<22:57:08,  1.94s/it]
Outer training loop:  15%|#4        | 7306/50000 [3:28:19<21:35:12,  1.82s/it]
Outer training loop:  15%|#4        | 7307/50000 [3:28:21<20:51:20,  1.76s/it]
Outer training loop:  15%|#4        | 7308/50000 [3:28:23<20:18:03,  1.71s/it]
Outer training loop:  15%|#4        | 7309/50000 [3:28:24<19:26:40,  1.64s/it]
Outer training loop:  15%|#4        | 7310/50000 [3:28:26<18:54:28,  1.59s/it]
Outer training loop:  15%|#4        | 7311/50000 [3:28:27<18:46:34,  1.58s/it]
Outer training loop:  15%|#4        | 7312/50000 [3:28:29<19:12:16,  1.62s/it]
Outer training loop:  15%|#4        | 7313/50000 [3:28:30<18:54:56,  1.60s/it]
Outer training loop:  15%|#4        | 7314/50000 [3:28:32<18:20:33,  1.55s/it]
Outer training loop:  15%|#4        | 7315/50000 [3:28:34<18:46:51,  1.58s/it]
Outer training loop:  15%|#4        | 7316/50000 [3:28:35<18:44:39,  1.58s/it]
Outer training loop:  15%|#4        | 7317/50000 [3:28:37<18:33:49,  1.57s/it]
Outer training loop:  15%|#4        | 7318/50000 [3:28:38<18:28:18,  1.56s/it]
Outer training loop:  15%|#4        | 7319/50000 [3:28:40<18:18:55,  1.54s/it]
Outer training loop:  15%|#4        | 7320/50000 [3:28:41<18:11:07,  1.53s/it]
Outer training loop:  15%|#4        | 7321/50000 [3:28:43<18:48:25,  1.59s/it]
Outer training loop:  15%|#4        | 7322/50000 [3:28:44<18:30:00,  1.56s/it]
Outer training loop:  15%|#4        | 7323/50000 [3:28:46<18:26:00,  1.55s/it]
Outer training loop:  15%|#4        | 7324/50000 [3:28:47<18:18:26,  1.54s/it]
Outer training loop:  15%|#4        | 7325/50000 [3:28:49<18:50:11,  1.59s/it]
Outer training loop:  15%|#4        | 7326/50000 [3:28:51<18:31:03,  1.56s/it]
Outer training loop:  15%|#4        | 7327/50000 [3:28:52<18:33:53,  1.57s/it]
Outer training loop:  15%|#4        | 7328/50000 [3:28:54<18:10:00,  1.53s/it]
Outer training loop:  15%|#4        | 7329/50000 [3:28:55<18:22:40,  1.55s/it]
Outer training loop:  15%|#4        | 7330/50000 [3:28:57<18:19:45,  1.55s/it]
Outer training loop:  15%|#4        | 7331/50000 [3:28:58<18:38:06,  1.57s/it]
Outer training loop:  15%|#4        | 7332/50000 [3:29:00<18:22:46,  1.55s/it]
Outer training loop:  15%|#4        | 7333/50000 [3:29:02<18:33:27,  1.57s/it]
Outer training loop:  15%|#4        | 7334/50000 [3:29:03<18:30:38,  1.56s/it]
Outer training loop:  15%|#4        | 7335/50000 [3:29:05<18:39:32,  1.57s/it]
Outer training loop:  15%|#4        | 7336/50000 [3:29:06<18:19:33,  1.55s/it]
Outer training loop:  15%|#4        | 7337/50000 [3:29:08<18:16:42,  1.54s/it]
Outer training loop:  15%|#4        | 7338/50000 [3:29:09<18:25:57,  1.56s/it]
Outer training loop:  15%|#4        | 7339/50000 [3:29:11<18:45:39,  1.58s/it]
Outer training loop:  15%|#4        | 7340/50000 [3:29:12<18:31:06,  1.56s/it]
Outer training loop:  15%|#4        | 7341/50000 [3:29:14<18:17:49,  1.54s/it]
Outer training loop:  15%|#4        | 7342/50000 [3:29:16<18:21:45,  1.55s/it]
Outer training loop:  15%|#4        | 7343/50000 [3:29:17<18:04:06,  1.52s/it]
Outer training loop:  15%|#4        | 7344/50000 [3:29:19<18:29:11,  1.56s/it]
Outer training loop:  15%|#4        | 7345/50000 [3:29:20<18:56:18,  1.60s/it]
Outer training loop:  15%|#4        | 7346/50000 [3:29:22<18:36:57,  1.57s/it]
Outer training loop:  15%|#4        | 7347/50000 [3:29:23<18:14:47,  1.54s/it]
Outer training loop:  15%|#4        | 7348/50000 [3:29:25<18:28:37,  1.56s/it]
Outer training loop:  15%|#4        | 7349/50000 [3:29:26<18:18:19,  1.55s/it]
Outer training loop:  15%|#4        | 7350/50000 [3:29:28<18:11:56,  1.54s/it]
Outer training loop:  15%|#4        | 7351/50000 [3:29:30<18:34:08,  1.57s/it]
Outer training loop:  15%|#4        | 7352/50000 [3:29:31<18:22:51,  1.55s/it]
Outer training loop:  15%|#4        | 7353/50000 [3:29:33<18:46:33,  1.58s/it]
Outer training loop:  15%|#4        | 7354/50000 [3:29:34<18:42:22,  1.58s/it]
Outer training loop:  15%|#4        | 7355/50000 [3:29:36<19:14:34,  1.62s/it]
Outer training loop:  15%|#4        | 7356/50000 [3:29:38<18:50:47,  1.59s/it]
Outer training loop:  15%|#4        | 7357/50000 [3:29:39<18:43:46,  1.58s/it]
Outer training loop:  15%|#4        | 7358/50000 [3:29:41<18:36:21,  1.57s/it]
Outer training loop:  15%|#4        | 7359/50000 [3:29:42<18:42:56,  1.58s/it]
Outer training loop:  15%|#4        | 7360/50000 [3:29:44<18:29:06,  1.56s/it]
Outer training loop:  15%|#4        | 7361/50000 [3:29:45<18:51:10,  1.59s/it]
Outer training loop:  15%|#4        | 7362/50000 [3:29:47<18:41:13,  1.58s/it]
Outer training loop:  15%|#4        | 7363/50000 [3:29:48<18:26:01,  1.56s/it]
Outer training loop:  15%|#4        | 7364/50000 [3:29:50<18:45:47,  1.58s/it]
Outer training loop:  15%|#4        | 7365/50000 [3:29:52<18:51:55,  1.59s/it]
Outer training loop:  15%|#4        | 7366/50000 [3:29:53<18:36:07,  1.57s/it]
Outer training loop:  15%|#4        | 7367/50000 [3:29:55<18:27:44,  1.56s/it]
Outer training loop:  15%|#4        | 7368/50000 [3:29:56<18:27:56,  1.56s/it]
Outer training loop:  15%|#4        | 7369/50000 [3:29:58<18:30:18,  1.56s/it]
Outer training loop:  15%|#4        | 7370/50000 [3:30:00<18:56:49,  1.60s/it]
Outer training loop:  15%|#4        | 7371/50000 [3:30:01<18:55:18,  1.60s/it]
Outer training loop:  15%|#4        | 7372/50000 [3:30:03<18:30:30,  1.56s/it]
Outer training loop:  15%|#4        | 7373/50000 [3:30:04<18:26:13,  1.56s/it]
Outer training loop:  15%|#4        | 7374/50000 [3:30:06<18:39:16,  1.58s/it]
Outer training loop:  15%|#4        | 7375/50000 [3:30:07<18:50:13,  1.59s/it]
Outer training loop:  15%|#4        | 7376/50000 [3:30:09<18:28:36,  1.56s/it]
Outer training loop:  15%|#4        | 7377/50000 [3:30:10<18:11:25,  1.54s/it]
Outer training loop:  15%|#4        | 7378/50000 [3:30:12<18:16:27,  1.54s/it]
Outer training loop:  15%|#4        | 7379/50000 [3:30:13<17:58:34,  1.52s/it]
Outer training loop:  15%|#4        | 7380/50000 [3:30:15<18:10:17,  1.53s/it]
Outer training loop:  15%|#4        | 7381/50000 [3:30:17<18:02:24,  1.52s/it]
Outer training loop:  15%|#4        | 7382/50000 [3:30:18<18:36:08,  1.57s/it]
Outer training loop:  15%|#4        | 7383/50000 [3:30:20<18:27:46,  1.56s/it]
Outer training loop:  15%|#4        | 7384/50000 [3:30:21<18:28:55,  1.56s/it]
Outer training loop:  15%|#4        | 7385/50000 [3:30:23<18:09:47,  1.53s/it]
Outer training loop:  15%|#4        | 7386/50000 [3:30:24<18:37:51,  1.57s/it]
Outer training loop:  15%|#4        | 7387/50000 [3:30:26<18:15:46,  1.54s/it]
Outer training loop:  15%|#4        | 7388/50000 [3:30:27<18:22:38,  1.55s/it]
Outer training loop:  15%|#4        | 7389/50000 [3:30:29<18:12:39,  1.54s/it]
Outer training loop:  15%|#4        | 7390/50000 [3:30:31<18:23:59,  1.55s/it]
Outer training loop:  15%|#4        | 7391/50000 [3:30:32<18:23:41,  1.55s/it]
Outer training loop:  15%|#4        | 7392/50000 [3:30:34<18:59:39,  1.60s/it]
Outer training loop:  15%|#4        | 7393/50000 [3:30:35<18:42:49,  1.58s/it]
Outer training loop:  15%|#4        | 7394/50000 [3:30:37<18:56:37,  1.60s/it]
Outer training loop:  15%|#4        | 7395/50000 [3:30:38<18:25:24,  1.56s/it]
Outer training loop:  15%|#4        | 7396/50000 [3:30:40<18:44:13,  1.58s/it]
Outer training loop:  15%|#4        | 7397/50000 [3:30:42<18:30:10,  1.56s/it]
Outer training loop:  15%|#4        | 7398/50000 [3:30:43<18:20:54,  1.55s/it]
Outer training loop:  15%|#4        | 7399/50000 [3:30:45<18:03:30,  1.53s/it]
Outer training loop:  15%|#4        | 7400/50000 [3:30:46<18:12:29,  1.54s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.61s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.57s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.54s/it]

Outer training loop:  15%|#4        | 7401/50000 [3:30:52<34:24:51,  2.91s/it]
Outer training loop:  15%|#4        | 7402/50000 [3:30:54<29:59:47,  2.54s/it]
Outer training loop:  15%|#4        | 7403/50000 [3:30:56<26:44:08,  2.26s/it]
Outer training loop:  15%|#4        | 7404/50000 [3:30:57<24:05:53,  2.04s/it]
Outer training loop:  15%|#4        | 7405/50000 [3:30:59<22:17:55,  1.88s/it]
Outer training loop:  15%|#4        | 7406/50000 [3:31:00<20:59:39,  1.77s/it]
Outer training loop:  15%|#4        | 7407/50000 [3:31:02<20:25:17,  1.73s/it]
Outer training loop:  15%|#4        | 7408/50000 [3:31:03<19:26:30,  1.64s/it]
Outer training loop:  15%|#4        | 7409/50000 [3:31:05<19:38:14,  1.66s/it]
Outer training loop:  15%|#4        | 7410/50000 [3:31:06<19:08:15,  1.62s/it]
Outer training loop:  15%|#4        | 7411/50000 [3:31:08<19:06:19,  1.61s/it]
Outer training loop:  15%|#4        | 7412/50000 [3:31:10<18:45:15,  1.59s/it]
Outer training loop:  15%|#4        | 7413/50000 [3:31:11<18:45:53,  1.59s/it]
Outer training loop:  15%|#4        | 7414/50000 [3:31:13<18:35:02,  1.57s/it]
Outer training loop:  15%|#4        | 7415/50000 [3:31:14<18:38:55,  1.58s/it]
Outer training loop:  15%|#4        | 7416/50000 [3:31:16<18:19:40,  1.55s/it]
Outer training loop:  15%|#4        | 7417/50000 [3:31:17<18:01:32,  1.52s/it]
Outer training loop:  15%|#4        | 7418/50000 [3:31:19<18:16:19,  1.54s/it]
Outer training loop:  15%|#4        | 7419/50000 [3:31:20<18:10:53,  1.54s/it]
Outer training loop:  15%|#4        | 7420/50000 [3:31:22<18:09:33,  1.54s/it]
Outer training loop:  15%|#4        | 7421/50000 [3:31:23<17:49:01,  1.51s/it]
Outer training loop:  15%|#4        | 7422/50000 [3:31:25<18:23:58,  1.56s/it]
Outer training loop:  15%|#4        | 7423/50000 [3:31:26<18:10:54,  1.54s/it]
Outer training loop:  15%|#4        | 7424/50000 [3:31:28<18:36:59,  1.57s/it]
Outer training loop:  15%|#4        | 7425/50000 [3:31:30<18:17:49,  1.55s/it]
Outer training loop:  15%|#4        | 7426/50000 [3:31:31<18:11:49,  1.54s/it]
Outer training loop:  15%|#4        | 7427/50000 [3:31:33<18:03:35,  1.53s/it]
Outer training loop:  15%|#4        | 7428/50000 [3:31:34<18:19:16,  1.55s/it]
Outer training loop:  15%|#4        | 7429/50000 [3:31:36<18:07:23,  1.53s/it]
Outer training loop:  15%|#4        | 7430/50000 [3:31:37<18:30:25,  1.57s/it]
Outer training loop:  15%|#4        | 7431/50000 [3:31:39<18:18:19,  1.55s/it]
Outer training loop:  15%|#4        | 7432/50000 [3:31:40<18:31:11,  1.57s/it]
Outer training loop:  15%|#4        | 7433/50000 [3:31:42<18:24:28,  1.56s/it]
Outer training loop:  15%|#4        | 7434/50000 [3:31:44<18:35:54,  1.57s/it]
Outer training loop:  15%|#4        | 7435/50000 [3:31:45<18:06:31,  1.53s/it]
Outer training loop:  15%|#4        | 7436/50000 [3:31:47<17:54:33,  1.51s/it]
Outer training loop:  15%|#4        | 7437/50000 [3:31:48<18:18:53,  1.55s/it]
Outer training loop:  15%|#4        | 7438/50000 [3:31:50<18:06:13,  1.53s/it]
Outer training loop:  15%|#4        | 7439/50000 [3:31:51<18:28:16,  1.56s/it]
Outer training loop:  15%|#4        | 7440/50000 [3:31:53<18:17:49,  1.55s/it]
Outer training loop:  15%|#4        | 7441/50000 [3:31:54<18:18:54,  1.55s/it]
Outer training loop:  15%|#4        | 7442/50000 [3:31:56<18:03:09,  1.53s/it]
Outer training loop:  15%|#4        | 7443/50000 [3:31:57<18:17:36,  1.55s/it]
Outer training loop:  15%|#4        | 7444/50000 [3:31:59<18:08:33,  1.53s/it]
Outer training loop:  15%|#4        | 7445/50000 [3:32:01<18:26:22,  1.56s/it]
Outer training loop:  15%|#4        | 7446/50000 [3:32:02<18:36:05,  1.57s/it]
Outer training loop:  15%|#4        | 7447/50000 [3:32:04<18:22:30,  1.55s/it]
Outer training loop:  15%|#4        | 7448/50000 [3:32:05<17:59:42,  1.52s/it]
Outer training loop:  15%|#4        | 7449/50000 [3:32:07<18:13:26,  1.54s/it]
Outer training loop:  15%|#4        | 7450/50000 [3:32:08<18:07:55,  1.53s/it]
Outer training loop:  15%|#4        | 7451/50000 [3:32:10<18:18:19,  1.55s/it]
Outer training loop:  15%|#4        | 7452/50000 [3:32:12<18:59:01,  1.61s/it]
Outer training loop:  15%|#4        | 7453/50000 [3:32:13<18:52:12,  1.60s/it]
Outer training loop:  15%|#4        | 7454/50000 [3:32:15<18:40:32,  1.58s/it]
Outer training loop:  15%|#4        | 7455/50000 [3:32:16<19:03:19,  1.61s/it]
Outer training loop:  15%|#4        | 7456/50000 [3:32:18<18:58:57,  1.61s/it]
Outer training loop:  15%|#4        | 7457/50000 [3:32:20<18:52:30,  1.60s/it]
Outer training loop:  15%|#4        | 7458/50000 [3:32:21<18:40:27,  1.58s/it]
Outer training loop:  15%|#4        | 7459/50000 [3:32:23<18:47:07,  1.59s/it]
Outer training loop:  15%|#4        | 7460/50000 [3:32:24<18:32:56,  1.57s/it]
Outer training loop:  15%|#4        | 7461/50000 [3:32:26<18:45:12,  1.59s/it]
Outer training loop:  15%|#4        | 7462/50000 [3:32:27<18:35:09,  1.57s/it]
Outer training loop:  15%|#4        | 7463/50000 [3:32:29<18:51:03,  1.60s/it]
Outer training loop:  15%|#4        | 7464/50000 [3:32:31<18:48:27,  1.59s/it]
Outer training loop:  15%|#4        | 7465/50000 [3:32:32<18:51:09,  1.60s/it]
Outer training loop:  15%|#4        | 7466/50000 [3:32:34<18:39:23,  1.58s/it]
Outer training loop:  15%|#4        | 7467/50000 [3:32:35<18:42:20,  1.58s/it]
Outer training loop:  15%|#4        | 7468/50000 [3:32:37<18:31:50,  1.57s/it]
Outer training loop:  15%|#4        | 7469/50000 [3:32:38<18:38:29,  1.58s/it]
Outer training loop:  15%|#4        | 7470/50000 [3:32:40<18:30:59,  1.57s/it]
Outer training loop:  15%|#4        | 7471/50000 [3:32:42<18:38:11,  1.58s/it]
Outer training loop:  15%|#4        | 7472/50000 [3:32:43<18:44:24,  1.59s/it]
Outer training loop:  15%|#4        | 7473/50000 [3:32:45<18:42:21,  1.58s/it]
Outer training loop:  15%|#4        | 7474/50000 [3:32:46<18:23:24,  1.56s/it]
Outer training loop:  15%|#4        | 7475/50000 [3:32:48<18:39:40,  1.58s/it]
Outer training loop:  15%|#4        | 7476/50000 [3:32:49<18:24:47,  1.56s/it]
Outer training loop:  15%|#4        | 7477/50000 [3:32:51<18:25:44,  1.56s/it]
Outer training loop:  15%|#4        | 7478/50000 [3:32:52<17:59:19,  1.52s/it]
Outer training loop:  15%|#4        | 7479/50000 [3:32:54<18:07:05,  1.53s/it]
Outer training loop:  15%|#4        | 7480/50000 [3:32:56<18:01:13,  1.53s/it]
Outer training loop:  15%|#4        | 7481/50000 [3:32:57<18:09:15,  1.54s/it]
Outer training loop:  15%|#4        | 7482/50000 [3:32:59<18:06:20,  1.53s/it]
Outer training loop:  15%|#4        | 7483/50000 [3:33:00<18:17:01,  1.55s/it]
Outer training loop:  15%|#4        | 7484/50000 [3:33:02<18:00:48,  1.53s/it]
Outer training loop:  15%|#4        | 7485/50000 [3:33:03<18:16:16,  1.55s/it]
Outer training loop:  15%|#4        | 7486/50000 [3:33:05<18:17:39,  1.55s/it]
Outer training loop:  15%|#4        | 7487/50000 [3:33:06<18:26:37,  1.56s/it]
Outer training loop:  15%|#4        | 7488/50000 [3:33:08<18:25:12,  1.56s/it]
Outer training loop:  15%|#4        | 7489/50000 [3:33:10<18:29:13,  1.57s/it]
Outer training loop:  15%|#4        | 7490/50000 [3:33:11<18:14:07,  1.54s/it]
Outer training loop:  15%|#4        | 7491/50000 [3:33:13<18:24:47,  1.56s/it]
Outer training loop:  15%|#4        | 7492/50000 [3:33:14<18:14:24,  1.54s/it]
Outer training loop:  15%|#4        | 7493/50000 [3:33:16<18:28:17,  1.56s/it]
Outer training loop:  15%|#4        | 7494/50000 [3:33:17<18:23:14,  1.56s/it]
Outer training loop:  15%|#4        | 7495/50000 [3:33:19<18:48:54,  1.59s/it]
Outer training loop:  15%|#4        | 7496/50000 [3:33:21<18:49:13,  1.59s/it]
Outer training loop:  15%|#4        | 7497/50000 [3:33:22<19:02:07,  1.61s/it]
Outer training loop:  15%|#4        | 7498/50000 [3:33:24<18:35:45,  1.58s/it]
Outer training loop:  15%|#4        | 7499/50000 [3:33:25<18:48:57,  1.59s/it]
Outer training loop:  15%|#5        | 7500/50000 [3:33:27<18:32:41,  1.57s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.63s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.64s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.58s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:13,  1.62s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:07,  1.58s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:05,  1.57s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:03,  1.56s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:03,  1.59s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:00,  1.56s/it]

Evaluating policy:   8%|8         | 7/83 [00:10<02:00,  1.59s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:58,  1.58s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:55,  1.56s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:56,  1.59s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:54,  1.60s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:52,  1.59s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:50,  1.58s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:47,  1.55s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:43,  1.52s/it]

Evaluating policy:  19%|#9        | 16/83 [00:24<01:43,  1.54s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:39,  1.51s/it]

Evaluating policy:  22%|##1       | 18/83 [00:27<01:37,  1.49s/it]

Evaluating policy:  23%|##2       | 19/83 [00:29<01:38,  1.54s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:36,  1.53s/it]

Evaluating policy:  25%|##5       | 21/83 [00:32<01:34,  1.52s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:33,  1.54s/it]

Evaluating policy:  28%|##7       | 23/83 [00:35<01:32,  1.54s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:29,  1.53s/it]

Evaluating policy:  30%|###       | 25/83 [00:38<01:28,  1.53s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:28,  1.55s/it]

Evaluating policy:  33%|###2      | 27/83 [00:41<01:26,  1.55s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:24,  1.53s/it]

Evaluating policy:  35%|###4      | 29/83 [00:44<01:21,  1.52s/it]

Evaluating policy:  36%|###6      | 30/83 [00:46<01:23,  1.58s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:21,  1.56s/it]

Evaluating policy:  39%|###8      | 32/83 [00:49<01:18,  1.54s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:17,  1.55s/it]

Evaluating policy:  41%|####      | 34/83 [00:52<01:15,  1.53s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:15,  1.58s/it]

Evaluating policy:  43%|####3     | 36/83 [00:55<01:13,  1.56s/it]

Evaluating policy:  45%|####4     | 37/83 [00:57<01:09,  1.52s/it]

Evaluating policy:  46%|####5     | 38/83 [00:58<01:08,  1.52s/it]

Evaluating policy:  47%|####6     | 39/83 [01:00<01:07,  1.53s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:07,  1.57s/it]

Evaluating policy:  49%|####9     | 41/83 [01:03<01:04,  1.55s/it]

Evaluating policy:  51%|#####     | 42/83 [01:05<01:03,  1.54s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:06<01:01,  1.54s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:08<00:59,  1.53s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:09<01:00,  1.59s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:11<00:57,  1.55s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:12<00:55,  1.53s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:14<00:54,  1.55s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:15<00:51,  1.53s/it]

Evaluating policy:  60%|######    | 50/83 [01:17<00:50,  1.52s/it]

Evaluating policy:  61%|######1   | 51/83 [01:19<00:51,  1.62s/it]

Evaluating policy:  63%|######2   | 52/83 [01:20<00:49,  1.59s/it]

Evaluating policy:  64%|######3   | 53/83 [01:22<00:47,  1.57s/it]

Evaluating policy:  65%|######5   | 54/83 [01:23<00:45,  1.56s/it]

Evaluating policy:  66%|######6   | 55/83 [01:25<00:43,  1.54s/it]

Evaluating policy:  67%|######7   | 56/83 [01:26<00:41,  1.53s/it]

Evaluating policy:  69%|######8   | 57/83 [01:28<00:42,  1.65s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:40,  1.62s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:31<00:38,  1.59s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:33<00:36,  1.58s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:34<00:34,  1.56s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:36<00:32,  1.53s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:32,  1.62s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:39<00:29,  1.58s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:27,  1.55s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:42<00:26,  1.55s/it]

Evaluating policy:  81%|########  | 67/83 [01:44<00:24,  1.54s/it]

Evaluating policy:  82%|########1 | 68/83 [01:45<00:23,  1.57s/it]

Evaluating policy:  83%|########3 | 69/83 [01:47<00:21,  1.55s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:21,  1.66s/it]

Evaluating policy:  86%|########5 | 71/83 [01:50<00:19,  1.60s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:17,  1.57s/it]

Evaluating policy:  88%|########7 | 73/83 [01:53<00:15,  1.54s/it]

Evaluating policy:  89%|########9 | 74/83 [01:55<00:13,  1.53s/it]

Evaluating policy:  90%|######### | 75/83 [01:56<00:12,  1.52s/it]

Evaluating policy:  92%|#########1| 76/83 [01:58<00:10,  1.51s/it]

Evaluating policy:  93%|#########2| 77/83 [01:59<00:09,  1.52s/it]

Evaluating policy:  94%|#########3| 78/83 [02:01<00:08,  1.66s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.65s/it]

Evaluating policy:  96%|#########6| 80/83 [02:04<00:04,  1.60s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.56s/it]

Evaluating policy:  99%|#########8| 82/83 [02:07<00:01,  1.54s/it]

Evaluating policy: 100%|##########| 83/83 [02:09<00:00,  1.51s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<01:59,  1.46s/it]

Evaluating policy:   2%|2         | 2/83 [00:02<01:59,  1.47s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<01:58,  1.49s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:10,  1.65s/it]

Evaluating policy:   6%|6         | 5/83 [00:07<02:04,  1.60s/it]

Evaluating policy:   7%|7         | 6/83 [00:09<02:03,  1.60s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<01:58,  1.56s/it]

Evaluating policy:  10%|9         | 8/83 [00:12<01:56,  1.55s/it]

Evaluating policy:  11%|#         | 9/83 [00:14<01:55,  1.57s/it]

Evaluating policy:  12%|#2        | 10/83 [00:15<01:52,  1.53s/it]

Evaluating policy:  13%|#3        | 11/83 [00:17<01:49,  1.51s/it]

Evaluating policy:  14%|#4        | 12/83 [00:18<01:51,  1.57s/it]

Evaluating policy:  16%|#5        | 13/83 [00:20<01:49,  1.57s/it]

Evaluating policy:  17%|#6        | 14/83 [00:21<01:47,  1.56s/it]

Evaluating policy:  18%|#8        | 15/83 [00:23<01:52,  1.66s/it]

Evaluating policy:  19%|#9        | 16/83 [00:25<01:48,  1.62s/it]

Evaluating policy:  20%|##        | 17/83 [00:26<01:44,  1.58s/it]

Evaluating policy:  22%|##1       | 18/83 [00:28<01:45,  1.62s/it]

Evaluating policy:  23%|##2       | 19/83 [00:30<01:40,  1.58s/it]

Evaluating policy:  24%|##4       | 20/83 [00:31<01:36,  1.54s/it]

Evaluating policy:  25%|##5       | 21/83 [00:33<01:37,  1.58s/it]

Evaluating policy:  27%|##6       | 22/83 [00:34<01:35,  1.56s/it]

Evaluating policy:  28%|##7       | 23/83 [00:36<01:34,  1.57s/it]

Evaluating policy:  29%|##8       | 24/83 [00:37<01:31,  1.55s/it]

Evaluating policy:  30%|###       | 25/83 [00:39<01:31,  1.58s/it]

Evaluating policy:  31%|###1      | 26/83 [00:40<01:29,  1.57s/it]

Evaluating policy:  33%|###2      | 27/83 [00:42<01:26,  1.55s/it]

Evaluating policy:  34%|###3      | 28/83 [00:43<01:25,  1.55s/it]

Evaluating policy:  35%|###4      | 29/83 [00:45<01:25,  1.58s/it]

Evaluating policy:  36%|###6      | 30/83 [00:47<01:22,  1.56s/it]

Evaluating policy:  37%|###7      | 31/83 [00:48<01:19,  1.53s/it]

Evaluating policy:  39%|###8      | 32/83 [00:50<01:17,  1.52s/it]

Evaluating policy:  40%|###9      | 33/83 [00:51<01:19,  1.59s/it]

Evaluating policy:  41%|####      | 34/83 [00:53<01:16,  1.56s/it]

Evaluating policy:  42%|####2     | 35/83 [00:54<01:13,  1.54s/it]

Evaluating policy:  43%|####3     | 36/83 [00:56<01:12,  1.54s/it]

Evaluating policy:  45%|####4     | 37/83 [00:58<01:14,  1.61s/it]

Evaluating policy:  46%|####5     | 38/83 [00:59<01:11,  1.59s/it]

Evaluating policy:  47%|####6     | 39/83 [01:01<01:08,  1.55s/it]

Evaluating policy:  48%|####8     | 40/83 [01:02<01:06,  1.54s/it]

Evaluating policy:  49%|####9     | 41/83 [01:04<01:04,  1.53s/it]

Evaluating policy:  51%|#####     | 42/83 [01:06<01:07,  1.64s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:07<01:04,  1.62s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:09<01:02,  1.60s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:10<01:01,  1.61s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:12<00:59,  1.60s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:14<00:59,  1.66s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:15<00:55,  1.60s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:17<00:53,  1.56s/it]

Evaluating policy:  60%|######    | 50/83 [01:18<00:50,  1.53s/it]

Evaluating policy:  61%|######1   | 51/83 [01:20<00:49,  1.54s/it]

Evaluating policy:  63%|######2   | 52/83 [01:21<00:46,  1.51s/it]

Evaluating policy:  64%|######3   | 53/83 [01:23<00:47,  1.59s/it]

Evaluating policy:  65%|######5   | 54/83 [01:24<00:45,  1.57s/it]

Evaluating policy:  66%|######6   | 55/83 [01:26<00:43,  1.56s/it]

Evaluating policy:  67%|######7   | 56/83 [01:27<00:41,  1.54s/it]

Evaluating policy:  69%|######8   | 57/83 [01:29<00:39,  1.52s/it]

Evaluating policy:  70%|######9   | 58/83 [01:30<00:38,  1.52s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:32<00:35,  1.50s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:34<00:36,  1.61s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:35<00:34,  1.59s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:37<00:33,  1.58s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:38<00:31,  1.55s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:40<00:28,  1.52s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:41<00:27,  1.55s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:43<00:26,  1.56s/it]

Evaluating policy:  81%|########  | 67/83 [01:45<00:26,  1.67s/it]

Evaluating policy:  82%|########1 | 68/83 [01:46<00:24,  1.61s/it]

Evaluating policy:  83%|########3 | 69/83 [01:48<00:22,  1.58s/it]

Evaluating policy:  84%|########4 | 70/83 [01:49<00:20,  1.56s/it]

Evaluating policy:  86%|########5 | 71/83 [01:51<00:18,  1.56s/it]

Evaluating policy:  87%|########6 | 72/83 [01:52<00:17,  1.55s/it]

Evaluating policy:  88%|########7 | 73/83 [01:54<00:15,  1.53s/it]

Evaluating policy:  89%|########9 | 74/83 [01:56<00:14,  1.63s/it]

Evaluating policy:  90%|######### | 75/83 [01:57<00:12,  1.60s/it]

Evaluating policy:  92%|#########1| 76/83 [01:59<00:10,  1.56s/it]

Evaluating policy:  93%|#########2| 77/83 [02:00<00:09,  1.55s/it]

Evaluating policy:  94%|#########3| 78/83 [02:02<00:07,  1.52s/it]

Evaluating policy:  95%|#########5| 79/83 [02:03<00:06,  1.53s/it]

Evaluating policy:  96%|#########6| 80/83 [02:05<00:04,  1.53s/it]

Evaluating policy:  98%|#########7| 81/83 [02:06<00:03,  1.52s/it]

Evaluating policy:  99%|#########8| 82/83 [02:08<00:01,  1.66s/it]

Evaluating policy: 100%|##########| 83/83 [02:10<00:00,  1.60s/it]

Outer training loop:  15%|#5        | 7501/50000 [3:37:53<955:10:13, 80.91s/it]
Outer training loop:  15%|#5        | 7502/50000 [3:37:54<674:02:04, 57.10s/it]
Outer training loop:  15%|#5        | 7503/50000 [3:37:56<477:09:01, 40.42s/it]
Outer training loop:  15%|#5        | 7504/50000 [3:37:57<339:21:56, 28.75s/it]
Outer training loop:  15%|#5        | 7505/50000 [3:37:59<242:47:00, 20.57s/it]
Outer training loop:  15%|#5        | 7506/50000 [3:38:00<175:09:28, 14.84s/it]
Outer training loop:  15%|#5        | 7507/50000 [3:38:02<127:50:21, 10.83s/it]
Outer training loop:  15%|#5        | 7508/50000 [3:38:03<94:38:19,  8.02s/it] 
Outer training loop:  15%|#5        | 7509/50000 [3:38:05<73:46:39,  6.25s/it]
Outer training loop:  15%|#5        | 7510/50000 [3:38:07<56:58:54,  4.83s/it]
Outer training loop:  15%|#5        | 7511/50000 [3:38:09<45:40:47,  3.87s/it]
Outer training loop:  15%|#5        | 7512/50000 [3:38:10<37:19:34,  3.16s/it]
Outer training loop:  15%|#5        | 7513/50000 [3:38:12<31:55:35,  2.71s/it]
Outer training loop:  15%|#5        | 7514/50000 [3:38:13<27:14:38,  2.31s/it]
Outer training loop:  15%|#5        | 7515/50000 [3:38:15<24:36:52,  2.09s/it]
Outer training loop:  15%|#5        | 7516/50000 [3:38:16<22:52:37,  1.94s/it]
Outer training loop:  15%|#5        | 7517/50000 [3:38:18<21:21:24,  1.81s/it]
Outer training loop:  15%|#5        | 7518/50000 [3:38:19<20:25:48,  1.73s/it]
Outer training loop:  15%|#5        | 7519/50000 [3:38:21<19:33:13,  1.66s/it]
Outer training loop:  15%|#5        | 7520/50000 [3:38:22<19:10:51,  1.63s/it]
Outer training loop:  15%|#5        | 7521/50000 [3:38:24<18:48:23,  1.59s/it]
Outer training loop:  15%|#5        | 7522/50000 [3:38:26<19:07:05,  1.62s/it]
Outer training loop:  15%|#5        | 7523/50000 [3:38:27<19:06:18,  1.62s/it]
Outer training loop:  15%|#5        | 7524/50000 [3:38:29<18:32:29,  1.57s/it]
Outer training loop:  15%|#5        | 7525/50000 [3:38:30<18:39:26,  1.58s/it]
Outer training loop:  15%|#5        | 7526/50000 [3:38:32<18:42:47,  1.59s/it]
Outer training loop:  15%|#5        | 7527/50000 [3:38:33<18:33:36,  1.57s/it]
Outer training loop:  15%|#5        | 7528/50000 [3:38:35<18:45:54,  1.59s/it]
Outer training loop:  15%|#5        | 7529/50000 [3:38:37<18:48:51,  1.59s/it]
Outer training loop:  15%|#5        | 7530/50000 [3:38:38<18:24:31,  1.56s/it]
Outer training loop:  15%|#5        | 7531/50000 [3:38:40<18:31:04,  1.57s/it]
Outer training loop:  15%|#5        | 7532/50000 [3:38:41<18:49:19,  1.60s/it]
Outer training loop:  15%|#5        | 7533/50000 [3:38:43<19:33:35,  1.66s/it]
Outer training loop:  15%|#5        | 7534/50000 [3:38:45<19:22:55,  1.64s/it]
Outer training loop:  15%|#5        | 7535/50000 [3:38:46<18:59:06,  1.61s/it]
Outer training loop:  15%|#5        | 7536/50000 [3:38:48<19:06:00,  1.62s/it]
Outer training loop:  15%|#5        | 7537/50000 [3:38:50<19:07:53,  1.62s/it]
Outer training loop:  15%|#5        | 7538/50000 [3:38:51<18:25:55,  1.56s/it]
Outer training loop:  15%|#5        | 7539/50000 [3:38:52<18:02:54,  1.53s/it]
Outer training loop:  15%|#5        | 7540/50000 [3:38:54<18:28:54,  1.57s/it]
Outer training loop:  15%|#5        | 7541/50000 [3:38:56<18:18:13,  1.55s/it]
Outer training loop:  15%|#5        | 7542/50000 [3:38:57<18:26:11,  1.56s/it]
Outer training loop:  15%|#5        | 7543/50000 [3:38:59<18:14:35,  1.55s/it]
Outer training loop:  15%|#5        | 7544/50000 [3:39:00<18:23:45,  1.56s/it]
Outer training loop:  15%|#5        | 7545/50000 [3:39:02<18:25:57,  1.56s/it]
Outer training loop:  15%|#5        | 7546/50000 [3:39:03<18:26:20,  1.56s/it]
Outer training loop:  15%|#5        | 7547/50000 [3:39:05<18:22:42,  1.56s/it]
Outer training loop:  15%|#5        | 7548/50000 [3:39:07<18:19:17,  1.55s/it]
Outer training loop:  15%|#5        | 7549/50000 [3:39:08<17:54:38,  1.52s/it]
Outer training loop:  15%|#5        | 7550/50000 [3:39:10<18:19:48,  1.55s/it]
Outer training loop:  15%|#5        | 7551/50000 [3:39:11<18:15:20,  1.55s/it]
Outer training loop:  15%|#5        | 7552/50000 [3:39:13<18:20:01,  1.55s/it]
Outer training loop:  15%|#5        | 7553/50000 [3:39:14<17:54:57,  1.52s/it]
Outer training loop:  15%|#5        | 7554/50000 [3:39:16<18:20:37,  1.56s/it]
Outer training loop:  15%|#5        | 7555/50000 [3:39:17<17:59:33,  1.53s/it]
Outer training loop:  15%|#5        | 7556/50000 [3:39:19<18:33:45,  1.57s/it]
Outer training loop:  15%|#5        | 7557/50000 [3:39:21<18:40:55,  1.58s/it]
Outer training loop:  15%|#5        | 7558/50000 [3:39:22<18:37:25,  1.58s/it]
Outer training loop:  15%|#5        | 7559/50000 [3:39:24<18:38:05,  1.58s/it]
Outer training loop:  15%|#5        | 7560/50000 [3:39:25<18:38:02,  1.58s/it]
Outer training loop:  15%|#5        | 7561/50000 [3:39:27<19:02:43,  1.62s/it]
Outer training loop:  15%|#5        | 7562/50000 [3:39:29<18:43:33,  1.59s/it]
Outer training loop:  15%|#5        | 7563/50000 [3:39:30<18:42:43,  1.59s/it]
Outer training loop:  15%|#5        | 7564/50000 [3:39:32<18:21:25,  1.56s/it]
Outer training loop:  15%|#5        | 7565/50000 [3:39:33<18:41:19,  1.59s/it]
Outer training loop:  15%|#5        | 7566/50000 [3:39:35<18:36:38,  1.58s/it]
Outer training loop:  15%|#5        | 7567/50000 [3:39:36<18:27:17,  1.57s/it]
Outer training loop:  15%|#5        | 7568/50000 [3:39:38<18:09:41,  1.54s/it]
Outer training loop:  15%|#5        | 7569/50000 [3:39:39<18:35:11,  1.58s/it]
Outer training loop:  15%|#5        | 7570/50000 [3:39:41<18:12:57,  1.55s/it]
Outer training loop:  15%|#5        | 7571/50000 [3:39:43<18:24:17,  1.56s/it]
Outer training loop:  15%|#5        | 7572/50000 [3:39:44<18:20:13,  1.56s/it]
Outer training loop:  15%|#5        | 7573/50000 [3:39:46<18:08:45,  1.54s/it]
Outer training loop:  15%|#5        | 7574/50000 [3:39:47<18:17:09,  1.55s/it]
Outer training loop:  15%|#5        | 7575/50000 [3:39:49<18:08:28,  1.54s/it]
Outer training loop:  15%|#5        | 7576/50000 [3:39:50<18:19:43,  1.56s/it]
Outer training loop:  15%|#5        | 7577/50000 [3:39:52<18:24:46,  1.56s/it]
Outer training loop:  15%|#5        | 7578/50000 [3:39:53<18:01:55,  1.53s/it]
Outer training loop:  15%|#5        | 7579/50000 [3:39:55<17:53:33,  1.52s/it]
Outer training loop:  15%|#5        | 7580/50000 [3:39:56<18:10:11,  1.54s/it]
Outer training loop:  15%|#5        | 7581/50000 [3:39:58<18:07:57,  1.54s/it]
Outer training loop:  15%|#5        | 7582/50000 [3:40:00<18:20:44,  1.56s/it]
Outer training loop:  15%|#5        | 7583/50000 [3:40:01<18:42:27,  1.59s/it]
Outer training loop:  15%|#5        | 7584/50000 [3:40:03<19:01:17,  1.61s/it]
Outer training loop:  15%|#5        | 7585/50000 [3:40:05<19:07:34,  1.62s/it]
Outer training loop:  15%|#5        | 7586/50000 [3:40:06<18:47:17,  1.59s/it]
Outer training loop:  15%|#5        | 7587/50000 [3:40:08<18:52:05,  1.60s/it]
Outer training loop:  15%|#5        | 7588/50000 [3:40:09<18:46:33,  1.59s/it]
Outer training loop:  15%|#5        | 7589/50000 [3:40:11<18:58:51,  1.61s/it]
Outer training loop:  15%|#5        | 7590/50000 [3:40:12<18:45:40,  1.59s/it]
Outer training loop:  15%|#5        | 7591/50000 [3:40:14<18:55:58,  1.61s/it]
Outer training loop:  15%|#5        | 7592/50000 [3:40:16<18:40:48,  1.59s/it]
Outer training loop:  15%|#5        | 7593/50000 [3:40:17<18:27:24,  1.57s/it]
Outer training loop:  15%|#5        | 7594/50000 [3:40:19<18:38:49,  1.58s/it]
Outer training loop:  15%|#5        | 7595/50000 [3:40:20<18:29:52,  1.57s/it]
Outer training loop:  15%|#5        | 7596/50000 [3:40:22<18:36:35,  1.58s/it]
Outer training loop:  15%|#5        | 7597/50000 [3:40:23<18:20:39,  1.56s/it]
Outer training loop:  15%|#5        | 7598/50000 [3:40:25<18:21:28,  1.56s/it]
Outer training loop:  15%|#5        | 7599/50000 [3:40:26<18:14:54,  1.55s/it]
Outer training loop:  15%|#5        | 7600/50000 [3:40:28<18:15:25,  1.55s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.53s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.55s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.55s/it]

Outer training loop:  15%|#5        | 7601/50000 [3:40:34<34:41:43,  2.95s/it]
Outer training loop:  15%|#5        | 7602/50000 [3:40:36<29:39:33,  2.52s/it]
Outer training loop:  15%|#5        | 7603/50000 [3:40:37<26:33:59,  2.26s/it]
Outer training loop:  15%|#5        | 7604/50000 [3:40:39<23:45:33,  2.02s/it]
Outer training loop:  15%|#5        | 7605/50000 [3:40:40<22:21:35,  1.90s/it]
Outer training loop:  15%|#5        | 7606/50000 [3:40:42<21:14:29,  1.80s/it]
Outer training loop:  15%|#5        | 7607/50000 [3:40:44<20:26:51,  1.74s/it]
Outer training loop:  15%|#5        | 7608/50000 [3:40:45<19:54:46,  1.69s/it]
Outer training loop:  15%|#5        | 7609/50000 [3:40:47<18:56:27,  1.61s/it]
Outer training loop:  15%|#5        | 7610/50000 [3:40:48<19:20:06,  1.64s/it]
Outer training loop:  15%|#5        | 7611/50000 [3:40:50<18:59:55,  1.61s/it]
Outer training loop:  15%|#5        | 7612/50000 [3:40:51<18:40:00,  1.59s/it]
Outer training loop:  15%|#5        | 7613/50000 [3:40:53<18:24:18,  1.56s/it]
Outer training loop:  15%|#5        | 7614/50000 [3:40:55<18:47:03,  1.60s/it]
Outer training loop:  15%|#5        | 7615/50000 [3:40:56<18:31:35,  1.57s/it]
Outer training loop:  15%|#5        | 7616/50000 [3:40:58<18:34:25,  1.58s/it]
Outer training loop:  15%|#5        | 7617/50000 [3:40:59<18:06:28,  1.54s/it]
Outer training loop:  15%|#5        | 7618/50000 [3:41:01<18:13:03,  1.55s/it]
Outer training loop:  15%|#5        | 7619/50000 [3:41:02<18:15:16,  1.55s/it]
Outer training loop:  15%|#5        | 7620/50000 [3:41:04<18:01:48,  1.53s/it]
Outer training loop:  15%|#5        | 7621/50000 [3:41:05<18:21:12,  1.56s/it]
Outer training loop:  15%|#5        | 7622/50000 [3:41:07<18:05:48,  1.54s/it]
Outer training loop:  15%|#5        | 7623/50000 [3:41:08<18:17:45,  1.55s/it]
Outer training loop:  15%|#5        | 7624/50000 [3:41:10<18:15:48,  1.55s/it]
Outer training loop:  15%|#5        | 7625/50000 [3:41:12<18:19:39,  1.56s/it]
Outer training loop:  15%|#5        | 7626/50000 [3:41:13<18:05:11,  1.54s/it]
Outer training loop:  15%|#5        | 7627/50000 [3:41:15<18:20:46,  1.56s/it]
Outer training loop:  15%|#5        | 7628/50000 [3:41:16<18:14:59,  1.55s/it]
Outer training loop:  15%|#5        | 7629/50000 [3:41:18<18:20:54,  1.56s/it]
Outer training loop:  15%|#5        | 7630/50000 [3:41:20<18:50:58,  1.60s/it]
Outer training loop:  15%|#5        | 7631/50000 [3:41:21<18:50:31,  1.60s/it]
Outer training loop:  15%|#5        | 7632/50000 [3:41:23<18:30:00,  1.57s/it]
Outer training loop:  15%|#5        | 7633/50000 [3:41:24<18:24:32,  1.56s/it]
Outer training loop:  15%|#5        | 7634/50000 [3:41:26<18:10:24,  1.54s/it]
Outer training loop:  15%|#5        | 7635/50000 [3:41:27<18:26:26,  1.57s/it]
Outer training loop:  15%|#5        | 7636/50000 [3:41:29<18:08:32,  1.54s/it]
Outer training loop:  15%|#5        | 7637/50000 [3:41:30<18:43:06,  1.59s/it]
Outer training loop:  15%|#5        | 7638/50000 [3:41:32<18:29:34,  1.57s/it]
Outer training loop:  15%|#5        | 7639/50000 [3:41:34<18:44:55,  1.59s/it]
Outer training loop:  15%|#5        | 7640/50000 [3:41:35<18:31:30,  1.57s/it]
Outer training loop:  15%|#5        | 7641/50000 [3:41:37<18:40:31,  1.59s/it]
Outer training loop:  15%|#5        | 7642/50000 [3:41:38<18:17:34,  1.55s/it]
Outer training loop:  15%|#5        | 7643/50000 [3:41:40<18:29:22,  1.57s/it]
Outer training loop:  15%|#5        | 7644/50000 [3:41:41<18:13:21,  1.55s/it]
Outer training loop:  15%|#5        | 7645/50000 [3:41:43<18:24:23,  1.56s/it]
Outer training loop:  15%|#5        | 7646/50000 [3:41:45<18:36:12,  1.58s/it]
Outer training loop:  15%|#5        | 7647/50000 [3:41:46<19:19:34,  1.64s/it]
Outer training loop:  15%|#5        | 7648/50000 [3:41:48<19:12:14,  1.63s/it]
Outer training loop:  15%|#5        | 7649/50000 [3:41:49<18:39:03,  1.59s/it]
Outer training loop:  15%|#5        | 7650/50000 [3:41:51<19:44:50,  1.68s/it]
Outer training loop:  15%|#5        | 7651/50000 [3:41:53<19:01:07,  1.62s/it]
Outer training loop:  15%|#5        | 7652/50000 [3:41:54<18:23:35,  1.56s/it]
Outer training loop:  15%|#5        | 7653/50000 [3:41:56<17:39:38,  1.50s/it]
Outer training loop:  15%|#5        | 7654/50000 [3:41:57<17:46:17,  1.51s/it]
Outer training loop:  15%|#5        | 7655/50000 [3:41:59<17:16:14,  1.47s/it]
Outer training loop:  15%|#5        | 7656/50000 [3:42:00<17:29:07,  1.49s/it]
Outer training loop:  15%|#5        | 7657/50000 [3:42:01<16:36:55,  1.41s/it]
Outer training loop:  15%|#5        | 7658/50000 [3:42:02<15:46:47,  1.34s/it]
Outer training loop:  15%|#5        | 7659/50000 [3:42:04<16:34:35,  1.41s/it]
Outer training loop:  15%|#5        | 7660/50000 [3:42:05<16:03:33,  1.37s/it]
Outer training loop:  15%|#5        | 7661/50000 [3:42:06<15:18:36,  1.30s/it]
Outer training loop:  15%|#5        | 7662/50000 [3:42:08<15:11:18,  1.29s/it]
Outer training loop:  15%|#5        | 7663/50000 [3:42:09<14:32:33,  1.24s/it]
Outer training loop:  15%|#5        | 7664/50000 [3:42:10<14:00:16,  1.19s/it]
Outer training loop:  15%|#5        | 7665/50000 [3:42:11<13:57:50,  1.19s/it]
Outer training loop:  15%|#5        | 7666/50000 [3:42:12<14:39:32,  1.25s/it]
Outer training loop:  15%|#5        | 7667/50000 [3:42:14<14:01:56,  1.19s/it]
Outer training loop:  15%|#5        | 7668/50000 [3:42:15<14:36:06,  1.24s/it]
Outer training loop:  15%|#5        | 7669/50000 [3:42:16<15:24:40,  1.31s/it]
Outer training loop:  15%|#5        | 7670/50000 [3:42:18<16:31:23,  1.41s/it]
Outer training loop:  15%|#5        | 7671/50000 [3:42:19<16:29:36,  1.40s/it]
Outer training loop:  15%|#5        | 7672/50000 [3:42:21<16:24:51,  1.40s/it]
Outer training loop:  15%|#5        | 7673/50000 [3:42:22<16:53:11,  1.44s/it]
Outer training loop:  15%|#5        | 7674/50000 [3:42:24<17:21:14,  1.48s/it]
Outer training loop:  15%|#5        | 7675/50000 [3:42:25<16:16:45,  1.38s/it]
Outer training loop:  15%|#5        | 7676/50000 [3:42:26<15:37:49,  1.33s/it]
Outer training loop:  15%|#5        | 7677/50000 [3:42:28<16:05:47,  1.37s/it]
Outer training loop:  15%|#5        | 7678/50000 [3:42:29<16:51:42,  1.43s/it]
Outer training loop:  15%|#5        | 7679/50000 [3:42:31<17:07:12,  1.46s/it]
Outer training loop:  15%|#5        | 7680/50000 [3:42:32<17:10:10,  1.46s/it]
Outer training loop:  15%|#5        | 7681/50000 [3:42:33<15:44:06,  1.34s/it]
Outer training loop:  15%|#5        | 7682/50000 [3:42:34<14:57:57,  1.27s/it]
Outer training loop:  15%|#5        | 7683/50000 [3:42:36<16:07:35,  1.37s/it]
Outer training loop:  15%|#5        | 7684/50000 [3:42:37<14:55:02,  1.27s/it]
Outer training loop:  15%|#5        | 7685/50000 [3:42:38<15:19:28,  1.30s/it]
Outer training loop:  15%|#5        | 7686/50000 [3:42:40<15:26:40,  1.31s/it]
Outer training loop:  15%|#5        | 7687/50000 [3:42:41<14:32:22,  1.24s/it]
Outer training loop:  15%|#5        | 7688/50000 [3:42:42<15:28:58,  1.32s/it]
Outer training loop:  15%|#5        | 7689/50000 [3:42:44<16:07:15,  1.37s/it]
Outer training loop:  15%|#5        | 7690/50000 [3:42:46<17:02:41,  1.45s/it]
Outer training loop:  15%|#5        | 7691/50000 [3:42:47<16:20:07,  1.39s/it]
Outer training loop:  15%|#5        | 7692/50000 [3:42:48<16:48:17,  1.43s/it]
Outer training loop:  15%|#5        | 7693/50000 [3:42:50<17:16:57,  1.47s/it]
Outer training loop:  15%|#5        | 7694/50000 [3:42:51<16:49:27,  1.43s/it]
Outer training loop:  15%|#5        | 7695/50000 [3:42:53<17:19:06,  1.47s/it]
Outer training loop:  15%|#5        | 7696/50000 [3:42:55<18:34:12,  1.58s/it]
Outer training loop:  15%|#5        | 7697/50000 [3:42:56<18:10:16,  1.55s/it]
Outer training loop:  15%|#5        | 7698/50000 [3:42:57<17:43:16,  1.51s/it]
Outer training loop:  15%|#5        | 7699/50000 [3:42:59<18:23:10,  1.56s/it]
Outer training loop:  15%|#5        | 7700/50000 [3:43:01<18:19:14,  1.56s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:02,  1.48s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.52s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.46s/it]

Outer training loop:  15%|#5        | 7701/50000 [3:43:07<34:25:25,  2.93s/it]
Outer training loop:  15%|#5        | 7702/50000 [3:43:08<29:51:40,  2.54s/it]
Outer training loop:  15%|#5        | 7703/50000 [3:43:10<26:40:43,  2.27s/it]
Outer training loop:  15%|#5        | 7704/50000 [3:43:12<24:13:34,  2.06s/it]
Outer training loop:  15%|#5        | 7705/50000 [3:43:14<23:21:08,  1.99s/it]
Outer training loop:  15%|#5        | 7706/50000 [3:43:15<22:10:03,  1.89s/it]
Outer training loop:  15%|#5        | 7707/50000 [3:43:17<21:55:07,  1.87s/it]
Outer training loop:  15%|#5        | 7708/50000 [3:43:19<21:22:53,  1.82s/it]
Outer training loop:  15%|#5        | 7709/50000 [3:43:20<20:32:01,  1.75s/it]
Outer training loop:  15%|#5        | 7710/50000 [3:43:22<20:55:48,  1.78s/it]
Outer training loop:  15%|#5        | 7711/50000 [3:43:24<20:24:26,  1.74s/it]
Outer training loop:  15%|#5        | 7712/50000 [3:43:26<20:39:30,  1.76s/it]
Outer training loop:  15%|#5        | 7713/50000 [3:43:27<19:43:08,  1.68s/it]
Outer training loop:  15%|#5        | 7714/50000 [3:43:29<19:15:05,  1.64s/it]
Outer training loop:  15%|#5        | 7715/50000 [3:43:30<19:40:14,  1.67s/it]
Outer training loop:  15%|#5        | 7716/50000 [3:43:32<19:42:46,  1.68s/it]
Outer training loop:  15%|#5        | 7717/50000 [3:43:34<19:33:14,  1.66s/it]
Outer training loop:  15%|#5        | 7718/50000 [3:43:35<19:27:30,  1.66s/it]
Outer training loop:  15%|#5        | 7719/50000 [3:43:37<18:58:50,  1.62s/it]
Outer training loop:  15%|#5        | 7720/50000 [3:43:38<18:21:29,  1.56s/it]
Outer training loop:  15%|#5        | 7721/50000 [3:43:40<19:29:22,  1.66s/it]
Outer training loop:  15%|#5        | 7722/50000 [3:43:42<19:13:31,  1.64s/it]
Outer training loop:  15%|#5        | 7723/50000 [3:43:43<19:01:08,  1.62s/it]
Outer training loop:  15%|#5        | 7724/50000 [3:43:45<19:37:50,  1.67s/it]
Outer training loop:  15%|#5        | 7725/50000 [3:43:47<19:16:47,  1.64s/it]
Outer training loop:  15%|#5        | 7726/50000 [3:43:48<19:09:47,  1.63s/it]
Outer training loop:  15%|#5        | 7727/50000 [3:43:50<20:21:03,  1.73s/it]
Outer training loop:  15%|#5        | 7728/50000 [3:43:52<20:02:11,  1.71s/it]
Outer training loop:  15%|#5        | 7729/50000 [3:43:54<19:48:58,  1.69s/it]
Outer training loop:  15%|#5        | 7730/50000 [3:43:55<19:56:18,  1.70s/it]
Outer training loop:  15%|#5        | 7731/50000 [3:43:57<19:26:23,  1.66s/it]
Outer training loop:  15%|#5        | 7732/50000 [3:43:58<18:59:00,  1.62s/it]
Outer training loop:  15%|#5        | 7733/50000 [3:44:00<19:21:15,  1.65s/it]
Outer training loop:  15%|#5        | 7734/50000 [3:44:02<19:11:25,  1.63s/it]
Outer training loop:  15%|#5        | 7735/50000 [3:44:03<19:33:45,  1.67s/it]
Outer training loop:  15%|#5        | 7736/50000 [3:44:05<20:19:47,  1.73s/it]
Outer training loop:  15%|#5        | 7737/50000 [3:44:07<20:00:46,  1.70s/it]
Outer training loop:  15%|#5        | 7738/50000 [3:44:09<19:52:15,  1.69s/it]
Outer training loop:  15%|#5        | 7739/50000 [3:44:10<20:03:48,  1.71s/it]
Outer training loop:  15%|#5        | 7740/50000 [3:44:12<20:14:26,  1.72s/it]
Outer training loop:  15%|#5        | 7741/50000 [3:44:14<19:55:37,  1.70s/it]
Outer training loop:  15%|#5        | 7742/50000 [3:44:15<19:50:09,  1.69s/it]
Outer training loop:  15%|#5        | 7743/50000 [3:44:17<19:37:22,  1.67s/it]
Outer training loop:  15%|#5        | 7744/50000 [3:44:19<19:46:48,  1.69s/it]
Outer training loop:  15%|#5        | 7745/50000 [3:44:21<20:04:57,  1.71s/it]
Outer training loop:  15%|#5        | 7746/50000 [3:44:22<19:42:56,  1.68s/it]
Outer training loop:  15%|#5        | 7747/50000 [3:44:24<19:28:35,  1.66s/it]
Outer training loop:  15%|#5        | 7748/50000 [3:44:26<20:04:22,  1.71s/it]
Outer training loop:  15%|#5        | 7749/50000 [3:44:27<20:00:00,  1.70s/it]
Outer training loop:  16%|#5        | 7750/50000 [3:44:29<20:25:14,  1.74s/it]
Outer training loop:  16%|#5        | 7751/50000 [3:44:31<20:04:33,  1.71s/it]
Outer training loop:  16%|#5        | 7752/50000 [3:44:32<20:04:20,  1.71s/it]
Outer training loop:  16%|#5        | 7753/50000 [3:44:34<20:37:27,  1.76s/it]
Outer training loop:  16%|#5        | 7754/50000 [3:44:36<20:00:47,  1.71s/it]
Outer training loop:  16%|#5        | 7755/50000 [3:44:38<19:47:43,  1.69s/it]
Outer training loop:  16%|#5        | 7756/50000 [3:44:39<20:17:03,  1.73s/it]
Outer training loop:  16%|#5        | 7757/50000 [3:44:41<19:50:18,  1.69s/it]
Outer training loop:  16%|#5        | 7758/50000 [3:44:43<19:43:53,  1.68s/it]
Outer training loop:  16%|#5        | 7759/50000 [3:44:44<20:15:54,  1.73s/it]
Outer training loop:  16%|#5        | 7760/50000 [3:44:46<20:06:48,  1.71s/it]
Outer training loop:  16%|#5        | 7761/50000 [3:44:48<19:49:04,  1.69s/it]
Outer training loop:  16%|#5        | 7762/50000 [3:44:50<20:20:24,  1.73s/it]
Outer training loop:  16%|#5        | 7763/50000 [3:44:51<20:00:51,  1.71s/it]
Outer training loop:  16%|#5        | 7764/50000 [3:44:53<19:20:38,  1.65s/it]
Outer training loop:  16%|#5        | 7765/50000 [3:44:55<19:49:10,  1.69s/it]
Outer training loop:  16%|#5        | 7766/50000 [3:44:56<19:37:56,  1.67s/it]
Outer training loop:  16%|#5        | 7767/50000 [3:44:58<19:13:45,  1.64s/it]
Outer training loop:  16%|#5        | 7768/50000 [3:45:00<19:46:06,  1.69s/it]
Outer training loop:  16%|#5        | 7769/50000 [3:45:01<19:31:52,  1.66s/it]
Outer training loop:  16%|#5        | 7770/50000 [3:45:03<19:54:02,  1.70s/it]
Outer training loop:  16%|#5        | 7771/50000 [3:45:05<20:05:20,  1.71s/it]
Outer training loop:  16%|#5        | 7772/50000 [3:45:06<19:49:50,  1.69s/it]
Outer training loop:  16%|#5        | 7773/50000 [3:45:08<20:37:23,  1.76s/it]
Outer training loop:  16%|#5        | 7774/50000 [3:45:10<20:13:50,  1.72s/it]
Outer training loop:  16%|#5        | 7775/50000 [3:45:12<20:02:30,  1.71s/it]
Outer training loop:  16%|#5        | 7776/50000 [3:45:13<20:17:12,  1.73s/it]
Outer training loop:  16%|#5        | 7777/50000 [3:45:15<20:43:59,  1.77s/it]
Outer training loop:  16%|#5        | 7778/50000 [3:45:17<20:27:25,  1.74s/it]
Outer training loop:  16%|#5        | 7779/50000 [3:45:19<20:38:05,  1.76s/it]
Outer training loop:  16%|#5        | 7780/50000 [3:45:20<20:24:05,  1.74s/it]
Outer training loop:  16%|#5        | 7781/50000 [3:45:22<19:51:05,  1.69s/it]
Outer training loop:  16%|#5        | 7782/50000 [3:45:24<20:29:38,  1.75s/it]
Outer training loop:  16%|#5        | 7783/50000 [3:45:26<20:09:47,  1.72s/it]
Outer training loop:  16%|#5        | 7784/50000 [3:45:27<20:29:22,  1.75s/it]
Outer training loop:  16%|#5        | 7785/50000 [3:45:29<19:50:47,  1.69s/it]
Outer training loop:  16%|#5        | 7786/50000 [3:45:31<19:56:48,  1.70s/it]
Outer training loop:  16%|#5        | 7787/50000 [3:45:32<20:07:43,  1.72s/it]
Outer training loop:  16%|#5        | 7788/50000 [3:45:34<20:01:36,  1.71s/it]
Outer training loop:  16%|#5        | 7789/50000 [3:45:36<19:56:12,  1.70s/it]
Outer training loop:  16%|#5        | 7790/50000 [3:45:37<20:05:45,  1.71s/it]
Outer training loop:  16%|#5        | 7791/50000 [3:45:39<19:47:58,  1.69s/it]
Outer training loop:  16%|#5        | 7792/50000 [3:45:41<19:48:07,  1.69s/it]
Outer training loop:  16%|#5        | 7793/50000 [3:45:43<20:21:03,  1.74s/it]
Outer training loop:  16%|#5        | 7794/50000 [3:45:44<19:57:00,  1.70s/it]
Outer training loop:  16%|#5        | 7795/50000 [3:45:46<19:39:51,  1.68s/it]
Outer training loop:  16%|#5        | 7796/50000 [3:45:48<19:52:18,  1.70s/it]
Outer training loop:  16%|#5        | 7797/50000 [3:45:49<19:51:44,  1.69s/it]
Outer training loop:  16%|#5        | 7798/50000 [3:45:51<19:56:56,  1.70s/it]
Outer training loop:  16%|#5        | 7799/50000 [3:45:53<19:59:05,  1.70s/it]
Outer training loop:  16%|#5        | 7800/50000 [3:45:54<19:39:09,  1.68s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.73s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.66s/it]

Outer training loop:  16%|#5        | 7801/50000 [3:46:01<36:34:43,  3.12s/it]
Outer training loop:  16%|#5        | 7802/50000 [3:46:03<32:15:48,  2.75s/it]
Outer training loop:  16%|#5        | 7803/50000 [3:46:04<28:41:02,  2.45s/it]
Outer training loop:  16%|#5        | 7804/50000 [3:46:06<25:40:39,  2.19s/it]
Outer training loop:  16%|#5        | 7805/50000 [3:46:08<24:10:19,  2.06s/it]
Outer training loop:  16%|#5        | 7806/50000 [3:46:10<22:51:06,  1.95s/it]
Outer training loop:  16%|#5        | 7807/50000 [3:46:11<21:40:01,  1.85s/it]
Outer training loop:  16%|#5        | 7808/50000 [3:46:13<21:24:09,  1.83s/it]
Outer training loop:  16%|#5        | 7809/50000 [3:46:15<20:55:37,  1.79s/it]
Outer training loop:  16%|#5        | 7810/50000 [3:46:16<21:05:45,  1.80s/it]
Outer training loop:  16%|#5        | 7811/50000 [3:46:18<20:37:19,  1.76s/it]
Outer training loop:  16%|#5        | 7812/50000 [3:46:20<20:23:55,  1.74s/it]
Outer training loop:  16%|#5        | 7813/50000 [3:46:21<20:15:10,  1.73s/it]
Outer training loop:  16%|#5        | 7814/50000 [3:46:23<20:05:00,  1.71s/it]
Outer training loop:  16%|#5        | 7815/50000 [3:46:25<20:07:07,  1.72s/it]
Outer training loop:  16%|#5        | 7816/50000 [3:46:27<20:25:31,  1.74s/it]
Outer training loop:  16%|#5        | 7817/50000 [3:46:28<20:01:36,  1.71s/it]
Outer training loop:  16%|#5        | 7818/50000 [3:46:30<19:35:17,  1.67s/it]
Outer training loop:  16%|#5        | 7819/50000 [3:46:32<19:48:10,  1.69s/it]
Outer training loop:  16%|#5        | 7820/50000 [3:46:33<19:43:16,  1.68s/it]
Outer training loop:  16%|#5        | 7821/50000 [3:46:35<19:48:18,  1.69s/it]
Outer training loop:  16%|#5        | 7822/50000 [3:46:37<20:23:10,  1.74s/it]
Outer training loop:  16%|#5        | 7823/50000 [3:46:38<19:59:35,  1.71s/it]
Outer training loop:  16%|#5        | 7824/50000 [3:46:40<20:21:10,  1.74s/it]
Outer training loop:  16%|#5        | 7825/50000 [3:46:42<20:39:12,  1.76s/it]
Outer training loop:  16%|#5        | 7826/50000 [3:46:44<20:10:06,  1.72s/it]
Outer training loop:  16%|#5        | 7827/50000 [3:46:45<19:48:55,  1.69s/it]
Outer training loop:  16%|#5        | 7828/50000 [3:46:47<20:04:55,  1.71s/it]
Outer training loop:  16%|#5        | 7829/50000 [3:46:49<20:04:17,  1.71s/it]
Outer training loop:  16%|#5        | 7830/50000 [3:46:51<19:59:01,  1.71s/it]
Outer training loop:  16%|#5        | 7831/50000 [3:46:52<20:11:54,  1.72s/it]
Outer training loop:  16%|#5        | 7832/50000 [3:46:54<19:55:22,  1.70s/it]
Outer training loop:  16%|#5        | 7833/50000 [3:46:56<20:11:25,  1.72s/it]
Outer training loop:  16%|#5        | 7834/50000 [3:46:57<19:35:35,  1.67s/it]
Outer training loop:  16%|#5        | 7835/50000 [3:46:59<19:14:38,  1.64s/it]
Outer training loop:  16%|#5        | 7836/50000 [3:47:01<20:01:05,  1.71s/it]
Outer training loop:  16%|#5        | 7837/50000 [3:47:02<20:01:25,  1.71s/it]
Outer training loop:  16%|#5        | 7838/50000 [3:47:04<19:55:48,  1.70s/it]
Outer training loop:  16%|#5        | 7839/50000 [3:47:06<20:29:53,  1.75s/it]
Outer training loop:  16%|#5        | 7840/50000 [3:47:08<20:16:47,  1.73s/it]
Outer training loop:  16%|#5        | 7841/50000 [3:47:09<20:06:44,  1.72s/it]
Outer training loop:  16%|#5        | 7842/50000 [3:47:11<20:49:38,  1.78s/it]
Outer training loop:  16%|#5        | 7843/50000 [3:47:13<20:27:21,  1.75s/it]
Outer training loop:  16%|#5        | 7844/50000 [3:47:15<20:42:30,  1.77s/it]
Outer training loop:  16%|#5        | 7845/50000 [3:47:16<20:01:10,  1.71s/it]
Outer training loop:  16%|#5        | 7846/50000 [3:47:18<19:39:01,  1.68s/it]
Outer training loop:  16%|#5        | 7847/50000 [3:47:20<20:19:51,  1.74s/it]
Outer training loop:  16%|#5        | 7848/50000 [3:47:21<20:01:21,  1.71s/it]
Outer training loop:  16%|#5        | 7849/50000 [3:47:23<19:41:02,  1.68s/it]
Outer training loop:  16%|#5        | 7850/50000 [3:47:25<20:07:33,  1.72s/it]
Outer training loop:  16%|#5        | 7851/50000 [3:47:27<19:52:55,  1.70s/it]
Outer training loop:  16%|#5        | 7852/50000 [3:47:28<20:03:33,  1.71s/it]
Outer training loop:  16%|#5        | 7853/50000 [3:47:30<20:07:15,  1.72s/it]
Outer training loop:  16%|#5        | 7854/50000 [3:47:32<19:52:30,  1.70s/it]
Outer training loop:  16%|#5        | 7855/50000 [3:47:33<19:31:27,  1.67s/it]
Outer training loop:  16%|#5        | 7856/50000 [3:47:35<20:32:20,  1.75s/it]
Outer training loop:  16%|#5        | 7857/50000 [3:47:37<19:56:35,  1.70s/it]
Outer training loop:  16%|#5        | 7858/50000 [3:47:38<19:44:47,  1.69s/it]
Outer training loop:  16%|#5        | 7859/50000 [3:47:40<20:16:15,  1.73s/it]
Outer training loop:  16%|#5        | 7860/50000 [3:47:42<20:00:50,  1.71s/it]
Outer training loop:  16%|#5        | 7861/50000 [3:47:44<20:33:20,  1.76s/it]
Outer training loop:  16%|#5        | 7862/50000 [3:47:45<19:54:05,  1.70s/it]
Outer training loop:  16%|#5        | 7863/50000 [3:47:47<19:24:32,  1.66s/it]
Outer training loop:  16%|#5        | 7864/50000 [3:47:49<20:01:54,  1.71s/it]
Outer training loop:  16%|#5        | 7865/50000 [3:47:50<19:29:17,  1.67s/it]
Outer training loop:  16%|#5        | 7866/50000 [3:47:52<19:21:45,  1.65s/it]
Outer training loop:  16%|#5        | 7867/50000 [3:47:54<19:54:42,  1.70s/it]
Outer training loop:  16%|#5        | 7868/50000 [3:47:55<19:29:02,  1.66s/it]
Outer training loop:  16%|#5        | 7869/50000 [3:47:57<19:37:08,  1.68s/it]
Outer training loop:  16%|#5        | 7870/50000 [3:47:59<20:02:43,  1.71s/it]
Outer training loop:  16%|#5        | 7871/50000 [3:48:00<19:39:08,  1.68s/it]
Outer training loop:  16%|#5        | 7872/50000 [3:48:02<19:33:06,  1.67s/it]
Outer training loop:  16%|#5        | 7873/50000 [3:48:04<20:11:17,  1.73s/it]
Outer training loop:  16%|#5        | 7874/50000 [3:48:06<20:52:31,  1.78s/it]
Outer training loop:  16%|#5        | 7875/50000 [3:48:08<21:21:02,  1.82s/it]
Outer training loop:  16%|#5        | 7876/50000 [3:48:10<21:11:01,  1.81s/it]
Outer training loop:  16%|#5        | 7877/50000 [3:48:11<20:15:14,  1.73s/it]
Outer training loop:  16%|#5        | 7878/50000 [3:48:13<20:42:41,  1.77s/it]
Outer training loop:  16%|#5        | 7879/50000 [3:48:15<20:33:33,  1.76s/it]
Outer training loop:  16%|#5        | 7880/50000 [3:48:16<20:30:14,  1.75s/it]
Outer training loop:  16%|#5        | 7881/50000 [3:48:18<21:05:24,  1.80s/it]
Outer training loop:  16%|#5        | 7882/50000 [3:48:20<20:44:09,  1.77s/it]
Outer training loop:  16%|#5        | 7883/50000 [3:48:22<20:25:42,  1.75s/it]
Outer training loop:  16%|#5        | 7884/50000 [3:48:23<20:22:38,  1.74s/it]
Outer training loop:  16%|#5        | 7885/50000 [3:48:25<20:17:20,  1.73s/it]
Outer training loop:  16%|#5        | 7886/50000 [3:48:27<19:45:54,  1.69s/it]
Outer training loop:  16%|#5        | 7887/50000 [3:48:29<20:08:32,  1.72s/it]
Outer training loop:  16%|#5        | 7888/50000 [3:48:30<19:49:11,  1.69s/it]
Outer training loop:  16%|#5        | 7889/50000 [3:48:32<19:42:06,  1.68s/it]
Outer training loop:  16%|#5        | 7890/50000 [3:48:34<20:28:50,  1.75s/it]
Outer training loop:  16%|#5        | 7891/50000 [3:48:36<20:24:38,  1.74s/it]
Outer training loop:  16%|#5        | 7892/50000 [3:48:37<20:13:30,  1.73s/it]
Outer training loop:  16%|#5        | 7893/50000 [3:48:39<20:26:01,  1.75s/it]
Outer training loop:  16%|#5        | 7894/50000 [3:48:41<20:30:45,  1.75s/it]
Outer training loop:  16%|#5        | 7895/50000 [3:48:43<20:28:18,  1.75s/it]
Outer training loop:  16%|#5        | 7896/50000 [3:48:44<20:18:38,  1.74s/it]
Outer training loop:  16%|#5        | 7897/50000 [3:48:46<20:13:10,  1.73s/it]
Outer training loop:  16%|#5        | 7898/50000 [3:48:48<20:29:20,  1.75s/it]
Outer training loop:  16%|#5        | 7899/50000 [3:48:49<19:55:01,  1.70s/it]
Outer training loop:  16%|#5        | 7900/50000 [3:48:51<20:08:27,  1.72s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.62s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.57s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.58s/it]

Outer training loop:  16%|#5        | 7901/50000 [3:48:58<36:55:13,  3.16s/it]
Outer training loop:  16%|#5        | 7902/50000 [3:48:59<31:52:36,  2.73s/it]
Outer training loop:  16%|#5        | 7903/50000 [3:49:01<28:03:40,  2.40s/it]
Outer training loop:  16%|#5        | 7904/50000 [3:49:03<25:36:18,  2.19s/it]
Outer training loop:  16%|#5        | 7905/50000 [3:49:04<24:19:45,  2.08s/it]
Outer training loop:  16%|#5        | 7906/50000 [3:49:06<23:11:37,  1.98s/it]
Outer training loop:  16%|#5        | 7907/50000 [3:49:08<22:34:23,  1.93s/it]
Outer training loop:  16%|#5        | 7908/50000 [3:49:10<21:32:13,  1.84s/it]
Outer training loop:  16%|#5        | 7909/50000 [3:49:11<20:53:03,  1.79s/it]
Outer training loop:  16%|#5        | 7910/50000 [3:49:13<20:00:58,  1.71s/it]
Outer training loop:  16%|#5        | 7911/50000 [3:49:15<20:04:26,  1.72s/it]
Outer training loop:  16%|#5        | 7912/50000 [3:49:16<18:56:01,  1.62s/it]
Outer training loop:  16%|#5        | 7913/50000 [3:49:18<18:49:00,  1.61s/it]
Outer training loop:  16%|#5        | 7914/50000 [3:49:19<19:24:10,  1.66s/it]
Outer training loop:  16%|#5        | 7915/50000 [3:49:21<18:35:43,  1.59s/it]
Outer training loop:  16%|#5        | 7916/50000 [3:49:22<18:22:09,  1.57s/it]
Outer training loop:  16%|#5        | 7917/50000 [3:49:24<18:57:27,  1.62s/it]
Outer training loop:  16%|#5        | 7918/50000 [3:49:26<19:34:35,  1.67s/it]
Outer training loop:  16%|#5        | 7919/50000 [3:49:28<19:56:28,  1.71s/it]
Outer training loop:  16%|#5        | 7920/50000 [3:49:29<19:30:47,  1.67s/it]
Outer training loop:  16%|#5        | 7921/50000 [3:49:31<20:07:53,  1.72s/it]
Outer training loop:  16%|#5        | 7922/50000 [3:49:32<18:59:08,  1.62s/it]
Outer training loop:  16%|#5        | 7923/50000 [3:49:34<18:46:53,  1.61s/it]
Outer training loop:  16%|#5        | 7924/50000 [3:49:36<19:50:35,  1.70s/it]
Outer training loop:  16%|#5        | 7925/50000 [3:49:38<19:53:34,  1.70s/it]
Outer training loop:  16%|#5        | 7926/50000 [3:49:39<20:25:03,  1.75s/it]
Outer training loop:  16%|#5        | 7927/50000 [3:49:41<20:43:07,  1.77s/it]
Outer training loop:  16%|#5        | 7928/50000 [3:49:43<19:30:38,  1.67s/it]
Outer training loop:  16%|#5        | 7929/50000 [3:49:45<19:48:08,  1.69s/it]
Outer training loop:  16%|#5        | 7930/50000 [3:49:46<20:38:31,  1.77s/it]
Outer training loop:  16%|#5        | 7931/50000 [3:49:48<20:33:08,  1.76s/it]
Outer training loop:  16%|#5        | 7932/50000 [3:49:50<20:29:44,  1.75s/it]
Outer training loop:  16%|#5        | 7933/50000 [3:49:52<20:52:45,  1.79s/it]
Outer training loop:  16%|#5        | 7934/50000 [3:49:54<20:40:01,  1.77s/it]
Outer training loop:  16%|#5        | 7935/50000 [3:49:55<20:32:39,  1.76s/it]
Outer training loop:  16%|#5        | 7936/50000 [3:49:57<20:53:06,  1.79s/it]
Outer training loop:  16%|#5        | 7937/50000 [3:49:59<20:42:16,  1.77s/it]
Outer training loop:  16%|#5        | 7938/50000 [3:50:01<20:29:02,  1.75s/it]
Outer training loop:  16%|#5        | 7939/50000 [3:50:02<21:00:46,  1.80s/it]
Outer training loop:  16%|#5        | 7940/50000 [3:50:04<21:00:06,  1.80s/it]
Outer training loop:  16%|#5        | 7941/50000 [3:50:06<20:36:31,  1.76s/it]
Outer training loop:  16%|#5        | 7942/50000 [3:50:08<21:08:39,  1.81s/it]
Outer training loop:  16%|#5        | 7943/50000 [3:50:10<21:18:33,  1.82s/it]
Outer training loop:  16%|#5        | 7944/50000 [3:50:11<20:41:30,  1.77s/it]
Outer training loop:  16%|#5        | 7945/50000 [3:50:13<20:28:11,  1.75s/it]
Outer training loop:  16%|#5        | 7946/50000 [3:50:15<21:21:16,  1.83s/it]
Outer training loop:  16%|#5        | 7947/50000 [3:50:17<20:55:27,  1.79s/it]
Outer training loop:  16%|#5        | 7948/50000 [3:50:19<20:49:07,  1.78s/it]
Outer training loop:  16%|#5        | 7949/50000 [3:50:20<20:44:58,  1.78s/it]
Outer training loop:  16%|#5        | 7950/50000 [3:50:22<21:03:17,  1.80s/it]
Outer training loop:  16%|#5        | 7951/50000 [3:50:24<20:55:27,  1.79s/it]
Outer training loop:  16%|#5        | 7952/50000 [3:50:26<21:02:57,  1.80s/it]
Outer training loop:  16%|#5        | 7953/50000 [3:50:27<20:30:46,  1.76s/it]
Outer training loop:  16%|#5        | 7954/50000 [3:50:29<19:44:13,  1.69s/it]
Outer training loop:  16%|#5        | 7955/50000 [3:50:31<20:29:32,  1.75s/it]
Outer training loop:  16%|#5        | 7956/50000 [3:50:33<20:30:03,  1.76s/it]
Outer training loop:  16%|#5        | 7957/50000 [3:50:34<20:30:00,  1.76s/it]
Outer training loop:  16%|#5        | 7958/50000 [3:50:36<20:53:37,  1.79s/it]
Outer training loop:  16%|#5        | 7959/50000 [3:50:38<20:37:32,  1.77s/it]
Outer training loop:  16%|#5        | 7960/50000 [3:50:40<20:20:40,  1.74s/it]
Outer training loop:  16%|#5        | 7961/50000 [3:50:41<19:58:51,  1.71s/it]
Outer training loop:  16%|#5        | 7962/50000 [3:50:43<20:45:54,  1.78s/it]
Outer training loop:  16%|#5        | 7963/50000 [3:50:45<20:47:33,  1.78s/it]
Outer training loop:  16%|#5        | 7964/50000 [3:50:47<20:26:41,  1.75s/it]
Outer training loop:  16%|#5        | 7965/50000 [3:50:49<21:13:02,  1.82s/it]
Outer training loop:  16%|#5        | 7966/50000 [3:50:50<20:52:20,  1.79s/it]
Outer training loop:  16%|#5        | 7967/50000 [3:50:52<20:18:52,  1.74s/it]
Outer training loop:  16%|#5        | 7968/50000 [3:50:54<20:39:03,  1.77s/it]
Outer training loop:  16%|#5        | 7969/50000 [3:50:56<20:40:24,  1.77s/it]
Outer training loop:  16%|#5        | 7970/50000 [3:50:57<20:50:27,  1.79s/it]
Outer training loop:  16%|#5        | 7971/50000 [3:51:00<22:02:47,  1.89s/it]
Outer training loop:  16%|#5        | 7972/50000 [3:51:01<21:46:45,  1.87s/it]
Outer training loop:  16%|#5        | 7973/50000 [3:51:03<21:37:33,  1.85s/it]
Outer training loop:  16%|#5        | 7974/50000 [3:51:05<21:57:05,  1.88s/it]
Outer training loop:  16%|#5        | 7975/50000 [3:51:07<21:47:15,  1.87s/it]
Outer training loop:  16%|#5        | 7976/50000 [3:51:09<20:56:45,  1.79s/it]
Outer training loop:  16%|#5        | 7977/50000 [3:51:10<20:26:13,  1.75s/it]
Outer training loop:  16%|#5        | 7978/50000 [3:51:12<20:45:46,  1.78s/it]
Outer training loop:  16%|#5        | 7979/50000 [3:51:14<20:26:51,  1.75s/it]
Outer training loop:  16%|#5        | 7980/50000 [3:51:16<21:01:44,  1.80s/it]
Outer training loop:  16%|#5        | 7981/50000 [3:51:18<21:29:02,  1.84s/it]
Outer training loop:  16%|#5        | 7982/50000 [3:51:19<20:58:51,  1.80s/it]
Outer training loop:  16%|#5        | 7983/50000 [3:51:21<21:09:39,  1.81s/it]
Outer training loop:  16%|#5        | 7984/50000 [3:51:23<21:22:39,  1.83s/it]
Outer training loop:  16%|#5        | 7985/50000 [3:51:25<21:31:26,  1.84s/it]
Outer training loop:  16%|#5        | 7986/50000 [3:51:27<21:16:04,  1.82s/it]
Outer training loop:  16%|#5        | 7987/50000 [3:51:29<21:46:32,  1.87s/it]
Outer training loop:  16%|#5        | 7988/50000 [3:51:30<21:19:07,  1.83s/it]
Outer training loop:  16%|#5        | 7989/50000 [3:51:32<20:48:54,  1.78s/it]
Outer training loop:  16%|#5        | 7990/50000 [3:51:34<21:22:47,  1.83s/it]
Outer training loop:  16%|#5        | 7991/50000 [3:51:36<21:00:01,  1.80s/it]
Outer training loop:  16%|#5        | 7992/50000 [3:51:37<20:09:25,  1.73s/it]
Outer training loop:  16%|#5        | 7993/50000 [3:51:39<20:01:31,  1.72s/it]
Outer training loop:  16%|#5        | 7994/50000 [3:51:41<20:38:02,  1.77s/it]
Outer training loop:  16%|#5        | 7995/50000 [3:51:43<20:11:21,  1.73s/it]
Outer training loop:  16%|#5        | 7996/50000 [3:51:44<20:16:13,  1.74s/it]
Outer training loop:  16%|#5        | 7997/50000 [3:51:46<21:03:35,  1.80s/it]
Outer training loop:  16%|#5        | 7998/50000 [3:51:48<20:29:32,  1.76s/it]
Outer training loop:  16%|#5        | 7999/50000 [3:51:50<20:29:42,  1.76s/it]
Outer training loop:  16%|#6        | 8000/50000 [3:51:51<20:33:31,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.65s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.70s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:17,  1.68s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:17,  1.70s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:18,  1.73s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.78s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:18,  1.77s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:14,  1.75s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:16,  1.80s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:13,  1.78s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:13,  1.81s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:09,  1.77s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:12,  1.84s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.83s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.79s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:02,  1.77s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:03,  1.81s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:58,  1.77s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.80s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:54,  1.77s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:57,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:50,  1.79s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:46,  1.75s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:45,  1.76s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:46,  1.83s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:43,  1.81s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:42,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:39,  1.81s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:35,  1.77s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:56<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:32,  1.82s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:30,  1.80s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:26,  1.76s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:26,  1.83s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:22,  1.79s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:18,  1.75s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:17,  1.76s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.74s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:17,  1.85s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:15,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:12,  1.80s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:09,  1.79s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:06,  1.74s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:03,  1.71s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:00,  1.69s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:04,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:58,  1.76s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:55,  1.75s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:54,  1.75s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:52,  1.74s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:50,  1.75s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:48,  1.73s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:50,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:45,  1.81s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.79s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:41,  1.78s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:39,  1.78s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:37,  1.77s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.77s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:55<00:35,  1.89s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:30,  1.79s/it]

Evaluating policy:  81%|########  | 67/83 [02:00<00:29,  1.82s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:26,  1.80s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:24,  1.76s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:23,  1.80s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:21,  1.78s/it]

Evaluating policy:  87%|########6 | 72/83 [02:09<00:20,  1.89s/it]

Evaluating policy:  88%|########7 | 73/83 [02:11<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:14,  1.81s/it]

Evaluating policy:  92%|#########1| 76/83 [02:16<00:12,  1.78s/it]

Evaluating policy:  93%|#########2| 77/83 [02:18<00:10,  1.76s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.76s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.81s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.79s/it]

Evaluating policy:  98%|#########7| 81/83 [02:25<00:03,  1.94s/it]

Evaluating policy:  99%|#########8| 82/83 [02:27<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 83/83 [02:29<00:00,  1.80s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:14,  1.64s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:14,  1.66s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:13,  1.67s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:15,  1.72s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:13,  1.71s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:13,  1.73s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:12,  1.75s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:24,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:18,  1.88s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:12,  1.82s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:10,  1.81s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.80s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:08,  1.83s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:05,  1.82s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:02,  1.80s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:58,  1.78s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<02:00,  1.82s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:52,  1.76s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:49,  1.74s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:48,  1.75s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:50,  1.82s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:47,  1.80s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:44,  1.78s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:42,  1.77s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:39,  1.75s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:41,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:38,  1.80s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:35,  1.77s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.76s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:30,  1.74s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:30,  1.81s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:29,  1.83s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:26,  1.79s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:24,  1.80s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:23,  1.81s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:24,  1.87s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:20,  1.82s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:18,  1.82s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:15,  1.79s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:12,  1.77s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:10,  1.76s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.77s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:08,  1.86s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:02,  1.80s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.78s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:58,  1.76s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:53,  1.78s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:51,  1.76s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:49,  1.77s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:48,  1.79s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:46,  1.78s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:44,  1.80s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:48<00:42,  1.85s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:39,  1.79s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:37,  1.77s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:53<00:35,  1.77s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:33,  1.75s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:31,  1.75s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:29,  1.73s/it]

Evaluating policy:  81%|########  | 67/83 [02:00<00:30,  1.88s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:04<00:25,  1.81s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:23,  1.80s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:21,  1.79s/it]

Evaluating policy:  87%|########6 | 72/83 [02:09<00:19,  1.74s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.73s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:15,  1.71s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:13,  1.74s/it]

Evaluating policy:  92%|#########1| 76/83 [02:16<00:13,  1.87s/it]

Evaluating policy:  93%|#########2| 77/83 [02:18<00:10,  1.82s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.77s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.78s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.77s/it]

Evaluating policy:  98%|#########7| 81/83 [02:25<00:03,  1.73s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.76s/it]

Outer training loop:  16%|#6        | 8001/50000 [3:56:57<1082:37:54, 92.80s/it]
Outer training loop:  16%|#6        | 8002/50000 [3:56:58<764:21:55, 65.52s/it] 
Outer training loop:  16%|#6        | 8003/50000 [3:57:01<542:55:42, 46.54s/it]
Outer training loop:  16%|#6        | 8004/50000 [3:57:02<385:44:25, 33.07s/it]
Outer training loop:  16%|#6        | 8005/50000 [3:57:04<276:26:05, 23.70s/it]
Outer training loop:  16%|#6        | 8006/50000 [3:57:06<199:47:13, 17.13s/it]
Outer training loop:  16%|#6        | 8007/50000 [3:57:08<146:39:27, 12.57s/it]
Outer training loop:  16%|#6        | 8008/50000 [3:57:10<108:54:11,  9.34s/it]
Outer training loop:  16%|#6        | 8009/50000 [3:57:12<82:38:41,  7.09s/it] 
Outer training loop:  16%|#6        | 8010/50000 [3:57:13<64:16:57,  5.51s/it]
Outer training loop:  16%|#6        | 8011/50000 [3:57:15<51:13:54,  4.39s/it]
Outer training loop:  16%|#6        | 8012/50000 [3:57:17<41:58:33,  3.60s/it]
Outer training loop:  16%|#6        | 8013/50000 [3:57:19<35:48:02,  3.07s/it]
Outer training loop:  16%|#6        | 8014/50000 [3:57:21<31:58:27,  2.74s/it]
Outer training loop:  16%|#6        | 8015/50000 [3:57:23<28:31:36,  2.45s/it]
Outer training loop:  16%|#6        | 8016/50000 [3:57:24<25:48:30,  2.21s/it]
Outer training loop:  16%|#6        | 8017/50000 [3:57:26<24:26:44,  2.10s/it]
Outer training loop:  16%|#6        | 8018/50000 [3:57:28<23:05:07,  1.98s/it]
Outer training loop:  16%|#6        | 8019/50000 [3:57:29<22:15:42,  1.91s/it]
Outer training loop:  16%|#6        | 8020/50000 [3:57:31<22:09:36,  1.90s/it]
Outer training loop:  16%|#6        | 8021/50000 [3:57:33<21:47:05,  1.87s/it]
Outer training loop:  16%|#6        | 8022/50000 [3:57:35<21:31:30,  1.85s/it]
Outer training loop:  16%|#6        | 8023/50000 [3:57:37<21:55:00,  1.88s/it]
Outer training loop:  16%|#6        | 8024/50000 [3:57:39<21:37:43,  1.85s/it]
Outer training loop:  16%|#6        | 8025/50000 [3:57:41<21:34:14,  1.85s/it]
Outer training loop:  16%|#6        | 8026/50000 [3:57:42<21:33:42,  1.85s/it]
Outer training loop:  16%|#6        | 8027/50000 [3:57:44<21:34:28,  1.85s/it]
Outer training loop:  16%|#6        | 8028/50000 [3:57:46<21:06:24,  1.81s/it]
Outer training loop:  16%|#6        | 8029/50000 [3:57:48<20:44:53,  1.78s/it]
Outer training loop:  16%|#6        | 8030/50000 [3:57:50<21:21:44,  1.83s/it]
Outer training loop:  16%|#6        | 8031/50000 [3:57:51<20:53:26,  1.79s/it]
Outer training loop:  16%|#6        | 8032/50000 [3:57:53<20:47:42,  1.78s/it]
Outer training loop:  16%|#6        | 8033/50000 [3:57:55<20:57:38,  1.80s/it]
Outer training loop:  16%|#6        | 8034/50000 [3:57:57<20:27:57,  1.76s/it]
Outer training loop:  16%|#6        | 8035/50000 [3:57:58<20:36:58,  1.77s/it]
Outer training loop:  16%|#6        | 8036/50000 [3:58:00<21:10:11,  1.82s/it]
Outer training loop:  16%|#6        | 8037/50000 [3:58:02<20:47:37,  1.78s/it]
Outer training loop:  16%|#6        | 8038/50000 [3:58:04<20:37:56,  1.77s/it]
Outer training loop:  16%|#6        | 8039/50000 [3:58:06<21:18:51,  1.83s/it]
Outer training loop:  16%|#6        | 8040/50000 [3:58:07<21:13:34,  1.82s/it]
Outer training loop:  16%|#6        | 8041/50000 [3:58:09<20:46:58,  1.78s/it]
Outer training loop:  16%|#6        | 8042/50000 [3:58:11<21:05:49,  1.81s/it]
Outer training loop:  16%|#6        | 8043/50000 [3:58:13<20:30:17,  1.76s/it]
Outer training loop:  16%|#6        | 8044/50000 [3:58:14<20:12:12,  1.73s/it]
Outer training loop:  16%|#6        | 8045/50000 [3:58:16<21:06:02,  1.81s/it]
Outer training loop:  16%|#6        | 8046/50000 [3:58:18<20:50:02,  1.79s/it]
Outer training loop:  16%|#6        | 8047/50000 [3:58:20<20:48:11,  1.79s/it]
Outer training loop:  16%|#6        | 8048/50000 [3:58:22<20:43:41,  1.78s/it]
Outer training loop:  16%|#6        | 8049/50000 [3:58:24<21:13:18,  1.82s/it]
Outer training loop:  16%|#6        | 8050/50000 [3:58:25<20:40:06,  1.77s/it]
Outer training loop:  16%|#6        | 8051/50000 [3:58:27<20:11:26,  1.73s/it]
Outer training loop:  16%|#6        | 8052/50000 [3:58:29<21:41:20,  1.86s/it]
Outer training loop:  16%|#6        | 8053/50000 [3:58:31<21:22:47,  1.83s/it]
Outer training loop:  16%|#6        | 8054/50000 [3:58:33<21:02:05,  1.81s/it]
Outer training loop:  16%|#6        | 8055/50000 [3:58:35<22:07:51,  1.90s/it]
Outer training loop:  16%|#6        | 8056/50000 [3:58:36<21:30:47,  1.85s/it]
Outer training loop:  16%|#6        | 8057/50000 [3:58:38<20:48:43,  1.79s/it]
Outer training loop:  16%|#6        | 8058/50000 [3:58:40<21:22:25,  1.83s/it]
Outer training loop:  16%|#6        | 8059/50000 [3:58:42<21:07:49,  1.81s/it]
Outer training loop:  16%|#6        | 8060/50000 [3:58:43<20:49:58,  1.79s/it]
Outer training loop:  16%|#6        | 8061/50000 [3:58:45<21:07:34,  1.81s/it]
Outer training loop:  16%|#6        | 8062/50000 [3:58:47<21:14:02,  1.82s/it]
Outer training loop:  16%|#6        | 8063/50000 [3:58:49<21:24:37,  1.84s/it]
Outer training loop:  16%|#6        | 8064/50000 [3:58:51<20:55:47,  1.80s/it]
Outer training loop:  16%|#6        | 8065/50000 [3:58:53<21:08:58,  1.82s/it]
Outer training loop:  16%|#6        | 8066/50000 [3:58:54<20:45:23,  1.78s/it]
Outer training loop:  16%|#6        | 8067/50000 [3:58:56<20:37:30,  1.77s/it]
Outer training loop:  16%|#6        | 8068/50000 [3:58:58<20:43:07,  1.78s/it]
Outer training loop:  16%|#6        | 8069/50000 [3:59:00<20:28:21,  1.76s/it]
Outer training loop:  16%|#6        | 8070/50000 [3:59:01<20:43:24,  1.78s/it]
Outer training loop:  16%|#6        | 8071/50000 [3:59:03<20:31:01,  1.76s/it]
Outer training loop:  16%|#6        | 8072/50000 [3:59:05<20:50:36,  1.79s/it]
Outer training loop:  16%|#6        | 8073/50000 [3:59:07<20:45:33,  1.78s/it]
Outer training loop:  16%|#6        | 8074/50000 [3:59:08<20:20:11,  1.75s/it]
Outer training loop:  16%|#6        | 8075/50000 [3:59:10<21:06:36,  1.81s/it]
Outer training loop:  16%|#6        | 8076/50000 [3:59:12<20:29:23,  1.76s/it]
Outer training loop:  16%|#6        | 8077/50000 [3:59:14<20:15:43,  1.74s/it]
Outer training loop:  16%|#6        | 8078/50000 [3:59:16<20:42:32,  1.78s/it]
Outer training loop:  16%|#6        | 8079/50000 [3:59:17<20:38:16,  1.77s/it]
Outer training loop:  16%|#6        | 8080/50000 [3:59:19<20:40:17,  1.78s/it]
Outer training loop:  16%|#6        | 8081/50000 [3:59:21<21:29:44,  1.85s/it]
Outer training loop:  16%|#6        | 8082/50000 [3:59:23<21:04:03,  1.81s/it]
Outer training loop:  16%|#6        | 8083/50000 [3:59:25<20:40:15,  1.78s/it]
Outer training loop:  16%|#6        | 8084/50000 [3:59:26<20:50:45,  1.79s/it]
Outer training loop:  16%|#6        | 8085/50000 [3:59:28<20:52:18,  1.79s/it]
Outer training loop:  16%|#6        | 8086/50000 [3:59:30<20:21:10,  1.75s/it]
Outer training loop:  16%|#6        | 8087/50000 [3:59:32<20:34:04,  1.77s/it]
Outer training loop:  16%|#6        | 8088/50000 [3:59:34<21:22:27,  1.84s/it]
Outer training loop:  16%|#6        | 8089/50000 [3:59:35<21:07:28,  1.81s/it]
Outer training loop:  16%|#6        | 8090/50000 [3:59:37<20:34:33,  1.77s/it]
Outer training loop:  16%|#6        | 8091/50000 [3:59:39<20:20:32,  1.75s/it]
Outer training loop:  16%|#6        | 8092/50000 [3:59:41<20:56:29,  1.80s/it]
Outer training loop:  16%|#6        | 8093/50000 [3:59:42<20:13:00,  1.74s/it]
Outer training loop:  16%|#6        | 8094/50000 [3:59:44<20:04:49,  1.73s/it]
Outer training loop:  16%|#6        | 8095/50000 [3:59:46<20:42:27,  1.78s/it]
Outer training loop:  16%|#6        | 8096/50000 [3:59:48<20:34:56,  1.77s/it]
Outer training loop:  16%|#6        | 8097/50000 [3:59:49<20:20:42,  1.75s/it]
Outer training loop:  16%|#6        | 8098/50000 [3:59:51<20:48:21,  1.79s/it]
Outer training loop:  16%|#6        | 8099/50000 [3:59:53<20:43:10,  1.78s/it]
Outer training loop:  16%|#6        | 8100/50000 [3:59:55<20:21:53,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.00s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  16%|#6        | 8101/50000 [4:00:02<38:47:48,  3.33s/it]
Outer training loop:  16%|#6        | 8102/50000 [4:00:04<33:52:39,  2.91s/it]
Outer training loop:  16%|#6        | 8103/50000 [4:00:05<30:01:11,  2.58s/it]
Outer training loop:  16%|#6        | 8104/50000 [4:00:07<26:49:29,  2.30s/it]
Outer training loop:  16%|#6        | 8105/50000 [4:00:09<24:43:39,  2.12s/it]
Outer training loop:  16%|#6        | 8106/50000 [4:00:11<24:13:32,  2.08s/it]
Outer training loop:  16%|#6        | 8107/50000 [4:00:13<23:15:55,  2.00s/it]
Outer training loop:  16%|#6        | 8108/50000 [4:00:14<22:29:02,  1.93s/it]
Outer training loop:  16%|#6        | 8109/50000 [4:00:16<21:57:55,  1.89s/it]
Outer training loop:  16%|#6        | 8110/50000 [4:00:18<21:12:19,  1.82s/it]
Outer training loop:  16%|#6        | 8111/50000 [4:00:20<20:58:39,  1.80s/it]
Outer training loop:  16%|#6        | 8112/50000 [4:00:21<21:33:25,  1.85s/it]
Outer training loop:  16%|#6        | 8113/50000 [4:00:23<21:09:32,  1.82s/it]
Outer training loop:  16%|#6        | 8114/50000 [4:00:25<21:12:49,  1.82s/it]
Outer training loop:  16%|#6        | 8115/50000 [4:00:27<21:07:04,  1.82s/it]
Outer training loop:  16%|#6        | 8116/50000 [4:00:29<20:37:12,  1.77s/it]
Outer training loop:  16%|#6        | 8117/50000 [4:00:30<20:48:11,  1.79s/it]
Outer training loop:  16%|#6        | 8118/50000 [4:00:32<20:35:10,  1.77s/it]
Outer training loop:  16%|#6        | 8119/50000 [4:00:34<20:26:35,  1.76s/it]
Outer training loop:  16%|#6        | 8120/50000 [4:00:36<20:15:42,  1.74s/it]
Outer training loop:  16%|#6        | 8121/50000 [4:00:37<20:13:06,  1.74s/it]
Outer training loop:  16%|#6        | 8122/50000 [4:00:39<20:59:21,  1.80s/it]
Outer training loop:  16%|#6        | 8123/50000 [4:00:41<20:52:17,  1.79s/it]
Outer training loop:  16%|#6        | 8124/50000 [4:00:43<20:48:36,  1.79s/it]
Outer training loop:  16%|#6        | 8125/50000 [4:00:45<21:27:22,  1.84s/it]
Outer training loop:  16%|#6        | 8126/50000 [4:00:46<20:50:48,  1.79s/it]
Outer training loop:  16%|#6        | 8127/50000 [4:00:48<20:46:59,  1.79s/it]
Outer training loop:  16%|#6        | 8128/50000 [4:00:50<21:08:07,  1.82s/it]
Outer training loop:  16%|#6        | 8129/50000 [4:00:52<20:43:03,  1.78s/it]
Outer training loop:  16%|#6        | 8130/50000 [4:00:53<20:35:58,  1.77s/it]
Outer training loop:  16%|#6        | 8131/50000 [4:00:55<21:01:44,  1.81s/it]
Outer training loop:  16%|#6        | 8132/50000 [4:00:57<21:09:37,  1.82s/it]
Outer training loop:  16%|#6        | 8133/50000 [4:00:59<20:37:44,  1.77s/it]
Outer training loop:  16%|#6        | 8134/50000 [4:01:01<20:28:07,  1.76s/it]
Outer training loop:  16%|#6        | 8135/50000 [4:01:02<20:42:31,  1.78s/it]
Outer training loop:  16%|#6        | 8136/50000 [4:01:04<20:26:00,  1.76s/it]
Outer training loop:  16%|#6        | 8137/50000 [4:01:06<20:37:31,  1.77s/it]
Outer training loop:  16%|#6        | 8138/50000 [4:01:08<20:55:50,  1.80s/it]
Outer training loop:  16%|#6        | 8139/50000 [4:01:09<20:27:04,  1.76s/it]
Outer training loop:  16%|#6        | 8140/50000 [4:01:11<20:18:12,  1.75s/it]
Outer training loop:  16%|#6        | 8141/50000 [4:01:13<20:56:49,  1.80s/it]
Outer training loop:  16%|#6        | 8142/50000 [4:01:15<20:19:29,  1.75s/it]
Outer training loop:  16%|#6        | 8143/50000 [4:01:16<20:00:25,  1.72s/it]
Outer training loop:  16%|#6        | 8144/50000 [4:01:18<19:43:13,  1.70s/it]
Outer training loop:  16%|#6        | 8145/50000 [4:01:20<20:50:25,  1.79s/it]
Outer training loop:  16%|#6        | 8146/50000 [4:01:22<20:19:30,  1.75s/it]
Outer training loop:  16%|#6        | 8147/50000 [4:01:23<20:09:03,  1.73s/it]
Outer training loop:  16%|#6        | 8148/50000 [4:01:25<20:45:34,  1.79s/it]
Outer training loop:  16%|#6        | 8149/50000 [4:01:27<20:59:42,  1.81s/it]
Outer training loop:  16%|#6        | 8150/50000 [4:01:29<20:21:37,  1.75s/it]
Outer training loop:  16%|#6        | 8151/50000 [4:01:31<20:52:25,  1.80s/it]
Outer training loop:  16%|#6        | 8152/50000 [4:01:33<20:52:44,  1.80s/it]
Outer training loop:  16%|#6        | 8153/50000 [4:01:34<20:52:59,  1.80s/it]
Outer training loop:  16%|#6        | 8154/50000 [4:01:36<20:47:29,  1.79s/it]
Outer training loop:  16%|#6        | 8155/50000 [4:01:38<21:19:11,  1.83s/it]
Outer training loop:  16%|#6        | 8156/50000 [4:01:40<20:33:11,  1.77s/it]
Outer training loop:  16%|#6        | 8157/50000 [4:01:41<20:07:01,  1.73s/it]
Outer training loop:  16%|#6        | 8158/50000 [4:01:43<20:28:08,  1.76s/it]
Outer training loop:  16%|#6        | 8159/50000 [4:01:45<20:29:38,  1.76s/it]
Outer training loop:  16%|#6        | 8160/50000 [4:01:47<20:35:16,  1.77s/it]
Outer training loop:  16%|#6        | 8161/50000 [4:01:49<21:11:09,  1.82s/it]
Outer training loop:  16%|#6        | 8162/50000 [4:01:50<21:07:07,  1.82s/it]
Outer training loop:  16%|#6        | 8163/50000 [4:01:52<21:02:23,  1.81s/it]
Outer training loop:  16%|#6        | 8164/50000 [4:01:54<20:42:53,  1.78s/it]
Outer training loop:  16%|#6        | 8165/50000 [4:01:56<21:28:50,  1.85s/it]
Outer training loop:  16%|#6        | 8166/50000 [4:01:58<21:18:01,  1.83s/it]
Outer training loop:  16%|#6        | 8167/50000 [4:01:59<21:01:28,  1.81s/it]
Outer training loop:  16%|#6        | 8168/50000 [4:02:01<21:25:23,  1.84s/it]
Outer training loop:  16%|#6        | 8169/50000 [4:02:03<20:57:32,  1.80s/it]
Outer training loop:  16%|#6        | 8170/50000 [4:02:05<20:58:42,  1.81s/it]
Outer training loop:  16%|#6        | 8171/50000 [4:02:07<21:25:15,  1.84s/it]
Outer training loop:  16%|#6        | 8172/50000 [4:02:09<21:08:47,  1.82s/it]
Outer training loop:  16%|#6        | 8173/50000 [4:02:10<20:32:59,  1.77s/it]
Outer training loop:  16%|#6        | 8174/50000 [4:02:12<20:58:52,  1.81s/it]
Outer training loop:  16%|#6        | 8175/50000 [4:02:14<20:42:47,  1.78s/it]
Outer training loop:  16%|#6        | 8176/50000 [4:02:16<21:05:03,  1.81s/it]
Outer training loop:  16%|#6        | 8177/50000 [4:02:18<20:52:31,  1.80s/it]
Outer training loop:  16%|#6        | 8178/50000 [4:02:20<21:30:22,  1.85s/it]
Outer training loop:  16%|#6        | 8179/50000 [4:02:21<20:53:23,  1.80s/it]
Outer training loop:  16%|#6        | 8180/50000 [4:02:23<20:37:07,  1.77s/it]
Outer training loop:  16%|#6        | 8181/50000 [4:02:25<21:35:13,  1.86s/it]
Outer training loop:  16%|#6        | 8182/50000 [4:02:27<20:49:29,  1.79s/it]
Outer training loop:  16%|#6        | 8183/50000 [4:02:28<20:41:43,  1.78s/it]
Outer training loop:  16%|#6        | 8184/50000 [4:02:30<21:21:04,  1.84s/it]
Outer training loop:  16%|#6        | 8185/50000 [4:02:32<21:06:46,  1.82s/it]
Outer training loop:  16%|#6        | 8186/50000 [4:02:34<20:50:58,  1.80s/it]
Outer training loop:  16%|#6        | 8187/50000 [4:02:36<20:29:58,  1.76s/it]
Outer training loop:  16%|#6        | 8188/50000 [4:02:37<20:35:55,  1.77s/it]
Outer training loop:  16%|#6        | 8189/50000 [4:02:39<20:13:22,  1.74s/it]
Outer training loop:  16%|#6        | 8190/50000 [4:02:41<20:09:34,  1.74s/it]
Outer training loop:  16%|#6        | 8191/50000 [4:02:43<20:37:21,  1.78s/it]
Outer training loop:  16%|#6        | 8192/50000 [4:02:44<20:28:12,  1.76s/it]
Outer training loop:  16%|#6        | 8193/50000 [4:02:46<20:27:50,  1.76s/it]
Outer training loop:  16%|#6        | 8194/50000 [4:02:48<21:06:03,  1.82s/it]
Outer training loop:  16%|#6        | 8195/50000 [4:02:50<21:15:33,  1.83s/it]
Outer training loop:  16%|#6        | 8196/50000 [4:02:52<21:06:43,  1.82s/it]
Outer training loop:  16%|#6        | 8197/50000 [4:02:54<21:38:14,  1.86s/it]
Outer training loop:  16%|#6        | 8198/50000 [4:02:55<21:01:06,  1.81s/it]
Outer training loop:  16%|#6        | 8199/50000 [4:02:57<21:22:46,  1.84s/it]
Outer training loop:  16%|#6        | 8200/50000 [4:02:59<20:36:33,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.72s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  16%|#6        | 8201/50000 [4:03:06<39:06:06,  3.37s/it]
Outer training loop:  16%|#6        | 8202/50000 [4:03:08<33:20:00,  2.87s/it]
Outer training loop:  16%|#6        | 8203/50000 [4:03:09<29:10:45,  2.51s/it]
Outer training loop:  16%|#6        | 8204/50000 [4:03:11<26:09:10,  2.25s/it]
Outer training loop:  16%|#6        | 8205/50000 [4:03:13<24:51:04,  2.14s/it]
Outer training loop:  16%|#6        | 8206/50000 [4:03:14<23:00:39,  1.98s/it]
Outer training loop:  16%|#6        | 8207/50000 [4:03:16<22:11:43,  1.91s/it]
Outer training loop:  16%|#6        | 8208/50000 [4:03:18<22:15:21,  1.92s/it]
Outer training loop:  16%|#6        | 8209/50000 [4:03:20<21:47:01,  1.88s/it]
Outer training loop:  16%|#6        | 8210/50000 [4:03:22<21:20:05,  1.84s/it]
Outer training loop:  16%|#6        | 8211/50000 [4:03:24<21:27:40,  1.85s/it]
Outer training loop:  16%|#6        | 8212/50000 [4:03:25<21:13:37,  1.83s/it]
Outer training loop:  16%|#6        | 8213/50000 [4:03:27<20:44:26,  1.79s/it]
Outer training loop:  16%|#6        | 8214/50000 [4:03:29<21:01:47,  1.81s/it]
Outer training loop:  16%|#6        | 8215/50000 [4:03:31<21:00:37,  1.81s/it]
Outer training loop:  16%|#6        | 8216/50000 [4:03:32<20:38:26,  1.78s/it]
Outer training loop:  16%|#6        | 8217/50000 [4:03:34<20:51:56,  1.80s/it]
Outer training loop:  16%|#6        | 8218/50000 [4:03:36<21:29:47,  1.85s/it]
Outer training loop:  16%|#6        | 8219/50000 [4:03:38<21:13:36,  1.83s/it]
Outer training loop:  16%|#6        | 8220/50000 [4:03:40<21:01:53,  1.81s/it]
Outer training loop:  16%|#6        | 8221/50000 [4:03:42<21:08:13,  1.82s/it]
Outer training loop:  16%|#6        | 8222/50000 [4:03:43<20:49:47,  1.79s/it]
Outer training loop:  16%|#6        | 8223/50000 [4:03:45<20:32:22,  1.77s/it]
Outer training loop:  16%|#6        | 8224/50000 [4:03:47<20:26:38,  1.76s/it]
Outer training loop:  16%|#6        | 8225/50000 [4:03:49<21:28:55,  1.85s/it]
Outer training loop:  16%|#6        | 8226/50000 [4:03:51<21:02:36,  1.81s/it]
Outer training loop:  16%|#6        | 8227/50000 [4:03:52<20:52:52,  1.80s/it]
Outer training loop:  16%|#6        | 8228/50000 [4:03:54<20:52:38,  1.80s/it]
Outer training loop:  16%|#6        | 8229/50000 [4:03:56<21:08:50,  1.82s/it]
Outer training loop:  16%|#6        | 8230/50000 [4:03:58<20:59:02,  1.81s/it]
Outer training loop:  16%|#6        | 8231/50000 [4:04:00<20:52:16,  1.80s/it]
Outer training loop:  16%|#6        | 8232/50000 [4:04:01<20:30:21,  1.77s/it]
Outer training loop:  16%|#6        | 8233/50000 [4:04:03<20:00:36,  1.72s/it]
Outer training loop:  16%|#6        | 8234/50000 [4:04:05<20:34:05,  1.77s/it]
Outer training loop:  16%|#6        | 8235/50000 [4:04:07<20:35:05,  1.77s/it]
Outer training loop:  16%|#6        | 8236/50000 [4:04:08<20:17:10,  1.75s/it]
Outer training loop:  16%|#6        | 8237/50000 [4:04:10<20:36:21,  1.78s/it]
Outer training loop:  16%|#6        | 8238/50000 [4:04:12<20:24:22,  1.76s/it]
Outer training loop:  16%|#6        | 8239/50000 [4:04:14<20:22:12,  1.76s/it]
Outer training loop:  16%|#6        | 8240/50000 [4:04:15<20:03:55,  1.73s/it]
Outer training loop:  16%|#6        | 8241/50000 [4:04:17<20:35:21,  1.77s/it]
Outer training loop:  16%|#6        | 8242/50000 [4:04:19<20:35:15,  1.77s/it]
Outer training loop:  16%|#6        | 8243/50000 [4:04:21<20:07:14,  1.73s/it]
Outer training loop:  16%|#6        | 8244/50000 [4:04:22<20:47:17,  1.79s/it]
Outer training loop:  16%|#6        | 8245/50000 [4:04:24<20:12:19,  1.74s/it]
Outer training loop:  16%|#6        | 8246/50000 [4:04:26<20:46:50,  1.79s/it]
Outer training loop:  16%|#6        | 8247/50000 [4:04:28<21:03:31,  1.82s/it]
Outer training loop:  16%|#6        | 8248/50000 [4:04:30<20:41:04,  1.78s/it]
Outer training loop:  16%|#6        | 8249/50000 [4:04:31<20:24:36,  1.76s/it]
Outer training loop:  16%|#6        | 8250/50000 [4:04:33<20:41:53,  1.78s/it]
Outer training loop:  17%|#6        | 8251/50000 [4:04:35<20:24:18,  1.76s/it]
Outer training loop:  17%|#6        | 8252/50000 [4:04:36<20:03:23,  1.73s/it]
Outer training loop:  17%|#6        | 8253/50000 [4:04:38<20:29:56,  1.77s/it]
Outer training loop:  17%|#6        | 8254/50000 [4:04:40<20:04:04,  1.73s/it]
Outer training loop:  17%|#6        | 8255/50000 [4:04:42<19:55:50,  1.72s/it]
Outer training loop:  17%|#6        | 8256/50000 [4:04:43<19:55:43,  1.72s/it]
Outer training loop:  17%|#6        | 8257/50000 [4:04:45<20:22:32,  1.76s/it]
Outer training loop:  17%|#6        | 8258/50000 [4:04:47<20:13:26,  1.74s/it]
Outer training loop:  17%|#6        | 8259/50000 [4:04:49<20:07:18,  1.74s/it]
Outer training loop:  17%|#6        | 8260/50000 [4:04:51<20:43:35,  1.79s/it]
Outer training loop:  17%|#6        | 8261/50000 [4:04:52<20:30:17,  1.77s/it]
Outer training loop:  17%|#6        | 8262/50000 [4:04:54<20:01:11,  1.73s/it]
Outer training loop:  17%|#6        | 8263/50000 [4:04:56<20:30:32,  1.77s/it]
Outer training loop:  17%|#6        | 8264/50000 [4:04:58<20:20:06,  1.75s/it]
Outer training loop:  17%|#6        | 8265/50000 [4:04:59<20:27:20,  1.76s/it]
Outer training loop:  17%|#6        | 8266/50000 [4:05:01<21:05:29,  1.82s/it]
Outer training loop:  17%|#6        | 8267/50000 [4:05:03<20:31:47,  1.77s/it]
Outer training loop:  17%|#6        | 8268/50000 [4:05:05<20:09:09,  1.74s/it]
Outer training loop:  17%|#6        | 8269/50000 [4:05:06<20:11:49,  1.74s/it]
Outer training loop:  17%|#6        | 8270/50000 [4:05:08<20:29:49,  1.77s/it]
Outer training loop:  17%|#6        | 8271/50000 [4:05:10<20:28:27,  1.77s/it]
Outer training loop:  17%|#6        | 8272/50000 [4:05:12<20:49:49,  1.80s/it]
Outer training loop:  17%|#6        | 8273/50000 [4:05:14<20:36:08,  1.78s/it]
Outer training loop:  17%|#6        | 8274/50000 [4:05:15<20:41:59,  1.79s/it]
Outer training loop:  17%|#6        | 8275/50000 [4:05:17<20:27:33,  1.77s/it]
Outer training loop:  17%|#6        | 8276/50000 [4:05:19<20:59:40,  1.81s/it]
Outer training loop:  17%|#6        | 8277/50000 [4:05:21<20:53:21,  1.80s/it]
Outer training loop:  17%|#6        | 8278/50000 [4:05:23<20:45:55,  1.79s/it]
Outer training loop:  17%|#6        | 8279/50000 [4:05:24<21:20:54,  1.84s/it]
Outer training loop:  17%|#6        | 8280/50000 [4:05:26<21:05:07,  1.82s/it]
Outer training loop:  17%|#6        | 8281/50000 [4:05:28<21:06:28,  1.82s/it]
Outer training loop:  17%|#6        | 8282/50000 [4:05:30<21:12:19,  1.83s/it]
Outer training loop:  17%|#6        | 8283/50000 [4:05:32<20:44:41,  1.79s/it]
Outer training loop:  17%|#6        | 8284/50000 [4:05:33<20:50:13,  1.80s/it]
Outer training loop:  17%|#6        | 8285/50000 [4:05:35<21:29:23,  1.85s/it]
Outer training loop:  17%|#6        | 8286/50000 [4:05:37<21:04:53,  1.82s/it]
Outer training loop:  17%|#6        | 8287/50000 [4:05:39<20:35:05,  1.78s/it]
Outer training loop:  17%|#6        | 8288/50000 [4:05:41<20:55:01,  1.81s/it]
Outer training loop:  17%|#6        | 8289/50000 [4:05:42<20:27:02,  1.77s/it]
Outer training loop:  17%|#6        | 8290/50000 [4:05:44<20:19:41,  1.75s/it]
Outer training loop:  17%|#6        | 8291/50000 [4:05:46<20:16:10,  1.75s/it]
Outer training loop:  17%|#6        | 8292/50000 [4:05:48<20:32:38,  1.77s/it]
Outer training loop:  17%|#6        | 8293/50000 [4:05:49<20:27:09,  1.77s/it]
Outer training loop:  17%|#6        | 8294/50000 [4:05:51<20:21:23,  1.76s/it]
Outer training loop:  17%|#6        | 8295/50000 [4:05:53<20:51:11,  1.80s/it]
Outer training loop:  17%|#6        | 8296/50000 [4:05:55<21:09:04,  1.83s/it]
Outer training loop:  17%|#6        | 8297/50000 [4:05:57<20:52:04,  1.80s/it]
Outer training loop:  17%|#6        | 8298/50000 [4:05:59<21:23:42,  1.85s/it]
Outer training loop:  17%|#6        | 8299/50000 [4:06:00<20:52:27,  1.80s/it]
Outer training loop:  17%|#6        | 8300/50000 [4:06:02<20:30:32,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.64s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.68s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.65s/it]

Outer training loop:  17%|#6        | 8301/50000 [4:06:09<38:21:04,  3.31s/it]
Outer training loop:  17%|#6        | 8302/50000 [4:06:11<33:24:34,  2.88s/it]
Outer training loop:  17%|#6        | 8303/50000 [4:06:13<29:24:50,  2.54s/it]
Outer training loop:  17%|#6        | 8304/50000 [4:06:14<26:29:00,  2.29s/it]
Outer training loop:  17%|#6        | 8305/50000 [4:06:16<25:00:13,  2.16s/it]
Outer training loop:  17%|#6        | 8306/50000 [4:06:18<24:00:26,  2.07s/it]
Outer training loop:  17%|#6        | 8307/50000 [4:06:20<22:46:09,  1.97s/it]
Outer training loop:  17%|#6        | 8308/50000 [4:06:22<22:57:04,  1.98s/it]
Outer training loop:  17%|#6        | 8309/50000 [4:06:23<22:10:57,  1.92s/it]
Outer training loop:  17%|#6        | 8310/50000 [4:06:25<21:51:16,  1.89s/it]
Outer training loop:  17%|#6        | 8311/50000 [4:06:27<22:03:40,  1.91s/it]
Outer training loop:  17%|#6        | 8312/50000 [4:06:29<21:21:37,  1.84s/it]
Outer training loop:  17%|#6        | 8313/50000 [4:06:31<20:49:19,  1.80s/it]
Outer training loop:  17%|#6        | 8314/50000 [4:06:32<20:51:18,  1.80s/it]
Outer training loop:  17%|#6        | 8315/50000 [4:06:34<20:44:47,  1.79s/it]
Outer training loop:  17%|#6        | 8316/50000 [4:06:36<20:26:18,  1.77s/it]
Outer training loop:  17%|#6        | 8317/50000 [4:06:38<20:32:19,  1.77s/it]
Outer training loop:  17%|#6        | 8318/50000 [4:06:39<20:21:01,  1.76s/it]
Outer training loop:  17%|#6        | 8319/50000 [4:06:41<20:15:59,  1.75s/it]
Outer training loop:  17%|#6        | 8320/50000 [4:06:43<20:08:59,  1.74s/it]
Outer training loop:  17%|#6        | 8321/50000 [4:06:45<20:35:04,  1.78s/it]
Outer training loop:  17%|#6        | 8322/50000 [4:06:46<20:22:27,  1.76s/it]
Outer training loop:  17%|#6        | 8323/50000 [4:06:48<20:08:55,  1.74s/it]
Outer training loop:  17%|#6        | 8324/50000 [4:06:50<20:26:35,  1.77s/it]
Outer training loop:  17%|#6        | 8325/50000 [4:06:52<20:11:20,  1.74s/it]
Outer training loop:  17%|#6        | 8326/50000 [4:06:53<20:01:14,  1.73s/it]
Outer training loop:  17%|#6        | 8327/50000 [4:06:55<20:26:33,  1.77s/it]
Outer training loop:  17%|#6        | 8328/50000 [4:06:57<20:24:27,  1.76s/it]
Outer training loop:  17%|#6        | 8329/50000 [4:06:59<20:05:48,  1.74s/it]
Outer training loop:  17%|#6        | 8330/50000 [4:07:00<19:48:23,  1.71s/it]
Outer training loop:  17%|#6        | 8331/50000 [4:07:02<20:27:06,  1.77s/it]
Outer training loop:  17%|#6        | 8332/50000 [4:07:04<20:26:51,  1.77s/it]
Outer training loop:  17%|#6        | 8333/50000 [4:07:06<20:32:16,  1.77s/it]
Outer training loop:  17%|#6        | 8334/50000 [4:07:07<20:20:26,  1.76s/it]
Outer training loop:  17%|#6        | 8335/50000 [4:07:09<21:12:40,  1.83s/it]
Outer training loop:  17%|#6        | 8336/50000 [4:07:11<20:44:31,  1.79s/it]
Outer training loop:  17%|#6        | 8337/50000 [4:07:13<20:20:11,  1.76s/it]
Outer training loop:  17%|#6        | 8338/50000 [4:07:15<20:46:43,  1.80s/it]
Outer training loop:  17%|#6        | 8339/50000 [4:07:16<20:27:07,  1.77s/it]
Outer training loop:  17%|#6        | 8340/50000 [4:07:18<20:18:07,  1.75s/it]
Outer training loop:  17%|#6        | 8341/50000 [4:07:20<20:51:05,  1.80s/it]
Outer training loop:  17%|#6        | 8342/50000 [4:07:22<20:23:22,  1.76s/it]
Outer training loop:  17%|#6        | 8343/50000 [4:07:24<20:46:07,  1.79s/it]
Outer training loop:  17%|#6        | 8344/50000 [4:07:26<21:34:49,  1.87s/it]
Outer training loop:  17%|#6        | 8345/50000 [4:07:27<20:41:46,  1.79s/it]
Outer training loop:  17%|#6        | 8346/50000 [4:07:29<20:20:26,  1.76s/it]
Outer training loop:  17%|#6        | 8347/50000 [4:07:31<19:57:26,  1.72s/it]
Outer training loop:  17%|#6        | 8348/50000 [4:07:33<20:38:00,  1.78s/it]
Outer training loop:  17%|#6        | 8349/50000 [4:07:34<20:25:52,  1.77s/it]
Outer training loop:  17%|#6        | 8350/50000 [4:07:36<21:27:23,  1.85s/it]
Outer training loop:  17%|#6        | 8351/50000 [4:07:38<20:55:26,  1.81s/it]
Outer training loop:  17%|#6        | 8352/50000 [4:07:40<20:28:19,  1.77s/it]
Outer training loop:  17%|#6        | 8353/50000 [4:07:41<20:26:37,  1.77s/it]
Outer training loop:  17%|#6        | 8354/50000 [4:07:43<20:50:54,  1.80s/it]
Outer training loop:  17%|#6        | 8355/50000 [4:07:45<20:21:12,  1.76s/it]
Outer training loop:  17%|#6        | 8356/50000 [4:07:47<20:12:55,  1.75s/it]
Outer training loop:  17%|#6        | 8357/50000 [4:07:49<20:55:57,  1.81s/it]
Outer training loop:  17%|#6        | 8358/50000 [4:07:50<20:43:54,  1.79s/it]
Outer training loop:  17%|#6        | 8359/50000 [4:07:52<20:28:14,  1.77s/it]
Outer training loop:  17%|#6        | 8360/50000 [4:07:54<20:03:19,  1.73s/it]
Outer training loop:  17%|#6        | 8361/50000 [4:07:56<20:49:51,  1.80s/it]
Outer training loop:  17%|#6        | 8362/50000 [4:07:57<20:27:55,  1.77s/it]
Outer training loop:  17%|#6        | 8363/50000 [4:07:59<20:39:23,  1.79s/it]
Outer training loop:  17%|#6        | 8364/50000 [4:08:01<21:12:14,  1.83s/it]
Outer training loop:  17%|#6        | 8365/50000 [4:08:03<20:27:00,  1.77s/it]
Outer training loop:  17%|#6        | 8366/50000 [4:08:05<20:11:38,  1.75s/it]
Outer training loop:  17%|#6        | 8367/50000 [4:08:06<20:33:16,  1.78s/it]
Outer training loop:  17%|#6        | 8368/50000 [4:08:08<20:26:11,  1.77s/it]
Outer training loop:  17%|#6        | 8369/50000 [4:08:10<20:05:59,  1.74s/it]
Outer training loop:  17%|#6        | 8370/50000 [4:08:11<19:54:36,  1.72s/it]
Outer training loop:  17%|#6        | 8371/50000 [4:08:13<20:13:19,  1.75s/it]
Outer training loop:  17%|#6        | 8372/50000 [4:08:15<20:13:15,  1.75s/it]
Outer training loop:  17%|#6        | 8373/50000 [4:08:17<20:08:55,  1.74s/it]
Outer training loop:  17%|#6        | 8374/50000 [4:08:19<20:36:50,  1.78s/it]
Outer training loop:  17%|#6        | 8375/50000 [4:08:20<20:32:17,  1.78s/it]
Outer training loop:  17%|#6        | 8376/50000 [4:08:22<20:11:45,  1.75s/it]
Outer training loop:  17%|#6        | 8377/50000 [4:08:24<20:32:01,  1.78s/it]
Outer training loop:  17%|#6        | 8378/50000 [4:08:26<20:22:21,  1.76s/it]
Outer training loop:  17%|#6        | 8379/50000 [4:08:27<19:58:58,  1.73s/it]
Outer training loop:  17%|#6        | 8380/50000 [4:08:29<19:50:40,  1.72s/it]
Outer training loop:  17%|#6        | 8381/50000 [4:08:31<20:51:54,  1.80s/it]
Outer training loop:  17%|#6        | 8382/50000 [4:08:33<20:28:53,  1.77s/it]
Outer training loop:  17%|#6        | 8383/50000 [4:08:34<20:27:28,  1.77s/it]
Outer training loop:  17%|#6        | 8384/50000 [4:08:36<20:54:27,  1.81s/it]
Outer training loop:  17%|#6        | 8385/50000 [4:08:38<20:29:24,  1.77s/it]
Outer training loop:  17%|#6        | 8386/50000 [4:08:40<20:26:18,  1.77s/it]
Outer training loop:  17%|#6        | 8387/50000 [4:08:42<21:04:06,  1.82s/it]
Outer training loop:  17%|#6        | 8388/50000 [4:08:43<20:42:48,  1.79s/it]
Outer training loop:  17%|#6        | 8389/50000 [4:08:45<20:21:44,  1.76s/it]
Outer training loop:  17%|#6        | 8390/50000 [4:08:47<19:57:37,  1.73s/it]
Outer training loop:  17%|#6        | 8391/50000 [4:08:49<20:29:30,  1.77s/it]
Outer training loop:  17%|#6        | 8392/50000 [4:08:50<20:19:00,  1.76s/it]
Outer training loop:  17%|#6        | 8393/50000 [4:08:52<20:47:44,  1.80s/it]
Outer training loop:  17%|#6        | 8394/50000 [4:08:54<21:11:41,  1.83s/it]
Outer training loop:  17%|#6        | 8395/50000 [4:08:56<20:40:12,  1.79s/it]
Outer training loop:  17%|#6        | 8396/50000 [4:08:58<20:13:13,  1.75s/it]
Outer training loop:  17%|#6        | 8397/50000 [4:08:59<20:49:40,  1.80s/it]
Outer training loop:  17%|#6        | 8398/50000 [4:09:01<20:59:41,  1.82s/it]
Outer training loop:  17%|#6        | 8399/50000 [4:09:03<20:48:10,  1.80s/it]
Outer training loop:  17%|#6        | 8400/50000 [4:09:05<21:06:29,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.71s/it]

Outer training loop:  17%|#6        | 8401/50000 [4:09:12<38:32:59,  3.34s/it]
Outer training loop:  17%|#6        | 8402/50000 [4:09:14<32:56:34,  2.85s/it]
Outer training loop:  17%|#6        | 8403/50000 [4:09:15<29:04:43,  2.52s/it]
Outer training loop:  17%|#6        | 8404/50000 [4:09:17<26:43:39,  2.31s/it]
Outer training loop:  17%|#6        | 8405/50000 [4:09:19<25:01:46,  2.17s/it]
Outer training loop:  17%|#6        | 8406/50000 [4:09:21<23:42:33,  2.05s/it]
Outer training loop:  17%|#6        | 8407/50000 [4:09:23<23:17:12,  2.02s/it]
Outer training loop:  17%|#6        | 8408/50000 [4:09:24<22:17:43,  1.93s/it]
Outer training loop:  17%|#6        | 8409/50000 [4:09:26<21:13:31,  1.84s/it]
Outer training loop:  17%|#6        | 8410/50000 [4:09:28<20:36:59,  1.78s/it]
Outer training loop:  17%|#6        | 8411/50000 [4:09:30<20:51:06,  1.80s/it]
Outer training loop:  17%|#6        | 8412/50000 [4:09:31<20:29:09,  1.77s/it]
Outer training loop:  17%|#6        | 8413/50000 [4:09:33<20:15:36,  1.75s/it]
Outer training loop:  17%|#6        | 8414/50000 [4:09:35<21:02:21,  1.82s/it]
Outer training loop:  17%|#6        | 8415/50000 [4:09:37<20:24:06,  1.77s/it]
Outer training loop:  17%|#6        | 8416/50000 [4:09:38<20:10:37,  1.75s/it]
Outer training loop:  17%|#6        | 8417/50000 [4:09:40<20:02:23,  1.73s/it]
Outer training loop:  17%|#6        | 8418/50000 [4:09:42<20:51:19,  1.81s/it]
Outer training loop:  17%|#6        | 8419/50000 [4:09:44<20:26:56,  1.77s/it]
Outer training loop:  17%|#6        | 8420/50000 [4:09:45<20:22:09,  1.76s/it]
Outer training loop:  17%|#6        | 8421/50000 [4:09:47<20:50:01,  1.80s/it]
Outer training loop:  17%|#6        | 8422/50000 [4:09:49<20:27:06,  1.77s/it]
Outer training loop:  17%|#6        | 8423/50000 [4:09:51<20:03:18,  1.74s/it]
Outer training loop:  17%|#6        | 8424/50000 [4:09:52<20:24:30,  1.77s/it]
Outer training loop:  17%|#6        | 8425/50000 [4:09:54<20:26:18,  1.77s/it]
Outer training loop:  17%|#6        | 8426/50000 [4:09:56<20:06:34,  1.74s/it]
Outer training loop:  17%|#6        | 8427/50000 [4:09:58<20:25:57,  1.77s/it]
Outer training loop:  17%|#6        | 8428/50000 [4:10:00<20:44:00,  1.80s/it]
Outer training loop:  17%|#6        | 8429/50000 [4:10:01<20:24:31,  1.77s/it]
Outer training loop:  17%|#6        | 8430/50000 [4:10:03<20:32:03,  1.78s/it]
Outer training loop:  17%|#6        | 8431/50000 [4:10:05<21:16:20,  1.84s/it]
Outer training loop:  17%|#6        | 8432/50000 [4:10:07<20:50:49,  1.81s/it]
Outer training loop:  17%|#6        | 8433/50000 [4:10:09<20:25:55,  1.77s/it]
Outer training loop:  17%|#6        | 8434/50000 [4:10:10<20:15:51,  1.76s/it]
Outer training loop:  17%|#6        | 8435/50000 [4:10:12<20:54:22,  1.81s/it]
Outer training loop:  17%|#6        | 8436/50000 [4:10:14<20:38:59,  1.79s/it]
Outer training loop:  17%|#6        | 8437/50000 [4:10:16<20:11:57,  1.75s/it]
Outer training loop:  17%|#6        | 8438/50000 [4:10:18<21:03:16,  1.82s/it]
Outer training loop:  17%|#6        | 8439/50000 [4:10:19<20:59:42,  1.82s/it]
Outer training loop:  17%|#6        | 8440/50000 [4:10:21<21:29:56,  1.86s/it]
Outer training loop:  17%|#6        | 8441/50000 [4:10:23<21:24:25,  1.85s/it]
Outer training loop:  17%|#6        | 8442/50000 [4:10:25<21:00:30,  1.82s/it]
Outer training loop:  17%|#6        | 8443/50000 [4:10:27<20:48:06,  1.80s/it]
Outer training loop:  17%|#6        | 8444/50000 [4:10:28<20:15:47,  1.76s/it]
Outer training loop:  17%|#6        | 8445/50000 [4:10:30<20:52:16,  1.81s/it]
Outer training loop:  17%|#6        | 8446/50000 [4:10:32<20:12:21,  1.75s/it]
Outer training loop:  17%|#6        | 8447/50000 [4:10:34<20:29:06,  1.77s/it]
Outer training loop:  17%|#6        | 8448/50000 [4:10:36<21:02:19,  1.82s/it]
Outer training loop:  17%|#6        | 8449/50000 [4:10:37<20:52:40,  1.81s/it]
Outer training loop:  17%|#6        | 8450/50000 [4:10:39<20:31:53,  1.78s/it]
Outer training loop:  17%|#6        | 8451/50000 [4:10:41<21:05:57,  1.83s/it]
Outer training loop:  17%|#6        | 8452/50000 [4:10:43<20:44:58,  1.80s/it]
Outer training loop:  17%|#6        | 8453/50000 [4:10:45<20:28:17,  1.77s/it]
Outer training loop:  17%|#6        | 8454/50000 [4:10:46<20:14:46,  1.75s/it]
Outer training loop:  17%|#6        | 8455/50000 [4:10:48<21:02:42,  1.82s/it]
Outer training loop:  17%|#6        | 8456/50000 [4:10:50<20:46:47,  1.80s/it]
Outer training loop:  17%|#6        | 8457/50000 [4:10:52<20:37:09,  1.79s/it]
Outer training loop:  17%|#6        | 8458/50000 [4:10:54<20:46:56,  1.80s/it]
Outer training loop:  17%|#6        | 8459/50000 [4:10:55<20:44:47,  1.80s/it]
Outer training loop:  17%|#6        | 8460/50000 [4:10:57<20:20:41,  1.76s/it]
Outer training loop:  17%|#6        | 8461/50000 [4:10:59<20:43:25,  1.80s/it]
Outer training loop:  17%|#6        | 8462/50000 [4:11:01<20:27:28,  1.77s/it]
Outer training loop:  17%|#6        | 8463/50000 [4:11:02<20:32:07,  1.78s/it]
Outer training loop:  17%|#6        | 8464/50000 [4:11:04<20:05:28,  1.74s/it]
Outer training loop:  17%|#6        | 8465/50000 [4:11:06<20:58:17,  1.82s/it]
Outer training loop:  17%|#6        | 8466/50000 [4:11:08<20:35:21,  1.78s/it]
Outer training loop:  17%|#6        | 8467/50000 [4:11:09<20:13:51,  1.75s/it]
Outer training loop:  17%|#6        | 8468/50000 [4:11:11<20:58:59,  1.82s/it]
Outer training loop:  17%|#6        | 8469/50000 [4:11:13<20:18:49,  1.76s/it]
Outer training loop:  17%|#6        | 8470/50000 [4:11:15<19:57:09,  1.73s/it]
Outer training loop:  17%|#6        | 8471/50000 [4:11:17<20:25:32,  1.77s/it]
Outer training loop:  17%|#6        | 8472/50000 [4:11:18<19:54:02,  1.73s/it]
Outer training loop:  17%|#6        | 8473/50000 [4:11:20<20:23:31,  1.77s/it]
Outer training loop:  17%|#6        | 8474/50000 [4:11:22<20:51:16,  1.81s/it]
Outer training loop:  17%|#6        | 8475/50000 [4:11:24<20:49:55,  1.81s/it]
Outer training loop:  17%|#6        | 8476/50000 [4:11:26<20:45:51,  1.80s/it]
Outer training loop:  17%|#6        | 8477/50000 [4:11:27<20:47:06,  1.80s/it]
Outer training loop:  17%|#6        | 8478/50000 [4:11:29<19:56:15,  1.73s/it]
Outer training loop:  17%|#6        | 8479/50000 [4:11:31<20:06:18,  1.74s/it]
Outer training loop:  17%|#6        | 8480/50000 [4:11:32<19:49:16,  1.72s/it]
Outer training loop:  17%|#6        | 8481/50000 [4:11:34<20:33:19,  1.78s/it]
Outer training loop:  17%|#6        | 8482/50000 [4:11:36<20:20:14,  1.76s/it]
Outer training loop:  17%|#6        | 8483/50000 [4:11:38<20:00:16,  1.73s/it]
Outer training loop:  17%|#6        | 8484/50000 [4:11:39<20:20:36,  1.76s/it]
Outer training loop:  17%|#6        | 8485/50000 [4:11:41<20:16:24,  1.76s/it]
Outer training loop:  17%|#6        | 8486/50000 [4:11:43<19:54:38,  1.73s/it]
Outer training loop:  17%|#6        | 8487/50000 [4:11:45<20:27:37,  1.77s/it]
Outer training loop:  17%|#6        | 8488/50000 [4:11:47<20:20:46,  1.76s/it]
Outer training loop:  17%|#6        | 8489/50000 [4:11:48<20:12:48,  1.75s/it]
Outer training loop:  17%|#6        | 8490/50000 [4:11:50<21:13:27,  1.84s/it]
Outer training loop:  17%|#6        | 8491/50000 [4:11:52<20:33:06,  1.78s/it]
Outer training loop:  17%|#6        | 8492/50000 [4:11:54<20:09:32,  1.75s/it]
Outer training loop:  17%|#6        | 8493/50000 [4:11:55<20:09:40,  1.75s/it]
Outer training loop:  17%|#6        | 8494/50000 [4:11:57<20:37:49,  1.79s/it]
Outer training loop:  17%|#6        | 8495/50000 [4:11:59<20:15:35,  1.76s/it]
Outer training loop:  17%|#6        | 8496/50000 [4:12:01<20:01:50,  1.74s/it]
Outer training loop:  17%|#6        | 8497/50000 [4:12:03<20:46:13,  1.80s/it]
Outer training loop:  17%|#6        | 8498/50000 [4:12:04<20:39:36,  1.79s/it]
Outer training loop:  17%|#6        | 8499/50000 [4:12:06<20:18:25,  1.76s/it]
Outer training loop:  17%|#7        | 8500/50000 [4:12:08<20:35:03,  1.79s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.63s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.70s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.74s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:19,  1.75s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:22,  1.80s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.79s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.78s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:12,  1.75s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:13,  1.81s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:10,  1.78s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:14,  1.86s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:08,  1.81s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.79s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:01,  1.76s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<01:58,  1.74s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:58,  1.77s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:56,  1.76s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:51,  1.72s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:50,  1.73s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:55,  1.83s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:49,  1.77s/it]

Evaluating policy:  27%|##6       | 22/83 [00:38<01:45,  1.73s/it]

Evaluating policy:  28%|##7       | 23/83 [00:40<01:45,  1.76s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:42,  1.74s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:45,  1.81s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.78s/it]

Evaluating policy:  33%|###2      | 27/83 [00:47<01:39,  1.77s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:36,  1.76s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:33,  1.73s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:35,  1.80s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:32,  1.77s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:29,  1.75s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:26,  1.74s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:23,  1.71s/it]

Evaluating policy:  42%|####2     | 35/83 [01:01<01:21,  1.70s/it]

Evaluating policy:  43%|####3     | 36/83 [01:03<01:27,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:23,  1.81s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:19,  1.76s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:17,  1.77s/it]

Evaluating policy:  48%|####8     | 40/83 [01:10<01:14,  1.73s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:12,  1.72s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:11,  1.75s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:13,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:10,  1.81s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:07,  1.78s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:04,  1.74s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:02,  1.73s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:24<01:00,  1.72s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<00:57,  1.69s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:59,  1.79s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:56,  1.77s/it]

Evaluating policy:  63%|######2   | 52/83 [01:31<00:54,  1.75s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:52,  1.74s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.73s/it]

Evaluating policy:  67%|######7   | 56/83 [01:38<00:46,  1.74s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:49,  1.91s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:46,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.79s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.78s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:47<00:38,  1.77s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:36,  1.75s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:34,  1.74s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:32,  1.72s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:30,  1.78s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:27,  1.74s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:26,  1.74s/it]

Evaluating policy:  83%|########3 | 69/83 [02:01<00:23,  1.71s/it]

Evaluating policy:  84%|########4 | 70/83 [02:03<00:22,  1.73s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:20,  1.71s/it]

Evaluating policy:  87%|########6 | 72/83 [02:06<00:18,  1.69s/it]

Evaluating policy:  88%|########7 | 73/83 [02:08<00:16,  1.69s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:14,  1.82s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.78s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.78s/it]

Evaluating policy:  94%|#########3| 78/83 [02:17<00:08,  1.75s/it]

Evaluating policy:  95%|#########5| 79/83 [02:19<00:07,  1.80s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.79s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.76s/it]

Evaluating policy:  99%|#########8| 82/83 [02:24<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 83/83 [02:26<00:00,  1.70s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:56,  2.15s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:41,  1.99s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:34,  1.94s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:34,  1.95s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.87s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:20,  1.83s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:21,  1.86s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:12,  1.78s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:08,  1.76s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:06,  1.76s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.82s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.79s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:01,  1.77s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:00,  1.77s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:03,  1.84s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.79s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:54,  1.77s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:51,  1.74s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:53,  1.79s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:50,  1.78s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:46,  1.75s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:44,  1.75s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:42,  1.74s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:46,  1.83s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.79s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:37,  1.74s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:35,  1.74s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:34,  1.74s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.76s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:35,  1.83s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:31,  1.79s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:28,  1.76s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:26,  1.76s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:23,  1.75s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:21,  1.74s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:20,  1.79s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.73s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:15,  1.75s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:12,  1.73s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:11,  1.75s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:10,  1.80s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:08,  1.80s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:05,  1.77s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:03,  1.77s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:01,  1.74s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<00:59,  1.75s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<01:00,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:57,  1.80s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:54,  1.77s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:53,  1.79s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:49,  1.76s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:47,  1.78s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:48,  1.88s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:46,  1.85s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:43,  1.81s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:41,  1.79s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:38,  1.75s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:36,  1.72s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:34,  1.73s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:33,  1.74s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:30,  1.80s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:28,  1.79s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:26,  1.77s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.77s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.75s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:20,  1.74s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:18,  1.71s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.71s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.84s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.81s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.78s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.74s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:06,  1.74s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.73s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.73s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.75s/it]

Outer training loop:  17%|#7        | 8501/50000 [4:17:09<1054:53:44, 91.51s/it]
Outer training loop:  17%|#7        | 8502/50000 [4:17:11<746:33:10, 64.76s/it] 
Outer training loop:  17%|#7        | 8503/50000 [4:17:13<528:25:46, 45.84s/it]
Outer training loop:  17%|#7        | 8504/50000 [4:17:15<375:59:29, 32.62s/it]
Outer training loop:  17%|#7        | 8505/50000 [4:17:16<269:12:24, 23.36s/it]
Outer training loop:  17%|#7        | 8506/50000 [4:17:18<195:13:31, 16.94s/it]
Outer training loop:  17%|#7        | 8507/50000 [4:17:20<142:47:45, 12.39s/it]
Outer training loop:  17%|#7        | 8508/50000 [4:17:22<105:53:06,  9.19s/it]
Outer training loop:  17%|#7        | 8509/50000 [4:17:24<80:30:19,  6.99s/it] 
Outer training loop:  17%|#7        | 8510/50000 [4:17:25<62:47:09,  5.45s/it]
Outer training loop:  17%|#7        | 8511/50000 [4:17:27<49:47:29,  4.32s/it]
Outer training loop:  17%|#7        | 8512/50000 [4:17:29<41:27:42,  3.60s/it]
Outer training loop:  17%|#7        | 8513/50000 [4:17:31<34:50:35,  3.02s/it]
Outer training loop:  17%|#7        | 8514/50000 [4:17:32<30:15:22,  2.63s/it]
Outer training loop:  17%|#7        | 8515/50000 [4:17:34<28:04:30,  2.44s/it]
Outer training loop:  17%|#7        | 8516/50000 [4:17:36<25:40:09,  2.23s/it]
Outer training loop:  17%|#7        | 8517/50000 [4:17:38<23:46:02,  2.06s/it]
Outer training loop:  17%|#7        | 8518/50000 [4:17:40<23:07:14,  2.01s/it]
Outer training loop:  17%|#7        | 8519/50000 [4:17:41<22:09:28,  1.92s/it]
Outer training loop:  17%|#7        | 8520/50000 [4:17:43<21:24:59,  1.86s/it]
Outer training loop:  17%|#7        | 8521/50000 [4:17:45<21:10:25,  1.84s/it]
Outer training loop:  17%|#7        | 8522/50000 [4:17:47<21:37:56,  1.88s/it]
Outer training loop:  17%|#7        | 8523/50000 [4:17:49<20:57:46,  1.82s/it]
Outer training loop:  17%|#7        | 8524/50000 [4:17:50<20:33:12,  1.78s/it]
Outer training loop:  17%|#7        | 8525/50000 [4:17:52<20:23:13,  1.77s/it]
Outer training loop:  17%|#7        | 8526/50000 [4:17:54<20:29:43,  1.78s/it]
Outer training loop:  17%|#7        | 8527/50000 [4:17:56<20:10:11,  1.75s/it]
Outer training loop:  17%|#7        | 8528/50000 [4:17:57<20:48:04,  1.81s/it]
Outer training loop:  17%|#7        | 8529/50000 [4:17:59<20:43:04,  1.80s/it]
Outer training loop:  17%|#7        | 8530/50000 [4:18:01<20:19:28,  1.76s/it]
Outer training loop:  17%|#7        | 8531/50000 [4:18:03<20:09:49,  1.75s/it]
Outer training loop:  17%|#7        | 8532/50000 [4:18:05<20:40:48,  1.80s/it]
Outer training loop:  17%|#7        | 8533/50000 [4:18:06<20:39:28,  1.79s/it]
Outer training loop:  17%|#7        | 8534/50000 [4:18:08<20:41:47,  1.80s/it]
Outer training loop:  17%|#7        | 8535/50000 [4:18:10<20:55:52,  1.82s/it]
Outer training loop:  17%|#7        | 8536/50000 [4:18:12<20:32:23,  1.78s/it]
Outer training loop:  17%|#7        | 8537/50000 [4:18:13<20:21:34,  1.77s/it]
Outer training loop:  17%|#7        | 8538/50000 [4:18:15<20:29:42,  1.78s/it]
Outer training loop:  17%|#7        | 8539/50000 [4:18:17<20:28:41,  1.78s/it]
Outer training loop:  17%|#7        | 8540/50000 [4:18:19<20:02:42,  1.74s/it]
Outer training loop:  17%|#7        | 8541/50000 [4:18:20<19:51:24,  1.72s/it]
Outer training loop:  17%|#7        | 8542/50000 [4:18:22<20:21:29,  1.77s/it]
Outer training loop:  17%|#7        | 8543/50000 [4:18:24<20:12:44,  1.76s/it]
Outer training loop:  17%|#7        | 8544/50000 [4:18:26<20:31:35,  1.78s/it]
Outer training loop:  17%|#7        | 8545/50000 [4:18:28<20:48:06,  1.81s/it]
Outer training loop:  17%|#7        | 8546/50000 [4:18:29<20:42:24,  1.80s/it]
Outer training loop:  17%|#7        | 8547/50000 [4:18:31<20:30:27,  1.78s/it]
Outer training loop:  17%|#7        | 8548/50000 [4:18:33<21:00:29,  1.82s/it]
Outer training loop:  17%|#7        | 8549/50000 [4:18:35<21:13:20,  1.84s/it]
Outer training loop:  17%|#7        | 8550/50000 [4:18:37<20:32:16,  1.78s/it]
Outer training loop:  17%|#7        | 8551/50000 [4:18:39<20:53:21,  1.81s/it]
Outer training loop:  17%|#7        | 8552/50000 [4:18:40<20:42:59,  1.80s/it]
Outer training loop:  17%|#7        | 8553/50000 [4:18:42<20:15:40,  1.76s/it]
Outer training loop:  17%|#7        | 8554/50000 [4:18:44<19:52:36,  1.73s/it]
Outer training loop:  17%|#7        | 8555/50000 [4:18:45<20:25:13,  1.77s/it]
Outer training loop:  17%|#7        | 8556/50000 [4:18:47<19:57:22,  1.73s/it]
Outer training loop:  17%|#7        | 8557/50000 [4:18:49<20:06:12,  1.75s/it]
Outer training loop:  17%|#7        | 8558/50000 [4:18:51<20:45:53,  1.80s/it]
Outer training loop:  17%|#7        | 8559/50000 [4:18:53<20:32:50,  1.78s/it]
Outer training loop:  17%|#7        | 8560/50000 [4:18:54<20:13:56,  1.76s/it]
Outer training loop:  17%|#7        | 8561/50000 [4:18:56<20:37:26,  1.79s/it]
Outer training loop:  17%|#7        | 8562/50000 [4:18:58<20:14:12,  1.76s/it]
Outer training loop:  17%|#7        | 8563/50000 [4:19:00<20:21:18,  1.77s/it]
Outer training loop:  17%|#7        | 8564/50000 [4:19:01<20:13:53,  1.76s/it]
Outer training loop:  17%|#7        | 8565/50000 [4:19:03<20:38:32,  1.79s/it]
Outer training loop:  17%|#7        | 8566/50000 [4:19:05<20:08:03,  1.75s/it]
Outer training loop:  17%|#7        | 8567/50000 [4:19:07<20:09:13,  1.75s/it]
Outer training loop:  17%|#7        | 8568/50000 [4:19:09<20:43:04,  1.80s/it]
Outer training loop:  17%|#7        | 8569/50000 [4:19:10<20:25:00,  1.77s/it]
Outer training loop:  17%|#7        | 8570/50000 [4:19:12<19:48:02,  1.72s/it]
Outer training loop:  17%|#7        | 8571/50000 [4:19:14<21:02:27,  1.83s/it]
Outer training loop:  17%|#7        | 8572/50000 [4:19:16<20:41:45,  1.80s/it]
Outer training loop:  17%|#7        | 8573/50000 [4:19:17<20:20:35,  1.77s/it]
Outer training loop:  17%|#7        | 8574/50000 [4:19:19<20:07:21,  1.75s/it]
Outer training loop:  17%|#7        | 8575/50000 [4:19:21<20:20:22,  1.77s/it]
Outer training loop:  17%|#7        | 8576/50000 [4:19:23<20:12:45,  1.76s/it]
Outer training loop:  17%|#7        | 8577/50000 [4:19:24<19:57:56,  1.74s/it]
Outer training loop:  17%|#7        | 8578/50000 [4:19:26<20:38:09,  1.79s/it]
Outer training loop:  17%|#7        | 8579/50000 [4:19:28<20:10:22,  1.75s/it]
Outer training loop:  17%|#7        | 8580/50000 [4:19:30<20:07:12,  1.75s/it]
Outer training loop:  17%|#7        | 8581/50000 [4:19:31<20:25:05,  1.77s/it]
Outer training loop:  17%|#7        | 8582/50000 [4:19:33<20:33:30,  1.79s/it]
Outer training loop:  17%|#7        | 8583/50000 [4:19:35<20:17:37,  1.76s/it]
Outer training loop:  17%|#7        | 8584/50000 [4:19:37<20:53:24,  1.82s/it]
Outer training loop:  17%|#7        | 8585/50000 [4:19:39<21:17:11,  1.85s/it]
Outer training loop:  17%|#7        | 8586/50000 [4:19:41<20:41:54,  1.80s/it]
Outer training loop:  17%|#7        | 8587/50000 [4:19:42<21:12:17,  1.84s/it]
Outer training loop:  17%|#7        | 8588/50000 [4:19:44<20:55:46,  1.82s/it]
Outer training loop:  17%|#7        | 8589/50000 [4:19:46<20:59:02,  1.82s/it]
Outer training loop:  17%|#7        | 8590/50000 [4:19:48<20:46:16,  1.81s/it]
Outer training loop:  17%|#7        | 8591/50000 [4:19:50<21:33:59,  1.87s/it]
Outer training loop:  17%|#7        | 8592/50000 [4:19:52<20:48:12,  1.81s/it]
Outer training loop:  17%|#7        | 8593/50000 [4:19:53<20:31:29,  1.78s/it]
Outer training loop:  17%|#7        | 8594/50000 [4:19:55<20:38:45,  1.80s/it]
Outer training loop:  17%|#7        | 8595/50000 [4:19:57<20:16:17,  1.76s/it]
Outer training loop:  17%|#7        | 8596/50000 [4:19:58<20:00:28,  1.74s/it]
Outer training loop:  17%|#7        | 8597/50000 [4:20:00<20:37:20,  1.79s/it]
Outer training loop:  17%|#7        | 8598/50000 [4:20:02<19:55:58,  1.73s/it]
Outer training loop:  17%|#7        | 8599/50000 [4:20:04<20:18:07,  1.77s/it]
Outer training loop:  17%|#7        | 8600/50000 [4:20:06<20:30:19,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.67s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  17%|#7        | 8601/50000 [4:20:13<39:10:27,  3.41s/it]
Outer training loop:  17%|#7        | 8602/50000 [4:20:15<33:36:35,  2.92s/it]
Outer training loop:  17%|#7        | 8603/50000 [4:20:16<29:42:11,  2.58s/it]
Outer training loop:  17%|#7        | 8604/50000 [4:20:18<27:41:28,  2.41s/it]
Outer training loop:  17%|#7        | 8605/50000 [4:20:20<25:15:11,  2.20s/it]
Outer training loop:  17%|#7        | 8606/50000 [4:20:22<23:44:12,  2.06s/it]
Outer training loop:  17%|#7        | 8607/50000 [4:20:24<22:34:18,  1.96s/it]
Outer training loop:  17%|#7        | 8608/50000 [4:20:25<22:11:26,  1.93s/it]
Outer training loop:  17%|#7        | 8609/50000 [4:20:27<21:33:57,  1.88s/it]
Outer training loop:  17%|#7        | 8610/50000 [4:20:29<21:08:55,  1.84s/it]
Outer training loop:  17%|#7        | 8611/50000 [4:20:31<21:17:34,  1.85s/it]
Outer training loop:  17%|#7        | 8612/50000 [4:20:33<20:52:53,  1.82s/it]
Outer training loop:  17%|#7        | 8613/50000 [4:20:34<20:43:10,  1.80s/it]
Outer training loop:  17%|#7        | 8614/50000 [4:20:36<21:12:37,  1.85s/it]
Outer training loop:  17%|#7        | 8615/50000 [4:20:38<20:45:23,  1.81s/it]
Outer training loop:  17%|#7        | 8616/50000 [4:20:40<20:05:34,  1.75s/it]
Outer training loop:  17%|#7        | 8617/50000 [4:20:41<19:47:33,  1.72s/it]
Outer training loop:  17%|#7        | 8618/50000 [4:20:43<20:55:53,  1.82s/it]
Outer training loop:  17%|#7        | 8619/50000 [4:20:45<20:14:12,  1.76s/it]
Outer training loop:  17%|#7        | 8620/50000 [4:20:47<19:49:04,  1.72s/it]
Outer training loop:  17%|#7        | 8621/50000 [4:20:48<20:19:19,  1.77s/it]
Outer training loop:  17%|#7        | 8622/50000 [4:20:50<20:21:22,  1.77s/it]
Outer training loop:  17%|#7        | 8623/50000 [4:20:52<20:04:33,  1.75s/it]
Outer training loop:  17%|#7        | 8624/50000 [4:20:54<20:19:54,  1.77s/it]
Outer training loop:  17%|#7        | 8625/50000 [4:20:56<20:29:50,  1.78s/it]
Outer training loop:  17%|#7        | 8626/50000 [4:20:57<20:33:50,  1.79s/it]
Outer training loop:  17%|#7        | 8627/50000 [4:20:59<20:12:30,  1.76s/it]
Outer training loop:  17%|#7        | 8628/50000 [4:21:01<20:30:31,  1.78s/it]
Outer training loop:  17%|#7        | 8629/50000 [4:21:03<20:09:06,  1.75s/it]
Outer training loop:  17%|#7        | 8630/50000 [4:21:04<19:46:36,  1.72s/it]
Outer training loop:  17%|#7        | 8631/50000 [4:21:06<20:14:47,  1.76s/it]
Outer training loop:  17%|#7        | 8632/50000 [4:21:08<20:25:16,  1.78s/it]
Outer training loop:  17%|#7        | 8633/50000 [4:21:10<20:24:41,  1.78s/it]
Outer training loop:  17%|#7        | 8634/50000 [4:21:11<20:29:25,  1.78s/it]
Outer training loop:  17%|#7        | 8635/50000 [4:21:13<20:20:40,  1.77s/it]
Outer training loop:  17%|#7        | 8636/50000 [4:21:15<20:12:45,  1.76s/it]
Outer training loop:  17%|#7        | 8637/50000 [4:21:17<20:24:08,  1.78s/it]
Outer training loop:  17%|#7        | 8638/50000 [4:21:18<20:01:36,  1.74s/it]
Outer training loop:  17%|#7        | 8639/50000 [4:21:20<20:13:42,  1.76s/it]
Outer training loop:  17%|#7        | 8640/50000 [4:21:22<19:52:39,  1.73s/it]
Outer training loop:  17%|#7        | 8641/50000 [4:21:24<20:37:41,  1.80s/it]
Outer training loop:  17%|#7        | 8642/50000 [4:21:26<20:20:19,  1.77s/it]
Outer training loop:  17%|#7        | 8643/50000 [4:21:27<20:02:35,  1.74s/it]
Outer training loop:  17%|#7        | 8644/50000 [4:21:29<20:33:11,  1.79s/it]
Outer training loop:  17%|#7        | 8645/50000 [4:21:31<20:05:36,  1.75s/it]
Outer training loop:  17%|#7        | 8646/50000 [4:21:33<20:11:33,  1.76s/it]
Outer training loop:  17%|#7        | 8647/50000 [4:21:34<20:50:52,  1.81s/it]
Outer training loop:  17%|#7        | 8648/50000 [4:21:36<20:41:39,  1.80s/it]
Outer training loop:  17%|#7        | 8649/50000 [4:21:38<20:29:40,  1.78s/it]
Outer training loop:  17%|#7        | 8650/50000 [4:21:40<21:09:59,  1.84s/it]
Outer training loop:  17%|#7        | 8651/50000 [4:21:42<20:41:32,  1.80s/it]
Outer training loop:  17%|#7        | 8652/50000 [4:21:43<20:22:56,  1.77s/it]
Outer training loop:  17%|#7        | 8653/50000 [4:21:45<20:45:09,  1.81s/it]
Outer training loop:  17%|#7        | 8654/50000 [4:21:47<20:39:13,  1.80s/it]
Outer training loop:  17%|#7        | 8655/50000 [4:21:49<20:16:54,  1.77s/it]
Outer training loop:  17%|#7        | 8656/50000 [4:21:50<19:51:38,  1.73s/it]
Outer training loop:  17%|#7        | 8657/50000 [4:21:52<20:12:40,  1.76s/it]
Outer training loop:  17%|#7        | 8658/50000 [4:21:54<19:59:18,  1.74s/it]
Outer training loop:  17%|#7        | 8659/50000 [4:21:56<20:02:39,  1.75s/it]
Outer training loop:  17%|#7        | 8660/50000 [4:21:58<20:27:04,  1.78s/it]
Outer training loop:  17%|#7        | 8661/50000 [4:21:59<20:11:27,  1.76s/it]
Outer training loop:  17%|#7        | 8662/50000 [4:22:01<20:11:38,  1.76s/it]
Outer training loop:  17%|#7        | 8663/50000 [4:22:03<19:52:08,  1.73s/it]
Outer training loop:  17%|#7        | 8664/50000 [4:22:05<20:51:49,  1.82s/it]
Outer training loop:  17%|#7        | 8665/50000 [4:22:06<20:28:52,  1.78s/it]
Outer training loop:  17%|#7        | 8666/50000 [4:22:08<20:03:29,  1.75s/it]
Outer training loop:  17%|#7        | 8667/50000 [4:22:10<20:42:27,  1.80s/it]
Outer training loop:  17%|#7        | 8668/50000 [4:22:12<20:27:27,  1.78s/it]
Outer training loop:  17%|#7        | 8669/50000 [4:22:13<20:14:02,  1.76s/it]
Outer training loop:  17%|#7        | 8670/50000 [4:22:15<19:34:50,  1.71s/it]
Outer training loop:  17%|#7        | 8671/50000 [4:22:17<20:15:41,  1.76s/it]
Outer training loop:  17%|#7        | 8672/50000 [4:22:19<20:08:42,  1.75s/it]
Outer training loop:  17%|#7        | 8673/50000 [4:22:20<20:10:56,  1.76s/it]
Outer training loop:  17%|#7        | 8674/50000 [4:22:22<20:43:30,  1.81s/it]
Outer training loop:  17%|#7        | 8675/50000 [4:22:24<20:22:59,  1.78s/it]
Outer training loop:  17%|#7        | 8676/50000 [4:22:26<20:26:55,  1.78s/it]
Outer training loop:  17%|#7        | 8677/50000 [4:22:28<20:52:54,  1.82s/it]
Outer training loop:  17%|#7        | 8678/50000 [4:22:29<20:38:39,  1.80s/it]
Outer training loop:  17%|#7        | 8679/50000 [4:22:31<20:29:47,  1.79s/it]
Outer training loop:  17%|#7        | 8680/50000 [4:22:33<20:54:13,  1.82s/it]
Outer training loop:  17%|#7        | 8681/50000 [4:22:35<21:02:32,  1.83s/it]
Outer training loop:  17%|#7        | 8682/50000 [4:22:37<20:14:50,  1.76s/it]
Outer training loop:  17%|#7        | 8683/50000 [4:22:38<20:04:51,  1.75s/it]
Outer training loop:  17%|#7        | 8684/50000 [4:22:40<20:41:16,  1.80s/it]
Outer training loop:  17%|#7        | 8685/50000 [4:22:42<20:13:21,  1.76s/it]
Outer training loop:  17%|#7        | 8686/50000 [4:22:44<20:16:50,  1.77s/it]
Outer training loop:  17%|#7        | 8687/50000 [4:22:46<20:47:43,  1.81s/it]
Outer training loop:  17%|#7        | 8688/50000 [4:22:47<20:41:00,  1.80s/it]
Outer training loop:  17%|#7        | 8689/50000 [4:22:49<20:50:30,  1.82s/it]
Outer training loop:  17%|#7        | 8690/50000 [4:22:51<21:06:48,  1.84s/it]
Outer training loop:  17%|#7        | 8691/50000 [4:22:53<20:11:02,  1.76s/it]
Outer training loop:  17%|#7        | 8692/50000 [4:22:54<19:50:51,  1.73s/it]
Outer training loop:  17%|#7        | 8693/50000 [4:22:56<19:40:40,  1.71s/it]
Outer training loop:  17%|#7        | 8694/50000 [4:22:58<20:23:18,  1.78s/it]
Outer training loop:  17%|#7        | 8695/50000 [4:23:00<20:28:44,  1.78s/it]
Outer training loop:  17%|#7        | 8696/50000 [4:23:01<20:05:55,  1.75s/it]
Outer training loop:  17%|#7        | 8697/50000 [4:23:03<20:28:48,  1.79s/it]
Outer training loop:  17%|#7        | 8698/50000 [4:23:05<20:15:59,  1.77s/it]
Outer training loop:  17%|#7        | 8699/50000 [4:23:07<19:49:28,  1.73s/it]
Outer training loop:  17%|#7        | 8700/50000 [4:23:08<19:30:16,  1.70s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.63s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.69s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.66s/it]

Outer training loop:  17%|#7        | 8701/50000 [4:23:15<37:39:12,  3.28s/it]
Outer training loop:  17%|#7        | 8702/50000 [4:23:17<32:36:19,  2.84s/it]
Outer training loop:  17%|#7        | 8703/50000 [4:23:19<28:45:06,  2.51s/it]
Outer training loop:  17%|#7        | 8704/50000 [4:23:20<25:47:16,  2.25s/it]
Outer training loop:  17%|#7        | 8705/50000 [4:23:22<24:19:30,  2.12s/it]
Outer training loop:  17%|#7        | 8706/50000 [4:23:24<23:03:54,  2.01s/it]
Outer training loop:  17%|#7        | 8707/50000 [4:23:26<21:54:35,  1.91s/it]
Outer training loop:  17%|#7        | 8708/50000 [4:23:28<21:45:01,  1.90s/it]
Outer training loop:  17%|#7        | 8709/50000 [4:23:29<21:02:56,  1.84s/it]
Outer training loop:  17%|#7        | 8710/50000 [4:23:31<20:47:09,  1.81s/it]
Outer training loop:  17%|#7        | 8711/50000 [4:23:33<20:23:33,  1.78s/it]
Outer training loop:  17%|#7        | 8712/50000 [4:23:35<21:04:03,  1.84s/it]
Outer training loop:  17%|#7        | 8713/50000 [4:23:36<20:38:48,  1.80s/it]
Outer training loop:  17%|#7        | 8714/50000 [4:23:38<20:22:39,  1.78s/it]
Outer training loop:  17%|#7        | 8715/50000 [4:23:40<21:19:49,  1.86s/it]
Outer training loop:  17%|#7        | 8716/50000 [4:23:42<20:32:19,  1.79s/it]
Outer training loop:  17%|#7        | 8717/50000 [4:23:44<20:15:32,  1.77s/it]
Outer training loop:  17%|#7        | 8718/50000 [4:23:45<20:09:26,  1.76s/it]
Outer training loop:  17%|#7        | 8719/50000 [4:23:47<20:24:21,  1.78s/it]
Outer training loop:  17%|#7        | 8720/50000 [4:23:49<20:16:50,  1.77s/it]
Outer training loop:  17%|#7        | 8721/50000 [4:23:51<20:14:22,  1.77s/it]
Outer training loop:  17%|#7        | 8722/50000 [4:23:53<20:47:23,  1.81s/it]
Outer training loop:  17%|#7        | 8723/50000 [4:23:54<20:26:17,  1.78s/it]
Outer training loop:  17%|#7        | 8724/50000 [4:23:56<20:32:46,  1.79s/it]
Outer training loop:  17%|#7        | 8725/50000 [4:23:58<20:33:52,  1.79s/it]
Outer training loop:  17%|#7        | 8726/50000 [4:24:00<20:10:56,  1.76s/it]
Outer training loop:  17%|#7        | 8727/50000 [4:24:01<19:53:05,  1.73s/it]
Outer training loop:  17%|#7        | 8728/50000 [4:24:03<20:52:26,  1.82s/it]
Outer training loop:  17%|#7        | 8729/50000 [4:24:05<20:53:03,  1.82s/it]
Outer training loop:  17%|#7        | 8730/50000 [4:24:07<20:49:21,  1.82s/it]
Outer training loop:  17%|#7        | 8731/50000 [4:24:09<20:50:04,  1.82s/it]
Outer training loop:  17%|#7        | 8732/50000 [4:24:10<20:30:41,  1.79s/it]
Outer training loop:  17%|#7        | 8733/50000 [4:24:12<19:59:03,  1.74s/it]
Outer training loop:  17%|#7        | 8734/50000 [4:24:14<19:48:15,  1.73s/it]
Outer training loop:  17%|#7        | 8735/50000 [4:24:16<20:28:31,  1.79s/it]
Outer training loop:  17%|#7        | 8736/50000 [4:24:17<20:01:42,  1.75s/it]
Outer training loop:  17%|#7        | 8737/50000 [4:24:19<19:56:47,  1.74s/it]
Outer training loop:  17%|#7        | 8738/50000 [4:24:21<20:22:50,  1.78s/it]
Outer training loop:  17%|#7        | 8739/50000 [4:24:23<20:02:46,  1.75s/it]
Outer training loop:  17%|#7        | 8740/50000 [4:24:24<20:06:51,  1.75s/it]
Outer training loop:  17%|#7        | 8741/50000 [4:24:26<20:11:32,  1.76s/it]
Outer training loop:  17%|#7        | 8742/50000 [4:24:28<20:31:30,  1.79s/it]
Outer training loop:  17%|#7        | 8743/50000 [4:24:30<20:10:41,  1.76s/it]
Outer training loop:  17%|#7        | 8744/50000 [4:24:31<19:51:36,  1.73s/it]
Outer training loop:  17%|#7        | 8745/50000 [4:24:33<20:30:29,  1.79s/it]
Outer training loop:  17%|#7        | 8746/50000 [4:24:35<20:18:43,  1.77s/it]
Outer training loop:  17%|#7        | 8747/50000 [4:24:37<20:16:10,  1.77s/it]
Outer training loop:  17%|#7        | 8748/50000 [4:24:38<19:53:21,  1.74s/it]
Outer training loop:  17%|#7        | 8749/50000 [4:24:40<20:15:57,  1.77s/it]
Outer training loop:  18%|#7        | 8750/50000 [4:24:42<20:14:40,  1.77s/it]
Outer training loop:  18%|#7        | 8751/50000 [4:24:44<20:15:07,  1.77s/it]
Outer training loop:  18%|#7        | 8752/50000 [4:24:46<20:29:35,  1.79s/it]
Outer training loop:  18%|#7        | 8753/50000 [4:24:47<20:21:53,  1.78s/it]
Outer training loop:  18%|#7        | 8754/50000 [4:24:49<20:21:23,  1.78s/it]
Outer training loop:  18%|#7        | 8755/50000 [4:24:51<20:58:21,  1.83s/it]
Outer training loop:  18%|#7        | 8756/50000 [4:24:53<20:33:40,  1.79s/it]
Outer training loop:  18%|#7        | 8757/50000 [4:24:54<20:00:32,  1.75s/it]
Outer training loop:  18%|#7        | 8758/50000 [4:24:56<20:33:45,  1.79s/it]
Outer training loop:  18%|#7        | 8759/50000 [4:24:58<20:37:24,  1.80s/it]
Outer training loop:  18%|#7        | 8760/50000 [4:25:00<20:33:12,  1.79s/it]
Outer training loop:  18%|#7        | 8761/50000 [4:25:02<20:05:59,  1.75s/it]
Outer training loop:  18%|#7        | 8762/50000 [4:25:03<20:21:22,  1.78s/it]
Outer training loop:  18%|#7        | 8763/50000 [4:25:05<20:24:38,  1.78s/it]
Outer training loop:  18%|#7        | 8764/50000 [4:25:07<20:05:04,  1.75s/it]
Outer training loop:  18%|#7        | 8765/50000 [4:25:09<21:13:02,  1.85s/it]
Outer training loop:  18%|#7        | 8766/50000 [4:25:11<20:47:18,  1.81s/it]
Outer training loop:  18%|#7        | 8767/50000 [4:25:12<20:24:37,  1.78s/it]
Outer training loop:  18%|#7        | 8768/50000 [4:25:14<20:41:00,  1.81s/it]
Outer training loop:  18%|#7        | 8769/50000 [4:25:16<20:33:13,  1.79s/it]
Outer training loop:  18%|#7        | 8770/50000 [4:25:18<20:20:40,  1.78s/it]
Outer training loop:  18%|#7        | 8771/50000 [4:25:20<21:09:24,  1.85s/it]
Outer training loop:  18%|#7        | 8772/50000 [4:25:22<20:54:11,  1.83s/it]
Outer training loop:  18%|#7        | 8773/50000 [4:25:23<20:47:24,  1.82s/it]
Outer training loop:  18%|#7        | 8774/50000 [4:25:25<21:31:31,  1.88s/it]
Outer training loop:  18%|#7        | 8775/50000 [4:25:27<21:06:37,  1.84s/it]
Outer training loop:  18%|#7        | 8776/50000 [4:25:29<20:45:31,  1.81s/it]
Outer training loop:  18%|#7        | 8777/50000 [4:25:31<21:06:35,  1.84s/it]
Outer training loop:  18%|#7        | 8778/50000 [4:25:33<20:31:52,  1.79s/it]
Outer training loop:  18%|#7        | 8779/50000 [4:25:34<20:12:15,  1.76s/it]
Outer training loop:  18%|#7        | 8780/50000 [4:25:36<19:48:38,  1.73s/it]
Outer training loop:  18%|#7        | 8781/50000 [4:25:38<19:53:38,  1.74s/it]
Outer training loop:  18%|#7        | 8782/50000 [4:25:39<20:07:20,  1.76s/it]
Outer training loop:  18%|#7        | 8783/50000 [4:25:41<19:57:00,  1.74s/it]
Outer training loop:  18%|#7        | 8784/50000 [4:25:43<20:24:14,  1.78s/it]
Outer training loop:  18%|#7        | 8785/50000 [4:25:45<20:01:18,  1.75s/it]
Outer training loop:  18%|#7        | 8786/50000 [4:25:46<19:59:27,  1.75s/it]
Outer training loop:  18%|#7        | 8787/50000 [4:25:48<19:44:39,  1.72s/it]
Outer training loop:  18%|#7        | 8788/50000 [4:25:50<20:18:32,  1.77s/it]
Outer training loop:  18%|#7        | 8789/50000 [4:25:52<19:54:28,  1.74s/it]
Outer training loop:  18%|#7        | 8790/50000 [4:25:53<20:05:03,  1.75s/it]
Outer training loop:  18%|#7        | 8791/50000 [4:25:55<20:43:30,  1.81s/it]
Outer training loop:  18%|#7        | 8792/50000 [4:25:57<20:43:37,  1.81s/it]
Outer training loop:  18%|#7        | 8793/50000 [4:25:59<20:17:56,  1.77s/it]
Outer training loop:  18%|#7        | 8794/50000 [4:26:01<20:48:08,  1.82s/it]
Outer training loop:  18%|#7        | 8795/50000 [4:26:02<20:21:10,  1.78s/it]
Outer training loop:  18%|#7        | 8796/50000 [4:26:04<19:47:28,  1.73s/it]
Outer training loop:  18%|#7        | 8797/50000 [4:26:06<20:13:18,  1.77s/it]
Outer training loop:  18%|#7        | 8798/50000 [4:26:08<20:07:25,  1.76s/it]
Outer training loop:  18%|#7        | 8799/50000 [4:26:09<19:44:50,  1.73s/it]
Outer training loop:  18%|#7        | 8800/50000 [4:26:11<20:14:21,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.69s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  18%|#7        | 8801/50000 [4:26:18<37:30:31,  3.28s/it]
Outer training loop:  18%|#7        | 8802/50000 [4:26:20<32:06:56,  2.81s/it]
Outer training loop:  18%|#7        | 8803/50000 [4:26:21<28:20:38,  2.48s/it]
Outer training loop:  18%|#7        | 8804/50000 [4:26:23<25:38:36,  2.24s/it]
Outer training loop:  18%|#7        | 8805/50000 [4:26:25<24:09:37,  2.11s/it]
Outer training loop:  18%|#7        | 8806/50000 [4:26:27<22:39:25,  1.98s/it]
Outer training loop:  18%|#7        | 8807/50000 [4:26:28<21:36:24,  1.89s/it]
Outer training loop:  18%|#7        | 8808/50000 [4:26:30<21:34:19,  1.89s/it]
Outer training loop:  18%|#7        | 8809/50000 [4:26:32<20:50:17,  1.82s/it]
Outer training loop:  18%|#7        | 8810/50000 [4:26:34<20:50:45,  1.82s/it]
Outer training loop:  18%|#7        | 8811/50000 [4:26:36<21:07:03,  1.85s/it]
Outer training loop:  18%|#7        | 8812/50000 [4:26:37<21:21:17,  1.87s/it]
Outer training loop:  18%|#7        | 8813/50000 [4:26:39<20:31:53,  1.79s/it]
Outer training loop:  18%|#7        | 8814/50000 [4:26:41<20:20:36,  1.78s/it]
Outer training loop:  18%|#7        | 8815/50000 [4:26:43<20:51:01,  1.82s/it]
Outer training loop:  18%|#7        | 8816/50000 [4:26:45<20:44:45,  1.81s/it]
Outer training loop:  18%|#7        | 8817/50000 [4:26:46<20:23:01,  1.78s/it]
Outer training loop:  18%|#7        | 8818/50000 [4:26:48<20:50:18,  1.82s/it]
Outer training loop:  18%|#7        | 8819/50000 [4:26:50<20:47:55,  1.82s/it]
Outer training loop:  18%|#7        | 8820/50000 [4:26:52<20:27:49,  1.79s/it]
Outer training loop:  18%|#7        | 8821/50000 [4:26:53<20:10:56,  1.76s/it]
Outer training loop:  18%|#7        | 8822/50000 [4:26:55<21:00:17,  1.84s/it]
Outer training loop:  18%|#7        | 8823/50000 [4:26:57<20:48:02,  1.82s/it]
Outer training loop:  18%|#7        | 8824/50000 [4:26:59<20:32:23,  1.80s/it]
Outer training loop:  18%|#7        | 8825/50000 [4:27:01<20:43:59,  1.81s/it]
Outer training loop:  18%|#7        | 8826/50000 [4:27:03<20:36:29,  1.80s/it]
Outer training loop:  18%|#7        | 8827/50000 [4:27:04<20:06:57,  1.76s/it]
Outer training loop:  18%|#7        | 8828/50000 [4:27:06<20:37:25,  1.80s/it]
Outer training loop:  18%|#7        | 8829/50000 [4:27:08<20:38:20,  1.80s/it]
Outer training loop:  18%|#7        | 8830/50000 [4:27:10<20:07:37,  1.76s/it]
Outer training loop:  18%|#7        | 8831/50000 [4:27:12<20:46:05,  1.82s/it]
Outer training loop:  18%|#7        | 8832/50000 [4:27:13<20:33:31,  1.80s/it]
Outer training loop:  18%|#7        | 8833/50000 [4:27:15<20:13:49,  1.77s/it]
Outer training loop:  18%|#7        | 8834/50000 [4:27:17<20:09:42,  1.76s/it]
Outer training loop:  18%|#7        | 8835/50000 [4:27:19<20:53:16,  1.83s/it]
Outer training loop:  18%|#7        | 8836/50000 [4:27:20<20:35:04,  1.80s/it]
Outer training loop:  18%|#7        | 8837/50000 [4:27:22<19:49:05,  1.73s/it]
Outer training loop:  18%|#7        | 8838/50000 [4:27:24<20:37:43,  1.80s/it]
Outer training loop:  18%|#7        | 8839/50000 [4:27:26<20:09:31,  1.76s/it]
Outer training loop:  18%|#7        | 8840/50000 [4:27:27<19:56:33,  1.74s/it]
Outer training loop:  18%|#7        | 8841/50000 [4:27:29<20:18:22,  1.78s/it]
Outer training loop:  18%|#7        | 8842/50000 [4:27:31<20:23:11,  1.78s/it]
Outer training loop:  18%|#7        | 8843/50000 [4:27:33<20:15:11,  1.77s/it]
Outer training loop:  18%|#7        | 8844/50000 [4:27:34<20:03:59,  1.76s/it]
Outer training loop:  18%|#7        | 8845/50000 [4:27:36<20:28:41,  1.79s/it]
Outer training loop:  18%|#7        | 8846/50000 [4:27:38<20:05:20,  1.76s/it]
Outer training loop:  18%|#7        | 8847/50000 [4:27:40<19:53:07,  1.74s/it]
Outer training loop:  18%|#7        | 8848/50000 [4:27:41<19:58:08,  1.75s/it]
Outer training loop:  18%|#7        | 8849/50000 [4:27:43<19:45:58,  1.73s/it]
Outer training loop:  18%|#7        | 8850/50000 [4:27:45<19:32:40,  1.71s/it]
Outer training loop:  18%|#7        | 8851/50000 [4:27:47<19:29:07,  1.70s/it]
Outer training loop:  18%|#7        | 8852/50000 [4:27:49<20:28:20,  1.79s/it]
Outer training loop:  18%|#7        | 8853/50000 [4:27:50<20:09:29,  1.76s/it]
Outer training loop:  18%|#7        | 8854/50000 [4:27:52<20:11:10,  1.77s/it]
Outer training loop:  18%|#7        | 8855/50000 [4:27:54<20:31:59,  1.80s/it]
Outer training loop:  18%|#7        | 8856/50000 [4:27:56<20:41:16,  1.81s/it]
Outer training loop:  18%|#7        | 8857/50000 [4:27:57<20:13:21,  1.77s/it]
Outer training loop:  18%|#7        | 8858/50000 [4:27:59<20:26:05,  1.79s/it]
Outer training loop:  18%|#7        | 8859/50000 [4:28:01<20:04:16,  1.76s/it]
Outer training loop:  18%|#7        | 8860/50000 [4:28:03<20:11:18,  1.77s/it]
Outer training loop:  18%|#7        | 8861/50000 [4:28:05<20:48:59,  1.82s/it]
Outer training loop:  18%|#7        | 8862/50000 [4:28:06<20:58:16,  1.84s/it]
Outer training loop:  18%|#7        | 8863/50000 [4:28:08<20:23:08,  1.78s/it]
Outer training loop:  18%|#7        | 8864/50000 [4:28:10<20:39:53,  1.81s/it]
Outer training loop:  18%|#7        | 8865/50000 [4:28:12<20:07:38,  1.76s/it]
Outer training loop:  18%|#7        | 8866/50000 [4:28:14<20:23:06,  1.78s/it]
Outer training loop:  18%|#7        | 8867/50000 [4:28:15<21:04:38,  1.84s/it]
Outer training loop:  18%|#7        | 8868/50000 [4:28:17<20:28:50,  1.79s/it]
Outer training loop:  18%|#7        | 8869/50000 [4:28:19<20:12:33,  1.77s/it]
Outer training loop:  18%|#7        | 8870/50000 [4:28:21<19:58:33,  1.75s/it]
Outer training loop:  18%|#7        | 8871/50000 [4:28:22<20:23:49,  1.79s/it]
Outer training loop:  18%|#7        | 8872/50000 [4:28:24<20:12:57,  1.77s/it]
Outer training loop:  18%|#7        | 8873/50000 [4:28:26<19:42:30,  1.73s/it]
Outer training loop:  18%|#7        | 8874/50000 [4:28:28<20:08:05,  1.76s/it]
Outer training loop:  18%|#7        | 8875/50000 [4:28:29<20:07:03,  1.76s/it]
Outer training loop:  18%|#7        | 8876/50000 [4:28:31<19:56:43,  1.75s/it]
Outer training loop:  18%|#7        | 8877/50000 [4:28:33<19:42:14,  1.72s/it]
Outer training loop:  18%|#7        | 8878/50000 [4:28:35<20:18:17,  1.78s/it]
Outer training loop:  18%|#7        | 8879/50000 [4:28:36<20:06:57,  1.76s/it]
Outer training loop:  18%|#7        | 8880/50000 [4:28:38<19:49:07,  1.74s/it]
Outer training loop:  18%|#7        | 8881/50000 [4:28:40<20:25:32,  1.79s/it]
Outer training loop:  18%|#7        | 8882/50000 [4:28:42<20:20:47,  1.78s/it]
Outer training loop:  18%|#7        | 8883/50000 [4:28:44<20:13:31,  1.77s/it]
Outer training loop:  18%|#7        | 8884/50000 [4:28:45<20:41:14,  1.81s/it]
Outer training loop:  18%|#7        | 8885/50000 [4:28:47<20:24:41,  1.79s/it]
Outer training loop:  18%|#7        | 8886/50000 [4:28:49<19:55:00,  1.74s/it]
Outer training loop:  18%|#7        | 8887/50000 [4:28:50<19:39:54,  1.72s/it]
Outer training loop:  18%|#7        | 8888/50000 [4:28:52<20:27:08,  1.79s/it]
Outer training loop:  18%|#7        | 8889/50000 [4:28:54<20:33:24,  1.80s/it]
Outer training loop:  18%|#7        | 8890/50000 [4:28:56<20:07:19,  1.76s/it]
Outer training loop:  18%|#7        | 8891/50000 [4:28:58<19:52:00,  1.74s/it]
Outer training loop:  18%|#7        | 8892/50000 [4:29:00<20:33:34,  1.80s/it]
Outer training loop:  18%|#7        | 8893/50000 [4:29:01<20:18:49,  1.78s/it]
Outer training loop:  18%|#7        | 8894/50000 [4:29:03<20:12:56,  1.77s/it]
Outer training loop:  18%|#7        | 8895/50000 [4:29:05<20:30:55,  1.80s/it]
Outer training loop:  18%|#7        | 8896/50000 [4:29:07<20:05:20,  1.76s/it]
Outer training loop:  18%|#7        | 8897/50000 [4:29:08<19:43:49,  1.73s/it]
Outer training loop:  18%|#7        | 8898/50000 [4:29:10<19:33:04,  1.71s/it]
Outer training loop:  18%|#7        | 8899/50000 [4:29:12<20:19:29,  1.78s/it]
Outer training loop:  18%|#7        | 8900/50000 [4:29:14<20:08:52,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  18%|#7        | 8901/50000 [4:29:21<38:07:44,  3.34s/it]
Outer training loop:  18%|#7        | 8902/50000 [4:29:22<32:37:41,  2.86s/it]
Outer training loop:  18%|#7        | 8903/50000 [4:29:24<29:15:15,  2.56s/it]
Outer training loop:  18%|#7        | 8904/50000 [4:29:26<26:31:36,  2.32s/it]
Outer training loop:  18%|#7        | 8905/50000 [4:29:28<24:28:17,  2.14s/it]
Outer training loop:  18%|#7        | 8906/50000 [4:29:29<22:49:48,  2.00s/it]
Outer training loop:  18%|#7        | 8907/50000 [4:29:31<22:22:33,  1.96s/it]
Outer training loop:  18%|#7        | 8908/50000 [4:29:33<22:03:42,  1.93s/it]
Outer training loop:  18%|#7        | 8909/50000 [4:29:35<22:46:03,  1.99s/it]
Outer training loop:  18%|#7        | 8910/50000 [4:29:37<22:20:34,  1.96s/it]
Outer training loop:  18%|#7        | 8911/50000 [4:29:39<21:37:56,  1.90s/it]
Outer training loop:  18%|#7        | 8912/50000 [4:29:41<21:17:27,  1.87s/it]
Outer training loop:  18%|#7        | 8913/50000 [4:29:43<21:34:23,  1.89s/it]
Outer training loop:  18%|#7        | 8914/50000 [4:29:44<21:18:55,  1.87s/it]
Outer training loop:  18%|#7        | 8915/50000 [4:29:46<20:45:12,  1.82s/it]
Outer training loop:  18%|#7        | 8916/50000 [4:29:48<21:16:01,  1.86s/it]
Outer training loop:  18%|#7        | 8917/50000 [4:29:50<20:52:20,  1.83s/it]
Outer training loop:  18%|#7        | 8918/50000 [4:29:52<20:32:02,  1.80s/it]
Outer training loop:  18%|#7        | 8919/50000 [4:29:53<20:39:38,  1.81s/it]
Outer training loop:  18%|#7        | 8920/50000 [4:29:55<20:03:46,  1.76s/it]
Outer training loop:  18%|#7        | 8921/50000 [4:29:57<19:43:48,  1.73s/it]
Outer training loop:  18%|#7        | 8922/50000 [4:29:59<20:06:34,  1.76s/it]
Outer training loop:  18%|#7        | 8923/50000 [4:30:00<20:04:04,  1.76s/it]
Outer training loop:  18%|#7        | 8924/50000 [4:30:02<20:11:25,  1.77s/it]
Outer training loop:  18%|#7        | 8925/50000 [4:30:04<20:02:20,  1.76s/it]
Outer training loop:  18%|#7        | 8926/50000 [4:30:06<20:37:58,  1.81s/it]
Outer training loop:  18%|#7        | 8927/50000 [4:30:07<20:10:33,  1.77s/it]
Outer training loop:  18%|#7        | 8928/50000 [4:30:09<19:46:23,  1.73s/it]
Outer training loop:  18%|#7        | 8929/50000 [4:30:11<20:21:40,  1.78s/it]
Outer training loop:  18%|#7        | 8930/50000 [4:30:13<20:25:47,  1.79s/it]
Outer training loop:  18%|#7        | 8931/50000 [4:30:15<20:33:44,  1.80s/it]
Outer training loop:  18%|#7        | 8932/50000 [4:30:16<20:52:43,  1.83s/it]
Outer training loop:  18%|#7        | 8933/50000 [4:30:18<20:25:55,  1.79s/it]
Outer training loop:  18%|#7        | 8934/50000 [4:30:20<20:23:10,  1.79s/it]
Outer training loop:  18%|#7        | 8935/50000 [4:30:22<20:37:31,  1.81s/it]
Outer training loop:  18%|#7        | 8936/50000 [4:30:23<20:00:42,  1.75s/it]
Outer training loop:  18%|#7        | 8937/50000 [4:30:25<19:57:56,  1.75s/it]
Outer training loop:  18%|#7        | 8938/50000 [4:30:27<19:35:58,  1.72s/it]
Outer training loop:  18%|#7        | 8939/50000 [4:30:29<20:46:50,  1.82s/it]
Outer training loop:  18%|#7        | 8940/50000 [4:30:31<20:31:06,  1.80s/it]
Outer training loop:  18%|#7        | 8941/50000 [4:30:32<20:23:28,  1.79s/it]
Outer training loop:  18%|#7        | 8942/50000 [4:30:34<20:37:52,  1.81s/it]
Outer training loop:  18%|#7        | 8943/50000 [4:30:36<20:18:05,  1.78s/it]
Outer training loop:  18%|#7        | 8944/50000 [4:30:38<20:30:11,  1.80s/it]
Outer training loop:  18%|#7        | 8945/50000 [4:30:40<20:54:05,  1.83s/it]
Outer training loop:  18%|#7        | 8946/50000 [4:30:41<20:32:25,  1.80s/it]
Outer training loop:  18%|#7        | 8947/50000 [4:30:43<20:03:44,  1.76s/it]
Outer training loop:  18%|#7        | 8948/50000 [4:30:45<19:40:58,  1.73s/it]
Outer training loop:  18%|#7        | 8949/50000 [4:30:47<20:19:48,  1.78s/it]
Outer training loop:  18%|#7        | 8950/50000 [4:30:48<20:00:50,  1.76s/it]
Outer training loop:  18%|#7        | 8951/50000 [4:30:50<19:57:20,  1.75s/it]
Outer training loop:  18%|#7        | 8952/50000 [4:30:52<20:29:54,  1.80s/it]
Outer training loop:  18%|#7        | 8953/50000 [4:30:54<20:10:43,  1.77s/it]
Outer training loop:  18%|#7        | 8954/50000 [4:30:55<20:04:32,  1.76s/it]
Outer training loop:  18%|#7        | 8955/50000 [4:30:57<20:05:02,  1.76s/it]
Outer training loop:  18%|#7        | 8956/50000 [4:30:59<19:46:43,  1.73s/it]
Outer training loop:  18%|#7        | 8957/50000 [4:31:01<19:52:07,  1.74s/it]
Outer training loop:  18%|#7        | 8958/50000 [4:31:03<21:04:26,  1.85s/it]
Outer training loop:  18%|#7        | 8959/50000 [4:31:05<21:28:08,  1.88s/it]
Outer training loop:  18%|#7        | 8960/50000 [4:31:06<21:04:36,  1.85s/it]
Outer training loop:  18%|#7        | 8961/50000 [4:31:08<21:15:29,  1.86s/it]
Outer training loop:  18%|#7        | 8962/50000 [4:31:10<20:45:41,  1.82s/it]
Outer training loop:  18%|#7        | 8963/50000 [4:31:12<20:26:13,  1.79s/it]
Outer training loop:  18%|#7        | 8964/50000 [4:31:14<20:48:03,  1.82s/it]
Outer training loop:  18%|#7        | 8965/50000 [4:31:15<20:29:03,  1.80s/it]
Outer training loop:  18%|#7        | 8966/50000 [4:31:17<20:14:49,  1.78s/it]
Outer training loop:  18%|#7        | 8967/50000 [4:31:19<19:59:05,  1.75s/it]
Outer training loop:  18%|#7        | 8968/50000 [4:31:21<20:06:36,  1.76s/it]
Outer training loop:  18%|#7        | 8969/50000 [4:31:22<20:02:14,  1.76s/it]
Outer training loop:  18%|#7        | 8970/50000 [4:31:24<19:53:02,  1.74s/it]
Outer training loop:  18%|#7        | 8971/50000 [4:31:26<20:12:37,  1.77s/it]
Outer training loop:  18%|#7        | 8972/50000 [4:31:28<19:53:52,  1.75s/it]
Outer training loop:  18%|#7        | 8973/50000 [4:31:29<19:51:03,  1.74s/it]
Outer training loop:  18%|#7        | 8974/50000 [4:31:31<20:24:56,  1.79s/it]
Outer training loop:  18%|#7        | 8975/50000 [4:31:33<20:34:45,  1.81s/it]
Outer training loop:  18%|#7        | 8976/50000 [4:31:35<20:12:25,  1.77s/it]
Outer training loop:  18%|#7        | 8977/50000 [4:31:36<19:44:45,  1.73s/it]
Outer training loop:  18%|#7        | 8978/50000 [4:31:38<20:07:53,  1.77s/it]
Outer training loop:  18%|#7        | 8979/50000 [4:31:40<19:53:39,  1.75s/it]
Outer training loop:  18%|#7        | 8980/50000 [4:31:42<19:53:44,  1.75s/it]
Outer training loop:  18%|#7        | 8981/50000 [4:31:44<20:18:30,  1.78s/it]
Outer training loop:  18%|#7        | 8982/50000 [4:31:45<20:14:02,  1.78s/it]
Outer training loop:  18%|#7        | 8983/50000 [4:31:47<20:07:15,  1.77s/it]
Outer training loop:  18%|#7        | 8984/50000 [4:31:49<20:57:31,  1.84s/it]
Outer training loop:  18%|#7        | 8985/50000 [4:31:51<20:34:33,  1.81s/it]
Outer training loop:  18%|#7        | 8986/50000 [4:31:53<20:05:29,  1.76s/it]
Outer training loop:  18%|#7        | 8987/50000 [4:31:54<20:21:06,  1.79s/it]
Outer training loop:  18%|#7        | 8988/50000 [4:31:56<19:58:18,  1.75s/it]
Outer training loop:  18%|#7        | 8989/50000 [4:31:58<19:45:19,  1.73s/it]
Outer training loop:  18%|#7        | 8990/50000 [4:32:00<20:00:00,  1.76s/it]
Outer training loop:  18%|#7        | 8991/50000 [4:32:01<19:51:54,  1.74s/it]
Outer training loop:  18%|#7        | 8992/50000 [4:32:03<19:54:33,  1.75s/it]
Outer training loop:  18%|#7        | 8993/50000 [4:32:05<19:35:09,  1.72s/it]
Outer training loop:  18%|#7        | 8994/50000 [4:32:07<20:13:27,  1.78s/it]
Outer training loop:  18%|#7        | 8995/50000 [4:32:08<19:44:13,  1.73s/it]
Outer training loop:  18%|#7        | 8996/50000 [4:32:10<19:53:05,  1.75s/it]
Outer training loop:  18%|#7        | 8997/50000 [4:32:12<20:27:16,  1.80s/it]
Outer training loop:  18%|#7        | 8998/50000 [4:32:14<19:55:07,  1.75s/it]
Outer training loop:  18%|#7        | 8999/50000 [4:32:15<19:33:57,  1.72s/it]
Outer training loop:  18%|#8        | 9000/50000 [4:32:17<19:32:59,  1.72s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:21,  1.72s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:19,  1.72s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.79s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:17,  1.75s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:21,  1.82s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:17,  1.79s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.83s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:17,  1.83s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:16,  1.84s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:10,  1.79s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.88s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:11,  1.85s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:06,  1.81s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:03,  1.79s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:05,  1.84s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:59,  1.78s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:56,  1.76s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:53,  1.74s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:57,  1.83s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:53,  1.81s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:49,  1.77s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:46,  1.75s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:45,  1.76s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:44,  1.80s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.78s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:39,  1.78s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:37,  1.77s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:33,  1.74s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:37,  1.85s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:36,  1.85s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:31,  1.80s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:30,  1.80s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:26,  1.77s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:24,  1.79s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:20,  1.74s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:17,  1.72s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.75s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.73s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:11,  1.70s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:14,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:11,  1.78s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:07,  1.74s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:04,  1.71s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:01,  1.67s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:00,  1.68s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<00:58,  1.67s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.77s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:57,  1.74s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:55,  1.73s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:53,  1.73s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:51,  1.72s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:49,  1.69s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.74s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:47,  1.74s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:46,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:43,  1.80s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:41,  1.80s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:39,  1.79s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:37,  1.76s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:34,  1.73s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:30,  1.78s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.75s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:26,  1.77s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.74s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.75s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:20,  1.73s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:18,  1.72s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:18,  1.87s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.81s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.80s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.79s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.78s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.82s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.80s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.77s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.89s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:12,  1.62s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:13,  1.65s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:13,  1.67s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:14,  1.70s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:11,  1.69s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:12,  1.72s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:13,  1.76s/it]

Evaluating policy:  10%|9         | 8/83 [00:13<02:10,  1.74s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:07,  1.72s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:14,  1.87s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.82s/it]

Evaluating policy:  16%|#5        | 13/83 [00:22<02:03,  1.77s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:05,  1.82s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:02,  1.80s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:00,  1.80s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:56,  1.77s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:58,  1.82s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:54,  1.79s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:51,  1.77s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:49,  1.77s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:49,  1.79s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:48,  1.85s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:44,  1.81s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.79s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:43,  1.85s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:40,  1.83s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.80s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:34,  1.78s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.76s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:29,  1.76s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:31,  1.84s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:28,  1.81s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:26,  1.80s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:26,  1.83s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:21,  1.78s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:17,  1.73s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:14,  1.70s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:15,  1.80s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:13,  1.80s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:11,  1.79s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:08,  1.77s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:07,  1.78s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:08,  1.86s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:06,  1.83s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:02,  1.79s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:57,  1.75s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:56,  1.75s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:55,  1.78s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:56,  1.89s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:49,  1.78s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:48,  1.79s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:45,  1.76s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:43,  1.76s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:41,  1.75s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:40,  1.75s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:41,  1.87s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:38,  1.82s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.78s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:33,  1.76s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:31,  1.74s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.73s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.73s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:26,  1.74s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:26,  1.90s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:23,  1.84s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:21,  1.79s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:19,  1.78s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.78s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:15,  1.78s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:14,  1.77s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.76s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.74s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.71s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.89s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.83s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.77s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.79s/it]

Outer training loop:  18%|#8        | 9001/50000 [4:37:21<1052:10:05, 92.39s/it]
Outer training loop:  18%|#8        | 9002/50000 [4:37:23<742:24:31, 65.19s/it] 
Outer training loop:  18%|#8        | 9003/50000 [4:37:24<525:52:33, 46.18s/it]
Outer training loop:  18%|#8        | 9004/50000 [4:37:26<374:18:10, 32.87s/it]
Outer training loop:  18%|#8        | 9005/50000 [4:37:28<268:22:18, 23.57s/it]
Outer training loop:  18%|#8        | 9006/50000 [4:37:30<193:19:09, 16.98s/it]
Outer training loop:  18%|#8        | 9007/50000 [4:37:32<142:26:39, 12.51s/it]
Outer training loop:  18%|#8        | 9008/50000 [4:37:34<105:58:37,  9.31s/it]
Outer training loop:  18%|#8        | 9009/50000 [4:37:35<80:09:51,  7.04s/it] 
Outer training loop:  18%|#8        | 9010/50000 [4:37:37<62:29:56,  5.49s/it]
Outer training loop:  18%|#8        | 9011/50000 [4:37:39<49:59:34,  4.39s/it]
Outer training loop:  18%|#8        | 9012/50000 [4:37:41<40:47:13,  3.58s/it]
Outer training loop:  18%|#8        | 9013/50000 [4:37:43<35:27:04,  3.11s/it]
Outer training loop:  18%|#8        | 9014/50000 [4:37:44<30:37:16,  2.69s/it]
Outer training loop:  18%|#8        | 9015/50000 [4:37:46<27:19:21,  2.40s/it]
Outer training loop:  18%|#8        | 9016/50000 [4:37:48<25:04:44,  2.20s/it]
Outer training loop:  18%|#8        | 9017/50000 [4:37:50<24:04:41,  2.12s/it]
Outer training loop:  18%|#8        | 9018/50000 [4:37:52<22:39:58,  1.99s/it]
Outer training loop:  18%|#8        | 9019/50000 [4:37:53<21:58:47,  1.93s/it]
Outer training loop:  18%|#8        | 9020/50000 [4:37:55<21:38:29,  1.90s/it]
Outer training loop:  18%|#8        | 9021/50000 [4:37:57<20:44:13,  1.82s/it]
Outer training loop:  18%|#8        | 9022/50000 [4:37:58<20:18:04,  1.78s/it]
Outer training loop:  18%|#8        | 9023/50000 [4:38:00<20:00:49,  1.76s/it]
Outer training loop:  18%|#8        | 9024/50000 [4:38:02<20:24:03,  1.79s/it]
Outer training loop:  18%|#8        | 9025/50000 [4:38:04<19:58:14,  1.75s/it]
Outer training loop:  18%|#8        | 9026/50000 [4:38:05<20:00:19,  1.76s/it]
Outer training loop:  18%|#8        | 9027/50000 [4:38:07<20:13:02,  1.78s/it]
Outer training loop:  18%|#8        | 9028/50000 [4:38:09<19:46:24,  1.74s/it]
Outer training loop:  18%|#8        | 9029/50000 [4:38:11<19:12:46,  1.69s/it]
Outer training loop:  18%|#8        | 9030/50000 [4:38:13<20:22:28,  1.79s/it]
Outer training loop:  18%|#8        | 9031/50000 [4:38:14<20:03:19,  1.76s/it]
Outer training loop:  18%|#8        | 9032/50000 [4:38:16<19:48:20,  1.74s/it]
Outer training loop:  18%|#8        | 9033/50000 [4:38:18<19:43:46,  1.73s/it]
Outer training loop:  18%|#8        | 9034/50000 [4:38:20<20:06:09,  1.77s/it]
Outer training loop:  18%|#8        | 9035/50000 [4:38:21<19:59:23,  1.76s/it]
Outer training loop:  18%|#8        | 9036/50000 [4:38:23<19:54:21,  1.75s/it]
Outer training loop:  18%|#8        | 9037/50000 [4:38:25<20:21:04,  1.79s/it]
Outer training loop:  18%|#8        | 9038/50000 [4:38:26<19:51:02,  1.74s/it]
Outer training loop:  18%|#8        | 9039/50000 [4:38:28<19:29:26,  1.71s/it]
Outer training loop:  18%|#8        | 9040/50000 [4:38:30<20:40:07,  1.82s/it]
Outer training loop:  18%|#8        | 9041/50000 [4:38:32<20:09:28,  1.77s/it]
Outer training loop:  18%|#8        | 9042/50000 [4:38:34<19:56:20,  1.75s/it]
Outer training loop:  18%|#8        | 9043/50000 [4:38:36<20:34:48,  1.81s/it]
Outer training loop:  18%|#8        | 9044/50000 [4:38:37<20:12:20,  1.78s/it]
Outer training loop:  18%|#8        | 9045/50000 [4:38:39<20:03:24,  1.76s/it]
Outer training loop:  18%|#8        | 9046/50000 [4:38:41<21:11:56,  1.86s/it]
Outer training loop:  18%|#8        | 9047/50000 [4:38:43<20:36:10,  1.81s/it]
Outer training loop:  18%|#8        | 9048/50000 [4:38:44<20:26:53,  1.80s/it]
Outer training loop:  18%|#8        | 9049/50000 [4:38:46<20:37:09,  1.81s/it]
Outer training loop:  18%|#8        | 9050/50000 [4:38:48<20:21:56,  1.79s/it]
Outer training loop:  18%|#8        | 9051/50000 [4:38:50<20:11:04,  1.77s/it]
Outer training loop:  18%|#8        | 9052/50000 [4:38:51<19:48:35,  1.74s/it]
Outer training loop:  18%|#8        | 9053/50000 [4:38:53<20:16:58,  1.78s/it]
Outer training loop:  18%|#8        | 9054/50000 [4:38:55<20:04:54,  1.77s/it]
Outer training loop:  18%|#8        | 9055/50000 [4:38:57<19:24:53,  1.71s/it]
Outer training loop:  18%|#8        | 9056/50000 [4:38:59<20:00:22,  1.76s/it]
Outer training loop:  18%|#8        | 9057/50000 [4:39:00<20:04:37,  1.77s/it]
Outer training loop:  18%|#8        | 9058/50000 [4:39:02<20:34:58,  1.81s/it]
Outer training loop:  18%|#8        | 9059/50000 [4:39:04<21:01:59,  1.85s/it]
Outer training loop:  18%|#8        | 9060/50000 [4:39:06<20:31:59,  1.81s/it]
Outer training loop:  18%|#8        | 9061/50000 [4:39:08<20:13:18,  1.78s/it]
Outer training loop:  18%|#8        | 9062/50000 [4:39:10<20:59:28,  1.85s/it]
Outer training loop:  18%|#8        | 9063/50000 [4:39:11<20:23:24,  1.79s/it]
Outer training loop:  18%|#8        | 9064/50000 [4:39:13<19:55:38,  1.75s/it]
Outer training loop:  18%|#8        | 9065/50000 [4:39:15<19:38:09,  1.73s/it]
Outer training loop:  18%|#8        | 9066/50000 [4:39:17<20:22:25,  1.79s/it]
Outer training loop:  18%|#8        | 9067/50000 [4:39:18<20:12:01,  1.78s/it]
Outer training loop:  18%|#8        | 9068/50000 [4:39:20<20:08:51,  1.77s/it]
Outer training loop:  18%|#8        | 9069/50000 [4:39:22<20:40:58,  1.82s/it]
Outer training loop:  18%|#8        | 9070/50000 [4:39:24<20:20:06,  1.79s/it]
Outer training loop:  18%|#8        | 9071/50000 [4:39:25<20:09:47,  1.77s/it]
Outer training loop:  18%|#8        | 9072/50000 [4:39:27<19:57:51,  1.76s/it]
Outer training loop:  18%|#8        | 9073/50000 [4:39:29<20:24:22,  1.79s/it]
Outer training loop:  18%|#8        | 9074/50000 [4:39:31<20:01:35,  1.76s/it]
Outer training loop:  18%|#8        | 9075/50000 [4:39:32<19:32:01,  1.72s/it]
Outer training loop:  18%|#8        | 9076/50000 [4:39:34<20:15:15,  1.78s/it]
Outer training loop:  18%|#8        | 9077/50000 [4:39:36<20:04:46,  1.77s/it]
Outer training loop:  18%|#8        | 9078/50000 [4:39:38<19:47:39,  1.74s/it]
Outer training loop:  18%|#8        | 9079/50000 [4:39:39<19:54:54,  1.75s/it]
Outer training loop:  18%|#8        | 9080/50000 [4:39:41<19:43:57,  1.74s/it]
Outer training loop:  18%|#8        | 9081/50000 [4:39:43<19:42:45,  1.73s/it]
Outer training loop:  18%|#8        | 9082/50000 [4:39:45<20:12:29,  1.78s/it]
Outer training loop:  18%|#8        | 9083/50000 [4:39:46<19:55:27,  1.75s/it]
Outer training loop:  18%|#8        | 9084/50000 [4:39:48<19:43:28,  1.74s/it]
Outer training loop:  18%|#8        | 9085/50000 [4:39:50<19:26:38,  1.71s/it]
Outer training loop:  18%|#8        | 9086/50000 [4:39:52<19:55:05,  1.75s/it]
Outer training loop:  18%|#8        | 9087/50000 [4:39:53<19:48:44,  1.74s/it]
Outer training loop:  18%|#8        | 9088/50000 [4:39:55<19:40:34,  1.73s/it]
Outer training loop:  18%|#8        | 9089/50000 [4:39:57<20:10:08,  1.77s/it]
Outer training loop:  18%|#8        | 9090/50000 [4:39:59<20:22:45,  1.79s/it]
Outer training loop:  18%|#8        | 9091/50000 [4:40:00<19:30:29,  1.72s/it]
Outer training loop:  18%|#8        | 9092/50000 [4:40:02<19:15:39,  1.69s/it]
Outer training loop:  18%|#8        | 9093/50000 [4:40:04<20:18:25,  1.79s/it]
Outer training loop:  18%|#8        | 9094/50000 [4:40:06<20:06:49,  1.77s/it]
Outer training loop:  18%|#8        | 9095/50000 [4:40:07<19:59:13,  1.76s/it]
Outer training loop:  18%|#8        | 9096/50000 [4:40:09<20:50:55,  1.83s/it]
Outer training loop:  18%|#8        | 9097/50000 [4:40:11<20:49:44,  1.83s/it]
Outer training loop:  18%|#8        | 9098/50000 [4:40:13<20:19:31,  1.79s/it]
Outer training loop:  18%|#8        | 9099/50000 [4:40:15<20:46:16,  1.83s/it]
Outer training loop:  18%|#8        | 9100/50000 [4:40:17<20:13:45,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  18%|#8        | 9101/50000 [4:40:24<38:06:07,  3.35s/it]
Outer training loop:  18%|#8        | 9102/50000 [4:40:25<32:28:51,  2.86s/it]
Outer training loop:  18%|#8        | 9103/50000 [4:40:27<28:56:48,  2.55s/it]
Outer training loop:  18%|#8        | 9104/50000 [4:40:29<26:06:56,  2.30s/it]
Outer training loop:  18%|#8        | 9105/50000 [4:40:30<23:58:55,  2.11s/it]
Outer training loop:  18%|#8        | 9106/50000 [4:40:32<22:40:04,  2.00s/it]
Outer training loop:  18%|#8        | 9107/50000 [4:40:34<22:14:37,  1.96s/it]
Outer training loop:  18%|#8        | 9108/50000 [4:40:36<21:41:25,  1.91s/it]
Outer training loop:  18%|#8        | 9109/50000 [4:40:38<20:56:30,  1.84s/it]
Outer training loop:  18%|#8        | 9110/50000 [4:40:39<21:07:01,  1.86s/it]
Outer training loop:  18%|#8        | 9111/50000 [4:40:41<20:29:51,  1.80s/it]
Outer training loop:  18%|#8        | 9112/50000 [4:40:43<20:00:03,  1.76s/it]
Outer training loop:  18%|#8        | 9113/50000 [4:40:45<20:17:56,  1.79s/it]
Outer training loop:  18%|#8        | 9114/50000 [4:40:46<20:20:31,  1.79s/it]
Outer training loop:  18%|#8        | 9115/50000 [4:40:48<20:21:06,  1.79s/it]
Outer training loop:  18%|#8        | 9116/50000 [4:40:50<19:47:40,  1.74s/it]
Outer training loop:  18%|#8        | 9117/50000 [4:40:52<20:08:17,  1.77s/it]
Outer training loop:  18%|#8        | 9118/50000 [4:40:53<20:01:05,  1.76s/it]
Outer training loop:  18%|#8        | 9119/50000 [4:40:55<19:54:40,  1.75s/it]
Outer training loop:  18%|#8        | 9120/50000 [4:40:57<20:18:30,  1.79s/it]
Outer training loop:  18%|#8        | 9121/50000 [4:40:59<20:09:42,  1.78s/it]
Outer training loop:  18%|#8        | 9122/50000 [4:41:01<20:06:23,  1.77s/it]
Outer training loop:  18%|#8        | 9123/50000 [4:41:03<20:40:43,  1.82s/it]
Outer training loop:  18%|#8        | 9124/50000 [4:41:04<20:20:00,  1.79s/it]
Outer training loop:  18%|#8        | 9125/50000 [4:41:06<20:05:53,  1.77s/it]
Outer training loop:  18%|#8        | 9126/50000 [4:41:08<20:22:44,  1.79s/it]
Outer training loop:  18%|#8        | 9127/50000 [4:41:09<19:57:43,  1.76s/it]
Outer training loop:  18%|#8        | 9128/50000 [4:41:11<19:39:13,  1.73s/it]
Outer training loop:  18%|#8        | 9129/50000 [4:41:13<19:41:35,  1.73s/it]
Outer training loop:  18%|#8        | 9130/50000 [4:41:15<20:01:07,  1.76s/it]
Outer training loop:  18%|#8        | 9131/50000 [4:41:16<19:40:28,  1.73s/it]
Outer training loop:  18%|#8        | 9132/50000 [4:41:18<19:15:50,  1.70s/it]
Outer training loop:  18%|#8        | 9133/50000 [4:41:20<20:23:01,  1.80s/it]
Outer training loop:  18%|#8        | 9134/50000 [4:41:22<20:14:32,  1.78s/it]
Outer training loop:  18%|#8        | 9135/50000 [4:41:23<19:58:41,  1.76s/it]
Outer training loop:  18%|#8        | 9136/50000 [4:41:25<20:37:40,  1.82s/it]
Outer training loop:  18%|#8        | 9137/50000 [4:41:27<21:00:26,  1.85s/it]
Outer training loop:  18%|#8        | 9138/50000 [4:41:29<20:29:11,  1.80s/it]
Outer training loop:  18%|#8        | 9139/50000 [4:41:31<20:21:07,  1.79s/it]
Outer training loop:  18%|#8        | 9140/50000 [4:41:33<20:52:07,  1.84s/it]
Outer training loop:  18%|#8        | 9141/50000 [4:41:35<20:45:12,  1.83s/it]
Outer training loop:  18%|#8        | 9142/50000 [4:41:36<20:11:01,  1.78s/it]
Outer training loop:  18%|#8        | 9143/50000 [4:41:38<20:25:06,  1.80s/it]
Outer training loop:  18%|#8        | 9144/50000 [4:41:40<20:33:10,  1.81s/it]
Outer training loop:  18%|#8        | 9145/50000 [4:41:42<20:05:06,  1.77s/it]
Outer training loop:  18%|#8        | 9146/50000 [4:41:43<20:32:19,  1.81s/it]
Outer training loop:  18%|#8        | 9147/50000 [4:41:45<20:41:55,  1.82s/it]
Outer training loop:  18%|#8        | 9148/50000 [4:41:47<20:29:19,  1.81s/it]
Outer training loop:  18%|#8        | 9149/50000 [4:41:49<20:50:19,  1.84s/it]
Outer training loop:  18%|#8        | 9150/50000 [4:41:51<20:50:57,  1.84s/it]
Outer training loop:  18%|#8        | 9151/50000 [4:41:53<20:23:25,  1.80s/it]
Outer training loop:  18%|#8        | 9152/50000 [4:41:54<20:36:03,  1.82s/it]
Outer training loop:  18%|#8        | 9153/50000 [4:41:56<20:04:55,  1.77s/it]
Outer training loop:  18%|#8        | 9154/50000 [4:41:58<19:43:41,  1.74s/it]
Outer training loop:  18%|#8        | 9155/50000 [4:42:00<19:46:32,  1.74s/it]
Outer training loop:  18%|#8        | 9156/50000 [4:42:01<20:15:13,  1.79s/it]
Outer training loop:  18%|#8        | 9157/50000 [4:42:03<20:07:01,  1.77s/it]
Outer training loop:  18%|#8        | 9158/50000 [4:42:05<20:08:15,  1.78s/it]
Outer training loop:  18%|#8        | 9159/50000 [4:42:07<20:23:14,  1.80s/it]
Outer training loop:  18%|#8        | 9160/50000 [4:42:08<20:08:14,  1.78s/it]
Outer training loop:  18%|#8        | 9161/50000 [4:42:10<19:45:41,  1.74s/it]
Outer training loop:  18%|#8        | 9162/50000 [4:42:12<19:51:09,  1.75s/it]
Outer training loop:  18%|#8        | 9163/50000 [4:42:14<20:11:31,  1.78s/it]
Outer training loop:  18%|#8        | 9164/50000 [4:42:16<20:13:15,  1.78s/it]
Outer training loop:  18%|#8        | 9165/50000 [4:42:17<20:01:30,  1.77s/it]
Outer training loop:  18%|#8        | 9166/50000 [4:42:19<20:25:22,  1.80s/it]
Outer training loop:  18%|#8        | 9167/50000 [4:42:21<19:57:01,  1.76s/it]
Outer training loop:  18%|#8        | 9168/50000 [4:42:23<19:48:10,  1.75s/it]
Outer training loop:  18%|#8        | 9169/50000 [4:42:24<20:25:54,  1.80s/it]
Outer training loop:  18%|#8        | 9170/50000 [4:42:26<20:08:20,  1.78s/it]
Outer training loop:  18%|#8        | 9171/50000 [4:42:28<19:47:52,  1.75s/it]
Outer training loop:  18%|#8        | 9172/50000 [4:42:30<19:49:14,  1.75s/it]
Outer training loop:  18%|#8        | 9173/50000 [4:42:32<20:45:19,  1.83s/it]
Outer training loop:  18%|#8        | 9174/50000 [4:42:33<20:34:56,  1.81s/it]
Outer training loop:  18%|#8        | 9175/50000 [4:42:35<20:16:45,  1.79s/it]
Outer training loop:  18%|#8        | 9176/50000 [4:42:37<20:08:07,  1.78s/it]
Outer training loop:  18%|#8        | 9177/50000 [4:42:39<20:07:21,  1.77s/it]
Outer training loop:  18%|#8        | 9178/50000 [4:42:40<19:46:22,  1.74s/it]
Outer training loop:  18%|#8        | 9179/50000 [4:42:42<19:34:56,  1.73s/it]
Outer training loop:  18%|#8        | 9180/50000 [4:42:44<19:59:47,  1.76s/it]
Outer training loop:  18%|#8        | 9181/50000 [4:42:46<20:02:51,  1.77s/it]
Outer training loop:  18%|#8        | 9182/50000 [4:42:47<19:36:13,  1.73s/it]
Outer training loop:  18%|#8        | 9183/50000 [4:42:49<19:57:03,  1.76s/it]
Outer training loop:  18%|#8        | 9184/50000 [4:42:51<19:54:28,  1.76s/it]
Outer training loop:  18%|#8        | 9185/50000 [4:42:53<19:58:02,  1.76s/it]
Outer training loop:  18%|#8        | 9186/50000 [4:42:55<20:30:16,  1.81s/it]
Outer training loop:  18%|#8        | 9187/50000 [4:42:57<20:58:00,  1.85s/it]
Outer training loop:  18%|#8        | 9188/50000 [4:42:58<20:34:53,  1.82s/it]
Outer training loop:  18%|#8        | 9189/50000 [4:43:00<20:41:48,  1.83s/it]
Outer training loop:  18%|#8        | 9190/50000 [4:43:02<20:14:00,  1.78s/it]
Outer training loop:  18%|#8        | 9191/50000 [4:43:03<19:43:23,  1.74s/it]
Outer training loop:  18%|#8        | 9192/50000 [4:43:05<20:04:37,  1.77s/it]
Outer training loop:  18%|#8        | 9193/50000 [4:43:07<20:18:51,  1.79s/it]
Outer training loop:  18%|#8        | 9194/50000 [4:43:09<19:57:47,  1.76s/it]
Outer training loop:  18%|#8        | 9195/50000 [4:43:11<20:14:30,  1.79s/it]
Outer training loop:  18%|#8        | 9196/50000 [4:43:13<21:04:07,  1.86s/it]
Outer training loop:  18%|#8        | 9197/50000 [4:43:15<21:06:12,  1.86s/it]
Outer training loop:  18%|#8        | 9198/50000 [4:43:16<20:55:31,  1.85s/it]
Outer training loop:  18%|#8        | 9199/50000 [4:43:18<21:07:07,  1.86s/it]
Outer training loop:  18%|#8        | 9200/50000 [4:43:20<20:45:58,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.90s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  18%|#8        | 9201/50000 [4:43:27<38:45:56,  3.42s/it]
Outer training loop:  18%|#8        | 9202/50000 [4:43:29<33:33:36,  2.96s/it]
Outer training loop:  18%|#8        | 9203/50000 [4:43:31<29:16:54,  2.58s/it]
Outer training loop:  18%|#8        | 9204/50000 [4:43:32<26:06:58,  2.30s/it]
Outer training loop:  18%|#8        | 9205/50000 [4:43:34<24:07:04,  2.13s/it]
Outer training loop:  18%|#8        | 9206/50000 [4:43:36<23:38:58,  2.09s/it]
Outer training loop:  18%|#8        | 9207/50000 [4:43:38<22:06:08,  1.95s/it]
Outer training loop:  18%|#8        | 9208/50000 [4:43:39<21:12:02,  1.87s/it]
Outer training loop:  18%|#8        | 9209/50000 [4:43:41<20:43:34,  1.83s/it]
Outer training loop:  18%|#8        | 9210/50000 [4:43:43<21:03:31,  1.86s/it]
Outer training loop:  18%|#8        | 9211/50000 [4:43:45<20:29:10,  1.81s/it]
Outer training loop:  18%|#8        | 9212/50000 [4:43:46<19:57:26,  1.76s/it]
Outer training loop:  18%|#8        | 9213/50000 [4:43:48<20:21:05,  1.80s/it]
Outer training loop:  18%|#8        | 9214/50000 [4:43:50<19:39:16,  1.73s/it]
Outer training loop:  18%|#8        | 9215/50000 [4:43:52<19:37:21,  1.73s/it]
Outer training loop:  18%|#8        | 9216/50000 [4:43:53<20:00:23,  1.77s/it]
Outer training loop:  18%|#8        | 9217/50000 [4:43:55<20:16:51,  1.79s/it]
Outer training loop:  18%|#8        | 9218/50000 [4:43:57<19:50:36,  1.75s/it]
Outer training loop:  18%|#8        | 9219/50000 [4:43:59<19:36:50,  1.73s/it]
Outer training loop:  18%|#8        | 9220/50000 [4:44:00<19:47:04,  1.75s/it]
Outer training loop:  18%|#8        | 9221/50000 [4:44:02<19:51:13,  1.75s/it]
Outer training loop:  18%|#8        | 9222/50000 [4:44:04<20:00:39,  1.77s/it]
Outer training loop:  18%|#8        | 9223/50000 [4:44:06<20:41:14,  1.83s/it]
Outer training loop:  18%|#8        | 9224/50000 [4:44:08<20:30:23,  1.81s/it]
Outer training loop:  18%|#8        | 9225/50000 [4:44:10<20:25:11,  1.80s/it]
Outer training loop:  18%|#8        | 9226/50000 [4:44:11<20:05:24,  1.77s/it]
Outer training loop:  18%|#8        | 9227/50000 [4:44:13<20:45:45,  1.83s/it]
Outer training loop:  18%|#8        | 9228/50000 [4:44:15<20:20:28,  1.80s/it]
Outer training loop:  18%|#8        | 9229/50000 [4:44:17<20:11:16,  1.78s/it]
Outer training loop:  18%|#8        | 9230/50000 [4:44:19<20:48:42,  1.84s/it]
Outer training loop:  18%|#8        | 9231/50000 [4:44:20<20:38:31,  1.82s/it]
Outer training loop:  18%|#8        | 9232/50000 [4:44:22<20:08:54,  1.78s/it]
Outer training loop:  18%|#8        | 9233/50000 [4:44:24<20:17:28,  1.79s/it]
Outer training loop:  18%|#8        | 9234/50000 [4:44:26<20:55:50,  1.85s/it]
Outer training loop:  18%|#8        | 9235/50000 [4:44:28<20:37:29,  1.82s/it]
Outer training loop:  18%|#8        | 9236/50000 [4:44:29<20:39:10,  1.82s/it]
Outer training loop:  18%|#8        | 9237/50000 [4:44:31<20:11:38,  1.78s/it]
Outer training loop:  18%|#8        | 9238/50000 [4:44:33<19:50:28,  1.75s/it]
Outer training loop:  18%|#8        | 9239/50000 [4:44:35<20:24:06,  1.80s/it]
Outer training loop:  18%|#8        | 9240/50000 [4:44:36<20:00:16,  1.77s/it]
Outer training loop:  18%|#8        | 9241/50000 [4:44:38<20:11:54,  1.78s/it]
Outer training loop:  18%|#8        | 9242/50000 [4:44:40<20:09:53,  1.78s/it]
Outer training loop:  18%|#8        | 9243/50000 [4:44:42<20:34:39,  1.82s/it]
Outer training loop:  18%|#8        | 9244/50000 [4:44:44<20:17:16,  1.79s/it]
Outer training loop:  18%|#8        | 9245/50000 [4:44:45<20:11:41,  1.78s/it]
Outer training loop:  18%|#8        | 9246/50000 [4:44:47<20:16:22,  1.79s/it]
Outer training loop:  18%|#8        | 9247/50000 [4:44:49<20:04:59,  1.77s/it]
Outer training loop:  18%|#8        | 9248/50000 [4:44:51<20:16:33,  1.79s/it]
Outer training loop:  18%|#8        | 9249/50000 [4:44:53<20:28:04,  1.81s/it]
Outer training loop:  18%|#8        | 9250/50000 [4:44:54<20:19:20,  1.80s/it]
Outer training loop:  19%|#8        | 9251/50000 [4:44:56<19:51:00,  1.75s/it]
Outer training loop:  19%|#8        | 9252/50000 [4:44:58<19:54:18,  1.76s/it]
Outer training loop:  19%|#8        | 9253/50000 [4:45:00<19:33:53,  1.73s/it]
Outer training loop:  19%|#8        | 9254/50000 [4:45:01<19:27:49,  1.72s/it]
Outer training loop:  19%|#8        | 9255/50000 [4:45:03<20:15:30,  1.79s/it]
Outer training loop:  19%|#8        | 9256/50000 [4:45:05<20:21:35,  1.80s/it]
Outer training loop:  19%|#8        | 9257/50000 [4:45:07<20:13:47,  1.79s/it]
Outer training loop:  19%|#8        | 9258/50000 [4:45:09<20:57:13,  1.85s/it]
Outer training loop:  19%|#8        | 9259/50000 [4:45:10<20:24:21,  1.80s/it]
Outer training loop:  19%|#8        | 9260/50000 [4:45:12<20:13:12,  1.79s/it]
Outer training loop:  19%|#8        | 9261/50000 [4:45:14<20:53:06,  1.85s/it]
Outer training loop:  19%|#8        | 9262/50000 [4:45:16<20:18:47,  1.80s/it]
Outer training loop:  19%|#8        | 9263/50000 [4:45:18<20:25:47,  1.81s/it]
Outer training loop:  19%|#8        | 9264/50000 [4:45:20<20:45:34,  1.83s/it]
Outer training loop:  19%|#8        | 9265/50000 [4:45:21<20:31:48,  1.81s/it]
Outer training loop:  19%|#8        | 9266/50000 [4:45:23<20:14:28,  1.79s/it]
Outer training loop:  19%|#8        | 9267/50000 [4:45:25<20:11:01,  1.78s/it]
Outer training loop:  19%|#8        | 9268/50000 [4:45:27<20:29:00,  1.81s/it]
Outer training loop:  19%|#8        | 9269/50000 [4:45:28<20:19:13,  1.80s/it]
Outer training loop:  19%|#8        | 9270/50000 [4:45:30<19:50:22,  1.75s/it]
Outer training loop:  19%|#8        | 9271/50000 [4:45:32<20:14:24,  1.79s/it]
Outer training loop:  19%|#8        | 9272/50000 [4:45:34<19:46:45,  1.75s/it]
Outer training loop:  19%|#8        | 9273/50000 [4:45:35<19:43:44,  1.74s/it]
Outer training loop:  19%|#8        | 9274/50000 [4:45:37<20:07:36,  1.78s/it]
Outer training loop:  19%|#8        | 9275/50000 [4:45:39<19:55:31,  1.76s/it]
Outer training loop:  19%|#8        | 9276/50000 [4:45:41<19:34:45,  1.73s/it]
Outer training loop:  19%|#8        | 9277/50000 [4:45:43<20:24:41,  1.80s/it]
Outer training loop:  19%|#8        | 9278/50000 [4:45:44<20:34:29,  1.82s/it]
Outer training loop:  19%|#8        | 9279/50000 [4:45:46<20:22:12,  1.80s/it]
Outer training loop:  19%|#8        | 9280/50000 [4:45:48<20:15:42,  1.79s/it]
Outer training loop:  19%|#8        | 9281/50000 [4:45:50<20:43:10,  1.83s/it]
Outer training loop:  19%|#8        | 9282/50000 [4:45:52<20:14:53,  1.79s/it]
Outer training loop:  19%|#8        | 9283/50000 [4:45:53<20:13:27,  1.79s/it]
Outer training loop:  19%|#8        | 9284/50000 [4:45:55<20:51:00,  1.84s/it]
Outer training loop:  19%|#8        | 9285/50000 [4:45:57<20:32:57,  1.82s/it]
Outer training loop:  19%|#8        | 9286/50000 [4:45:59<20:08:34,  1.78s/it]
Outer training loop:  19%|#8        | 9287/50000 [4:46:01<21:01:26,  1.86s/it]
Outer training loop:  19%|#8        | 9288/50000 [4:46:03<20:52:28,  1.85s/it]
Outer training loop:  19%|#8        | 9289/50000 [4:46:04<20:36:34,  1.82s/it]
Outer training loop:  19%|#8        | 9290/50000 [4:46:06<20:51:40,  1.84s/it]
Outer training loop:  19%|#8        | 9291/50000 [4:46:08<21:08:05,  1.87s/it]
Outer training loop:  19%|#8        | 9292/50000 [4:46:10<20:26:38,  1.81s/it]
Outer training loop:  19%|#8        | 9293/50000 [4:46:12<20:18:57,  1.80s/it]
Outer training loop:  19%|#8        | 9294/50000 [4:46:14<20:53:44,  1.85s/it]
Outer training loop:  19%|#8        | 9295/50000 [4:46:15<20:34:15,  1.82s/it]
Outer training loop:  19%|#8        | 9296/50000 [4:46:17<19:59:56,  1.77s/it]
Outer training loop:  19%|#8        | 9297/50000 [4:46:19<20:42:45,  1.83s/it]
Outer training loop:  19%|#8        | 9298/50000 [4:46:21<20:39:57,  1.83s/it]
Outer training loop:  19%|#8        | 9299/50000 [4:46:23<20:35:29,  1.82s/it]
Outer training loop:  19%|#8        | 9300/50000 [4:46:24<20:03:04,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.73s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  19%|#8        | 9301/50000 [4:46:31<38:18:37,  3.39s/it]
Outer training loop:  19%|#8        | 9302/50000 [4:46:33<32:41:23,  2.89s/it]
Outer training loop:  19%|#8        | 9303/50000 [4:46:35<28:49:35,  2.55s/it]
Outer training loop:  19%|#8        | 9304/50000 [4:46:37<26:33:48,  2.35s/it]
Outer training loop:  19%|#8        | 9305/50000 [4:46:39<24:31:18,  2.17s/it]
Outer training loop:  19%|#8        | 9306/50000 [4:46:40<23:33:33,  2.08s/it]
Outer training loop:  19%|#8        | 9307/50000 [4:46:42<22:31:39,  1.99s/it]
Outer training loop:  19%|#8        | 9308/50000 [4:46:44<21:19:59,  1.89s/it]
Outer training loop:  19%|#8        | 9309/50000 [4:46:46<21:06:29,  1.87s/it]
Outer training loop:  19%|#8        | 9310/50000 [4:46:47<20:16:35,  1.79s/it]
Outer training loop:  19%|#8        | 9311/50000 [4:46:49<20:46:43,  1.84s/it]
Outer training loop:  19%|#8        | 9312/50000 [4:46:51<20:46:17,  1.84s/it]
Outer training loop:  19%|#8        | 9313/50000 [4:46:53<20:24:58,  1.81s/it]
Outer training loop:  19%|#8        | 9314/50000 [4:46:55<20:14:49,  1.79s/it]
Outer training loop:  19%|#8        | 9315/50000 [4:46:57<20:43:05,  1.83s/it]
Outer training loop:  19%|#8        | 9316/50000 [4:46:58<20:32:31,  1.82s/it]
Outer training loop:  19%|#8        | 9317/50000 [4:47:00<20:14:06,  1.79s/it]
Outer training loop:  19%|#8        | 9318/50000 [4:47:02<20:28:17,  1.81s/it]
Outer training loop:  19%|#8        | 9319/50000 [4:47:04<20:23:32,  1.80s/it]
Outer training loop:  19%|#8        | 9320/50000 [4:47:05<20:16:28,  1.79s/it]
Outer training loop:  19%|#8        | 9321/50000 [4:47:07<20:24:23,  1.81s/it]
Outer training loop:  19%|#8        | 9322/50000 [4:47:09<20:17:50,  1.80s/it]
Outer training loop:  19%|#8        | 9323/50000 [4:47:11<20:01:11,  1.77s/it]
Outer training loop:  19%|#8        | 9324/50000 [4:47:13<20:27:29,  1.81s/it]
Outer training loop:  19%|#8        | 9325/50000 [4:47:14<19:55:24,  1.76s/it]
Outer training loop:  19%|#8        | 9326/50000 [4:47:16<19:40:35,  1.74s/it]
Outer training loop:  19%|#8        | 9327/50000 [4:47:18<19:49:59,  1.76s/it]
Outer training loop:  19%|#8        | 9328/50000 [4:47:20<20:29:29,  1.81s/it]
Outer training loop:  19%|#8        | 9329/50000 [4:47:21<20:05:23,  1.78s/it]
Outer training loop:  19%|#8        | 9330/50000 [4:47:23<19:44:56,  1.75s/it]
Outer training loop:  19%|#8        | 9331/50000 [4:47:25<21:09:48,  1.87s/it]
Outer training loop:  19%|#8        | 9332/50000 [4:47:27<20:39:03,  1.83s/it]
Outer training loop:  19%|#8        | 9333/50000 [4:47:29<20:15:17,  1.79s/it]
Outer training loop:  19%|#8        | 9334/50000 [4:47:30<19:57:06,  1.77s/it]
Outer training loop:  19%|#8        | 9335/50000 [4:47:32<20:15:54,  1.79s/it]
Outer training loop:  19%|#8        | 9336/50000 [4:47:34<19:55:15,  1.76s/it]
Outer training loop:  19%|#8        | 9337/50000 [4:47:36<19:38:32,  1.74s/it]
Outer training loop:  19%|#8        | 9338/50000 [4:47:38<20:09:05,  1.78s/it]
Outer training loop:  19%|#8        | 9339/50000 [4:47:39<20:09:06,  1.78s/it]
Outer training loop:  19%|#8        | 9340/50000 [4:47:41<20:22:45,  1.80s/it]
Outer training loop:  19%|#8        | 9341/50000 [4:47:43<21:00:06,  1.86s/it]
Outer training loop:  19%|#8        | 9342/50000 [4:47:45<20:39:54,  1.83s/it]
Outer training loop:  19%|#8        | 9343/50000 [4:47:47<20:26:32,  1.81s/it]
Outer training loop:  19%|#8        | 9344/50000 [4:47:48<20:14:49,  1.79s/it]
Outer training loop:  19%|#8        | 9345/50000 [4:47:50<20:28:55,  1.81s/it]
Outer training loop:  19%|#8        | 9346/50000 [4:47:52<20:21:08,  1.80s/it]
Outer training loop:  19%|#8        | 9347/50000 [4:47:54<19:55:20,  1.76s/it]
Outer training loop:  19%|#8        | 9348/50000 [4:47:56<20:31:15,  1.82s/it]
Outer training loop:  19%|#8        | 9349/50000 [4:47:58<20:40:11,  1.83s/it]
Outer training loop:  19%|#8        | 9350/50000 [4:48:00<20:56:54,  1.86s/it]
Outer training loop:  19%|#8        | 9351/50000 [4:48:01<20:39:03,  1.83s/it]
Outer training loop:  19%|#8        | 9352/50000 [4:48:03<20:12:24,  1.79s/it]
Outer training loop:  19%|#8        | 9353/50000 [4:48:05<19:51:07,  1.76s/it]
Outer training loop:  19%|#8        | 9354/50000 [4:48:06<20:05:36,  1.78s/it]
Outer training loop:  19%|#8        | 9355/50000 [4:48:08<20:08:44,  1.78s/it]
Outer training loop:  19%|#8        | 9356/50000 [4:48:10<20:03:21,  1.78s/it]
Outer training loop:  19%|#8        | 9357/50000 [4:48:12<20:20:32,  1.80s/it]
Outer training loop:  19%|#8        | 9358/50000 [4:48:14<20:11:17,  1.79s/it]
Outer training loop:  19%|#8        | 9359/50000 [4:48:15<19:52:12,  1.76s/it]
Outer training loop:  19%|#8        | 9360/50000 [4:48:17<19:47:25,  1.75s/it]
Outer training loop:  19%|#8        | 9361/50000 [4:48:19<20:37:00,  1.83s/it]
Outer training loop:  19%|#8        | 9362/50000 [4:48:21<20:12:52,  1.79s/it]
Outer training loop:  19%|#8        | 9363/50000 [4:48:23<19:54:41,  1.76s/it]
Outer training loop:  19%|#8        | 9364/50000 [4:48:24<20:27:43,  1.81s/it]
Outer training loop:  19%|#8        | 9365/50000 [4:48:26<20:15:07,  1.79s/it]
Outer training loop:  19%|#8        | 9366/50000 [4:48:28<20:09:20,  1.79s/it]
Outer training loop:  19%|#8        | 9367/50000 [4:48:30<20:28:42,  1.81s/it]
Outer training loop:  19%|#8        | 9368/50000 [4:48:31<19:58:09,  1.77s/it]
Outer training loop:  19%|#8        | 9369/50000 [4:48:33<19:51:30,  1.76s/it]
Outer training loop:  19%|#8        | 9370/50000 [4:48:35<19:44:49,  1.75s/it]
Outer training loop:  19%|#8        | 9371/50000 [4:48:37<20:15:24,  1.79s/it]
Outer training loop:  19%|#8        | 9372/50000 [4:48:39<19:56:43,  1.77s/it]
Outer training loop:  19%|#8        | 9373/50000 [4:48:40<19:44:41,  1.75s/it]
Outer training loop:  19%|#8        | 9374/50000 [4:48:42<20:24:13,  1.81s/it]
Outer training loop:  19%|#8        | 9375/50000 [4:48:44<19:56:04,  1.77s/it]
Outer training loop:  19%|#8        | 9376/50000 [4:48:46<19:52:39,  1.76s/it]
Outer training loop:  19%|#8        | 9377/50000 [4:48:48<20:28:18,  1.81s/it]
Outer training loop:  19%|#8        | 9378/50000 [4:48:49<20:04:32,  1.78s/it]
Outer training loop:  19%|#8        | 9379/50000 [4:48:51<19:40:20,  1.74s/it]
Outer training loop:  19%|#8        | 9380/50000 [4:48:53<19:57:03,  1.77s/it]
Outer training loop:  19%|#8        | 9381/50000 [4:48:55<20:17:25,  1.80s/it]
Outer training loop:  19%|#8        | 9382/50000 [4:48:56<20:18:24,  1.80s/it]
Outer training loop:  19%|#8        | 9383/50000 [4:48:58<20:24:23,  1.81s/it]
Outer training loop:  19%|#8        | 9384/50000 [4:49:00<20:13:41,  1.79s/it]
Outer training loop:  19%|#8        | 9385/50000 [4:49:02<19:48:04,  1.76s/it]
Outer training loop:  19%|#8        | 9386/50000 [4:49:03<19:32:44,  1.73s/it]
Outer training loop:  19%|#8        | 9387/50000 [4:49:05<20:19:27,  1.80s/it]
Outer training loop:  19%|#8        | 9388/50000 [4:49:07<20:09:13,  1.79s/it]
Outer training loop:  19%|#8        | 9389/50000 [4:49:09<20:06:41,  1.78s/it]
Outer training loop:  19%|#8        | 9390/50000 [4:49:11<20:15:05,  1.80s/it]
Outer training loop:  19%|#8        | 9391/50000 [4:49:12<19:49:31,  1.76s/it]
Outer training loop:  19%|#8        | 9392/50000 [4:49:14<19:33:23,  1.73s/it]
Outer training loop:  19%|#8        | 9393/50000 [4:49:16<20:04:03,  1.78s/it]
Outer training loop:  19%|#8        | 9394/50000 [4:49:18<19:56:49,  1.77s/it]
Outer training loop:  19%|#8        | 9395/50000 [4:49:19<19:53:11,  1.76s/it]
Outer training loop:  19%|#8        | 9396/50000 [4:49:21<19:56:59,  1.77s/it]
Outer training loop:  19%|#8        | 9397/50000 [4:49:23<20:16:20,  1.80s/it]
Outer training loop:  19%|#8        | 9398/50000 [4:49:25<20:35:29,  1.83s/it]
Outer training loop:  19%|#8        | 9399/50000 [4:49:27<19:54:32,  1.77s/it]
Outer training loop:  19%|#8        | 9400/50000 [4:49:29<20:43:05,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.67s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  19%|#8        | 9401/50000 [4:49:36<38:01:47,  3.37s/it]
Outer training loop:  19%|#8        | 9402/50000 [4:49:37<32:40:33,  2.90s/it]
Outer training loop:  19%|#8        | 9403/50000 [4:49:39<28:53:32,  2.56s/it]
Outer training loop:  19%|#8        | 9404/50000 [4:49:41<26:42:17,  2.37s/it]
Outer training loop:  19%|#8        | 9405/50000 [4:49:43<24:25:13,  2.17s/it]
Outer training loop:  19%|#8        | 9406/50000 [4:49:44<22:34:05,  2.00s/it]
Outer training loop:  19%|#8        | 9407/50000 [4:49:46<21:29:29,  1.91s/it]
Outer training loop:  19%|#8        | 9408/50000 [4:49:48<21:19:09,  1.89s/it]
Outer training loop:  19%|#8        | 9409/50000 [4:49:50<20:37:47,  1.83s/it]
Outer training loop:  19%|#8        | 9410/50000 [4:49:51<20:27:58,  1.82s/it]
Outer training loop:  19%|#8        | 9411/50000 [4:49:53<20:32:31,  1.82s/it]
Outer training loop:  19%|#8        | 9412/50000 [4:49:55<20:01:52,  1.78s/it]
Outer training loop:  19%|#8        | 9413/50000 [4:49:57<19:49:09,  1.76s/it]
Outer training loop:  19%|#8        | 9414/50000 [4:49:58<19:38:51,  1.74s/it]
Outer training loop:  19%|#8        | 9415/50000 [4:50:00<20:23:19,  1.81s/it]
Outer training loop:  19%|#8        | 9416/50000 [4:50:02<19:49:49,  1.76s/it]
Outer training loop:  19%|#8        | 9417/50000 [4:50:04<19:32:20,  1.73s/it]
Outer training loop:  19%|#8        | 9418/50000 [4:50:05<20:04:53,  1.78s/it]
Outer training loop:  19%|#8        | 9419/50000 [4:50:07<19:33:49,  1.74s/it]
Outer training loop:  19%|#8        | 9420/50000 [4:50:09<19:19:59,  1.72s/it]
Outer training loop:  19%|#8        | 9421/50000 [4:50:11<19:49:33,  1.76s/it]
Outer training loop:  19%|#8        | 9422/50000 [4:50:12<19:15:44,  1.71s/it]
Outer training loop:  19%|#8        | 9423/50000 [4:50:14<19:16:33,  1.71s/it]
Outer training loop:  19%|#8        | 9424/50000 [4:50:16<20:10:46,  1.79s/it]
Outer training loop:  19%|#8        | 9425/50000 [4:50:17<19:38:36,  1.74s/it]
Outer training loop:  19%|#8        | 9426/50000 [4:50:19<19:44:05,  1.75s/it]
Outer training loop:  19%|#8        | 9427/50000 [4:50:21<19:31:43,  1.73s/it]
Outer training loop:  19%|#8        | 9428/50000 [4:50:23<20:45:18,  1.84s/it]
Outer training loop:  19%|#8        | 9429/50000 [4:50:25<20:44:50,  1.84s/it]
Outer training loop:  19%|#8        | 9430/50000 [4:50:27<19:59:50,  1.77s/it]
Outer training loop:  19%|#8        | 9431/50000 [4:50:29<20:54:49,  1.86s/it]
Outer training loop:  19%|#8        | 9432/50000 [4:50:30<20:47:00,  1.84s/it]
Outer training loop:  19%|#8        | 9433/50000 [4:50:32<20:18:58,  1.80s/it]
Outer training loop:  19%|#8        | 9434/50000 [4:50:34<20:12:26,  1.79s/it]
Outer training loop:  19%|#8        | 9435/50000 [4:50:36<20:18:14,  1.80s/it]
Outer training loop:  19%|#8        | 9436/50000 [4:50:37<19:46:29,  1.75s/it]
Outer training loop:  19%|#8        | 9437/50000 [4:50:39<19:29:54,  1.73s/it]
Outer training loop:  19%|#8        | 9438/50000 [4:50:41<20:12:37,  1.79s/it]
Outer training loop:  19%|#8        | 9439/50000 [4:50:43<19:47:23,  1.76s/it]
Outer training loop:  19%|#8        | 9440/50000 [4:50:44<19:46:32,  1.76s/it]
Outer training loop:  19%|#8        | 9441/50000 [4:50:46<20:00:04,  1.78s/it]
Outer training loop:  19%|#8        | 9442/50000 [4:50:48<20:36:02,  1.83s/it]
Outer training loop:  19%|#8        | 9443/50000 [4:50:50<20:32:28,  1.82s/it]
Outer training loop:  19%|#8        | 9444/50000 [4:50:52<20:41:20,  1.84s/it]
Outer training loop:  19%|#8        | 9445/50000 [4:50:54<20:51:09,  1.85s/it]
Outer training loop:  19%|#8        | 9446/50000 [4:50:55<20:33:04,  1.82s/it]
Outer training loop:  19%|#8        | 9447/50000 [4:50:57<20:04:05,  1.78s/it]
Outer training loop:  19%|#8        | 9448/50000 [4:50:59<20:39:46,  1.83s/it]
Outer training loop:  19%|#8        | 9449/50000 [4:51:01<20:27:31,  1.82s/it]
Outer training loop:  19%|#8        | 9450/50000 [4:51:03<20:03:13,  1.78s/it]
Outer training loop:  19%|#8        | 9451/50000 [4:51:05<20:37:11,  1.83s/it]
Outer training loop:  19%|#8        | 9452/50000 [4:51:06<20:19:21,  1.80s/it]
Outer training loop:  19%|#8        | 9453/50000 [4:51:08<19:50:38,  1.76s/it]
Outer training loop:  19%|#8        | 9454/50000 [4:51:10<19:28:57,  1.73s/it]
Outer training loop:  19%|#8        | 9455/50000 [4:51:11<19:55:44,  1.77s/it]
Outer training loop:  19%|#8        | 9456/50000 [4:51:13<19:36:31,  1.74s/it]
Outer training loop:  19%|#8        | 9457/50000 [4:51:15<19:33:48,  1.74s/it]
Outer training loop:  19%|#8        | 9458/50000 [4:51:17<19:55:06,  1.77s/it]
Outer training loop:  19%|#8        | 9459/50000 [4:51:18<20:00:51,  1.78s/it]
Outer training loop:  19%|#8        | 9460/50000 [4:51:20<19:32:45,  1.74s/it]
Outer training loop:  19%|#8        | 9461/50000 [4:51:22<19:59:44,  1.78s/it]
Outer training loop:  19%|#8        | 9462/50000 [4:51:24<19:45:37,  1.75s/it]
Outer training loop:  19%|#8        | 9463/50000 [4:51:25<19:34:33,  1.74s/it]
Outer training loop:  19%|#8        | 9464/50000 [4:51:27<19:38:02,  1.74s/it]
Outer training loop:  19%|#8        | 9465/50000 [4:51:29<20:21:31,  1.81s/it]
Outer training loop:  19%|#8        | 9466/50000 [4:51:31<19:55:49,  1.77s/it]
Outer training loop:  19%|#8        | 9467/50000 [4:51:32<19:21:38,  1.72s/it]
Outer training loop:  19%|#8        | 9468/50000 [4:51:34<19:52:01,  1.76s/it]
Outer training loop:  19%|#8        | 9469/50000 [4:51:36<19:30:28,  1.73s/it]
Outer training loop:  19%|#8        | 9470/50000 [4:51:38<19:06:17,  1.70s/it]
Outer training loop:  19%|#8        | 9471/50000 [4:51:39<19:38:23,  1.74s/it]
Outer training loop:  19%|#8        | 9472/50000 [4:51:41<19:20:38,  1.72s/it]
Outer training loop:  19%|#8        | 9473/50000 [4:51:43<19:21:59,  1.72s/it]
Outer training loop:  19%|#8        | 9474/50000 [4:51:44<19:02:14,  1.69s/it]
Outer training loop:  19%|#8        | 9475/50000 [4:51:46<19:27:20,  1.73s/it]
Outer training loop:  19%|#8        | 9476/50000 [4:51:48<19:19:38,  1.72s/it]
Outer training loop:  19%|#8        | 9477/50000 [4:51:50<19:22:13,  1.72s/it]
Outer training loop:  19%|#8        | 9478/50000 [4:51:52<20:32:49,  1.83s/it]
Outer training loop:  19%|#8        | 9479/50000 [4:51:53<20:00:49,  1.78s/it]
Outer training loop:  19%|#8        | 9480/50000 [4:51:55<19:41:30,  1.75s/it]
Outer training loop:  19%|#8        | 9481/50000 [4:51:57<19:31:26,  1.73s/it]
Outer training loop:  19%|#8        | 9482/50000 [4:51:59<20:14:11,  1.80s/it]
Outer training loop:  19%|#8        | 9483/50000 [4:52:00<20:03:59,  1.78s/it]
Outer training loop:  19%|#8        | 9484/50000 [4:52:02<19:47:57,  1.76s/it]
Outer training loop:  19%|#8        | 9485/50000 [4:52:04<20:16:47,  1.80s/it]
Outer training loop:  19%|#8        | 9486/50000 [4:52:06<19:56:52,  1.77s/it]
Outer training loop:  19%|#8        | 9487/50000 [4:52:07<19:38:07,  1.74s/it]
Outer training loop:  19%|#8        | 9488/50000 [4:52:09<19:53:00,  1.77s/it]
Outer training loop:  19%|#8        | 9489/50000 [4:52:11<19:33:18,  1.74s/it]
Outer training loop:  19%|#8        | 9490/50000 [4:52:13<19:35:56,  1.74s/it]
Outer training loop:  19%|#8        | 9491/50000 [4:52:14<19:34:17,  1.74s/it]
Outer training loop:  19%|#8        | 9492/50000 [4:52:16<20:10:50,  1.79s/it]
Outer training loop:  19%|#8        | 9493/50000 [4:52:18<20:14:45,  1.80s/it]
Outer training loop:  19%|#8        | 9494/50000 [4:52:20<20:03:45,  1.78s/it]
Outer training loop:  19%|#8        | 9495/50000 [4:52:22<20:38:58,  1.84s/it]
Outer training loop:  19%|#8        | 9496/50000 [4:52:24<20:12:15,  1.80s/it]
Outer training loop:  19%|#8        | 9497/50000 [4:52:25<20:07:27,  1.79s/it]
Outer training loop:  19%|#8        | 9498/50000 [4:52:27<20:05:51,  1.79s/it]
Outer training loop:  19%|#8        | 9499/50000 [4:52:29<20:05:20,  1.79s/it]
Outer training loop:  19%|#9        | 9500/50000 [4:52:31<20:02:41,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:31,  1.85s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:26,  1.81s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.80s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:18,  1.76s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:20,  1.80s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.77s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:12,  1.75s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.80s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:14,  1.82s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:09,  1.78s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:07,  1.77s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.83s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:03,  1.77s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:01,  1.77s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<01:58,  1.75s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:55,  1.72s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:57,  1.78s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:53,  1.75s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:52,  1.76s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:51,  1.76s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:52,  1.81s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:48,  1.78s/it]

Evaluating policy:  28%|##7       | 23/83 [00:40<01:46,  1.78s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.79s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:40,  1.76s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:43,  1.85s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:40,  1.83s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:35,  1.77s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.77s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:32,  1.78s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:35,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:30,  1.82s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:26,  1.77s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:23,  1.74s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:22,  1.75s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:19,  1.72s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:21,  1.81s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:19,  1.80s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:15,  1.75s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:13,  1.76s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:12,  1.78s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:09,  1.75s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:07,  1.72s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:04,  1.75s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:01,  1.72s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<00:59,  1.71s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<00:57,  1.68s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:55,  1.68s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:53,  1.69s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:56,  1.81s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:52,  1.76s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:49,  1.71s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:47,  1.70s/it]

Evaluating policy:  67%|######7   | 56/83 [01:38<00:45,  1.70s/it]

Evaluating policy:  69%|######8   | 57/83 [01:40<00:44,  1.73s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:43,  1.73s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:43<00:40,  1.70s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:42,  1.85s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:47<00:40,  1.84s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:37,  1.80s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:35,  1.76s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:52<00:32,  1.73s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:54<00:30,  1.70s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:29,  1.72s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:27,  1.70s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:01<00:24,  1.77s/it]

Evaluating policy:  84%|########4 | 70/83 [02:03<00:22,  1.75s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:20,  1.73s/it]

Evaluating policy:  87%|########6 | 72/83 [02:06<00:18,  1.72s/it]

Evaluating policy:  88%|########7 | 73/83 [02:08<00:17,  1.71s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:15,  1.77s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:14,  1.78s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.76s/it]

Evaluating policy:  93%|#########2| 77/83 [02:15<00:10,  1.73s/it]

Evaluating policy:  94%|#########3| 78/83 [02:17<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:19<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.86s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.81s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 83/83 [02:26<00:00,  1.71s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:15,  1.65s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:14,  1.66s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:13,  1.67s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.89s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:22,  1.83s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:19,  1.81s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:15,  1.79s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:16,  1.82s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:12,  1.79s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:07,  1.75s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:06,  1.76s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.82s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:04,  1.78s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:02,  1.77s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<01:59,  1.76s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:02,  1.83s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:59,  1.81s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:52,  1.75s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:55,  1.83s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:50,  1.78s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:46,  1.75s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:44,  1.74s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:41,  1.72s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:40,  1.74s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:43,  1.82s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:39,  1.77s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:36,  1.76s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:33,  1.73s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:31,  1.73s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:33,  1.80s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:30,  1.77s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:27,  1.75s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:24,  1.73s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:23,  1.73s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:26,  1.85s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:23,  1.81s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:19,  1.76s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.74s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:15,  1.76s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:15,  1.80s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:13,  1.84s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:10,  1.80s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:05,  1.77s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:04,  1.78s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:02,  1.78s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:05,  1.92s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<01:02,  1.90s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:57,  1.85s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:55,  1.84s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:52,  1.80s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:50,  1.81s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:51,  1.91s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:45,  1.83s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:43,  1.81s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:41,  1.79s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:38,  1.76s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:36,  1.73s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:34,  1.73s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:55<00:35,  1.89s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:32,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:30,  1.80s/it]

Evaluating policy:  81%|########  | 67/83 [02:00<00:28,  1.81s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:26,  1.77s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:24,  1.78s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:23,  1.78s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:21,  1.76s/it]

Evaluating policy:  87%|########6 | 72/83 [02:09<00:20,  1.87s/it]

Evaluating policy:  88%|########7 | 73/83 [02:11<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:16,  1.79s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:14,  1.79s/it]

Evaluating policy:  92%|#########1| 76/83 [02:16<00:12,  1.78s/it]

Evaluating policy:  93%|#########2| 77/83 [02:18<00:10,  1.77s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.75s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.81s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.81s/it]

Evaluating policy:  98%|#########7| 81/83 [02:25<00:03,  1.94s/it]

Evaluating policy:  99%|#########8| 82/83 [02:27<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:29<00:00,  1.86s/it]

Outer training loop:  19%|#9        | 9501/50000 [4:57:34<1037:10:05, 92.20s/it]
Outer training loop:  19%|#9        | 9502/50000 [4:57:36<731:45:17, 65.05s/it] 
Outer training loop:  19%|#9        | 9503/50000 [4:57:37<518:18:13, 46.07s/it]
Outer training loop:  19%|#9        | 9504/50000 [4:57:39<368:31:19, 32.76s/it]
Outer training loop:  19%|#9        | 9505/50000 [4:57:41<263:41:23, 23.44s/it]
Outer training loop:  19%|#9        | 9506/50000 [4:57:42<190:14:22, 16.91s/it]
Outer training loop:  19%|#9        | 9507/50000 [4:57:44<139:40:47, 12.42s/it]
Outer training loop:  19%|#9        | 9508/50000 [4:57:46<103:26:47,  9.20s/it]
Outer training loop:  19%|#9        | 9509/50000 [4:57:49<80:53:09,  7.19s/it] 
Outer training loop:  19%|#9        | 9510/50000 [4:57:50<62:34:16,  5.56s/it]
Outer training loop:  19%|#9        | 9511/50000 [4:57:52<49:52:05,  4.43s/it]
Outer training loop:  19%|#9        | 9512/50000 [4:57:54<40:56:54,  3.64s/it]
Outer training loop:  19%|#9        | 9513/50000 [4:57:56<34:43:27,  3.09s/it]
Outer training loop:  19%|#9        | 9514/50000 [4:57:57<30:07:22,  2.68s/it]
Outer training loop:  19%|#9        | 9515/50000 [4:57:59<27:23:07,  2.44s/it]
Outer training loop:  19%|#9        | 9516/50000 [4:58:01<25:47:18,  2.29s/it]
Outer training loop:  19%|#9        | 9517/50000 [4:58:03<24:08:44,  2.15s/it]
Outer training loop:  19%|#9        | 9518/50000 [4:58:05<22:37:02,  2.01s/it]
Outer training loop:  19%|#9        | 9519/50000 [4:58:07<22:09:11,  1.97s/it]
Outer training loop:  19%|#9        | 9520/50000 [4:58:08<21:29:06,  1.91s/it]
Outer training loop:  19%|#9        | 9521/50000 [4:58:10<21:02:29,  1.87s/it]
Outer training loop:  19%|#9        | 9522/50000 [4:58:12<21:22:32,  1.90s/it]
Outer training loop:  19%|#9        | 9523/50000 [4:58:14<20:29:57,  1.82s/it]
Outer training loop:  19%|#9        | 9524/50000 [4:58:15<20:06:05,  1.79s/it]
Outer training loop:  19%|#9        | 9525/50000 [4:58:17<20:16:51,  1.80s/it]
Outer training loop:  19%|#9        | 9526/50000 [4:58:19<19:56:58,  1.77s/it]
Outer training loop:  19%|#9        | 9527/50000 [4:58:21<20:08:00,  1.79s/it]
Outer training loop:  19%|#9        | 9528/50000 [4:58:23<19:52:08,  1.77s/it]
Outer training loop:  19%|#9        | 9529/50000 [4:58:24<20:21:13,  1.81s/it]
Outer training loop:  19%|#9        | 9530/50000 [4:58:26<20:09:59,  1.79s/it]
Outer training loop:  19%|#9        | 9531/50000 [4:58:28<19:55:32,  1.77s/it]
Outer training loop:  19%|#9        | 9532/50000 [4:58:30<20:26:09,  1.82s/it]
Outer training loop:  19%|#9        | 9533/50000 [4:58:32<20:04:50,  1.79s/it]
Outer training loop:  19%|#9        | 9534/50000 [4:58:33<20:12:14,  1.80s/it]
Outer training loop:  19%|#9        | 9535/50000 [4:58:35<19:52:35,  1.77s/it]
Outer training loop:  19%|#9        | 9536/50000 [4:58:37<20:03:29,  1.78s/it]
Outer training loop:  19%|#9        | 9537/50000 [4:58:39<20:02:33,  1.78s/it]
Outer training loop:  19%|#9        | 9538/50000 [4:58:40<19:45:45,  1.76s/it]
Outer training loop:  19%|#9        | 9539/50000 [4:58:42<20:29:07,  1.82s/it]
Outer training loop:  19%|#9        | 9540/50000 [4:58:44<20:00:29,  1.78s/it]
Outer training loop:  19%|#9        | 9541/50000 [4:58:46<19:35:18,  1.74s/it]
Outer training loop:  19%|#9        | 9542/50000 [4:58:48<19:58:50,  1.78s/it]
Outer training loop:  19%|#9        | 9543/50000 [4:58:49<19:55:40,  1.77s/it]
Outer training loop:  19%|#9        | 9544/50000 [4:58:51<20:07:49,  1.79s/it]
Outer training loop:  19%|#9        | 9545/50000 [4:58:53<19:51:00,  1.77s/it]
Outer training loop:  19%|#9        | 9546/50000 [4:58:55<20:13:03,  1.80s/it]
Outer training loop:  19%|#9        | 9547/50000 [4:58:56<19:46:08,  1.76s/it]
Outer training loop:  19%|#9        | 9548/50000 [4:58:58<19:30:03,  1.74s/it]
Outer training loop:  19%|#9        | 9549/50000 [4:59:00<19:41:31,  1.75s/it]
Outer training loop:  19%|#9        | 9550/50000 [4:59:02<19:21:18,  1.72s/it]
Outer training loop:  19%|#9        | 9551/50000 [4:59:03<19:14:15,  1.71s/it]
Outer training loop:  19%|#9        | 9552/50000 [4:59:05<20:16:03,  1.80s/it]
Outer training loop:  19%|#9        | 9553/50000 [4:59:07<19:53:38,  1.77s/it]
Outer training loop:  19%|#9        | 9554/50000 [4:59:09<19:20:29,  1.72s/it]
Outer training loop:  19%|#9        | 9555/50000 [4:59:10<19:05:43,  1.70s/it]
Outer training loop:  19%|#9        | 9556/50000 [4:59:12<19:37:35,  1.75s/it]
Outer training loop:  19%|#9        | 9557/50000 [4:59:14<18:59:05,  1.69s/it]
Outer training loop:  19%|#9        | 9558/50000 [4:59:15<18:51:33,  1.68s/it]
Outer training loop:  19%|#9        | 9559/50000 [4:59:17<20:28:29,  1.82s/it]
Outer training loop:  19%|#9        | 9560/50000 [4:59:19<20:02:07,  1.78s/it]
Outer training loop:  19%|#9        | 9561/50000 [4:59:21<19:45:52,  1.76s/it]
Outer training loop:  19%|#9        | 9562/50000 [4:59:23<19:52:20,  1.77s/it]
Outer training loop:  19%|#9        | 9563/50000 [4:59:25<20:31:25,  1.83s/it]
Outer training loop:  19%|#9        | 9564/50000 [4:59:26<20:20:56,  1.81s/it]
Outer training loop:  19%|#9        | 9565/50000 [4:59:28<20:10:36,  1.80s/it]
Outer training loop:  19%|#9        | 9566/50000 [4:59:30<20:19:45,  1.81s/it]
Outer training loop:  19%|#9        | 9567/50000 [4:59:32<20:03:37,  1.79s/it]
Outer training loop:  19%|#9        | 9568/50000 [4:59:33<19:54:01,  1.77s/it]
Outer training loop:  19%|#9        | 9569/50000 [4:59:35<20:39:22,  1.84s/it]
Outer training loop:  19%|#9        | 9570/50000 [4:59:37<20:11:22,  1.80s/it]
Outer training loop:  19%|#9        | 9571/50000 [4:59:39<20:11:43,  1.80s/it]
Outer training loop:  19%|#9        | 9572/50000 [4:59:41<20:31:24,  1.83s/it]
Outer training loop:  19%|#9        | 9573/50000 [4:59:43<20:34:37,  1.83s/it]
Outer training loop:  19%|#9        | 9574/50000 [4:59:44<20:02:39,  1.78s/it]
Outer training loop:  19%|#9        | 9575/50000 [4:59:46<19:39:09,  1.75s/it]
Outer training loop:  19%|#9        | 9576/50000 [4:59:48<20:26:38,  1.82s/it]
Outer training loop:  19%|#9        | 9577/50000 [4:59:50<20:13:26,  1.80s/it]
Outer training loop:  19%|#9        | 9578/50000 [4:59:51<19:39:00,  1.75s/it]
Outer training loop:  19%|#9        | 9579/50000 [4:59:53<19:34:33,  1.74s/it]
Outer training loop:  19%|#9        | 9580/50000 [4:59:55<20:23:26,  1.82s/it]
Outer training loop:  19%|#9        | 9581/50000 [4:59:57<20:08:47,  1.79s/it]
Outer training loop:  19%|#9        | 9582/50000 [4:59:59<19:54:43,  1.77s/it]
Outer training loop:  19%|#9        | 9583/50000 [5:00:00<20:16:53,  1.81s/it]
Outer training loop:  19%|#9        | 9584/50000 [5:00:02<20:17:01,  1.81s/it]
Outer training loop:  19%|#9        | 9585/50000 [5:00:04<19:50:16,  1.77s/it]
Outer training loop:  19%|#9        | 9586/50000 [5:00:06<20:14:28,  1.80s/it]
Outer training loop:  19%|#9        | 9587/50000 [5:00:07<19:47:17,  1.76s/it]
Outer training loop:  19%|#9        | 9588/50000 [5:00:09<19:25:55,  1.73s/it]
Outer training loop:  19%|#9        | 9589/50000 [5:00:11<19:34:31,  1.74s/it]
Outer training loop:  19%|#9        | 9590/50000 [5:00:13<19:55:12,  1.77s/it]
Outer training loop:  19%|#9        | 9591/50000 [5:00:14<19:42:07,  1.76s/it]
Outer training loop:  19%|#9        | 9592/50000 [5:00:16<19:43:23,  1.76s/it]
Outer training loop:  19%|#9        | 9593/50000 [5:00:18<20:00:20,  1.78s/it]
Outer training loop:  19%|#9        | 9594/50000 [5:00:20<19:52:08,  1.77s/it]
Outer training loop:  19%|#9        | 9595/50000 [5:00:21<19:24:14,  1.73s/it]
Outer training loop:  19%|#9        | 9596/50000 [5:00:23<20:00:14,  1.78s/it]
Outer training loop:  19%|#9        | 9597/50000 [5:00:25<19:57:25,  1.78s/it]
Outer training loop:  19%|#9        | 9598/50000 [5:00:27<20:01:50,  1.78s/it]
Outer training loop:  19%|#9        | 9599/50000 [5:00:29<20:35:23,  1.83s/it]
Outer training loop:  19%|#9        | 9600/50000 [5:00:31<20:28:18,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  19%|#9        | 9601/50000 [5:00:38<37:58:42,  3.38s/it]
Outer training loop:  19%|#9        | 9602/50000 [5:00:39<32:13:01,  2.87s/it]
Outer training loop:  19%|#9        | 9603/50000 [5:00:41<28:51:27,  2.57s/it]
Outer training loop:  19%|#9        | 9604/50000 [5:00:43<25:45:24,  2.30s/it]
Outer training loop:  19%|#9        | 9605/50000 [5:00:45<23:33:37,  2.10s/it]
Outer training loop:  19%|#9        | 9606/50000 [5:00:46<23:03:15,  2.05s/it]
Outer training loop:  19%|#9        | 9607/50000 [5:00:48<21:54:28,  1.95s/it]
Outer training loop:  19%|#9        | 9608/50000 [5:00:50<20:46:28,  1.85s/it]
Outer training loop:  19%|#9        | 9609/50000 [5:00:52<20:59:50,  1.87s/it]
Outer training loop:  19%|#9        | 9610/50000 [5:00:53<20:17:50,  1.81s/it]
Outer training loop:  19%|#9        | 9611/50000 [5:00:55<20:10:16,  1.80s/it]
Outer training loop:  19%|#9        | 9612/50000 [5:00:57<19:41:36,  1.76s/it]
Outer training loop:  19%|#9        | 9613/50000 [5:00:59<20:36:18,  1.84s/it]
Outer training loop:  19%|#9        | 9614/50000 [5:01:00<19:56:03,  1.78s/it]
Outer training loop:  19%|#9        | 9615/50000 [5:01:02<20:10:22,  1.80s/it]
Outer training loop:  19%|#9        | 9616/50000 [5:01:04<20:55:15,  1.86s/it]
Outer training loop:  19%|#9        | 9617/50000 [5:01:06<20:36:43,  1.84s/it]
Outer training loop:  19%|#9        | 9618/50000 [5:01:08<19:52:22,  1.77s/it]
Outer training loop:  19%|#9        | 9619/50000 [5:01:10<19:59:20,  1.78s/it]
Outer training loop:  19%|#9        | 9620/50000 [5:01:11<20:13:36,  1.80s/it]
Outer training loop:  19%|#9        | 9621/50000 [5:01:13<20:16:09,  1.81s/it]
Outer training loop:  19%|#9        | 9622/50000 [5:01:15<19:31:44,  1.74s/it]
Outer training loop:  19%|#9        | 9623/50000 [5:01:17<20:18:09,  1.81s/it]
Outer training loop:  19%|#9        | 9624/50000 [5:01:19<20:01:37,  1.79s/it]
Outer training loop:  19%|#9        | 9625/50000 [5:01:20<19:43:29,  1.76s/it]
Outer training loop:  19%|#9        | 9626/50000 [5:01:22<19:18:07,  1.72s/it]
Outer training loop:  19%|#9        | 9627/50000 [5:01:24<19:54:56,  1.78s/it]
Outer training loop:  19%|#9        | 9628/50000 [5:01:25<19:49:38,  1.77s/it]
Outer training loop:  19%|#9        | 9629/50000 [5:01:27<20:03:39,  1.79s/it]
Outer training loop:  19%|#9        | 9630/50000 [5:01:29<20:10:41,  1.80s/it]
Outer training loop:  19%|#9        | 9631/50000 [5:01:31<19:57:32,  1.78s/it]
Outer training loop:  19%|#9        | 9632/50000 [5:01:33<19:51:38,  1.77s/it]
Outer training loop:  19%|#9        | 9633/50000 [5:01:35<20:28:48,  1.83s/it]
Outer training loop:  19%|#9        | 9634/50000 [5:01:36<20:12:46,  1.80s/it]
Outer training loop:  19%|#9        | 9635/50000 [5:01:38<19:58:26,  1.78s/it]
Outer training loop:  19%|#9        | 9636/50000 [5:01:40<20:21:44,  1.82s/it]
Outer training loop:  19%|#9        | 9637/50000 [5:01:42<19:39:24,  1.75s/it]
Outer training loop:  19%|#9        | 9638/50000 [5:01:43<19:06:00,  1.70s/it]
Outer training loop:  19%|#9        | 9639/50000 [5:01:45<19:40:21,  1.75s/it]
Outer training loop:  19%|#9        | 9640/50000 [5:01:47<19:20:39,  1.73s/it]
Outer training loop:  19%|#9        | 9641/50000 [5:01:48<19:15:05,  1.72s/it]
Outer training loop:  19%|#9        | 9642/50000 [5:01:50<19:16:38,  1.72s/it]
Outer training loop:  19%|#9        | 9643/50000 [5:01:52<19:36:01,  1.75s/it]
Outer training loop:  19%|#9        | 9644/50000 [5:01:54<19:54:03,  1.78s/it]
Outer training loop:  19%|#9        | 9645/50000 [5:01:55<19:35:56,  1.75s/it]
Outer training loop:  19%|#9        | 9646/50000 [5:01:57<19:53:26,  1.77s/it]
Outer training loop:  19%|#9        | 9647/50000 [5:01:59<19:16:16,  1.72s/it]
Outer training loop:  19%|#9        | 9648/50000 [5:02:01<19:21:03,  1.73s/it]
Outer training loop:  19%|#9        | 9649/50000 [5:02:02<19:24:46,  1.73s/it]
Outer training loop:  19%|#9        | 9650/50000 [5:02:04<20:11:17,  1.80s/it]
Outer training loop:  19%|#9        | 9651/50000 [5:02:06<20:17:40,  1.81s/it]
Outer training loop:  19%|#9        | 9652/50000 [5:02:08<20:14:47,  1.81s/it]
Outer training loop:  19%|#9        | 9653/50000 [5:02:10<20:25:52,  1.82s/it]
Outer training loop:  19%|#9        | 9654/50000 [5:02:12<19:56:32,  1.78s/it]
Outer training loop:  19%|#9        | 9655/50000 [5:02:13<19:24:47,  1.73s/it]
Outer training loop:  19%|#9        | 9656/50000 [5:02:15<19:54:18,  1.78s/it]
Outer training loop:  19%|#9        | 9657/50000 [5:02:17<20:45:48,  1.85s/it]
Outer training loop:  19%|#9        | 9658/50000 [5:02:19<20:44:32,  1.85s/it]
Outer training loop:  19%|#9        | 9659/50000 [5:02:21<20:33:00,  1.83s/it]
Outer training loop:  19%|#9        | 9660/50000 [5:02:23<20:33:02,  1.83s/it]
Outer training loop:  19%|#9        | 9661/50000 [5:02:24<19:54:21,  1.78s/it]
Outer training loop:  19%|#9        | 9662/50000 [5:02:26<19:47:40,  1.77s/it]
Outer training loop:  19%|#9        | 9663/50000 [5:02:28<19:48:57,  1.77s/it]
Outer training loop:  19%|#9        | 9664/50000 [5:02:30<20:21:14,  1.82s/it]
Outer training loop:  19%|#9        | 9665/50000 [5:02:31<20:02:17,  1.79s/it]
Outer training loop:  19%|#9        | 9666/50000 [5:02:33<19:35:33,  1.75s/it]
Outer training loop:  19%|#9        | 9667/50000 [5:02:35<20:18:24,  1.81s/it]
Outer training loop:  19%|#9        | 9668/50000 [5:02:37<20:06:54,  1.80s/it]
Outer training loop:  19%|#9        | 9669/50000 [5:02:38<19:42:41,  1.76s/it]
Outer training loop:  19%|#9        | 9670/50000 [5:02:40<20:00:49,  1.79s/it]
Outer training loop:  19%|#9        | 9671/50000 [5:02:42<19:40:48,  1.76s/it]
Outer training loop:  19%|#9        | 9672/50000 [5:02:44<19:43:22,  1.76s/it]
Outer training loop:  19%|#9        | 9673/50000 [5:02:45<19:36:24,  1.75s/it]
Outer training loop:  19%|#9        | 9674/50000 [5:02:47<20:20:50,  1.82s/it]
Outer training loop:  19%|#9        | 9675/50000 [5:02:49<20:05:23,  1.79s/it]
Outer training loop:  19%|#9        | 9676/50000 [5:02:51<19:57:53,  1.78s/it]
Outer training loop:  19%|#9        | 9677/50000 [5:02:53<20:33:17,  1.84s/it]
Outer training loop:  19%|#9        | 9678/50000 [5:02:54<19:44:52,  1.76s/it]
Outer training loop:  19%|#9        | 9679/50000 [5:02:56<20:12:15,  1.80s/it]
Outer training loop:  19%|#9        | 9680/50000 [5:02:58<20:22:46,  1.82s/it]
Outer training loop:  19%|#9        | 9681/50000 [5:03:00<20:03:18,  1.79s/it]
Outer training loop:  19%|#9        | 9682/50000 [5:03:02<19:50:53,  1.77s/it]
Outer training loop:  19%|#9        | 9683/50000 [5:03:04<20:12:24,  1.80s/it]
Outer training loop:  19%|#9        | 9684/50000 [5:03:05<20:33:00,  1.84s/it]
Outer training loop:  19%|#9        | 9685/50000 [5:03:07<19:54:42,  1.78s/it]
Outer training loop:  19%|#9        | 9686/50000 [5:03:09<20:07:08,  1.80s/it]
Outer training loop:  19%|#9        | 9687/50000 [5:03:11<19:45:26,  1.76s/it]
Outer training loop:  19%|#9        | 9688/50000 [5:03:12<19:27:35,  1.74s/it]
Outer training loop:  19%|#9        | 9689/50000 [5:03:14<19:14:30,  1.72s/it]
Outer training loop:  19%|#9        | 9690/50000 [5:03:16<19:40:09,  1.76s/it]
Outer training loop:  19%|#9        | 9691/50000 [5:03:17<19:24:48,  1.73s/it]
Outer training loop:  19%|#9        | 9692/50000 [5:03:19<19:52:43,  1.78s/it]
Outer training loop:  19%|#9        | 9693/50000 [5:03:21<20:12:27,  1.80s/it]
Outer training loop:  19%|#9        | 9694/50000 [5:03:23<19:47:33,  1.77s/it]
Outer training loop:  19%|#9        | 9695/50000 [5:03:25<19:45:37,  1.76s/it]
Outer training loop:  19%|#9        | 9696/50000 [5:03:26<19:45:27,  1.76s/it]
Outer training loop:  19%|#9        | 9697/50000 [5:03:28<19:39:57,  1.76s/it]
Outer training loop:  19%|#9        | 9698/50000 [5:03:30<19:23:39,  1.73s/it]
Outer training loop:  19%|#9        | 9699/50000 [5:03:32<19:18:32,  1.72s/it]
Outer training loop:  19%|#9        | 9700/50000 [5:03:33<19:58:58,  1.79s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.64s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.69s/it]

Outer training loop:  19%|#9        | 9701/50000 [5:03:40<37:15:32,  3.33s/it]
Outer training loop:  19%|#9        | 9702/50000 [5:03:42<31:48:15,  2.84s/it]
Outer training loop:  19%|#9        | 9703/50000 [5:03:44<28:35:20,  2.55s/it]
Outer training loop:  19%|#9        | 9704/50000 [5:03:46<26:30:53,  2.37s/it]
Outer training loop:  19%|#9        | 9705/50000 [5:03:48<24:17:30,  2.17s/it]
Outer training loop:  19%|#9        | 9706/50000 [5:03:49<22:39:21,  2.02s/it]
Outer training loop:  19%|#9        | 9707/50000 [5:03:51<22:15:31,  1.99s/it]
Outer training loop:  19%|#9        | 9708/50000 [5:03:53<21:34:56,  1.93s/it]
Outer training loop:  19%|#9        | 9709/50000 [5:03:55<20:49:25,  1.86s/it]
Outer training loop:  19%|#9        | 9710/50000 [5:03:57<20:58:35,  1.87s/it]
Outer training loop:  19%|#9        | 9711/50000 [5:03:58<20:23:43,  1.82s/it]
Outer training loop:  19%|#9        | 9712/50000 [5:04:00<20:00:51,  1.79s/it]
Outer training loop:  19%|#9        | 9713/50000 [5:04:02<20:42:34,  1.85s/it]
Outer training loop:  19%|#9        | 9714/50000 [5:04:04<20:38:41,  1.84s/it]
Outer training loop:  19%|#9        | 9715/50000 [5:04:06<20:02:56,  1.79s/it]
Outer training loop:  19%|#9        | 9716/50000 [5:04:07<20:01:50,  1.79s/it]
Outer training loop:  19%|#9        | 9717/50000 [5:04:09<20:34:02,  1.84s/it]
Outer training loop:  19%|#9        | 9718/50000 [5:04:11<20:20:00,  1.82s/it]
Outer training loop:  19%|#9        | 9719/50000 [5:04:13<20:02:06,  1.79s/it]
Outer training loop:  19%|#9        | 9720/50000 [5:04:15<20:18:48,  1.82s/it]
Outer training loop:  19%|#9        | 9721/50000 [5:04:16<20:06:32,  1.80s/it]
Outer training loop:  19%|#9        | 9722/50000 [5:04:18<19:38:19,  1.76s/it]
Outer training loop:  19%|#9        | 9723/50000 [5:04:20<20:12:09,  1.81s/it]
Outer training loop:  19%|#9        | 9724/50000 [5:04:22<20:03:05,  1.79s/it]
Outer training loop:  19%|#9        | 9725/50000 [5:04:23<19:29:52,  1.74s/it]
Outer training loop:  19%|#9        | 9726/50000 [5:04:25<20:31:20,  1.83s/it]
Outer training loop:  19%|#9        | 9727/50000 [5:04:27<19:52:36,  1.78s/it]
Outer training loop:  19%|#9        | 9728/50000 [5:04:29<19:51:32,  1.78s/it]
Outer training loop:  19%|#9        | 9729/50000 [5:04:31<19:45:32,  1.77s/it]
Outer training loop:  19%|#9        | 9730/50000 [5:04:32<20:09:03,  1.80s/it]
Outer training loop:  19%|#9        | 9731/50000 [5:04:34<19:46:25,  1.77s/it]
Outer training loop:  19%|#9        | 9732/50000 [5:04:36<19:39:05,  1.76s/it]
Outer training loop:  19%|#9        | 9733/50000 [5:04:38<19:57:00,  1.78s/it]
Outer training loop:  19%|#9        | 9734/50000 [5:04:39<19:39:40,  1.76s/it]
Outer training loop:  19%|#9        | 9735/50000 [5:04:41<19:48:09,  1.77s/it]
Outer training loop:  19%|#9        | 9736/50000 [5:04:43<20:07:42,  1.80s/it]
Outer training loop:  19%|#9        | 9737/50000 [5:04:45<19:51:33,  1.78s/it]
Outer training loop:  19%|#9        | 9738/50000 [5:04:47<19:44:31,  1.77s/it]
Outer training loop:  19%|#9        | 9739/50000 [5:04:48<20:05:54,  1.80s/it]
Outer training loop:  19%|#9        | 9740/50000 [5:04:50<19:39:51,  1.76s/it]
Outer training loop:  19%|#9        | 9741/50000 [5:04:52<19:34:52,  1.75s/it]
Outer training loop:  19%|#9        | 9742/50000 [5:04:53<19:22:50,  1.73s/it]
Outer training loop:  19%|#9        | 9743/50000 [5:04:55<20:06:12,  1.80s/it]
Outer training loop:  19%|#9        | 9744/50000 [5:04:57<19:39:16,  1.76s/it]
Outer training loop:  19%|#9        | 9745/50000 [5:04:59<19:12:57,  1.72s/it]
Outer training loop:  19%|#9        | 9746/50000 [5:05:01<19:27:35,  1.74s/it]
Outer training loop:  19%|#9        | 9747/50000 [5:05:02<19:29:05,  1.74s/it]
Outer training loop:  19%|#9        | 9748/50000 [5:05:04<19:16:37,  1.72s/it]
Outer training loop:  19%|#9        | 9749/50000 [5:05:06<19:58:11,  1.79s/it]
Outer training loop:  20%|#9        | 9750/50000 [5:05:08<19:41:45,  1.76s/it]
Outer training loop:  20%|#9        | 9751/50000 [5:05:09<19:30:47,  1.75s/it]
Outer training loop:  20%|#9        | 9752/50000 [5:05:11<19:14:32,  1.72s/it]
Outer training loop:  20%|#9        | 9753/50000 [5:05:13<20:24:15,  1.83s/it]
Outer training loop:  20%|#9        | 9754/50000 [5:05:15<20:07:36,  1.80s/it]
Outer training loop:  20%|#9        | 9755/50000 [5:05:16<19:33:25,  1.75s/it]
Outer training loop:  20%|#9        | 9756/50000 [5:05:18<19:52:40,  1.78s/it]
Outer training loop:  20%|#9        | 9757/50000 [5:05:20<19:55:41,  1.78s/it]
Outer training loop:  20%|#9        | 9758/50000 [5:05:22<19:34:31,  1.75s/it]
Outer training loop:  20%|#9        | 9759/50000 [5:05:24<20:04:38,  1.80s/it]
Outer training loop:  20%|#9        | 9760/50000 [5:05:25<19:52:19,  1.78s/it]
Outer training loop:  20%|#9        | 9761/50000 [5:05:27<19:41:24,  1.76s/it]
Outer training loop:  20%|#9        | 9762/50000 [5:05:29<19:49:43,  1.77s/it]
Outer training loop:  20%|#9        | 9763/50000 [5:05:31<20:45:16,  1.86s/it]
Outer training loop:  20%|#9        | 9764/50000 [5:05:33<20:18:32,  1.82s/it]
Outer training loop:  20%|#9        | 9765/50000 [5:05:34<19:57:34,  1.79s/it]
Outer training loop:  20%|#9        | 9766/50000 [5:05:36<20:24:34,  1.83s/it]
Outer training loop:  20%|#9        | 9767/50000 [5:05:38<20:16:39,  1.81s/it]
Outer training loop:  20%|#9        | 9768/50000 [5:05:40<20:05:49,  1.80s/it]
Outer training loop:  20%|#9        | 9769/50000 [5:05:42<20:20:05,  1.82s/it]
Outer training loop:  20%|#9        | 9770/50000 [5:05:43<19:59:06,  1.79s/it]
Outer training loop:  20%|#9        | 9771/50000 [5:05:45<19:36:32,  1.75s/it]
Outer training loop:  20%|#9        | 9772/50000 [5:05:47<19:51:27,  1.78s/it]
Outer training loop:  20%|#9        | 9773/50000 [5:05:49<19:41:36,  1.76s/it]
Outer training loop:  20%|#9        | 9774/50000 [5:05:50<19:43:38,  1.77s/it]
Outer training loop:  20%|#9        | 9775/50000 [5:05:52<19:27:38,  1.74s/it]
Outer training loop:  20%|#9        | 9776/50000 [5:05:54<19:48:20,  1.77s/it]
Outer training loop:  20%|#9        | 9777/50000 [5:05:56<19:47:34,  1.77s/it]
Outer training loop:  20%|#9        | 9778/50000 [5:05:58<19:50:34,  1.78s/it]
Outer training loop:  20%|#9        | 9779/50000 [5:05:59<20:25:19,  1.83s/it]
Outer training loop:  20%|#9        | 9780/50000 [5:06:01<20:37:58,  1.85s/it]
Outer training loop:  20%|#9        | 9781/50000 [5:06:03<20:09:34,  1.80s/it]
Outer training loop:  20%|#9        | 9782/50000 [5:06:05<21:01:32,  1.88s/it]
Outer training loop:  20%|#9        | 9783/50000 [5:06:07<20:30:48,  1.84s/it]
Outer training loop:  20%|#9        | 9784/50000 [5:06:09<20:09:22,  1.80s/it]
Outer training loop:  20%|#9        | 9785/50000 [5:06:10<19:45:59,  1.77s/it]
Outer training loop:  20%|#9        | 9786/50000 [5:06:12<20:25:19,  1.83s/it]
Outer training loop:  20%|#9        | 9787/50000 [5:06:14<20:17:02,  1.82s/it]
Outer training loop:  20%|#9        | 9788/50000 [5:06:16<19:43:16,  1.77s/it]
Outer training loop:  20%|#9        | 9789/50000 [5:06:18<20:18:53,  1.82s/it]
Outer training loop:  20%|#9        | 9790/50000 [5:06:19<20:08:57,  1.80s/it]
Outer training loop:  20%|#9        | 9791/50000 [5:06:21<20:06:51,  1.80s/it]
Outer training loop:  20%|#9        | 9792/50000 [5:06:23<20:37:54,  1.85s/it]
Outer training loop:  20%|#9        | 9793/50000 [5:06:25<20:30:45,  1.84s/it]
Outer training loop:  20%|#9        | 9794/50000 [5:06:27<20:24:36,  1.83s/it]
Outer training loop:  20%|#9        | 9795/50000 [5:06:29<20:33:01,  1.84s/it]
Outer training loop:  20%|#9        | 9796/50000 [5:06:30<20:13:01,  1.81s/it]
Outer training loop:  20%|#9        | 9797/50000 [5:06:32<20:16:38,  1.82s/it]
Outer training loop:  20%|#9        | 9798/50000 [5:06:34<20:45:35,  1.86s/it]
Outer training loop:  20%|#9        | 9799/50000 [5:06:36<20:21:42,  1.82s/it]
Outer training loop:  20%|#9        | 9800/50000 [5:06:38<20:27:59,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.90s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  20%|#9        | 9801/50000 [5:06:45<38:35:13,  3.46s/it]
Outer training loop:  20%|#9        | 9802/50000 [5:06:47<33:09:09,  2.97s/it]
Outer training loop:  20%|#9        | 9803/50000 [5:06:49<29:26:43,  2.64s/it]
Outer training loop:  20%|#9        | 9804/50000 [5:06:50<26:23:22,  2.36s/it]
Outer training loop:  20%|#9        | 9805/50000 [5:06:52<24:15:10,  2.17s/it]
Outer training loop:  20%|#9        | 9806/50000 [5:06:54<23:24:52,  2.10s/it]
Outer training loop:  20%|#9        | 9807/50000 [5:06:56<22:15:51,  1.99s/it]
Outer training loop:  20%|#9        | 9808/50000 [5:06:58<21:42:58,  1.95s/it]
Outer training loop:  20%|#9        | 9809/50000 [5:07:00<21:47:03,  1.95s/it]
Outer training loop:  20%|#9        | 9810/50000 [5:07:01<20:54:13,  1.87s/it]
Outer training loop:  20%|#9        | 9811/50000 [5:07:03<20:36:03,  1.85s/it]
Outer training loop:  20%|#9        | 9812/50000 [5:07:05<20:43:41,  1.86s/it]
Outer training loop:  20%|#9        | 9813/50000 [5:07:07<20:22:58,  1.83s/it]
Outer training loop:  20%|#9        | 9814/50000 [5:07:09<20:22:46,  1.83s/it]
Outer training loop:  20%|#9        | 9815/50000 [5:07:10<19:46:41,  1.77s/it]
Outer training loop:  20%|#9        | 9816/50000 [5:07:12<19:57:59,  1.79s/it]
Outer training loop:  20%|#9        | 9817/50000 [5:07:14<19:17:43,  1.73s/it]
Outer training loop:  20%|#9        | 9818/50000 [5:07:15<19:17:46,  1.73s/it]
Outer training loop:  20%|#9        | 9819/50000 [5:07:17<19:44:18,  1.77s/it]
Outer training loop:  20%|#9        | 9820/50000 [5:07:19<19:48:51,  1.78s/it]
Outer training loop:  20%|#9        | 9821/50000 [5:07:21<20:00:25,  1.79s/it]
Outer training loop:  20%|#9        | 9822/50000 [5:07:23<20:19:31,  1.82s/it]
Outer training loop:  20%|#9        | 9823/50000 [5:07:24<19:50:21,  1.78s/it]
Outer training loop:  20%|#9        | 9824/50000 [5:07:26<19:46:37,  1.77s/it]
Outer training loop:  20%|#9        | 9825/50000 [5:07:28<20:21:08,  1.82s/it]
Outer training loop:  20%|#9        | 9826/50000 [5:07:30<20:03:57,  1.80s/it]
Outer training loop:  20%|#9        | 9827/50000 [5:07:32<19:42:46,  1.77s/it]
Outer training loop:  20%|#9        | 9828/50000 [5:07:33<19:37:56,  1.76s/it]
Outer training loop:  20%|#9        | 9829/50000 [5:07:35<20:08:33,  1.81s/it]
Outer training loop:  20%|#9        | 9830/50000 [5:07:37<19:23:13,  1.74s/it]
Outer training loop:  20%|#9        | 9831/50000 [5:07:39<19:30:07,  1.75s/it]
Outer training loop:  20%|#9        | 9832/50000 [5:07:40<19:50:56,  1.78s/it]
Outer training loop:  20%|#9        | 9833/50000 [5:07:42<19:50:51,  1.78s/it]
Outer training loop:  20%|#9        | 9834/50000 [5:07:44<19:38:40,  1.76s/it]
Outer training loop:  20%|#9        | 9835/50000 [5:07:46<19:25:00,  1.74s/it]
Outer training loop:  20%|#9        | 9836/50000 [5:07:47<20:01:09,  1.79s/it]
Outer training loop:  20%|#9        | 9837/50000 [5:07:49<20:15:40,  1.82s/it]
Outer training loop:  20%|#9        | 9838/50000 [5:07:51<20:07:49,  1.80s/it]
Outer training loop:  20%|#9        | 9839/50000 [5:07:53<20:19:32,  1.82s/it]
Outer training loop:  20%|#9        | 9840/50000 [5:07:55<19:52:29,  1.78s/it]
Outer training loop:  20%|#9        | 9841/50000 [5:07:56<19:32:47,  1.75s/it]
Outer training loop:  20%|#9        | 9842/50000 [5:07:58<19:57:59,  1.79s/it]
Outer training loop:  20%|#9        | 9843/50000 [5:08:00<19:34:42,  1.76s/it]
Outer training loop:  20%|#9        | 9844/50000 [5:08:02<19:29:12,  1.75s/it]
Outer training loop:  20%|#9        | 9845/50000 [5:08:03<19:21:32,  1.74s/it]
Outer training loop:  20%|#9        | 9846/50000 [5:08:05<20:02:22,  1.80s/it]
Outer training loop:  20%|#9        | 9847/50000 [5:08:07<19:37:31,  1.76s/it]
Outer training loop:  20%|#9        | 9848/50000 [5:08:09<19:07:43,  1.72s/it]
Outer training loop:  20%|#9        | 9849/50000 [5:08:10<19:40:18,  1.76s/it]
Outer training loop:  20%|#9        | 9850/50000 [5:08:12<20:27:44,  1.83s/it]
Outer training loop:  20%|#9        | 9851/50000 [5:08:14<19:57:40,  1.79s/it]
Outer training loop:  20%|#9        | 9852/50000 [5:08:16<19:52:08,  1.78s/it]
Outer training loop:  20%|#9        | 9853/50000 [5:08:18<20:08:18,  1.81s/it]
Outer training loop:  20%|#9        | 9854/50000 [5:08:20<20:03:40,  1.80s/it]
Outer training loop:  20%|#9        | 9855/50000 [5:08:21<19:44:36,  1.77s/it]
Outer training loop:  20%|#9        | 9856/50000 [5:08:23<20:10:22,  1.81s/it]
Outer training loop:  20%|#9        | 9857/50000 [5:08:25<19:40:48,  1.76s/it]
Outer training loop:  20%|#9        | 9858/50000 [5:08:26<19:22:44,  1.74s/it]
Outer training loop:  20%|#9        | 9859/50000 [5:08:28<19:50:16,  1.78s/it]
Outer training loop:  20%|#9        | 9860/50000 [5:08:30<19:38:29,  1.76s/it]
Outer training loop:  20%|#9        | 9861/50000 [5:08:32<19:46:55,  1.77s/it]
Outer training loop:  20%|#9        | 9862/50000 [5:08:34<19:53:07,  1.78s/it]
Outer training loop:  20%|#9        | 9863/50000 [5:08:36<20:09:49,  1.81s/it]
Outer training loop:  20%|#9        | 9864/50000 [5:08:37<19:58:05,  1.79s/it]
Outer training loop:  20%|#9        | 9865/50000 [5:08:39<19:57:06,  1.79s/it]
Outer training loop:  20%|#9        | 9866/50000 [5:08:41<20:20:34,  1.82s/it]
Outer training loop:  20%|#9        | 9867/50000 [5:08:43<19:59:34,  1.79s/it]
Outer training loop:  20%|#9        | 9868/50000 [5:08:44<19:20:54,  1.74s/it]
Outer training loop:  20%|#9        | 9869/50000 [5:08:46<19:57:24,  1.79s/it]
Outer training loop:  20%|#9        | 9870/50000 [5:08:48<19:38:54,  1.76s/it]
Outer training loop:  20%|#9        | 9871/50000 [5:08:50<19:42:35,  1.77s/it]
Outer training loop:  20%|#9        | 9872/50000 [5:08:51<19:30:34,  1.75s/it]
Outer training loop:  20%|#9        | 9873/50000 [5:08:53<19:49:52,  1.78s/it]
Outer training loop:  20%|#9        | 9874/50000 [5:08:55<19:41:56,  1.77s/it]
Outer training loop:  20%|#9        | 9875/50000 [5:08:57<19:31:02,  1.75s/it]
Outer training loop:  20%|#9        | 9876/50000 [5:08:59<19:53:48,  1.79s/it]
Outer training loop:  20%|#9        | 9877/50000 [5:09:00<19:27:44,  1.75s/it]
Outer training loop:  20%|#9        | 9878/50000 [5:09:02<19:23:45,  1.74s/it]
Outer training loop:  20%|#9        | 9879/50000 [5:09:04<19:55:47,  1.79s/it]
Outer training loop:  20%|#9        | 9880/50000 [5:09:06<19:53:18,  1.78s/it]
Outer training loop:  20%|#9        | 9881/50000 [5:09:07<19:39:55,  1.76s/it]
Outer training loop:  20%|#9        | 9882/50000 [5:09:09<20:11:56,  1.81s/it]
Outer training loop:  20%|#9        | 9883/50000 [5:09:11<19:53:43,  1.79s/it]
Outer training loop:  20%|#9        | 9884/50000 [5:09:13<19:42:24,  1.77s/it]
Outer training loop:  20%|#9        | 9885/50000 [5:09:15<19:43:12,  1.77s/it]
Outer training loop:  20%|#9        | 9886/50000 [5:09:16<20:03:28,  1.80s/it]
Outer training loop:  20%|#9        | 9887/50000 [5:09:18<19:47:20,  1.78s/it]
Outer training loop:  20%|#9        | 9888/50000 [5:09:20<19:46:08,  1.77s/it]
Outer training loop:  20%|#9        | 9889/50000 [5:09:22<20:08:03,  1.81s/it]
Outer training loop:  20%|#9        | 9890/50000 [5:09:23<19:51:45,  1.78s/it]
Outer training loop:  20%|#9        | 9891/50000 [5:09:25<19:40:41,  1.77s/it]
Outer training loop:  20%|#9        | 9892/50000 [5:09:27<19:21:01,  1.74s/it]
Outer training loop:  20%|#9        | 9893/50000 [5:09:29<19:53:53,  1.79s/it]
Outer training loop:  20%|#9        | 9894/50000 [5:09:30<19:13:46,  1.73s/it]
Outer training loop:  20%|#9        | 9895/50000 [5:09:32<19:01:58,  1.71s/it]
Outer training loop:  20%|#9        | 9896/50000 [5:09:34<19:52:10,  1.78s/it]
Outer training loop:  20%|#9        | 9897/50000 [5:09:36<19:30:34,  1.75s/it]
Outer training loop:  20%|#9        | 9898/50000 [5:09:37<19:16:59,  1.73s/it]
Outer training loop:  20%|#9        | 9899/50000 [5:09:39<20:08:06,  1.81s/it]
Outer training loop:  20%|#9        | 9900/50000 [5:09:41<20:36:29,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.61s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.67s/it]

Outer training loop:  20%|#9        | 9901/50000 [5:09:48<37:23:27,  3.36s/it]
Outer training loop:  20%|#9        | 9902/50000 [5:09:50<32:05:29,  2.88s/it]
Outer training loop:  20%|#9        | 9903/50000 [5:09:52<28:38:35,  2.57s/it]
Outer training loop:  20%|#9        | 9904/50000 [5:09:53<25:47:54,  2.32s/it]
Outer training loop:  20%|#9        | 9905/50000 [5:09:55<23:59:57,  2.15s/it]
Outer training loop:  20%|#9        | 9906/50000 [5:09:57<22:40:42,  2.04s/it]
Outer training loop:  20%|#9        | 9907/50000 [5:09:59<21:49:35,  1.96s/it]
Outer training loop:  20%|#9        | 9908/50000 [5:10:01<20:56:14,  1.88s/it]
Outer training loop:  20%|#9        | 9909/50000 [5:10:02<20:39:58,  1.86s/it]
Outer training loop:  20%|#9        | 9910/50000 [5:10:04<20:56:52,  1.88s/it]
Outer training loop:  20%|#9        | 9911/50000 [5:10:06<20:31:00,  1.84s/it]
Outer training loop:  20%|#9        | 9912/50000 [5:10:08<20:00:59,  1.80s/it]
Outer training loop:  20%|#9        | 9913/50000 [5:10:10<20:20:46,  1.83s/it]
Outer training loop:  20%|#9        | 9914/50000 [5:10:11<19:43:54,  1.77s/it]
Outer training loop:  20%|#9        | 9915/50000 [5:10:13<19:17:59,  1.73s/it]
Outer training loop:  20%|#9        | 9916/50000 [5:10:14<18:48:29,  1.69s/it]
Outer training loop:  20%|#9        | 9917/50000 [5:10:16<19:28:50,  1.75s/it]
Outer training loop:  20%|#9        | 9918/50000 [5:10:18<19:24:55,  1.74s/it]
Outer training loop:  20%|#9        | 9919/50000 [5:10:20<19:19:27,  1.74s/it]
Outer training loop:  20%|#9        | 9920/50000 [5:10:22<19:44:01,  1.77s/it]
Outer training loop:  20%|#9        | 9921/50000 [5:10:23<19:18:13,  1.73s/it]
Outer training loop:  20%|#9        | 9922/50000 [5:10:25<19:16:01,  1.73s/it]
Outer training loop:  20%|#9        | 9923/50000 [5:10:27<19:42:56,  1.77s/it]
Outer training loop:  20%|#9        | 9924/50000 [5:10:29<19:41:06,  1.77s/it]
Outer training loop:  20%|#9        | 9925/50000 [5:10:30<19:28:26,  1.75s/it]
Outer training loop:  20%|#9        | 9926/50000 [5:10:32<19:32:01,  1.75s/it]
Outer training loop:  20%|#9        | 9927/50000 [5:10:34<20:37:56,  1.85s/it]
Outer training loop:  20%|#9        | 9928/50000 [5:10:36<20:12:24,  1.82s/it]
Outer training loop:  20%|#9        | 9929/50000 [5:10:38<20:12:20,  1.82s/it]
Outer training loop:  20%|#9        | 9930/50000 [5:10:39<19:53:17,  1.79s/it]
Outer training loop:  20%|#9        | 9931/50000 [5:10:41<19:32:22,  1.76s/it]
Outer training loop:  20%|#9        | 9932/50000 [5:10:43<19:08:35,  1.72s/it]
Outer training loop:  20%|#9        | 9933/50000 [5:10:45<19:54:04,  1.79s/it]
Outer training loop:  20%|#9        | 9934/50000 [5:10:47<19:49:43,  1.78s/it]
Outer training loop:  20%|#9        | 9935/50000 [5:10:48<19:19:34,  1.74s/it]
Outer training loop:  20%|#9        | 9936/50000 [5:10:50<19:48:35,  1.78s/it]
Outer training loop:  20%|#9        | 9937/50000 [5:10:52<19:26:40,  1.75s/it]
Outer training loop:  20%|#9        | 9938/50000 [5:10:53<19:24:08,  1.74s/it]
Outer training loop:  20%|#9        | 9939/50000 [5:10:55<19:43:37,  1.77s/it]
Outer training loop:  20%|#9        | 9940/50000 [5:10:57<19:41:23,  1.77s/it]
Outer training loop:  20%|#9        | 9941/50000 [5:10:59<19:23:17,  1.74s/it]
Outer training loop:  20%|#9        | 9942/50000 [5:11:00<18:53:11,  1.70s/it]
Outer training loop:  20%|#9        | 9943/50000 [5:11:02<19:20:17,  1.74s/it]
Outer training loop:  20%|#9        | 9944/50000 [5:11:04<19:37:43,  1.76s/it]
Outer training loop:  20%|#9        | 9945/50000 [5:11:06<19:20:23,  1.74s/it]
Outer training loop:  20%|#9        | 9946/50000 [5:11:08<19:58:11,  1.79s/it]
Outer training loop:  20%|#9        | 9947/50000 [5:11:09<20:16:19,  1.82s/it]
Outer training loop:  20%|#9        | 9948/50000 [5:11:11<20:04:23,  1.80s/it]
Outer training loop:  20%|#9        | 9949/50000 [5:11:13<19:52:53,  1.79s/it]
Outer training loop:  20%|#9        | 9950/50000 [5:11:15<19:58:50,  1.80s/it]
Outer training loop:  20%|#9        | 9951/50000 [5:11:16<19:42:52,  1.77s/it]
Outer training loop:  20%|#9        | 9952/50000 [5:11:18<19:53:36,  1.79s/it]
Outer training loop:  20%|#9        | 9953/50000 [5:11:20<20:24:46,  1.83s/it]
Outer training loop:  20%|#9        | 9954/50000 [5:11:22<20:15:58,  1.82s/it]
Outer training loop:  20%|#9        | 9955/50000 [5:11:24<19:44:40,  1.78s/it]
Outer training loop:  20%|#9        | 9956/50000 [5:11:26<19:51:18,  1.79s/it]
Outer training loop:  20%|#9        | 9957/50000 [5:11:27<20:22:50,  1.83s/it]
Outer training loop:  20%|#9        | 9958/50000 [5:11:29<20:00:29,  1.80s/it]
Outer training loop:  20%|#9        | 9959/50000 [5:11:31<19:31:28,  1.76s/it]
Outer training loop:  20%|#9        | 9960/50000 [5:11:33<20:04:53,  1.81s/it]
Outer training loop:  20%|#9        | 9961/50000 [5:11:35<19:54:36,  1.79s/it]
Outer training loop:  20%|#9        | 9962/50000 [5:11:36<19:30:57,  1.75s/it]
Outer training loop:  20%|#9        | 9963/50000 [5:11:38<19:57:28,  1.79s/it]
Outer training loop:  20%|#9        | 9964/50000 [5:11:40<19:31:43,  1.76s/it]
Outer training loop:  20%|#9        | 9965/50000 [5:11:42<19:34:48,  1.76s/it]
Outer training loop:  20%|#9        | 9966/50000 [5:11:43<19:06:03,  1.72s/it]
Outer training loop:  20%|#9        | 9967/50000 [5:11:45<20:01:56,  1.80s/it]
Outer training loop:  20%|#9        | 9968/50000 [5:11:47<19:40:26,  1.77s/it]
Outer training loop:  20%|#9        | 9969/50000 [5:11:49<19:31:17,  1.76s/it]
Outer training loop:  20%|#9        | 9970/50000 [5:11:50<20:03:07,  1.80s/it]
Outer training loop:  20%|#9        | 9971/50000 [5:11:52<19:47:23,  1.78s/it]
Outer training loop:  20%|#9        | 9972/50000 [5:11:54<19:34:44,  1.76s/it]
Outer training loop:  20%|#9        | 9973/50000 [5:11:56<20:14:33,  1.82s/it]
Outer training loop:  20%|#9        | 9974/50000 [5:11:58<19:48:51,  1.78s/it]
Outer training loop:  20%|#9        | 9975/50000 [5:11:59<19:44:48,  1.78s/it]
Outer training loop:  20%|#9        | 9976/50000 [5:12:01<19:47:22,  1.78s/it]
Outer training loop:  20%|#9        | 9977/50000 [5:12:03<19:33:58,  1.76s/it]
Outer training loop:  20%|#9        | 9978/50000 [5:12:05<19:37:39,  1.77s/it]
Outer training loop:  20%|#9        | 9979/50000 [5:12:06<19:46:48,  1.78s/it]
Outer training loop:  20%|#9        | 9980/50000 [5:12:08<20:11:04,  1.82s/it]
Outer training loop:  20%|#9        | 9981/50000 [5:12:10<20:16:50,  1.82s/it]
Outer training loop:  20%|#9        | 9982/50000 [5:12:12<20:16:05,  1.82s/it]
Outer training loop:  20%|#9        | 9983/50000 [5:12:14<20:08:55,  1.81s/it]
Outer training loop:  20%|#9        | 9984/50000 [5:12:15<19:49:49,  1.78s/it]
Outer training loop:  20%|#9        | 9985/50000 [5:12:17<19:30:23,  1.75s/it]
Outer training loop:  20%|#9        | 9986/50000 [5:12:19<20:07:11,  1.81s/it]
Outer training loop:  20%|#9        | 9987/50000 [5:12:21<19:30:33,  1.76s/it]
Outer training loop:  20%|#9        | 9988/50000 [5:12:22<18:51:12,  1.70s/it]
Outer training loop:  20%|#9        | 9989/50000 [5:12:24<19:38:19,  1.77s/it]
Outer training loop:  20%|#9        | 9990/50000 [5:12:26<19:46:41,  1.78s/it]
Outer training loop:  20%|#9        | 9991/50000 [5:12:28<19:26:55,  1.75s/it]
Outer training loop:  20%|#9        | 9992/50000 [5:12:29<19:11:06,  1.73s/it]
Outer training loop:  20%|#9        | 9993/50000 [5:12:31<19:52:21,  1.79s/it]
Outer training loop:  20%|#9        | 9994/50000 [5:12:33<19:46:33,  1.78s/it]
Outer training loop:  20%|#9        | 9995/50000 [5:12:35<19:51:15,  1.79s/it]
Outer training loop:  20%|#9        | 9996/50000 [5:12:37<20:25:18,  1.84s/it]
Outer training loop:  20%|#9        | 9997/50000 [5:12:39<20:50:52,  1.88s/it]
Outer training loop:  20%|#9        | 9998/50000 [5:12:41<20:32:55,  1.85s/it]
Outer training loop:  20%|#9        | 9999/50000 [5:12:42<20:25:34,  1.84s/it]
Outer training loop:  20%|##        | 10000/50000 [5:12:44<20:12:39,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:17,  1.67s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:20,  1.74s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:20,  1.75s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:17,  1.75s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.76s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:18,  1.80s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:17,  1.81s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:19,  1.88s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:14,  1.84s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:10,  1.81s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.79s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:08,  1.83s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:03,  1.79s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:01,  1.78s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:57,  1.75s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:55,  1.74s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:55,  1.78s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:54,  1.78s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:51,  1.78s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:47,  1.73s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:44,  1.71s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:49,  1.82s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:45,  1.79s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.79s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.77s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:40,  1.79s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:42,  1.86s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.79s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:34,  1.77s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.76s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:29,  1.75s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:28,  1.80s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:24,  1.77s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:23,  1.78s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:19,  1.73s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:16,  1.71s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:20,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:16,  1.77s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:13,  1.75s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:12,  1.77s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:12,  1.82s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.79s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:05,  1.73s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:02,  1.68s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:04,  1.78s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:01,  1.77s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<00:59,  1.74s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:57,  1.74s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:55,  1.73s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:53,  1.72s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:51,  1.72s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:50,  1.79s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:48,  1.78s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:46,  1.79s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:44,  1.77s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.76s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.76s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:39,  1.79s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:40,  1.91s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:34,  1.80s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:31,  1.76s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.75s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.73s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:26,  1.74s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:23,  1.70s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:21,  1.80s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:19,  1.77s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.73s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:15,  1.74s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.76s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.79s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.78s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.77s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.77s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:16,  1.66s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:15,  1.67s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:14,  1.68s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:14,  1.71s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:12,  1.70s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:24,  1.87s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:21,  1.86s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:17,  1.83s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:13,  1.81s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.88s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.83s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.79s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:08,  1.86s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:03,  1.82s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:01,  1.81s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:59,  1.81s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:57,  1.80s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:57,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:54,  1.82s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:51,  1.79s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:48,  1.77s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:51,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:45,  1.81s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:42,  1.79s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:38,  1.75s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:37,  1.77s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:38,  1.82s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:35,  1.79s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:32,  1.77s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:29,  1.75s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:28,  1.77s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:27,  1.86s/it]

Evaluating policy:  45%|####4     | 37/83 [01:07<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:20,  1.79s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:17,  1.76s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:16,  1.77s/it]

Evaluating policy:  49%|####9     | 41/83 [01:14<01:19,  1.89s/it]

Evaluating policy:  51%|#####     | 42/83 [01:16<01:16,  1.87s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:18<01:13,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:11,  1.84s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:09,  1.82s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:23<01:06,  1.79s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:25<01:07,  1.89s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:27<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<01:01,  1.80s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:58,  1.78s/it]

Evaluating policy:  61%|######1   | 51/83 [01:32<00:57,  1.79s/it]

Evaluating policy:  63%|######2   | 52/83 [01:34<00:54,  1.76s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:51,  1.73s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:39<00:50,  1.79s/it]

Evaluating policy:  67%|######7   | 56/83 [01:41<00:48,  1.80s/it]

Evaluating policy:  69%|######8   | 57/83 [01:43<00:46,  1.79s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:44,  1.80s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:43,  1.79s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:48<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:50<00:43,  1.96s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:52<00:39,  1.89s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:54<00:36,  1.82s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:56<00:34,  1.81s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:57<00:31,  1.77s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:59<00:30,  1.77s/it]

Evaluating policy:  81%|########  | 67/83 [02:01<00:28,  1.78s/it]

Evaluating policy:  82%|########1 | 68/83 [02:03<00:26,  1.76s/it]

Evaluating policy:  83%|########3 | 69/83 [02:05<00:26,  1.91s/it]

Evaluating policy:  84%|########4 | 70/83 [02:06<00:23,  1.84s/it]

Evaluating policy:  86%|########5 | 71/83 [02:08<00:21,  1.81s/it]

Evaluating policy:  87%|########6 | 72/83 [02:10<00:19,  1.75s/it]

Evaluating policy:  88%|########7 | 73/83 [02:12<00:17,  1.76s/it]

Evaluating policy:  89%|########9 | 74/83 [02:13<00:15,  1.77s/it]

Evaluating policy:  90%|######### | 75/83 [02:15<00:14,  1.79s/it]

Evaluating policy:  92%|#########1| 76/83 [02:17<00:12,  1.78s/it]

Evaluating policy:  93%|#########2| 77/83 [02:19<00:10,  1.77s/it]

Evaluating policy:  94%|#########3| 78/83 [02:20<00:08,  1.74s/it]

Evaluating policy:  95%|#########5| 79/83 [02:23<00:07,  1.92s/it]

Evaluating policy:  96%|#########6| 80/83 [02:24<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:26<00:03,  1.81s/it]

Evaluating policy:  99%|#########8| 82/83 [02:28<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:30<00:00,  1.84s/it]

Outer training loop:  20%|##        | 10001/50000 [5:17:50<1033:18:50, 93.00s/it]
Outer training loop:  20%|##        | 10002/50000 [5:17:52<728:58:46, 65.61s/it] 
Outer training loop:  20%|##        | 10003/50000 [5:17:53<515:51:24, 46.43s/it]
Outer training loop:  20%|##        | 10004/50000 [5:17:55<367:05:25, 33.04s/it]
Outer training loop:  20%|##        | 10005/50000 [5:17:57<262:26:12, 23.62s/it]
Outer training loop:  20%|##        | 10006/50000 [5:17:59<191:11:31, 17.21s/it]
Outer training loop:  20%|##        | 10007/50000 [5:18:01<139:47:37, 12.58s/it]
Outer training loop:  20%|##        | 10008/50000 [5:18:03<103:53:45,  9.35s/it]
Outer training loop:  20%|##        | 10009/50000 [5:18:04<78:39:32,  7.08s/it] 
Outer training loop:  20%|##        | 10010/50000 [5:18:06<61:18:41,  5.52s/it]
Outer training loop:  20%|##        | 10011/50000 [5:18:08<48:41:23,  4.38s/it]
Outer training loop:  20%|##        | 10012/50000 [5:18:10<39:37:23,  3.57s/it]
Outer training loop:  20%|##        | 10013/50000 [5:18:12<34:25:30,  3.10s/it]
Outer training loop:  20%|##        | 10014/50000 [5:18:13<30:05:54,  2.71s/it]
Outer training loop:  20%|##        | 10015/50000 [5:18:15<26:53:53,  2.42s/it]
Outer training loop:  20%|##        | 10016/50000 [5:18:17<25:10:49,  2.27s/it]
Outer training loop:  20%|##        | 10017/50000 [5:18:19<23:50:05,  2.15s/it]
Outer training loop:  20%|##        | 10018/50000 [5:18:21<22:33:59,  2.03s/it]
Outer training loop:  20%|##        | 10019/50000 [5:18:23<21:44:37,  1.96s/it]
Outer training loop:  20%|##        | 10020/50000 [5:18:25<22:00:47,  1.98s/it]
Outer training loop:  20%|##        | 10021/50000 [5:18:26<21:00:00,  1.89s/it]
Outer training loop:  20%|##        | 10022/50000 [5:18:28<20:03:23,  1.81s/it]
Outer training loop:  20%|##        | 10023/50000 [5:18:30<20:26:53,  1.84s/it]
Outer training loop:  20%|##        | 10024/50000 [5:18:31<20:00:30,  1.80s/it]
Outer training loop:  20%|##        | 10025/50000 [5:18:33<19:46:44,  1.78s/it]
Outer training loop:  20%|##        | 10026/50000 [5:18:35<20:13:18,  1.82s/it]
Outer training loop:  20%|##        | 10027/50000 [5:18:37<20:05:17,  1.81s/it]
Outer training loop:  20%|##        | 10028/50000 [5:18:39<20:10:27,  1.82s/it]
Outer training loop:  20%|##        | 10029/50000 [5:18:41<20:31:59,  1.85s/it]
Outer training loop:  20%|##        | 10030/50000 [5:18:42<20:04:05,  1.81s/it]
Outer training loop:  20%|##        | 10031/50000 [5:18:44<19:36:11,  1.77s/it]
Outer training loop:  20%|##        | 10032/50000 [5:18:46<19:37:43,  1.77s/it]
Outer training loop:  20%|##        | 10033/50000 [5:18:48<20:41:01,  1.86s/it]
Outer training loop:  20%|##        | 10034/50000 [5:18:50<20:35:14,  1.85s/it]
Outer training loop:  20%|##        | 10035/50000 [5:18:52<20:22:50,  1.84s/it]
Outer training loop:  20%|##        | 10036/50000 [5:18:53<20:43:23,  1.87s/it]
Outer training loop:  20%|##        | 10037/50000 [5:18:55<20:49:22,  1.88s/it]
Outer training loop:  20%|##        | 10038/50000 [5:18:57<20:16:25,  1.83s/it]
Outer training loop:  20%|##        | 10039/50000 [5:18:59<20:04:36,  1.81s/it]
Outer training loop:  20%|##        | 10040/50000 [5:19:01<20:32:32,  1.85s/it]
Outer training loop:  20%|##        | 10041/50000 [5:19:03<20:31:12,  1.85s/it]
Outer training loop:  20%|##        | 10042/50000 [5:19:04<20:03:34,  1.81s/it]
Outer training loop:  20%|##        | 10043/50000 [5:19:06<20:45:54,  1.87s/it]
Outer training loop:  20%|##        | 10044/50000 [5:19:08<20:04:20,  1.81s/it]
Outer training loop:  20%|##        | 10045/50000 [5:19:10<19:47:01,  1.78s/it]
Outer training loop:  20%|##        | 10046/50000 [5:19:11<19:35:05,  1.76s/it]
Outer training loop:  20%|##        | 10047/50000 [5:19:13<20:02:19,  1.81s/it]
Outer training loop:  20%|##        | 10048/50000 [5:19:15<19:53:05,  1.79s/it]
Outer training loop:  20%|##        | 10049/50000 [5:19:17<19:55:06,  1.79s/it]
Outer training loop:  20%|##        | 10050/50000 [5:19:19<20:21:12,  1.83s/it]
Outer training loop:  20%|##        | 10051/50000 [5:19:21<20:18:35,  1.83s/it]
Outer training loop:  20%|##        | 10052/50000 [5:19:23<20:32:50,  1.85s/it]
Outer training loop:  20%|##        | 10053/50000 [5:19:24<20:35:25,  1.86s/it]
Outer training loop:  20%|##        | 10054/50000 [5:19:27<21:14:26,  1.91s/it]
Outer training loop:  20%|##        | 10055/50000 [5:19:28<21:14:05,  1.91s/it]
Outer training loop:  20%|##        | 10056/50000 [5:19:30<20:46:34,  1.87s/it]
Outer training loop:  20%|##        | 10057/50000 [5:19:32<20:52:49,  1.88s/it]
Outer training loop:  20%|##        | 10058/50000 [5:19:34<20:07:33,  1.81s/it]
Outer training loop:  20%|##        | 10059/50000 [5:19:36<19:52:42,  1.79s/it]
Outer training loop:  20%|##        | 10060/50000 [5:19:37<20:09:34,  1.82s/it]
Outer training loop:  20%|##        | 10061/50000 [5:19:39<20:21:51,  1.84s/it]
Outer training loop:  20%|##        | 10062/50000 [5:19:41<19:40:57,  1.77s/it]
Outer training loop:  20%|##        | 10063/50000 [5:19:43<19:38:42,  1.77s/it]
Outer training loop:  20%|##        | 10064/50000 [5:19:45<20:23:18,  1.84s/it]
Outer training loop:  20%|##        | 10065/50000 [5:19:46<19:59:53,  1.80s/it]
Outer training loop:  20%|##        | 10066/50000 [5:19:48<20:00:19,  1.80s/it]
Outer training loop:  20%|##        | 10067/50000 [5:19:50<20:07:25,  1.81s/it]
Outer training loop:  20%|##        | 10068/50000 [5:19:52<20:03:18,  1.81s/it]
Outer training loop:  20%|##        | 10069/50000 [5:19:54<20:06:18,  1.81s/it]
Outer training loop:  20%|##        | 10070/50000 [5:19:55<20:04:46,  1.81s/it]
Outer training loop:  20%|##        | 10071/50000 [5:19:57<20:22:40,  1.84s/it]
Outer training loop:  20%|##        | 10072/50000 [5:19:59<20:21:25,  1.84s/it]
Outer training loop:  20%|##        | 10073/50000 [5:20:01<20:01:13,  1.81s/it]
Outer training loop:  20%|##        | 10074/50000 [5:20:03<19:48:31,  1.79s/it]
Outer training loop:  20%|##        | 10075/50000 [5:20:05<20:24:33,  1.84s/it]
Outer training loop:  20%|##        | 10076/50000 [5:20:06<20:16:11,  1.83s/it]
Outer training loop:  20%|##        | 10077/50000 [5:20:08<20:06:45,  1.81s/it]
Outer training loop:  20%|##        | 10078/50000 [5:20:10<20:55:11,  1.89s/it]
Outer training loop:  20%|##        | 10079/50000 [5:20:12<20:25:15,  1.84s/it]
Outer training loop:  20%|##        | 10080/50000 [5:20:14<19:32:53,  1.76s/it]
Outer training loop:  20%|##        | 10081/50000 [5:20:15<19:10:33,  1.73s/it]
Outer training loop:  20%|##        | 10082/50000 [5:20:17<19:58:16,  1.80s/it]
Outer training loop:  20%|##        | 10083/50000 [5:20:19<19:58:38,  1.80s/it]
Outer training loop:  20%|##        | 10084/50000 [5:20:21<20:12:05,  1.82s/it]
Outer training loop:  20%|##        | 10085/50000 [5:20:23<20:31:12,  1.85s/it]
Outer training loop:  20%|##        | 10086/50000 [5:20:25<20:13:15,  1.82s/it]
Outer training loop:  20%|##        | 10087/50000 [5:20:26<20:08:24,  1.82s/it]
Outer training loop:  20%|##        | 10088/50000 [5:20:28<20:22:50,  1.84s/it]
Outer training loop:  20%|##        | 10089/50000 [5:20:30<19:53:51,  1.79s/it]
Outer training loop:  20%|##        | 10090/50000 [5:20:32<19:24:51,  1.75s/it]
Outer training loop:  20%|##        | 10091/50000 [5:20:33<19:43:11,  1.78s/it]
Outer training loop:  20%|##        | 10092/50000 [5:20:35<20:02:39,  1.81s/it]
Outer training loop:  20%|##        | 10093/50000 [5:20:37<20:05:50,  1.81s/it]
Outer training loop:  20%|##        | 10094/50000 [5:20:39<19:55:54,  1.80s/it]
Outer training loop:  20%|##        | 10095/50000 [5:20:41<20:44:14,  1.87s/it]
Outer training loop:  20%|##        | 10096/50000 [5:20:43<20:22:18,  1.84s/it]
Outer training loop:  20%|##        | 10097/50000 [5:20:44<19:56:33,  1.80s/it]
Outer training loop:  20%|##        | 10098/50000 [5:20:46<20:01:13,  1.81s/it]
Outer training loop:  20%|##        | 10099/50000 [5:20:48<20:13:10,  1.82s/it]
Outer training loop:  20%|##        | 10100/50000 [5:20:50<20:08:34,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.72s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.66s/it]

Outer training loop:  20%|##        | 10101/50000 [5:20:57<37:01:52,  3.34s/it]
Outer training loop:  20%|##        | 10102/50000 [5:20:59<32:27:58,  2.93s/it]
Outer training loop:  20%|##        | 10103/50000 [5:21:01<28:38:08,  2.58s/it]
Outer training loop:  20%|##        | 10104/50000 [5:21:02<26:14:31,  2.37s/it]
Outer training loop:  20%|##        | 10105/50000 [5:21:04<24:52:02,  2.24s/it]
Outer training loop:  20%|##        | 10106/50000 [5:21:06<23:17:22,  2.10s/it]
Outer training loop:  20%|##        | 10107/50000 [5:21:08<21:48:59,  1.97s/it]
Outer training loop:  20%|##        | 10108/50000 [5:21:10<21:02:21,  1.90s/it]
Outer training loop:  20%|##        | 10109/50000 [5:21:11<21:09:03,  1.91s/it]
Outer training loop:  20%|##        | 10110/50000 [5:21:13<20:46:18,  1.87s/it]
Outer training loop:  20%|##        | 10111/50000 [5:21:15<20:24:18,  1.84s/it]
Outer training loop:  20%|##        | 10112/50000 [5:21:17<20:25:11,  1.84s/it]
Outer training loop:  20%|##        | 10113/50000 [5:21:19<20:37:29,  1.86s/it]
Outer training loop:  20%|##        | 10114/50000 [5:21:20<20:12:28,  1.82s/it]
Outer training loop:  20%|##        | 10115/50000 [5:21:22<20:38:04,  1.86s/it]
Outer training loop:  20%|##        | 10116/50000 [5:21:24<20:04:33,  1.81s/it]
Outer training loop:  20%|##        | 10117/50000 [5:21:26<20:05:30,  1.81s/it]
Outer training loop:  20%|##        | 10118/50000 [5:21:28<20:01:26,  1.81s/it]
Outer training loop:  20%|##        | 10119/50000 [5:21:30<20:07:39,  1.82s/it]
Outer training loop:  20%|##        | 10120/50000 [5:21:31<19:41:49,  1.78s/it]
Outer training loop:  20%|##        | 10121/50000 [5:21:33<19:47:57,  1.79s/it]
Outer training loop:  20%|##        | 10122/50000 [5:21:35<20:21:36,  1.84s/it]
Outer training loop:  20%|##        | 10123/50000 [5:21:37<19:55:30,  1.80s/it]
Outer training loop:  20%|##        | 10124/50000 [5:21:38<19:31:17,  1.76s/it]
Outer training loop:  20%|##        | 10125/50000 [5:21:41<20:57:37,  1.89s/it]
Outer training loop:  20%|##        | 10126/50000 [5:21:42<20:22:40,  1.84s/it]
Outer training loop:  20%|##        | 10127/50000 [5:21:44<19:57:54,  1.80s/it]
Outer training loop:  20%|##        | 10128/50000 [5:21:46<20:39:21,  1.87s/it]
Outer training loop:  20%|##        | 10129/50000 [5:21:48<20:15:27,  1.83s/it]
Outer training loop:  20%|##        | 10130/50000 [5:21:50<20:22:56,  1.84s/it]
Outer training loop:  20%|##        | 10131/50000 [5:21:52<20:34:07,  1.86s/it]
Outer training loop:  20%|##        | 10132/50000 [5:21:53<19:54:54,  1.80s/it]
Outer training loop:  20%|##        | 10133/50000 [5:21:55<19:30:06,  1.76s/it]
Outer training loop:  20%|##        | 10134/50000 [5:21:57<19:42:42,  1.78s/it]
Outer training loop:  20%|##        | 10135/50000 [5:21:59<20:30:26,  1.85s/it]
Outer training loop:  20%|##        | 10136/50000 [5:22:00<19:58:46,  1.80s/it]
Outer training loop:  20%|##        | 10137/50000 [5:22:02<19:55:27,  1.80s/it]
Outer training loop:  20%|##        | 10138/50000 [5:22:04<20:17:02,  1.83s/it]
Outer training loop:  20%|##        | 10139/50000 [5:22:06<20:23:03,  1.84s/it]
Outer training loop:  20%|##        | 10140/50000 [5:22:08<19:54:44,  1.80s/it]
Outer training loop:  20%|##        | 10141/50000 [5:22:10<20:21:34,  1.84s/it]
Outer training loop:  20%|##        | 10142/50000 [5:22:11<19:45:39,  1.78s/it]
Outer training loop:  20%|##        | 10143/50000 [5:22:13<19:44:19,  1.78s/it]
Outer training loop:  20%|##        | 10144/50000 [5:22:15<20:32:13,  1.86s/it]
Outer training loop:  20%|##        | 10145/50000 [5:22:17<20:27:27,  1.85s/it]
Outer training loop:  20%|##        | 10146/50000 [5:22:19<20:12:08,  1.82s/it]
Outer training loop:  20%|##        | 10147/50000 [5:22:21<20:48:08,  1.88s/it]
Outer training loop:  20%|##        | 10148/50000 [5:22:22<20:32:28,  1.86s/it]
Outer training loop:  20%|##        | 10149/50000 [5:22:24<20:00:39,  1.81s/it]
Outer training loop:  20%|##        | 10150/50000 [5:22:26<20:27:27,  1.85s/it]
Outer training loop:  20%|##        | 10151/50000 [5:22:28<20:05:01,  1.81s/it]
Outer training loop:  20%|##        | 10152/50000 [5:22:30<20:26:23,  1.85s/it]
Outer training loop:  20%|##        | 10153/50000 [5:22:31<19:56:16,  1.80s/it]
Outer training loop:  20%|##        | 10154/50000 [5:22:33<20:32:40,  1.86s/it]
Outer training loop:  20%|##        | 10155/50000 [5:22:35<20:17:16,  1.83s/it]
Outer training loop:  20%|##        | 10156/50000 [5:22:37<20:12:44,  1.83s/it]
Outer training loop:  20%|##        | 10157/50000 [5:22:39<20:11:33,  1.82s/it]
Outer training loop:  20%|##        | 10158/50000 [5:22:41<20:06:40,  1.82s/it]
Outer training loop:  20%|##        | 10159/50000 [5:22:42<19:39:23,  1.78s/it]
Outer training loop:  20%|##        | 10160/50000 [5:22:44<19:59:14,  1.81s/it]
Outer training loop:  20%|##        | 10161/50000 [5:22:46<20:08:20,  1.82s/it]
Outer training loop:  20%|##        | 10162/50000 [5:22:48<19:49:55,  1.79s/it]
Outer training loop:  20%|##        | 10163/50000 [5:22:50<20:14:35,  1.83s/it]
Outer training loop:  20%|##        | 10164/50000 [5:22:52<20:16:16,  1.83s/it]
Outer training loop:  20%|##        | 10165/50000 [5:22:53<20:02:40,  1.81s/it]
Outer training loop:  20%|##        | 10166/50000 [5:22:55<20:13:45,  1.83s/it]
Outer training loop:  20%|##        | 10167/50000 [5:22:57<20:01:17,  1.81s/it]
Outer training loop:  20%|##        | 10168/50000 [5:22:59<19:19:13,  1.75s/it]
Outer training loop:  20%|##        | 10169/50000 [5:23:00<19:37:59,  1.77s/it]
Outer training loop:  20%|##        | 10170/50000 [5:23:02<19:52:44,  1.80s/it]
Outer training loop:  20%|##        | 10171/50000 [5:23:04<19:41:56,  1.78s/it]
Outer training loop:  20%|##        | 10172/50000 [5:23:06<20:01:02,  1.81s/it]
Outer training loop:  20%|##        | 10173/50000 [5:23:08<19:30:04,  1.76s/it]
Outer training loop:  20%|##        | 10174/50000 [5:23:09<19:45:19,  1.79s/it]
Outer training loop:  20%|##        | 10175/50000 [5:23:11<19:49:54,  1.79s/it]
Outer training loop:  20%|##        | 10176/50000 [5:23:13<19:41:28,  1.78s/it]
Outer training loop:  20%|##        | 10177/50000 [5:23:15<20:23:22,  1.84s/it]
Outer training loop:  20%|##        | 10178/50000 [5:23:17<20:13:13,  1.83s/it]
Outer training loop:  20%|##        | 10179/50000 [5:23:18<19:56:48,  1.80s/it]
Outer training loop:  20%|##        | 10180/50000 [5:23:20<20:05:52,  1.82s/it]
Outer training loop:  20%|##        | 10181/50000 [5:23:22<19:47:55,  1.79s/it]
Outer training loop:  20%|##        | 10182/50000 [5:23:24<19:25:46,  1.76s/it]
Outer training loop:  20%|##        | 10183/50000 [5:23:26<19:41:25,  1.78s/it]
Outer training loop:  20%|##        | 10184/50000 [5:23:28<20:20:35,  1.84s/it]
Outer training loop:  20%|##        | 10185/50000 [5:23:29<19:50:39,  1.79s/it]
Outer training loop:  20%|##        | 10186/50000 [5:23:31<19:47:02,  1.79s/it]
Outer training loop:  20%|##        | 10187/50000 [5:23:33<20:05:56,  1.82s/it]
Outer training loop:  20%|##        | 10188/50000 [5:23:35<19:58:43,  1.81s/it]
Outer training loop:  20%|##        | 10189/50000 [5:23:36<19:31:21,  1.77s/it]
Outer training loop:  20%|##        | 10190/50000 [5:23:38<20:04:18,  1.82s/it]
Outer training loop:  20%|##        | 10191/50000 [5:23:40<20:00:34,  1.81s/it]
Outer training loop:  20%|##        | 10192/50000 [5:23:42<19:35:26,  1.77s/it]
Outer training loop:  20%|##        | 10193/50000 [5:23:44<19:54:58,  1.80s/it]
Outer training loop:  20%|##        | 10194/50000 [5:23:45<19:29:15,  1.76s/it]
Outer training loop:  20%|##        | 10195/50000 [5:23:47<19:36:33,  1.77s/it]
Outer training loop:  20%|##        | 10196/50000 [5:23:49<19:25:28,  1.76s/it]
Outer training loop:  20%|##        | 10197/50000 [5:23:51<20:04:20,  1.82s/it]
Outer training loop:  20%|##        | 10198/50000 [5:23:53<19:53:19,  1.80s/it]
Outer training loop:  20%|##        | 10199/50000 [5:23:54<19:39:05,  1.78s/it]
Outer training loop:  20%|##        | 10200/50000 [5:23:56<20:01:00,  1.81s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.72s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  20%|##        | 10201/50000 [5:24:03<37:07:10,  3.36s/it]
Outer training loop:  20%|##        | 10202/50000 [5:24:05<31:48:13,  2.88s/it]
Outer training loop:  20%|##        | 10203/50000 [5:24:06<27:33:25,  2.49s/it]
Outer training loop:  20%|##        | 10204/50000 [5:24:08<25:09:54,  2.28s/it]
Outer training loop:  20%|##        | 10205/50000 [5:24:10<23:53:11,  2.16s/it]
Outer training loop:  20%|##        | 10206/50000 [5:24:12<22:50:13,  2.07s/it]
Outer training loop:  20%|##        | 10207/50000 [5:24:14<21:48:56,  1.97s/it]
Outer training loop:  20%|##        | 10208/50000 [5:24:16<21:17:23,  1.93s/it]
Outer training loop:  20%|##        | 10209/50000 [5:24:17<20:47:40,  1.88s/it]
Outer training loop:  20%|##        | 10210/50000 [5:24:19<20:31:10,  1.86s/it]
Outer training loop:  20%|##        | 10211/50000 [5:24:21<20:51:05,  1.89s/it]
Outer training loop:  20%|##        | 10212/50000 [5:24:23<20:05:10,  1.82s/it]
Outer training loop:  20%|##        | 10213/50000 [5:24:24<19:45:18,  1.79s/it]
Outer training loop:  20%|##        | 10214/50000 [5:24:26<20:24:17,  1.85s/it]
Outer training loop:  20%|##        | 10215/50000 [5:24:28<20:03:15,  1.81s/it]
Outer training loop:  20%|##        | 10216/50000 [5:24:30<19:40:50,  1.78s/it]
Outer training loop:  20%|##        | 10217/50000 [5:24:32<19:22:35,  1.75s/it]
Outer training loop:  20%|##        | 10218/50000 [5:24:33<19:59:59,  1.81s/it]
Outer training loop:  20%|##        | 10219/50000 [5:24:35<19:48:49,  1.79s/it]
Outer training loop:  20%|##        | 10220/50000 [5:24:37<19:36:08,  1.77s/it]
Outer training loop:  20%|##        | 10221/50000 [5:24:39<20:14:23,  1.83s/it]
Outer training loop:  20%|##        | 10222/50000 [5:24:41<20:39:48,  1.87s/it]
Outer training loop:  20%|##        | 10223/50000 [5:24:43<20:15:31,  1.83s/it]
Outer training loop:  20%|##        | 10224/50000 [5:24:45<20:26:22,  1.85s/it]
Outer training loop:  20%|##        | 10225/50000 [5:24:46<20:02:15,  1.81s/it]
Outer training loop:  20%|##        | 10226/50000 [5:24:48<19:51:03,  1.80s/it]
Outer training loop:  20%|##        | 10227/50000 [5:24:50<19:31:34,  1.77s/it]
Outer training loop:  20%|##        | 10228/50000 [5:24:52<19:59:06,  1.81s/it]
Outer training loop:  20%|##        | 10229/50000 [5:24:53<19:48:52,  1.79s/it]
Outer training loop:  20%|##        | 10230/50000 [5:24:55<19:52:06,  1.80s/it]
Outer training loop:  20%|##        | 10231/50000 [5:24:57<20:11:06,  1.83s/it]
Outer training loop:  20%|##        | 10232/50000 [5:24:59<19:39:26,  1.78s/it]
Outer training loop:  20%|##        | 10233/50000 [5:25:01<19:44:06,  1.79s/it]
Outer training loop:  20%|##        | 10234/50000 [5:25:02<20:03:11,  1.82s/it]
Outer training loop:  20%|##        | 10235/50000 [5:25:04<19:51:40,  1.80s/it]
Outer training loop:  20%|##        | 10236/50000 [5:25:06<19:40:52,  1.78s/it]
Outer training loop:  20%|##        | 10237/50000 [5:25:08<19:53:18,  1.80s/it]
Outer training loop:  20%|##        | 10238/50000 [5:25:09<19:34:57,  1.77s/it]
Outer training loop:  20%|##        | 10239/50000 [5:25:11<19:27:15,  1.76s/it]
Outer training loop:  20%|##        | 10240/50000 [5:25:13<19:45:06,  1.79s/it]
Outer training loop:  20%|##        | 10241/50000 [5:25:15<19:31:48,  1.77s/it]
Outer training loop:  20%|##        | 10242/50000 [5:25:17<19:42:50,  1.79s/it]
Outer training loop:  20%|##        | 10243/50000 [5:25:18<19:19:17,  1.75s/it]
Outer training loop:  20%|##        | 10244/50000 [5:25:20<19:49:58,  1.80s/it]
Outer training loop:  20%|##        | 10245/50000 [5:25:22<20:03:46,  1.82s/it]
Outer training loop:  20%|##        | 10246/50000 [5:25:24<19:30:49,  1.77s/it]
Outer training loop:  20%|##        | 10247/50000 [5:25:26<20:34:50,  1.86s/it]
Outer training loop:  20%|##        | 10248/50000 [5:25:27<19:57:29,  1.81s/it]
Outer training loop:  20%|##        | 10249/50000 [5:25:29<19:54:23,  1.80s/it]
Outer training loop:  20%|##        | 10250/50000 [5:25:31<20:02:39,  1.82s/it]
Outer training loop:  21%|##        | 10251/50000 [5:25:33<19:58:49,  1.81s/it]
Outer training loop:  21%|##        | 10252/50000 [5:25:35<19:47:40,  1.79s/it]
Outer training loop:  21%|##        | 10253/50000 [5:25:36<19:31:29,  1.77s/it]
Outer training loop:  21%|##        | 10254/50000 [5:25:38<20:08:09,  1.82s/it]
Outer training loop:  21%|##        | 10255/50000 [5:25:40<20:00:00,  1.81s/it]
Outer training loop:  21%|##        | 10256/50000 [5:25:42<19:42:15,  1.78s/it]
Outer training loop:  21%|##        | 10257/50000 [5:25:44<20:08:07,  1.82s/it]
Outer training loop:  21%|##        | 10258/50000 [5:25:46<19:54:17,  1.80s/it]
Outer training loop:  21%|##        | 10259/50000 [5:25:47<19:55:12,  1.80s/it]
Outer training loop:  21%|##        | 10260/50000 [5:25:49<20:28:37,  1.85s/it]
Outer training loop:  21%|##        | 10261/50000 [5:25:51<20:24:44,  1.85s/it]
Outer training loop:  21%|##        | 10262/50000 [5:25:53<20:01:39,  1.81s/it]
Outer training loop:  21%|##        | 10263/50000 [5:25:55<20:37:28,  1.87s/it]
Outer training loop:  21%|##        | 10264/50000 [5:25:57<20:31:27,  1.86s/it]
Outer training loop:  21%|##        | 10265/50000 [5:25:58<20:17:41,  1.84s/it]
Outer training loop:  21%|##        | 10266/50000 [5:26:00<19:56:58,  1.81s/it]
Outer training loop:  21%|##        | 10267/50000 [5:26:02<20:05:40,  1.82s/it]
Outer training loop:  21%|##        | 10268/50000 [5:26:04<19:53:42,  1.80s/it]
Outer training loop:  21%|##        | 10269/50000 [5:26:06<19:41:04,  1.78s/it]
Outer training loop:  21%|##        | 10270/50000 [5:26:07<19:32:08,  1.77s/it]
Outer training loop:  21%|##        | 10271/50000 [5:26:09<19:59:47,  1.81s/it]
Outer training loop:  21%|##        | 10272/50000 [5:26:11<20:22:11,  1.85s/it]
Outer training loop:  21%|##        | 10273/50000 [5:26:13<20:06:13,  1.82s/it]
Outer training loop:  21%|##        | 10274/50000 [5:26:15<20:25:11,  1.85s/it]
Outer training loop:  21%|##        | 10275/50000 [5:26:17<19:54:21,  1.80s/it]
Outer training loop:  21%|##        | 10276/50000 [5:26:18<19:58:49,  1.81s/it]
Outer training loop:  21%|##        | 10277/50000 [5:26:20<20:18:02,  1.84s/it]
Outer training loop:  21%|##        | 10278/50000 [5:26:22<20:03:45,  1.82s/it]
Outer training loop:  21%|##        | 10279/50000 [5:26:24<19:43:53,  1.79s/it]
Outer training loop:  21%|##        | 10280/50000 [5:26:26<20:15:04,  1.84s/it]
Outer training loop:  21%|##        | 10281/50000 [5:26:27<19:59:56,  1.81s/it]
Outer training loop:  21%|##        | 10282/50000 [5:26:29<19:37:57,  1.78s/it]
Outer training loop:  21%|##        | 10283/50000 [5:26:31<20:29:34,  1.86s/it]
Outer training loop:  21%|##        | 10284/50000 [5:26:33<20:47:17,  1.88s/it]
Outer training loop:  21%|##        | 10285/50000 [5:26:35<20:58:46,  1.90s/it]
Outer training loop:  21%|##        | 10286/50000 [5:26:37<21:10:40,  1.92s/it]
Outer training loop:  21%|##        | 10287/50000 [5:26:39<20:38:38,  1.87s/it]
Outer training loop:  21%|##        | 10288/50000 [5:26:41<20:20:51,  1.84s/it]
Outer training loop:  21%|##        | 10289/50000 [5:26:42<20:22:52,  1.85s/it]
Outer training loop:  21%|##        | 10290/50000 [5:26:44<20:38:43,  1.87s/it]
Outer training loop:  21%|##        | 10291/50000 [5:26:46<20:25:32,  1.85s/it]
Outer training loop:  21%|##        | 10292/50000 [5:26:48<20:04:04,  1.82s/it]
Outer training loop:  21%|##        | 10293/50000 [5:26:50<20:17:46,  1.84s/it]
Outer training loop:  21%|##        | 10294/50000 [5:26:52<20:00:26,  1.81s/it]
Outer training loop:  21%|##        | 10295/50000 [5:26:53<19:54:55,  1.81s/it]
Outer training loop:  21%|##        | 10296/50000 [5:26:55<19:42:23,  1.79s/it]
Outer training loop:  21%|##        | 10297/50000 [5:26:57<20:09:08,  1.83s/it]
Outer training loop:  21%|##        | 10298/50000 [5:26:59<20:20:12,  1.84s/it]
Outer training loop:  21%|##        | 10299/50000 [5:27:01<20:08:27,  1.83s/it]
Outer training loop:  21%|##        | 10300/50000 [5:27:03<20:42:29,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  21%|##        | 10301/50000 [5:27:10<37:40:12,  3.42s/it]
Outer training loop:  21%|##        | 10302/50000 [5:27:11<31:59:27,  2.90s/it]
Outer training loop:  21%|##        | 10303/50000 [5:27:13<27:58:35,  2.54s/it]
Outer training loop:  21%|##        | 10304/50000 [5:27:15<26:08:11,  2.37s/it]
Outer training loop:  21%|##        | 10305/50000 [5:27:17<24:19:41,  2.21s/it]
Outer training loop:  21%|##        | 10306/50000 [5:27:19<22:48:20,  2.07s/it]
Outer training loop:  21%|##        | 10307/50000 [5:27:20<21:34:26,  1.96s/it]
Outer training loop:  21%|##        | 10308/50000 [5:27:22<21:25:09,  1.94s/it]
Outer training loop:  21%|##        | 10309/50000 [5:27:24<20:33:35,  1.86s/it]
Outer training loop:  21%|##        | 10310/50000 [5:27:26<20:28:25,  1.86s/it]
Outer training loop:  21%|##        | 10311/50000 [5:27:28<20:41:31,  1.88s/it]
Outer training loop:  21%|##        | 10312/50000 [5:27:29<20:17:19,  1.84s/it]
Outer training loop:  21%|##        | 10313/50000 [5:27:31<19:57:35,  1.81s/it]
Outer training loop:  21%|##        | 10314/50000 [5:27:33<20:26:57,  1.86s/it]
Outer training loop:  21%|##        | 10315/50000 [5:27:35<20:44:14,  1.88s/it]
Outer training loop:  21%|##        | 10316/50000 [5:27:37<20:22:40,  1.85s/it]
Outer training loop:  21%|##        | 10317/50000 [5:27:39<20:54:54,  1.90s/it]
Outer training loop:  21%|##        | 10318/50000 [5:27:41<20:32:45,  1.86s/it]
Outer training loop:  21%|##        | 10319/50000 [5:27:42<20:26:13,  1.85s/it]
Outer training loop:  21%|##        | 10320/50000 [5:27:44<20:19:34,  1.84s/it]
Outer training loop:  21%|##        | 10321/50000 [5:27:46<20:50:13,  1.89s/it]
Outer training loop:  21%|##        | 10322/50000 [5:27:48<20:25:44,  1.85s/it]
Outer training loop:  21%|##        | 10323/50000 [5:27:50<20:27:00,  1.86s/it]
Outer training loop:  21%|##        | 10324/50000 [5:27:52<20:46:39,  1.89s/it]
Outer training loop:  21%|##        | 10325/50000 [5:27:54<20:28:35,  1.86s/it]
Outer training loop:  21%|##        | 10326/50000 [5:27:55<20:12:23,  1.83s/it]
Outer training loop:  21%|##        | 10327/50000 [5:27:57<20:03:37,  1.82s/it]
Outer training loop:  21%|##        | 10328/50000 [5:27:59<21:13:05,  1.93s/it]
Outer training loop:  21%|##        | 10329/50000 [5:28:01<20:43:54,  1.88s/it]
Outer training loop:  21%|##        | 10330/50000 [5:28:03<21:13:37,  1.93s/it]
Outer training loop:  21%|##        | 10331/50000 [5:28:05<21:18:06,  1.93s/it]
Outer training loop:  21%|##        | 10332/50000 [5:28:07<20:52:23,  1.89s/it]
Outer training loop:  21%|##        | 10333/50000 [5:28:09<20:30:22,  1.86s/it]
Outer training loop:  21%|##        | 10334/50000 [5:28:11<20:25:21,  1.85s/it]
Outer training loop:  21%|##        | 10335/50000 [5:28:12<20:33:31,  1.87s/it]
Outer training loop:  21%|##        | 10336/50000 [5:28:14<19:54:08,  1.81s/it]
Outer training loop:  21%|##        | 10337/50000 [5:28:16<20:04:04,  1.82s/it]
Outer training loop:  21%|##        | 10338/50000 [5:28:18<19:55:32,  1.81s/it]
Outer training loop:  21%|##        | 10339/50000 [5:28:19<19:29:59,  1.77s/it]
Outer training loop:  21%|##        | 10340/50000 [5:28:21<19:26:10,  1.76s/it]
Outer training loop:  21%|##        | 10341/50000 [5:28:23<19:53:07,  1.81s/it]
Outer training loop:  21%|##        | 10342/50000 [5:28:25<19:53:55,  1.81s/it]
Outer training loop:  21%|##        | 10343/50000 [5:28:27<19:35:51,  1.78s/it]
Outer training loop:  21%|##        | 10344/50000 [5:28:28<19:31:52,  1.77s/it]
Outer training loop:  21%|##        | 10345/50000 [5:28:30<19:45:18,  1.79s/it]
Outer training loop:  21%|##        | 10346/50000 [5:28:32<19:33:00,  1.77s/it]
Outer training loop:  21%|##        | 10347/50000 [5:28:34<19:46:52,  1.80s/it]
Outer training loop:  21%|##        | 10348/50000 [5:28:36<20:30:57,  1.86s/it]
Outer training loop:  21%|##        | 10349/50000 [5:28:38<19:55:23,  1.81s/it]
Outer training loop:  21%|##        | 10350/50000 [5:28:39<19:54:39,  1.81s/it]
Outer training loop:  21%|##        | 10351/50000 [5:28:41<19:42:34,  1.79s/it]
Outer training loop:  21%|##        | 10352/50000 [5:28:43<20:11:40,  1.83s/it]
Outer training loop:  21%|##        | 10353/50000 [5:28:45<19:38:35,  1.78s/it]
Outer training loop:  21%|##        | 10354/50000 [5:28:46<19:18:06,  1.75s/it]
Outer training loop:  21%|##        | 10355/50000 [5:28:48<19:47:37,  1.80s/it]
Outer training loop:  21%|##        | 10356/50000 [5:28:50<19:40:58,  1.79s/it]
Outer training loop:  21%|##        | 10357/50000 [5:28:52<19:25:27,  1.76s/it]
Outer training loop:  21%|##        | 10358/50000 [5:28:54<20:05:07,  1.82s/it]
Outer training loop:  21%|##        | 10359/50000 [5:28:55<19:55:23,  1.81s/it]
Outer training loop:  21%|##        | 10360/50000 [5:28:57<19:42:42,  1.79s/it]
Outer training loop:  21%|##        | 10361/50000 [5:28:59<20:28:11,  1.86s/it]
Outer training loop:  21%|##        | 10362/50000 [5:29:01<19:50:48,  1.80s/it]
Outer training loop:  21%|##        | 10363/50000 [5:29:03<19:38:27,  1.78s/it]
Outer training loop:  21%|##        | 10364/50000 [5:29:05<20:07:31,  1.83s/it]
Outer training loop:  21%|##        | 10365/50000 [5:29:06<19:53:06,  1.81s/it]
Outer training loop:  21%|##        | 10366/50000 [5:29:08<19:36:49,  1.78s/it]
Outer training loop:  21%|##        | 10367/50000 [5:29:10<19:27:48,  1.77s/it]
Outer training loop:  21%|##        | 10368/50000 [5:29:12<20:05:29,  1.83s/it]
Outer training loop:  21%|##        | 10369/50000 [5:29:14<20:10:23,  1.83s/it]
Outer training loop:  21%|##        | 10370/50000 [5:29:15<19:49:31,  1.80s/it]
Outer training loop:  21%|##        | 10371/50000 [5:29:17<19:37:23,  1.78s/it]
Outer training loop:  21%|##        | 10372/50000 [5:29:19<20:01:45,  1.82s/it]
Outer training loop:  21%|##        | 10373/50000 [5:29:21<20:14:00,  1.84s/it]
Outer training loop:  21%|##        | 10374/50000 [5:29:23<19:53:04,  1.81s/it]
Outer training loop:  21%|##        | 10375/50000 [5:29:24<20:12:00,  1.84s/it]
Outer training loop:  21%|##        | 10376/50000 [5:29:26<20:05:40,  1.83s/it]
Outer training loop:  21%|##        | 10377/50000 [5:29:28<19:49:04,  1.80s/it]
Outer training loop:  21%|##        | 10378/50000 [5:29:30<20:11:41,  1.83s/it]
Outer training loop:  21%|##        | 10379/50000 [5:29:32<19:57:43,  1.81s/it]
Outer training loop:  21%|##        | 10380/50000 [5:29:33<19:32:02,  1.77s/it]
Outer training loop:  21%|##        | 10381/50000 [5:29:35<19:16:01,  1.75s/it]
Outer training loop:  21%|##        | 10382/50000 [5:29:37<19:46:27,  1.80s/it]
Outer training loop:  21%|##        | 10383/50000 [5:29:39<19:35:42,  1.78s/it]
Outer training loop:  21%|##        | 10384/50000 [5:29:41<19:40:55,  1.79s/it]
Outer training loop:  21%|##        | 10385/50000 [5:29:42<20:09:32,  1.83s/it]
Outer training loop:  21%|##        | 10386/50000 [5:29:44<20:03:52,  1.82s/it]
Outer training loop:  21%|##        | 10387/50000 [5:29:46<19:48:57,  1.80s/it]
Outer training loop:  21%|##        | 10388/50000 [5:29:48<20:01:28,  1.82s/it]
Outer training loop:  21%|##        | 10389/50000 [5:29:50<19:59:16,  1.82s/it]
Outer training loop:  21%|##        | 10390/50000 [5:29:51<19:26:51,  1.77s/it]
Outer training loop:  21%|##        | 10391/50000 [5:29:53<19:35:39,  1.78s/it]
Outer training loop:  21%|##        | 10392/50000 [5:29:55<19:30:35,  1.77s/it]
Outer training loop:  21%|##        | 10393/50000 [5:29:57<19:05:05,  1.73s/it]
Outer training loop:  21%|##        | 10394/50000 [5:29:58<18:58:02,  1.72s/it]
Outer training loop:  21%|##        | 10395/50000 [5:30:00<19:28:21,  1.77s/it]
Outer training loop:  21%|##        | 10396/50000 [5:30:02<19:27:35,  1.77s/it]
Outer training loop:  21%|##        | 10397/50000 [5:30:04<19:31:25,  1.77s/it]
Outer training loop:  21%|##        | 10398/50000 [5:30:06<19:43:04,  1.79s/it]
Outer training loop:  21%|##        | 10399/50000 [5:30:07<19:54:08,  1.81s/it]
Outer training loop:  21%|##        | 10400/50000 [5:30:09<19:33:13,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.70s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]

Outer training loop:  21%|##        | 10401/50000 [5:30:16<37:15:29,  3.39s/it]
Outer training loop:  21%|##        | 10402/50000 [5:30:18<31:49:06,  2.89s/it]
Outer training loop:  21%|##        | 10403/50000 [5:30:20<28:07:23,  2.56s/it]
Outer training loop:  21%|##        | 10404/50000 [5:30:22<26:01:22,  2.37s/it]
Outer training loop:  21%|##        | 10405/50000 [5:30:23<23:41:59,  2.15s/it]
Outer training loop:  21%|##        | 10406/50000 [5:30:25<22:21:18,  2.03s/it]
Outer training loop:  21%|##        | 10407/50000 [5:30:27<21:26:30,  1.95s/it]
Outer training loop:  21%|##        | 10408/50000 [5:30:29<21:01:14,  1.91s/it]
Outer training loop:  21%|##        | 10409/50000 [5:30:30<20:21:18,  1.85s/it]
Outer training loop:  21%|##        | 10410/50000 [5:30:32<20:37:45,  1.88s/it]
Outer training loop:  21%|##        | 10411/50000 [5:30:34<20:14:11,  1.84s/it]
Outer training loop:  21%|##        | 10412/50000 [5:30:36<19:57:23,  1.81s/it]
Outer training loop:  21%|##        | 10413/50000 [5:30:38<19:46:13,  1.80s/it]
Outer training loop:  21%|##        | 10414/50000 [5:30:40<20:23:16,  1.85s/it]
Outer training loop:  21%|##        | 10415/50000 [5:30:41<19:58:28,  1.82s/it]
Outer training loop:  21%|##        | 10416/50000 [5:30:43<19:58:05,  1.82s/it]
Outer training loop:  21%|##        | 10417/50000 [5:30:45<20:17:56,  1.85s/it]
Outer training loop:  21%|##        | 10418/50000 [5:30:47<19:37:41,  1.79s/it]
Outer training loop:  21%|##        | 10419/50000 [5:30:48<19:18:37,  1.76s/it]
Outer training loop:  21%|##        | 10420/50000 [5:30:50<20:28:33,  1.86s/it]
Outer training loop:  21%|##        | 10421/50000 [5:30:52<20:04:22,  1.83s/it]
Outer training loop:  21%|##        | 10422/50000 [5:30:54<19:54:58,  1.81s/it]
Outer training loop:  21%|##        | 10423/50000 [5:30:56<19:50:26,  1.80s/it]
Outer training loop:  21%|##        | 10424/50000 [5:30:58<20:33:51,  1.87s/it]
Outer training loop:  21%|##        | 10425/50000 [5:31:00<20:20:01,  1.85s/it]
Outer training loop:  21%|##        | 10426/50000 [5:31:01<20:15:01,  1.84s/it]
Outer training loop:  21%|##        | 10427/50000 [5:31:03<19:58:42,  1.82s/it]
Outer training loop:  21%|##        | 10428/50000 [5:31:05<19:49:47,  1.80s/it]
Outer training loop:  21%|##        | 10429/50000 [5:31:07<19:22:14,  1.76s/it]
Outer training loop:  21%|##        | 10430/50000 [5:31:08<19:50:52,  1.81s/it]
Outer training loop:  21%|##        | 10431/50000 [5:31:10<19:32:45,  1.78s/it]
Outer training loop:  21%|##        | 10432/50000 [5:31:12<19:10:17,  1.74s/it]
Outer training loop:  21%|##        | 10433/50000 [5:31:14<19:46:12,  1.80s/it]
Outer training loop:  21%|##        | 10434/50000 [5:31:15<19:10:26,  1.74s/it]
Outer training loop:  21%|##        | 10435/50000 [5:31:17<19:05:11,  1.74s/it]
Outer training loop:  21%|##        | 10436/50000 [5:31:19<19:30:27,  1.78s/it]
Outer training loop:  21%|##        | 10437/50000 [5:31:21<19:14:40,  1.75s/it]
Outer training loop:  21%|##        | 10438/50000 [5:31:22<18:57:02,  1.72s/it]
Outer training loop:  21%|##        | 10439/50000 [5:31:24<18:59:23,  1.73s/it]
Outer training loop:  21%|##        | 10440/50000 [5:31:26<19:40:58,  1.79s/it]
Outer training loop:  21%|##        | 10441/50000 [5:31:28<19:32:24,  1.78s/it]
Outer training loop:  21%|##        | 10442/50000 [5:31:30<19:32:12,  1.78s/it]
Outer training loop:  21%|##        | 10443/50000 [5:31:31<19:34:14,  1.78s/it]
Outer training loop:  21%|##        | 10444/50000 [5:31:33<19:24:33,  1.77s/it]
Outer training loop:  21%|##        | 10445/50000 [5:31:35<19:07:53,  1.74s/it]
Outer training loop:  21%|##        | 10446/50000 [5:31:37<19:33:31,  1.78s/it]
Outer training loop:  21%|##        | 10447/50000 [5:31:38<19:31:37,  1.78s/it]
Outer training loop:  21%|##        | 10448/50000 [5:31:40<19:23:30,  1.77s/it]
Outer training loop:  21%|##        | 10449/50000 [5:31:42<19:03:32,  1.73s/it]
Outer training loop:  21%|##        | 10450/50000 [5:31:44<19:26:00,  1.77s/it]
Outer training loop:  21%|##        | 10451/50000 [5:31:45<19:14:41,  1.75s/it]
Outer training loop:  21%|##        | 10452/50000 [5:31:47<19:13:39,  1.75s/it]
Outer training loop:  21%|##        | 10453/50000 [5:31:49<19:37:56,  1.79s/it]
Outer training loop:  21%|##        | 10454/50000 [5:31:51<19:17:56,  1.76s/it]
Outer training loop:  21%|##        | 10455/50000 [5:31:52<19:05:43,  1.74s/it]
Outer training loop:  21%|##        | 10456/50000 [5:31:54<19:31:02,  1.78s/it]
Outer training loop:  21%|##        | 10457/50000 [5:31:56<19:18:18,  1.76s/it]
Outer training loop:  21%|##        | 10458/50000 [5:31:58<18:52:13,  1.72s/it]
Outer training loop:  21%|##        | 10459/50000 [5:31:59<19:28:50,  1.77s/it]
Outer training loop:  21%|##        | 10460/50000 [5:32:01<19:27:54,  1.77s/it]
Outer training loop:  21%|##        | 10461/50000 [5:32:03<19:12:25,  1.75s/it]
Outer training loop:  21%|##        | 10462/50000 [5:32:05<19:02:33,  1.73s/it]
Outer training loop:  21%|##        | 10463/50000 [5:32:06<19:27:05,  1.77s/it]
Outer training loop:  21%|##        | 10464/50000 [5:32:08<19:06:41,  1.74s/it]
Outer training loop:  21%|##        | 10465/50000 [5:32:10<19:03:01,  1.73s/it]
Outer training loop:  21%|##        | 10466/50000 [5:32:12<19:51:10,  1.81s/it]
Outer training loop:  21%|##        | 10467/50000 [5:32:14<19:19:57,  1.76s/it]
Outer training loop:  21%|##        | 10468/50000 [5:32:15<19:08:15,  1.74s/it]
Outer training loop:  21%|##        | 10469/50000 [5:32:17<19:32:35,  1.78s/it]
Outer training loop:  21%|##        | 10470/50000 [5:32:19<19:24:47,  1.77s/it]
Outer training loop:  21%|##        | 10471/50000 [5:32:21<19:19:08,  1.76s/it]
Outer training loop:  21%|##        | 10472/50000 [5:32:22<19:27:01,  1.77s/it]
Outer training loop:  21%|##        | 10473/50000 [5:32:24<19:13:30,  1.75s/it]
Outer training loop:  21%|##        | 10474/50000 [5:32:26<19:11:18,  1.75s/it]
Outer training loop:  21%|##        | 10475/50000 [5:32:28<19:06:26,  1.74s/it]
Outer training loop:  21%|##        | 10476/50000 [5:32:29<19:20:07,  1.76s/it]
Outer training loop:  21%|##        | 10477/50000 [5:32:31<19:18:45,  1.76s/it]
Outer training loop:  21%|##        | 10478/50000 [5:32:33<19:01:45,  1.73s/it]
Outer training loop:  21%|##        | 10479/50000 [5:32:35<19:51:10,  1.81s/it]
Outer training loop:  21%|##        | 10480/50000 [5:32:36<19:31:40,  1.78s/it]
Outer training loop:  21%|##        | 10481/50000 [5:32:38<19:31:09,  1.78s/it]
Outer training loop:  21%|##        | 10482/50000 [5:32:40<19:46:57,  1.80s/it]
Outer training loop:  21%|##        | 10483/50000 [5:32:42<19:36:05,  1.79s/it]
Outer training loop:  21%|##        | 10484/50000 [5:32:44<19:20:30,  1.76s/it]
Outer training loop:  21%|##        | 10485/50000 [5:32:45<19:47:31,  1.80s/it]
Outer training loop:  21%|##        | 10486/50000 [5:32:47<19:33:11,  1.78s/it]
Outer training loop:  21%|##        | 10487/50000 [5:32:49<19:29:37,  1.78s/it]
Outer training loop:  21%|##        | 10488/50000 [5:32:51<20:02:37,  1.83s/it]
Outer training loop:  21%|##        | 10489/50000 [5:32:53<19:43:28,  1.80s/it]
Outer training loop:  21%|##        | 10490/50000 [5:32:54<19:13:03,  1.75s/it]
Outer training loop:  21%|##        | 10491/50000 [5:32:56<19:10:05,  1.75s/it]
Outer training loop:  21%|##        | 10492/50000 [5:32:58<19:56:25,  1.82s/it]
Outer training loop:  21%|##        | 10493/50000 [5:33:00<19:42:03,  1.80s/it]
Outer training loop:  21%|##        | 10494/50000 [5:33:01<19:37:55,  1.79s/it]
Outer training loop:  21%|##        | 10495/50000 [5:33:03<19:42:45,  1.80s/it]
Outer training loop:  21%|##        | 10496/50000 [5:33:05<19:35:00,  1.78s/it]
Outer training loop:  21%|##        | 10497/50000 [5:33:07<19:19:18,  1.76s/it]
Outer training loop:  21%|##        | 10498/50000 [5:33:09<19:52:15,  1.81s/it]
Outer training loop:  21%|##        | 10499/50000 [5:33:10<19:37:13,  1.79s/it]
Outer training loop:  21%|##1       | 10500/50000 [5:33:12<18:56:27,  1.73s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.65s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.65s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:35,  1.90s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:28,  1.84s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.80s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:18,  1.76s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.79s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:17,  1.78s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:13,  1.76s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:14,  1.82s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:09,  1.77s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:07,  1.77s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:12,  1.87s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:06,  1.81s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:03,  1.79s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<01:59,  1.76s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:54,  1.72s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.79s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:55,  1.77s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:53,  1.77s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:53,  1.80s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:54,  1.84s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:49,  1.80s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:48,  1.81s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:44,  1.78s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:42,  1.77s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:44,  1.84s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:43,  1.84s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:40,  1.83s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.78s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.76s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.77s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:29,  1.80s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:27,  1.78s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:23,  1.74s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:21,  1.74s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:18,  1.71s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:21,  1.81s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:18,  1.79s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.73s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:12,  1.73s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:11,  1.75s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:09,  1.73s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:06,  1.72s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:08,  1.80s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:05,  1.76s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:02,  1.73s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<00:59,  1.71s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<00:57,  1.68s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:55,  1.68s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:53,  1.68s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:55,  1.80s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:52,  1.76s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.72s/it]

Evaluating policy:  67%|######7   | 56/83 [01:38<00:46,  1.72s/it]

Evaluating policy:  69%|######8   | 57/83 [01:40<00:44,  1.72s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:42,  1.71s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:43<00:40,  1.69s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:47<00:39,  1.80s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:38,  1.82s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:35,  1.78s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:32,  1.73s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:54<00:30,  1.71s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:28,  1.71s/it]

Evaluating policy:  81%|########  | 67/83 [01:57<00:26,  1.68s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:01<00:25,  1.80s/it]

Evaluating policy:  84%|########4 | 70/83 [02:03<00:23,  1.78s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:20,  1.73s/it]

Evaluating policy:  87%|########6 | 72/83 [02:06<00:19,  1.73s/it]

Evaluating policy:  88%|########7 | 73/83 [02:08<00:16,  1.70s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:15,  1.72s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:13,  1.74s/it]

Evaluating policy:  92%|#########1| 76/83 [02:13<00:12,  1.75s/it]

Evaluating policy:  93%|#########2| 77/83 [02:15<00:10,  1.74s/it]

Evaluating policy:  94%|#########3| 78/83 [02:17<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:19<00:07,  1.87s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.84s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.80s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 83/83 [02:26<00:00,  1.73s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:15,  1.65s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:15,  1.67s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:14,  1.68s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:16,  1.73s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.88s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:20,  1.83s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.83s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:18,  1.85s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:13,  1.81s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:09,  1.78s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:09,  1.80s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:10,  1.83s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.79s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:04,  1.80s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:00,  1.78s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:59,  1.78s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:59,  1.82s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:53,  1.77s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:50,  1.76s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:52,  1.81s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:49,  1.79s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:46,  1.77s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:43,  1.75s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:40,  1.73s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:43,  1.82s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:38,  1.76s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:36,  1.76s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.79s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:34,  1.79s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:30,  1.74s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:32,  1.81s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:29,  1.79s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:26,  1.76s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:24,  1.76s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:23,  1.77s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:21,  1.77s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:19,  1.82s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:16,  1.79s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:14,  1.77s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:13,  1.79s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:10,  1.77s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:10,  1.86s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:07,  1.82s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:04,  1.79s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:01,  1.77s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.78s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:57,  1.74s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:53,  1.78s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:48,  1.73s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:47,  1.75s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:44,  1.72s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:43,  1.82s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:41,  1.78s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:38,  1.77s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:36,  1.73s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:34,  1.73s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:33,  1.75s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:30,  1.72s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:31,  1.84s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:28,  1.81s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:26,  1.77s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:24,  1.78s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.76s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:20,  1.74s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:18,  1.71s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.72s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:15,  1.70s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.86s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.81s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.79s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.75s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.81s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.78s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.73s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.72s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.72s/it]

Outer training loop:  21%|##1       | 10501/50000 [5:38:13<1005:21:13, 91.63s/it]
Outer training loop:  21%|##1       | 10502/50000 [5:38:15<709:36:01, 64.68s/it] 
Outer training loop:  21%|##1       | 10503/50000 [5:38:17<504:03:49, 45.94s/it]
Outer training loop:  21%|##1       | 10504/50000 [5:38:19<358:28:10, 32.67s/it]
Outer training loop:  21%|##1       | 10505/50000 [5:38:21<256:56:29, 23.42s/it]
Outer training loop:  21%|##1       | 10506/50000 [5:38:23<186:11:43, 16.97s/it]
Outer training loop:  21%|##1       | 10507/50000 [5:38:25<136:05:41, 12.41s/it]
Outer training loop:  21%|##1       | 10508/50000 [5:38:26<101:02:09,  9.21s/it]
Outer training loop:  21%|##1       | 10509/50000 [5:38:28<76:42:01,  6.99s/it] 
Outer training loop:  21%|##1       | 10510/50000 [5:38:30<60:48:15,  5.54s/it]
Outer training loop:  21%|##1       | 10511/50000 [5:38:32<48:12:08,  4.39s/it]
Outer training loop:  21%|##1       | 10512/50000 [5:38:34<39:26:35,  3.60s/it]
Outer training loop:  21%|##1       | 10513/50000 [5:38:36<33:35:28,  3.06s/it]
Outer training loop:  21%|##1       | 10514/50000 [5:38:37<28:50:50,  2.63s/it]
Outer training loop:  21%|##1       | 10515/50000 [5:38:39<25:46:40,  2.35s/it]
Outer training loop:  21%|##1       | 10516/50000 [5:38:41<24:03:20,  2.19s/it]
Outer training loop:  21%|##1       | 10517/50000 [5:38:43<22:33:15,  2.06s/it]
Outer training loop:  21%|##1       | 10518/50000 [5:38:44<21:30:58,  1.96s/it]
Outer training loop:  21%|##1       | 10519/50000 [5:38:46<20:48:25,  1.90s/it]
Outer training loop:  21%|##1       | 10520/50000 [5:38:48<20:57:36,  1.91s/it]
Outer training loop:  21%|##1       | 10521/50000 [5:38:50<20:31:37,  1.87s/it]
Outer training loop:  21%|##1       | 10522/50000 [5:38:51<19:51:42,  1.81s/it]
Outer training loop:  21%|##1       | 10523/50000 [5:38:53<19:40:19,  1.79s/it]
Outer training loop:  21%|##1       | 10524/50000 [5:38:55<19:27:59,  1.78s/it]
Outer training loop:  21%|##1       | 10525/50000 [5:38:57<19:06:51,  1.74s/it]
Outer training loop:  21%|##1       | 10526/50000 [5:38:58<19:32:25,  1.78s/it]
Outer training loop:  21%|##1       | 10527/50000 [5:39:00<19:06:01,  1.74s/it]
Outer training loop:  21%|##1       | 10528/50000 [5:39:02<19:00:54,  1.73s/it]
Outer training loop:  21%|##1       | 10529/50000 [5:39:03<18:45:35,  1.71s/it]
Outer training loop:  21%|##1       | 10530/50000 [5:39:05<19:19:23,  1.76s/it]
Outer training loop:  21%|##1       | 10531/50000 [5:39:07<19:00:55,  1.73s/it]
Outer training loop:  21%|##1       | 10532/50000 [5:39:09<19:11:09,  1.75s/it]
Outer training loop:  21%|##1       | 10533/50000 [5:39:11<19:17:27,  1.76s/it]
Outer training loop:  21%|##1       | 10534/50000 [5:39:12<18:50:09,  1.72s/it]
Outer training loop:  21%|##1       | 10535/50000 [5:39:14<18:47:20,  1.71s/it]
Outer training loop:  21%|##1       | 10536/50000 [5:39:16<19:29:14,  1.78s/it]
Outer training loop:  21%|##1       | 10537/50000 [5:39:18<19:14:52,  1.76s/it]
Outer training loop:  21%|##1       | 10538/50000 [5:39:19<19:09:46,  1.75s/it]
Outer training loop:  21%|##1       | 10539/50000 [5:39:21<20:05:34,  1.83s/it]
Outer training loop:  21%|##1       | 10540/50000 [5:39:23<19:45:00,  1.80s/it]
Outer training loop:  21%|##1       | 10541/50000 [5:39:25<19:31:14,  1.78s/it]
Outer training loop:  21%|##1       | 10542/50000 [5:39:26<19:19:33,  1.76s/it]
Outer training loop:  21%|##1       | 10543/50000 [5:39:28<19:20:06,  1.76s/it]
Outer training loop:  21%|##1       | 10544/50000 [5:39:30<18:59:12,  1.73s/it]
Outer training loop:  21%|##1       | 10545/50000 [5:39:32<18:53:25,  1.72s/it]
Outer training loop:  21%|##1       | 10546/50000 [5:39:33<18:57:21,  1.73s/it]
Outer training loop:  21%|##1       | 10547/50000 [5:39:36<20:34:31,  1.88s/it]
Outer training loop:  21%|##1       | 10548/50000 [5:39:37<20:09:31,  1.84s/it]
Outer training loop:  21%|##1       | 10549/50000 [5:39:39<19:45:20,  1.80s/it]
Outer training loop:  21%|##1       | 10550/50000 [5:39:41<20:20:07,  1.86s/it]
Outer training loop:  21%|##1       | 10551/50000 [5:39:43<19:53:19,  1.81s/it]
Outer training loop:  21%|##1       | 10552/50000 [5:39:44<19:26:40,  1.77s/it]
Outer training loop:  21%|##1       | 10553/50000 [5:39:46<19:53:11,  1.81s/it]
Outer training loop:  21%|##1       | 10554/50000 [5:39:48<19:42:45,  1.80s/it]
Outer training loop:  21%|##1       | 10555/50000 [5:39:50<19:30:05,  1.78s/it]
Outer training loop:  21%|##1       | 10556/50000 [5:39:52<19:25:45,  1.77s/it]
Outer training loop:  21%|##1       | 10557/50000 [5:39:53<19:48:29,  1.81s/it]
Outer training loop:  21%|##1       | 10558/50000 [5:39:55<20:02:15,  1.83s/it]
Outer training loop:  21%|##1       | 10559/50000 [5:39:57<19:32:44,  1.78s/it]
Outer training loop:  21%|##1       | 10560/50000 [5:39:59<20:03:47,  1.83s/it]
Outer training loop:  21%|##1       | 10561/50000 [5:40:01<19:36:45,  1.79s/it]
Outer training loop:  21%|##1       | 10562/50000 [5:40:02<19:18:28,  1.76s/it]
Outer training loop:  21%|##1       | 10563/50000 [5:40:04<19:30:24,  1.78s/it]
Outer training loop:  21%|##1       | 10564/50000 [5:40:06<19:42:16,  1.80s/it]
Outer training loop:  21%|##1       | 10565/50000 [5:40:08<18:57:25,  1.73s/it]
Outer training loop:  21%|##1       | 10566/50000 [5:40:09<18:52:08,  1.72s/it]
Outer training loop:  21%|##1       | 10567/50000 [5:40:11<18:32:07,  1.69s/it]
Outer training loop:  21%|##1       | 10568/50000 [5:40:13<19:19:32,  1.76s/it]
Outer training loop:  21%|##1       | 10569/50000 [5:40:15<19:13:30,  1.76s/it]
Outer training loop:  21%|##1       | 10570/50000 [5:40:16<19:19:10,  1.76s/it]
Outer training loop:  21%|##1       | 10571/50000 [5:40:18<19:50:12,  1.81s/it]
Outer training loop:  21%|##1       | 10572/50000 [5:40:20<19:40:21,  1.80s/it]
Outer training loop:  21%|##1       | 10573/50000 [5:40:22<19:16:27,  1.76s/it]
Outer training loop:  21%|##1       | 10574/50000 [5:40:24<19:36:06,  1.79s/it]
Outer training loop:  21%|##1       | 10575/50000 [5:40:25<19:35:28,  1.79s/it]
Outer training loop:  21%|##1       | 10576/50000 [5:40:27<19:28:48,  1.78s/it]
Outer training loop:  21%|##1       | 10577/50000 [5:40:29<19:35:36,  1.79s/it]
Outer training loop:  21%|##1       | 10578/50000 [5:40:31<19:01:34,  1.74s/it]
Outer training loop:  21%|##1       | 10579/50000 [5:40:32<18:33:24,  1.69s/it]
Outer training loop:  21%|##1       | 10580/50000 [5:40:34<18:29:51,  1.69s/it]
Outer training loop:  21%|##1       | 10581/50000 [5:40:36<18:57:53,  1.73s/it]
Outer training loop:  21%|##1       | 10582/50000 [5:40:37<18:50:38,  1.72s/it]
Outer training loop:  21%|##1       | 10583/50000 [5:40:39<18:35:19,  1.70s/it]
Outer training loop:  21%|##1       | 10584/50000 [5:40:41<19:00:06,  1.74s/it]
Outer training loop:  21%|##1       | 10585/50000 [5:40:42<18:33:44,  1.70s/it]
Outer training loop:  21%|##1       | 10586/50000 [5:40:44<18:31:41,  1.69s/it]
Outer training loop:  21%|##1       | 10587/50000 [5:40:46<18:57:26,  1.73s/it]
Outer training loop:  21%|##1       | 10588/50000 [5:40:48<18:41:12,  1.71s/it]
Outer training loop:  21%|##1       | 10589/50000 [5:40:49<18:37:19,  1.70s/it]
Outer training loop:  21%|##1       | 10590/50000 [5:40:51<19:40:01,  1.80s/it]
Outer training loop:  21%|##1       | 10591/50000 [5:40:53<19:08:14,  1.75s/it]
Outer training loop:  21%|##1       | 10592/50000 [5:40:55<19:10:08,  1.75s/it]
Outer training loop:  21%|##1       | 10593/50000 [5:40:56<19:12:52,  1.76s/it]
Outer training loop:  21%|##1       | 10594/50000 [5:40:58<19:27:16,  1.78s/it]
Outer training loop:  21%|##1       | 10595/50000 [5:41:00<18:58:27,  1.73s/it]
Outer training loop:  21%|##1       | 10596/50000 [5:41:02<18:40:34,  1.71s/it]
Outer training loop:  21%|##1       | 10597/50000 [5:41:04<19:56:53,  1.82s/it]
Outer training loop:  21%|##1       | 10598/50000 [5:41:06<20:01:34,  1.83s/it]
Outer training loop:  21%|##1       | 10599/50000 [5:41:07<20:03:59,  1.83s/it]
Outer training loop:  21%|##1       | 10600/50000 [5:41:09<20:32:16,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.72s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]

Outer training loop:  21%|##1       | 10601/50000 [5:41:16<37:10:10,  3.40s/it]
Outer training loop:  21%|##1       | 10602/50000 [5:41:18<31:13:49,  2.85s/it]
Outer training loop:  21%|##1       | 10603/50000 [5:41:20<27:39:00,  2.53s/it]
Outer training loop:  21%|##1       | 10604/50000 [5:41:22<25:48:04,  2.36s/it]
Outer training loop:  21%|##1       | 10605/50000 [5:41:23<23:25:59,  2.14s/it]
Outer training loop:  21%|##1       | 10606/50000 [5:41:25<22:06:00,  2.02s/it]
Outer training loop:  21%|##1       | 10607/50000 [5:41:27<22:03:57,  2.02s/it]
Outer training loop:  21%|##1       | 10608/50000 [5:41:29<21:01:18,  1.92s/it]
Outer training loop:  21%|##1       | 10609/50000 [5:41:30<20:27:44,  1.87s/it]
Outer training loop:  21%|##1       | 10610/50000 [5:41:32<20:09:05,  1.84s/it]
Outer training loop:  21%|##1       | 10611/50000 [5:41:34<20:05:29,  1.84s/it]
Outer training loop:  21%|##1       | 10612/50000 [5:41:36<19:34:47,  1.79s/it]
Outer training loop:  21%|##1       | 10613/50000 [5:41:38<19:40:33,  1.80s/it]
Outer training loop:  21%|##1       | 10614/50000 [5:41:39<19:38:14,  1.79s/it]
Outer training loop:  21%|##1       | 10615/50000 [5:41:41<19:14:01,  1.76s/it]
Outer training loop:  21%|##1       | 10616/50000 [5:41:43<19:07:55,  1.75s/it]
Outer training loop:  21%|##1       | 10617/50000 [5:41:45<19:46:53,  1.81s/it]
Outer training loop:  21%|##1       | 10618/50000 [5:41:46<19:40:28,  1.80s/it]
Outer training loop:  21%|##1       | 10619/50000 [5:41:48<19:25:27,  1.78s/it]
Outer training loop:  21%|##1       | 10620/50000 [5:41:50<19:19:38,  1.77s/it]
Outer training loop:  21%|##1       | 10621/50000 [5:41:52<20:09:17,  1.84s/it]
Outer training loop:  21%|##1       | 10622/50000 [5:41:54<19:45:57,  1.81s/it]
Outer training loop:  21%|##1       | 10623/50000 [5:41:55<19:30:53,  1.78s/it]
Outer training loop:  21%|##1       | 10624/50000 [5:41:57<19:53:38,  1.82s/it]
Outer training loop:  21%|##1       | 10625/50000 [5:41:59<19:43:44,  1.80s/it]
Outer training loop:  21%|##1       | 10626/50000 [5:42:01<19:38:00,  1.80s/it]
Outer training loop:  21%|##1       | 10627/50000 [5:42:03<19:50:58,  1.81s/it]
Outer training loop:  21%|##1       | 10628/50000 [5:42:04<19:39:29,  1.80s/it]
Outer training loop:  21%|##1       | 10629/50000 [5:42:06<19:14:43,  1.76s/it]
Outer training loop:  21%|##1       | 10630/50000 [5:42:08<18:46:46,  1.72s/it]
Outer training loop:  21%|##1       | 10631/50000 [5:42:10<19:31:40,  1.79s/it]
Outer training loop:  21%|##1       | 10632/50000 [5:42:12<19:43:22,  1.80s/it]
Outer training loop:  21%|##1       | 10633/50000 [5:42:13<19:25:04,  1.78s/it]
Outer training loop:  21%|##1       | 10634/50000 [5:42:15<19:37:05,  1.79s/it]
Outer training loop:  21%|##1       | 10635/50000 [5:42:17<18:55:04,  1.73s/it]
Outer training loop:  21%|##1       | 10636/50000 [5:42:18<18:34:59,  1.70s/it]
Outer training loop:  21%|##1       | 10637/50000 [5:42:20<19:11:31,  1.76s/it]
Outer training loop:  21%|##1       | 10638/50000 [5:42:22<19:24:13,  1.77s/it]
Outer training loop:  21%|##1       | 10639/50000 [5:42:24<18:57:32,  1.73s/it]
Outer training loop:  21%|##1       | 10640/50000 [5:42:25<18:52:13,  1.73s/it]
Outer training loop:  21%|##1       | 10641/50000 [5:42:27<19:15:25,  1.76s/it]
Outer training loop:  21%|##1       | 10642/50000 [5:42:29<18:50:05,  1.72s/it]
Outer training loop:  21%|##1       | 10643/50000 [5:42:30<18:32:58,  1.70s/it]
Outer training loop:  21%|##1       | 10644/50000 [5:42:32<19:44:52,  1.81s/it]
Outer training loop:  21%|##1       | 10645/50000 [5:42:34<19:22:18,  1.77s/it]
Outer training loop:  21%|##1       | 10646/50000 [5:42:36<19:21:55,  1.77s/it]
Outer training loop:  21%|##1       | 10647/50000 [5:42:38<19:19:37,  1.77s/it]
Outer training loop:  21%|##1       | 10648/50000 [5:42:40<19:57:34,  1.83s/it]
Outer training loop:  21%|##1       | 10649/50000 [5:42:41<19:40:23,  1.80s/it]
Outer training loop:  21%|##1       | 10650/50000 [5:42:43<19:32:37,  1.79s/it]
Outer training loop:  21%|##1       | 10651/50000 [5:42:45<20:01:45,  1.83s/it]
Outer training loop:  21%|##1       | 10652/50000 [5:42:47<19:38:17,  1.80s/it]
Outer training loop:  21%|##1       | 10653/50000 [5:42:49<19:20:53,  1.77s/it]
Outer training loop:  21%|##1       | 10654/50000 [5:42:50<19:56:30,  1.82s/it]
Outer training loop:  21%|##1       | 10655/50000 [5:42:52<19:27:18,  1.78s/it]
Outer training loop:  21%|##1       | 10656/50000 [5:42:54<19:08:13,  1.75s/it]
Outer training loop:  21%|##1       | 10657/50000 [5:42:56<19:05:18,  1.75s/it]
Outer training loop:  21%|##1       | 10658/50000 [5:42:58<20:07:31,  1.84s/it]
Outer training loop:  21%|##1       | 10659/50000 [5:42:59<19:39:08,  1.80s/it]
Outer training loop:  21%|##1       | 10660/50000 [5:43:01<19:22:01,  1.77s/it]
Outer training loop:  21%|##1       | 10661/50000 [5:43:03<19:26:51,  1.78s/it]
Outer training loop:  21%|##1       | 10662/50000 [5:43:05<19:19:30,  1.77s/it]
Outer training loop:  21%|##1       | 10663/50000 [5:43:06<18:54:06,  1.73s/it]
Outer training loop:  21%|##1       | 10664/50000 [5:43:08<18:45:23,  1.72s/it]
Outer training loop:  21%|##1       | 10665/50000 [5:43:10<19:08:10,  1.75s/it]
Outer training loop:  21%|##1       | 10666/50000 [5:43:11<18:55:23,  1.73s/it]
Outer training loop:  21%|##1       | 10667/50000 [5:43:13<18:59:49,  1.74s/it]
Outer training loop:  21%|##1       | 10668/50000 [5:43:15<19:26:26,  1.78s/it]
Outer training loop:  21%|##1       | 10669/50000 [5:43:17<19:19:22,  1.77s/it]
Outer training loop:  21%|##1       | 10670/50000 [5:43:18<18:51:16,  1.73s/it]
Outer training loop:  21%|##1       | 10671/50000 [5:43:20<19:47:52,  1.81s/it]
Outer training loop:  21%|##1       | 10672/50000 [5:43:22<19:36:42,  1.80s/it]
Outer training loop:  21%|##1       | 10673/50000 [5:43:24<19:14:49,  1.76s/it]
Outer training loop:  21%|##1       | 10674/50000 [5:43:26<19:02:12,  1.74s/it]
Outer training loop:  21%|##1       | 10675/50000 [5:43:28<19:50:50,  1.82s/it]
Outer training loop:  21%|##1       | 10676/50000 [5:43:29<19:32:19,  1.79s/it]
Outer training loop:  21%|##1       | 10677/50000 [5:43:31<19:01:41,  1.74s/it]
Outer training loop:  21%|##1       | 10678/50000 [5:43:33<19:19:10,  1.77s/it]
Outer training loop:  21%|##1       | 10679/50000 [5:43:35<19:38:58,  1.80s/it]
Outer training loop:  21%|##1       | 10680/50000 [5:43:36<19:06:18,  1.75s/it]
Outer training loop:  21%|##1       | 10681/50000 [5:43:38<19:50:22,  1.82s/it]
Outer training loop:  21%|##1       | 10682/50000 [5:43:40<19:35:44,  1.79s/it]
Outer training loop:  21%|##1       | 10683/50000 [5:43:42<19:22:32,  1.77s/it]
Outer training loop:  21%|##1       | 10684/50000 [5:43:43<19:19:07,  1.77s/it]
Outer training loop:  21%|##1       | 10685/50000 [5:43:45<19:18:33,  1.77s/it]
Outer training loop:  21%|##1       | 10686/50000 [5:43:47<19:12:08,  1.76s/it]
Outer training loop:  21%|##1       | 10687/50000 [5:43:49<19:01:22,  1.74s/it]
Outer training loop:  21%|##1       | 10688/50000 [5:43:51<19:29:51,  1.79s/it]
Outer training loop:  21%|##1       | 10689/50000 [5:43:52<19:24:34,  1.78s/it]
Outer training loop:  21%|##1       | 10690/50000 [5:43:54<19:09:55,  1.76s/it]
Outer training loop:  21%|##1       | 10691/50000 [5:43:56<19:34:26,  1.79s/it]
Outer training loop:  21%|##1       | 10692/50000 [5:43:58<19:02:23,  1.74s/it]
Outer training loop:  21%|##1       | 10693/50000 [5:43:59<18:50:58,  1.73s/it]
Outer training loop:  21%|##1       | 10694/50000 [5:44:01<19:04:02,  1.75s/it]
Outer training loop:  21%|##1       | 10695/50000 [5:44:03<19:40:27,  1.80s/it]
Outer training loop:  21%|##1       | 10696/50000 [5:44:05<19:06:39,  1.75s/it]
Outer training loop:  21%|##1       | 10697/50000 [5:44:06<19:40:18,  1.80s/it]
Outer training loop:  21%|##1       | 10698/50000 [5:44:08<19:46:43,  1.81s/it]
Outer training loop:  21%|##1       | 10699/50000 [5:44:10<19:39:40,  1.80s/it]
Outer training loop:  21%|##1       | 10700/50000 [5:44:12<19:46:43,  1.81s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.67s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.73s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  21%|##1       | 10701/50000 [5:44:19<37:25:38,  3.43s/it]
Outer training loop:  21%|##1       | 10702/50000 [5:44:21<32:07:39,  2.94s/it]
Outer training loop:  21%|##1       | 10703/50000 [5:44:23<27:57:36,  2.56s/it]
Outer training loop:  21%|##1       | 10704/50000 [5:44:24<25:28:33,  2.33s/it]
Outer training loop:  21%|##1       | 10705/50000 [5:44:26<24:10:51,  2.22s/it]
Outer training loop:  21%|##1       | 10706/50000 [5:44:28<22:32:10,  2.06s/it]
Outer training loop:  21%|##1       | 10707/50000 [5:44:30<21:09:31,  1.94s/it]
Outer training loop:  21%|##1       | 10708/50000 [5:44:32<20:58:56,  1.92s/it]
Outer training loop:  21%|##1       | 10709/50000 [5:44:33<20:15:02,  1.86s/it]
Outer training loop:  21%|##1       | 10710/50000 [5:44:35<20:12:05,  1.85s/it]
Outer training loop:  21%|##1       | 10711/50000 [5:44:37<19:56:41,  1.83s/it]
Outer training loop:  21%|##1       | 10712/50000 [5:44:39<20:10:04,  1.85s/it]
Outer training loop:  21%|##1       | 10713/50000 [5:44:41<20:08:55,  1.85s/it]
Outer training loop:  21%|##1       | 10714/50000 [5:44:42<19:40:35,  1.80s/it]
Outer training loop:  21%|##1       | 10715/50000 [5:44:44<19:55:17,  1.83s/it]
Outer training loop:  21%|##1       | 10716/50000 [5:44:46<19:37:57,  1.80s/it]
Outer training loop:  21%|##1       | 10717/50000 [5:44:48<19:11:20,  1.76s/it]
Outer training loop:  21%|##1       | 10718/50000 [5:44:49<19:24:18,  1.78s/it]
Outer training loop:  21%|##1       | 10719/50000 [5:44:51<19:03:01,  1.75s/it]
Outer training loop:  21%|##1       | 10720/50000 [5:44:53<18:36:34,  1.71s/it]
Outer training loop:  21%|##1       | 10721/50000 [5:44:54<18:41:39,  1.71s/it]
Outer training loop:  21%|##1       | 10722/50000 [5:44:56<19:21:32,  1.77s/it]
Outer training loop:  21%|##1       | 10723/50000 [5:44:58<18:48:11,  1.72s/it]
Outer training loop:  21%|##1       | 10724/50000 [5:45:00<19:00:28,  1.74s/it]
Outer training loop:  21%|##1       | 10725/50000 [5:45:02<19:32:50,  1.79s/it]
Outer training loop:  21%|##1       | 10726/50000 [5:45:03<19:11:20,  1.76s/it]
Outer training loop:  21%|##1       | 10727/50000 [5:45:05<19:00:27,  1.74s/it]
Outer training loop:  21%|##1       | 10728/50000 [5:45:07<18:44:31,  1.72s/it]
Outer training loop:  21%|##1       | 10729/50000 [5:45:09<19:22:40,  1.78s/it]
Outer training loop:  21%|##1       | 10730/50000 [5:45:10<19:08:11,  1.75s/it]
Outer training loop:  21%|##1       | 10731/50000 [5:45:12<19:05:04,  1.75s/it]
Outer training loop:  21%|##1       | 10732/50000 [5:45:14<19:16:21,  1.77s/it]
Outer training loop:  21%|##1       | 10733/50000 [5:45:16<18:59:55,  1.74s/it]
Outer training loop:  21%|##1       | 10734/50000 [5:45:17<18:51:43,  1.73s/it]
Outer training loop:  21%|##1       | 10735/50000 [5:45:19<19:16:22,  1.77s/it]
Outer training loop:  21%|##1       | 10736/50000 [5:45:21<19:00:46,  1.74s/it]
Outer training loop:  21%|##1       | 10737/50000 [5:45:22<18:34:32,  1.70s/it]
Outer training loop:  21%|##1       | 10738/50000 [5:45:24<19:21:12,  1.77s/it]
Outer training loop:  21%|##1       | 10739/50000 [5:45:26<18:57:32,  1.74s/it]
Outer training loop:  21%|##1       | 10740/50000 [5:45:28<19:04:45,  1.75s/it]
Outer training loop:  21%|##1       | 10741/50000 [5:45:30<19:28:00,  1.79s/it]
Outer training loop:  21%|##1       | 10742/50000 [5:45:32<19:58:33,  1.83s/it]
Outer training loop:  21%|##1       | 10743/50000 [5:45:33<19:28:04,  1.79s/it]
Outer training loop:  21%|##1       | 10744/50000 [5:45:35<19:11:27,  1.76s/it]
Outer training loop:  21%|##1       | 10745/50000 [5:45:37<19:54:18,  1.83s/it]
Outer training loop:  21%|##1       | 10746/50000 [5:45:39<19:45:16,  1.81s/it]
Outer training loop:  21%|##1       | 10747/50000 [5:45:40<19:30:16,  1.79s/it]
Outer training loop:  21%|##1       | 10748/50000 [5:45:42<19:14:53,  1.77s/it]
Outer training loop:  21%|##1       | 10749/50000 [5:45:44<19:32:12,  1.79s/it]
Outer training loop:  22%|##1       | 10750/50000 [5:45:46<19:04:44,  1.75s/it]
Outer training loop:  22%|##1       | 10751/50000 [5:45:47<18:40:14,  1.71s/it]
Outer training loop:  22%|##1       | 10752/50000 [5:45:49<19:16:04,  1.77s/it]
Outer training loop:  22%|##1       | 10753/50000 [5:45:51<18:59:29,  1.74s/it]
Outer training loop:  22%|##1       | 10754/50000 [5:45:53<18:55:20,  1.74s/it]
Outer training loop:  22%|##1       | 10755/50000 [5:45:55<19:31:46,  1.79s/it]
Outer training loop:  22%|##1       | 10756/50000 [5:45:56<19:27:32,  1.79s/it]
Outer training loop:  22%|##1       | 10757/50000 [5:45:58<18:52:45,  1.73s/it]
Outer training loop:  22%|##1       | 10758/50000 [5:46:00<18:48:11,  1.72s/it]
Outer training loop:  22%|##1       | 10759/50000 [5:46:02<19:18:26,  1.77s/it]
Outer training loop:  22%|##1       | 10760/50000 [5:46:03<19:16:03,  1.77s/it]
Outer training loop:  22%|##1       | 10761/50000 [5:46:05<19:26:57,  1.78s/it]
Outer training loop:  22%|##1       | 10762/50000 [5:46:07<19:45:06,  1.81s/it]
Outer training loop:  22%|##1       | 10763/50000 [5:46:09<19:27:02,  1.78s/it]
Outer training loop:  22%|##1       | 10764/50000 [5:46:10<19:30:33,  1.79s/it]
Outer training loop:  22%|##1       | 10765/50000 [5:46:12<19:09:45,  1.76s/it]
Outer training loop:  22%|##1       | 10766/50000 [5:46:14<19:39:54,  1.80s/it]
Outer training loop:  22%|##1       | 10767/50000 [5:46:16<19:23:13,  1.78s/it]
Outer training loop:  22%|##1       | 10768/50000 [5:46:18<19:19:49,  1.77s/it]
Outer training loop:  22%|##1       | 10769/50000 [5:46:19<19:36:09,  1.80s/it]
Outer training loop:  22%|##1       | 10770/50000 [5:46:21<19:36:18,  1.80s/it]
Outer training loop:  22%|##1       | 10771/50000 [5:46:23<19:19:24,  1.77s/it]
Outer training loop:  22%|##1       | 10772/50000 [5:46:25<19:46:57,  1.82s/it]
Outer training loop:  22%|##1       | 10773/50000 [5:46:27<19:25:32,  1.78s/it]
Outer training loop:  22%|##1       | 10774/50000 [5:46:28<19:22:23,  1.78s/it]
Outer training loop:  22%|##1       | 10775/50000 [5:46:30<19:19:10,  1.77s/it]
Outer training loop:  22%|##1       | 10776/50000 [5:46:32<19:43:33,  1.81s/it]
Outer training loop:  22%|##1       | 10777/50000 [5:46:34<19:27:02,  1.79s/it]
Outer training loop:  22%|##1       | 10778/50000 [5:46:35<19:09:16,  1.76s/it]
Outer training loop:  22%|##1       | 10779/50000 [5:46:37<19:12:30,  1.76s/it]
Outer training loop:  22%|##1       | 10780/50000 [5:46:39<19:13:30,  1.76s/it]
Outer training loop:  22%|##1       | 10781/50000 [5:46:41<19:11:29,  1.76s/it]
Outer training loop:  22%|##1       | 10782/50000 [5:46:43<19:27:25,  1.79s/it]
Outer training loop:  22%|##1       | 10783/50000 [5:46:44<19:18:50,  1.77s/it]
Outer training loop:  22%|##1       | 10784/50000 [5:46:46<19:32:20,  1.79s/it]
Outer training loop:  22%|##1       | 10785/50000 [5:46:48<19:13:18,  1.76s/it]
Outer training loop:  22%|##1       | 10786/50000 [5:46:50<19:26:22,  1.78s/it]
Outer training loop:  22%|##1       | 10787/50000 [5:46:51<19:23:59,  1.78s/it]
Outer training loop:  22%|##1       | 10788/50000 [5:46:53<19:09:52,  1.76s/it]
Outer training loop:  22%|##1       | 10789/50000 [5:46:55<20:02:44,  1.84s/it]
Outer training loop:  22%|##1       | 10790/50000 [5:46:57<19:38:27,  1.80s/it]
Outer training loop:  22%|##1       | 10791/50000 [5:46:59<19:48:54,  1.82s/it]
Outer training loop:  22%|##1       | 10792/50000 [5:47:01<20:26:33,  1.88s/it]
Outer training loop:  22%|##1       | 10793/50000 [5:47:03<20:03:04,  1.84s/it]
Outer training loop:  22%|##1       | 10794/50000 [5:47:04<19:40:47,  1.81s/it]
Outer training loop:  22%|##1       | 10795/50000 [5:47:06<20:26:32,  1.88s/it]
Outer training loop:  22%|##1       | 10796/50000 [5:47:08<20:13:42,  1.86s/it]
Outer training loop:  22%|##1       | 10797/50000 [5:47:10<20:06:58,  1.85s/it]
Outer training loop:  22%|##1       | 10798/50000 [5:47:12<19:29:57,  1.79s/it]
Outer training loop:  22%|##1       | 10799/50000 [5:47:13<19:35:52,  1.80s/it]
Outer training loop:  22%|##1       | 10800/50000 [5:47:15<18:55:30,  1.74s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.87s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  22%|##1       | 10801/50000 [5:47:22<35:42:33,  3.28s/it]
Outer training loop:  22%|##1       | 10802/50000 [5:47:24<31:17:03,  2.87s/it]
Outer training loop:  22%|##1       | 10803/50000 [5:47:25<27:26:53,  2.52s/it]
Outer training loop:  22%|##1       | 10804/50000 [5:47:27<24:41:03,  2.27s/it]
Outer training loop:  22%|##1       | 10805/50000 [5:47:29<22:43:09,  2.09s/it]
Outer training loop:  22%|##1       | 10806/50000 [5:47:31<22:01:42,  2.02s/it]
Outer training loop:  22%|##1       | 10807/50000 [5:47:32<21:10:45,  1.95s/it]
Outer training loop:  22%|##1       | 10808/50000 [5:47:34<20:22:42,  1.87s/it]
Outer training loop:  22%|##1       | 10809/50000 [5:47:36<20:17:56,  1.86s/it]
Outer training loop:  22%|##1       | 10810/50000 [5:47:38<19:49:35,  1.82s/it]
Outer training loop:  22%|##1       | 10811/50000 [5:47:39<19:36:44,  1.80s/it]
Outer training loop:  22%|##1       | 10812/50000 [5:47:41<20:01:07,  1.84s/it]
Outer training loop:  22%|##1       | 10813/50000 [5:47:43<19:22:50,  1.78s/it]
Outer training loop:  22%|##1       | 10814/50000 [5:47:45<18:58:11,  1.74s/it]
Outer training loop:  22%|##1       | 10815/50000 [5:47:46<18:50:50,  1.73s/it]
Outer training loop:  22%|##1       | 10816/50000 [5:47:48<19:14:39,  1.77s/it]
Outer training loop:  22%|##1       | 10817/50000 [5:47:50<19:12:32,  1.76s/it]
Outer training loop:  22%|##1       | 10818/50000 [5:47:52<18:40:31,  1.72s/it]
Outer training loop:  22%|##1       | 10819/50000 [5:47:54<19:21:15,  1.78s/it]
Outer training loop:  22%|##1       | 10820/50000 [5:47:55<19:03:23,  1.75s/it]
Outer training loop:  22%|##1       | 10821/50000 [5:47:57<19:06:42,  1.76s/it]
Outer training loop:  22%|##1       | 10822/50000 [5:47:59<19:24:04,  1.78s/it]
Outer training loop:  22%|##1       | 10823/50000 [5:48:01<19:23:02,  1.78s/it]
Outer training loop:  22%|##1       | 10824/50000 [5:48:02<18:53:07,  1.74s/it]
Outer training loop:  22%|##1       | 10825/50000 [5:48:04<18:39:05,  1.71s/it]
Outer training loop:  22%|##1       | 10826/50000 [5:48:06<19:20:04,  1.78s/it]
Outer training loop:  22%|##1       | 10827/50000 [5:48:08<19:00:57,  1.75s/it]
Outer training loop:  22%|##1       | 10828/50000 [5:48:09<19:16:00,  1.77s/it]
Outer training loop:  22%|##1       | 10829/50000 [5:48:11<19:19:10,  1.78s/it]
Outer training loop:  22%|##1       | 10830/50000 [5:48:13<18:58:26,  1.74s/it]
Outer training loop:  22%|##1       | 10831/50000 [5:48:15<19:13:42,  1.77s/it]
Outer training loop:  22%|##1       | 10832/50000 [5:48:16<19:19:31,  1.78s/it]
Outer training loop:  22%|##1       | 10833/50000 [5:48:18<18:55:41,  1.74s/it]
Outer training loop:  22%|##1       | 10834/50000 [5:48:20<19:07:49,  1.76s/it]
Outer training loop:  22%|##1       | 10835/50000 [5:48:22<19:33:35,  1.80s/it]
Outer training loop:  22%|##1       | 10836/50000 [5:48:24<19:33:21,  1.80s/it]
Outer training loop:  22%|##1       | 10837/50000 [5:48:25<19:30:37,  1.79s/it]
Outer training loop:  22%|##1       | 10838/50000 [5:48:28<20:52:51,  1.92s/it]
Outer training loop:  22%|##1       | 10839/50000 [5:48:29<20:11:26,  1.86s/it]
Outer training loop:  22%|##1       | 10840/50000 [5:48:31<20:04:39,  1.85s/it]
Outer training loop:  22%|##1       | 10841/50000 [5:48:33<19:20:31,  1.78s/it]
Outer training loop:  22%|##1       | 10842/50000 [5:48:35<20:00:05,  1.84s/it]
Outer training loop:  22%|##1       | 10843/50000 [5:48:36<19:37:44,  1.80s/it]
Outer training loop:  22%|##1       | 10844/50000 [5:48:38<19:39:52,  1.81s/it]
Outer training loop:  22%|##1       | 10845/50000 [5:48:40<19:46:01,  1.82s/it]
Outer training loop:  22%|##1       | 10846/50000 [5:48:42<19:20:29,  1.78s/it]
Outer training loop:  22%|##1       | 10847/50000 [5:48:43<19:01:40,  1.75s/it]
Outer training loop:  22%|##1       | 10848/50000 [5:48:45<19:51:01,  1.83s/it]
Outer training loop:  22%|##1       | 10849/50000 [5:48:47<19:36:24,  1.80s/it]
Outer training loop:  22%|##1       | 10850/50000 [5:48:49<19:54:47,  1.83s/it]
Outer training loop:  22%|##1       | 10851/50000 [5:48:51<19:46:30,  1.82s/it]
Outer training loop:  22%|##1       | 10852/50000 [5:48:53<20:37:50,  1.90s/it]
Outer training loop:  22%|##1       | 10853/50000 [5:48:55<19:43:26,  1.81s/it]
Outer training loop:  22%|##1       | 10854/50000 [5:48:56<19:38:15,  1.81s/it]
Outer training loop:  22%|##1       | 10855/50000 [5:48:58<19:57:12,  1.84s/it]
Outer training loop:  22%|##1       | 10856/50000 [5:49:00<19:23:19,  1.78s/it]
Outer training loop:  22%|##1       | 10857/50000 [5:49:02<18:55:16,  1.74s/it]
Outer training loop:  22%|##1       | 10858/50000 [5:49:03<19:05:48,  1.76s/it]
Outer training loop:  22%|##1       | 10859/50000 [5:49:05<19:39:10,  1.81s/it]
Outer training loop:  22%|##1       | 10860/50000 [5:49:07<19:20:57,  1.78s/it]
Outer training loop:  22%|##1       | 10861/50000 [5:49:09<19:19:22,  1.78s/it]
Outer training loop:  22%|##1       | 10862/50000 [5:49:11<19:29:49,  1.79s/it]
Outer training loop:  22%|##1       | 10863/50000 [5:49:12<19:37:42,  1.81s/it]
Outer training loop:  22%|##1       | 10864/50000 [5:49:14<19:53:26,  1.83s/it]
Outer training loop:  22%|##1       | 10865/50000 [5:49:16<20:31:36,  1.89s/it]
Outer training loop:  22%|##1       | 10866/50000 [5:49:18<19:47:37,  1.82s/it]
Outer training loop:  22%|##1       | 10867/50000 [5:49:20<19:59:36,  1.84s/it]
Outer training loop:  22%|##1       | 10868/50000 [5:49:22<19:46:06,  1.82s/it]
Outer training loop:  22%|##1       | 10869/50000 [5:49:23<19:41:26,  1.81s/it]
Outer training loop:  22%|##1       | 10870/50000 [5:49:25<19:30:52,  1.80s/it]
Outer training loop:  22%|##1       | 10871/50000 [5:49:27<19:12:07,  1.77s/it]
Outer training loop:  22%|##1       | 10872/50000 [5:49:29<19:18:50,  1.78s/it]
Outer training loop:  22%|##1       | 10873/50000 [5:49:31<19:35:54,  1.80s/it]
Outer training loop:  22%|##1       | 10874/50000 [5:49:32<19:36:07,  1.80s/it]
Outer training loop:  22%|##1       | 10875/50000 [5:49:34<19:27:14,  1.79s/it]
Outer training loop:  22%|##1       | 10876/50000 [5:49:36<19:55:57,  1.83s/it]
Outer training loop:  22%|##1       | 10877/50000 [5:49:38<19:19:20,  1.78s/it]
Outer training loop:  22%|##1       | 10878/50000 [5:49:39<19:02:43,  1.75s/it]
Outer training loop:  22%|##1       | 10879/50000 [5:49:41<19:12:16,  1.77s/it]
Outer training loop:  22%|##1       | 10880/50000 [5:49:43<19:06:36,  1.76s/it]
Outer training loop:  22%|##1       | 10881/50000 [5:49:45<18:27:36,  1.70s/it]
Outer training loop:  22%|##1       | 10882/50000 [5:49:46<19:07:59,  1.76s/it]
Outer training loop:  22%|##1       | 10883/50000 [5:49:48<19:11:29,  1.77s/it]
Outer training loop:  22%|##1       | 10884/50000 [5:49:50<19:25:23,  1.79s/it]
Outer training loop:  22%|##1       | 10885/50000 [5:49:52<19:11:53,  1.77s/it]
Outer training loop:  22%|##1       | 10886/50000 [5:49:54<19:38:02,  1.81s/it]
Outer training loop:  22%|##1       | 10887/50000 [5:49:55<19:21:57,  1.78s/it]
Outer training loop:  22%|##1       | 10888/50000 [5:49:57<19:46:33,  1.82s/it]
Outer training loop:  22%|##1       | 10889/50000 [5:49:59<19:57:52,  1.84s/it]
Outer training loop:  22%|##1       | 10890/50000 [5:50:01<19:28:41,  1.79s/it]
Outer training loop:  22%|##1       | 10891/50000 [5:50:03<19:07:36,  1.76s/it]
Outer training loop:  22%|##1       | 10892/50000 [5:50:04<19:32:15,  1.80s/it]
Outer training loop:  22%|##1       | 10893/50000 [5:50:06<19:24:50,  1.79s/it]
Outer training loop:  22%|##1       | 10894/50000 [5:50:08<19:29:13,  1.79s/it]
Outer training loop:  22%|##1       | 10895/50000 [5:50:10<19:25:00,  1.79s/it]
Outer training loop:  22%|##1       | 10896/50000 [5:50:12<19:39:02,  1.81s/it]
Outer training loop:  22%|##1       | 10897/50000 [5:50:13<19:15:53,  1.77s/it]
Outer training loop:  22%|##1       | 10898/50000 [5:50:15<19:08:36,  1.76s/it]
Outer training loop:  22%|##1       | 10899/50000 [5:50:17<19:29:05,  1.79s/it]
Outer training loop:  22%|##1       | 10900/50000 [5:50:19<19:15:55,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  22%|##1       | 10901/50000 [5:50:26<36:05:15,  3.32s/it]
Outer training loop:  22%|##1       | 10902/50000 [5:50:27<31:11:27,  2.87s/it]
Outer training loop:  22%|##1       | 10903/50000 [5:50:29<28:15:34,  2.60s/it]
Outer training loop:  22%|##1       | 10904/50000 [5:50:31<25:15:27,  2.33s/it]
Outer training loop:  22%|##1       | 10905/50000 [5:50:33<23:36:03,  2.17s/it]
Outer training loop:  22%|##1       | 10906/50000 [5:50:35<23:02:46,  2.12s/it]
Outer training loop:  22%|##1       | 10907/50000 [5:50:37<21:51:29,  2.01s/it]
Outer training loop:  22%|##1       | 10908/50000 [5:50:38<20:47:42,  1.92s/it]
Outer training loop:  22%|##1       | 10909/50000 [5:50:40<20:57:16,  1.93s/it]
Outer training loop:  22%|##1       | 10910/50000 [5:50:42<20:09:15,  1.86s/it]
Outer training loop:  22%|##1       | 10911/50000 [5:50:44<19:35:50,  1.80s/it]
Outer training loop:  22%|##1       | 10912/50000 [5:50:45<19:09:10,  1.76s/it]
Outer training loop:  22%|##1       | 10913/50000 [5:50:47<19:40:11,  1.81s/it]
Outer training loop:  22%|##1       | 10914/50000 [5:50:49<19:23:39,  1.79s/it]
Outer training loop:  22%|##1       | 10915/50000 [5:50:51<19:10:48,  1.77s/it]
Outer training loop:  22%|##1       | 10916/50000 [5:50:52<19:01:53,  1.75s/it]
Outer training loop:  22%|##1       | 10917/50000 [5:50:54<19:26:49,  1.79s/it]
Outer training loop:  22%|##1       | 10918/50000 [5:50:56<19:00:49,  1.75s/it]
Outer training loop:  22%|##1       | 10919/50000 [5:50:58<18:33:31,  1.71s/it]
Outer training loop:  22%|##1       | 10920/50000 [5:50:59<18:55:12,  1.74s/it]
Outer training loop:  22%|##1       | 10921/50000 [5:51:01<18:36:50,  1.71s/it]
Outer training loop:  22%|##1       | 10922/50000 [5:51:03<18:38:21,  1.72s/it]
Outer training loop:  22%|##1       | 10923/50000 [5:51:05<19:27:47,  1.79s/it]
Outer training loop:  22%|##1       | 10924/50000 [5:51:06<19:04:39,  1.76s/it]
Outer training loop:  22%|##1       | 10925/50000 [5:51:08<18:56:01,  1.74s/it]
Outer training loop:  22%|##1       | 10926/50000 [5:51:10<18:48:56,  1.73s/it]
Outer training loop:  22%|##1       | 10927/50000 [5:51:12<19:21:31,  1.78s/it]
Outer training loop:  22%|##1       | 10928/50000 [5:51:13<18:52:31,  1.74s/it]
Outer training loop:  22%|##1       | 10929/50000 [5:51:15<18:50:11,  1.74s/it]
Outer training loop:  22%|##1       | 10930/50000 [5:51:17<19:13:01,  1.77s/it]
Outer training loop:  22%|##1       | 10931/50000 [5:51:19<19:03:49,  1.76s/it]
Outer training loop:  22%|##1       | 10932/50000 [5:51:20<18:53:11,  1.74s/it]
Outer training loop:  22%|##1       | 10933/50000 [5:51:22<19:31:30,  1.80s/it]
Outer training loop:  22%|##1       | 10934/50000 [5:51:24<19:02:12,  1.75s/it]
Outer training loop:  22%|##1       | 10935/50000 [5:51:26<19:24:49,  1.79s/it]
Outer training loop:  22%|##1       | 10936/50000 [5:51:28<19:09:39,  1.77s/it]
Outer training loop:  22%|##1       | 10937/50000 [5:51:29<19:38:47,  1.81s/it]
Outer training loop:  22%|##1       | 10938/50000 [5:51:31<19:09:22,  1.77s/it]
Outer training loop:  22%|##1       | 10939/50000 [5:51:33<19:24:32,  1.79s/it]
Outer training loop:  22%|##1       | 10940/50000 [5:51:35<20:02:48,  1.85s/it]
Outer training loop:  22%|##1       | 10941/50000 [5:51:37<19:38:36,  1.81s/it]
Outer training loop:  22%|##1       | 10942/50000 [5:51:38<19:27:11,  1.79s/it]
Outer training loop:  22%|##1       | 10943/50000 [5:51:40<19:04:16,  1.76s/it]
Outer training loop:  22%|##1       | 10944/50000 [5:51:42<19:45:31,  1.82s/it]
Outer training loop:  22%|##1       | 10945/50000 [5:51:44<19:22:12,  1.79s/it]
Outer training loop:  22%|##1       | 10946/50000 [5:51:45<18:51:19,  1.74s/it]
Outer training loop:  22%|##1       | 10947/50000 [5:51:47<19:31:57,  1.80s/it]
Outer training loop:  22%|##1       | 10948/50000 [5:51:49<19:19:34,  1.78s/it]
Outer training loop:  22%|##1       | 10949/50000 [5:51:51<19:04:21,  1.76s/it]
Outer training loop:  22%|##1       | 10950/50000 [5:51:53<18:59:06,  1.75s/it]
Outer training loop:  22%|##1       | 10951/50000 [5:51:54<19:21:17,  1.78s/it]
Outer training loop:  22%|##1       | 10952/50000 [5:51:56<19:13:39,  1.77s/it]
Outer training loop:  22%|##1       | 10953/50000 [5:51:58<19:23:34,  1.79s/it]
Outer training loop:  22%|##1       | 10954/50000 [5:52:00<19:22:33,  1.79s/it]
Outer training loop:  22%|##1       | 10955/50000 [5:52:01<18:55:49,  1.75s/it]
Outer training loop:  22%|##1       | 10956/50000 [5:52:03<18:38:30,  1.72s/it]
Outer training loop:  22%|##1       | 10957/50000 [5:52:05<19:18:01,  1.78s/it]
Outer training loop:  22%|##1       | 10958/50000 [5:52:07<19:05:41,  1.76s/it]
Outer training loop:  22%|##1       | 10959/50000 [5:52:08<18:46:41,  1.73s/it]
Outer training loop:  22%|##1       | 10960/50000 [5:52:10<19:31:24,  1.80s/it]
Outer training loop:  22%|##1       | 10961/50000 [5:52:12<19:20:39,  1.78s/it]
Outer training loop:  22%|##1       | 10962/50000 [5:52:14<19:14:57,  1.78s/it]
Outer training loop:  22%|##1       | 10963/50000 [5:52:16<19:03:17,  1.76s/it]
Outer training loop:  22%|##1       | 10964/50000 [5:52:17<19:24:28,  1.79s/it]
Outer training loop:  22%|##1       | 10965/50000 [5:52:19<19:08:30,  1.77s/it]
Outer training loop:  22%|##1       | 10966/50000 [5:52:21<18:58:54,  1.75s/it]
Outer training loop:  22%|##1       | 10967/50000 [5:52:23<19:14:03,  1.77s/it]
Outer training loop:  22%|##1       | 10968/50000 [5:52:24<18:49:05,  1.74s/it]
Outer training loop:  22%|##1       | 10969/50000 [5:52:26<18:46:34,  1.73s/it]
Outer training loop:  22%|##1       | 10970/50000 [5:52:28<19:26:52,  1.79s/it]
Outer training loop:  22%|##1       | 10971/50000 [5:52:30<18:55:15,  1.75s/it]
Outer training loop:  22%|##1       | 10972/50000 [5:52:31<18:45:22,  1.73s/it]
Outer training loop:  22%|##1       | 10973/50000 [5:52:33<19:03:38,  1.76s/it]
Outer training loop:  22%|##1       | 10974/50000 [5:52:35<19:48:23,  1.83s/it]
Outer training loop:  22%|##1       | 10975/50000 [5:52:37<19:17:52,  1.78s/it]
Outer training loop:  22%|##1       | 10976/50000 [5:52:38<18:45:17,  1.73s/it]
Outer training loop:  22%|##1       | 10977/50000 [5:52:40<19:14:27,  1.78s/it]
Outer training loop:  22%|##1       | 10978/50000 [5:52:42<19:14:35,  1.78s/it]
Outer training loop:  22%|##1       | 10979/50000 [5:52:44<19:05:43,  1.76s/it]
Outer training loop:  22%|##1       | 10980/50000 [5:52:46<19:25:08,  1.79s/it]
Outer training loop:  22%|##1       | 10981/50000 [5:52:47<18:51:49,  1.74s/it]
Outer training loop:  22%|##1       | 10982/50000 [5:52:49<18:34:42,  1.71s/it]
Outer training loop:  22%|##1       | 10983/50000 [5:52:51<18:50:21,  1.74s/it]
Outer training loop:  22%|##1       | 10984/50000 [5:52:52<18:46:37,  1.73s/it]
Outer training loop:  22%|##1       | 10985/50000 [5:52:54<19:23:54,  1.79s/it]
Outer training loop:  22%|##1       | 10986/50000 [5:52:56<19:31:05,  1.80s/it]
Outer training loop:  22%|##1       | 10987/50000 [5:52:58<19:16:09,  1.78s/it]
Outer training loop:  22%|##1       | 10988/50000 [5:53:00<18:57:16,  1.75s/it]
Outer training loop:  22%|##1       | 10989/50000 [5:53:01<18:28:43,  1.71s/it]
Outer training loop:  22%|##1       | 10990/50000 [5:53:03<19:11:43,  1.77s/it]
Outer training loop:  22%|##1       | 10991/50000 [5:53:05<18:48:33,  1.74s/it]
Outer training loop:  22%|##1       | 10992/50000 [5:53:06<18:44:27,  1.73s/it]
Outer training loop:  22%|##1       | 10993/50000 [5:53:08<19:28:11,  1.80s/it]
Outer training loop:  22%|##1       | 10994/50000 [5:53:10<19:17:49,  1.78s/it]
Outer training loop:  22%|##1       | 10995/50000 [5:53:12<19:05:01,  1.76s/it]
Outer training loop:  22%|##1       | 10996/50000 [5:53:14<19:40:58,  1.82s/it]
Outer training loop:  22%|##1       | 10997/50000 [5:53:15<19:06:09,  1.76s/it]
Outer training loop:  22%|##1       | 10998/50000 [5:53:17<19:15:10,  1.78s/it]
Outer training loop:  22%|##1       | 10999/50000 [5:53:19<19:04:10,  1.76s/it]
Outer training loop:  22%|##2       | 11000/50000 [5:53:21<19:25:04,  1.79s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.65s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.68s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:16,  1.67s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:15,  1.67s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:19,  1.75s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:17,  1.74s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:15,  1.74s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:12,  1.72s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:14,  1.77s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:11,  1.76s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:09,  1.75s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:09,  1.78s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:08,  1.79s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:06,  1.78s/it]

Evaluating policy:  16%|#5        | 13/83 [00:22<02:02,  1.76s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:00,  1.74s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:02,  1.80s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:56,  1.74s/it]

Evaluating policy:  20%|##        | 17/83 [00:29<01:54,  1.74s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:52,  1.72s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:53,  1.78s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:51,  1.77s/it]

Evaluating policy:  25%|##5       | 21/83 [00:36<01:47,  1.73s/it]

Evaluating policy:  27%|##6       | 22/83 [00:38<01:43,  1.70s/it]

Evaluating policy:  28%|##7       | 23/83 [00:40<01:42,  1.71s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:45,  1.78s/it]

Evaluating policy:  30%|###       | 25/83 [00:43<01:43,  1.78s/it]

Evaluating policy:  31%|###1      | 26/83 [00:45<01:40,  1.76s/it]

Evaluating policy:  33%|###2      | 27/83 [00:47<01:39,  1.77s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:36,  1.76s/it]

Evaluating policy:  35%|###4      | 29/83 [00:50<01:32,  1.71s/it]

Evaluating policy:  36%|###6      | 30/83 [00:52<01:35,  1.80s/it]

Evaluating policy:  37%|###7      | 31/83 [00:54<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:34,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:30,  1.81s/it]

Evaluating policy:  41%|####      | 34/83 [00:59<01:25,  1.75s/it]

Evaluating policy:  42%|####2     | 35/83 [01:01<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:03<01:24,  1.80s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:20,  1.75s/it]

Evaluating policy:  46%|####5     | 38/83 [01:06<01:18,  1.73s/it]

Evaluating policy:  47%|####6     | 39/83 [01:08<01:17,  1.76s/it]

Evaluating policy:  48%|####8     | 40/83 [01:10<01:14,  1.73s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:12,  1.72s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:12,  1.82s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:17<01:09,  1.79s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:06,  1.75s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:03,  1.71s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:22<01:01,  1.71s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:24<00:59,  1.70s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<01:01,  1.81s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:58,  1.77s/it]

Evaluating policy:  61%|######1   | 51/83 [01:29<00:56,  1.77s/it]

Evaluating policy:  63%|######2   | 52/83 [01:31<00:54,  1.75s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:51,  1.73s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:50,  1.73s/it]

Evaluating policy:  66%|######6   | 55/83 [01:36<00:48,  1.73s/it]

Evaluating policy:  67%|######7   | 56/83 [01:38<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:40<00:47,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:45,  1.81s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.78s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:45<00:40,  1.74s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:47<00:38,  1.73s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:36,  1.73s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:50<00:34,  1.72s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:54<00:32,  1.78s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:29,  1.74s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:27,  1.72s/it]

Evaluating policy:  82%|########1 | 68/83 [01:59<00:25,  1.71s/it]

Evaluating policy:  83%|########3 | 69/83 [02:01<00:23,  1.68s/it]

Evaluating policy:  84%|########4 | 70/83 [02:03<00:22,  1.70s/it]

Evaluating policy:  86%|########5 | 71/83 [02:04<00:20,  1.70s/it]

Evaluating policy:  87%|########6 | 72/83 [02:06<00:18,  1.69s/it]

Evaluating policy:  88%|########7 | 73/83 [02:08<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:16,  1.81s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:14,  1.81s/it]

Evaluating policy:  92%|#########1| 76/83 [02:13<00:12,  1.79s/it]

Evaluating policy:  93%|#########2| 77/83 [02:15<00:10,  1.78s/it]

Evaluating policy:  94%|#########3| 78/83 [02:17<00:08,  1.75s/it]

Evaluating policy:  95%|#########5| 79/83 [02:19<00:07,  1.77s/it]

Evaluating policy:  96%|#########6| 80/83 [02:20<00:05,  1.76s/it]

Evaluating policy:  98%|#########7| 81/83 [02:22<00:03,  1.79s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 83/83 [02:26<00:00,  1.84s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:14,  1.64s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:14,  1.66s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:13,  1.67s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:14,  1.70s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:12,  1.69s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:12,  1.72s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:11,  1.72s/it]

Evaluating policy:  10%|9         | 8/83 [00:13<02:08,  1.72s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:15,  1.86s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:12,  1.84s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.80s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:08,  1.84s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:06,  1.83s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:02,  1.80s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:59,  1.78s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<02:01,  1.84s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:57,  1.81s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:53,  1.77s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:50,  1.76s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:53,  1.83s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:49,  1.80s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:48,  1.80s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:44,  1.77s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:41,  1.75s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:44,  1.84s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:39,  1.77s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:36,  1.76s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:33,  1.73s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:31,  1.72s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:29,  1.71s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:31,  1.80s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:28,  1.77s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:25,  1.75s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:23,  1.74s/it]

Evaluating policy:  43%|####3     | 36/83 [01:03<01:22,  1.75s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:18,  1.75s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:15,  1.72s/it]

Evaluating policy:  48%|####8     | 40/83 [01:10<01:14,  1.73s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:12,  1.74s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:12,  1.76s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:10,  1.76s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:09,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:06,  1.80s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:03,  1.78s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:02,  1.80s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:57,  1.75s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:53,  1.78s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.74s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:47,  1.74s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:45,  1.74s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:43,  1.76s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:45,  1.90s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:42,  1.85s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:39,  1.80s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:37,  1.77s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:34,  1.75s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:32,  1.73s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:31,  1.74s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:29,  1.82s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:26,  1.77s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.78s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.76s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:21,  1.76s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:18,  1.71s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.70s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:15,  1.71s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:13,  1.73s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:13,  1.87s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.80s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.74s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:06,  1.74s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.73s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.71s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.76s/it]

Outer training loop:  22%|##2       | 11001/50000 [5:58:22<991:32:42, 91.53s/it]
Outer training loop:  22%|##2       | 11002/50000 [5:58:23<699:30:34, 64.57s/it]
Outer training loop:  22%|##2       | 11003/50000 [5:58:25<495:07:36, 45.71s/it]
Outer training loop:  22%|##2       | 11004/50000 [5:58:27<353:49:50, 32.66s/it]
Outer training loop:  22%|##2       | 11005/50000 [5:58:29<252:58:44, 23.35s/it]
Outer training loop:  22%|##2       | 11006/50000 [5:58:31<182:42:08, 16.87s/it]
Outer training loop:  22%|##2       | 11007/50000 [5:58:33<134:19:52, 12.40s/it]
Outer training loop:  22%|##2       | 11008/50000 [5:58:35<100:10:12,  9.25s/it]
Outer training loop:  22%|##2       | 11009/50000 [5:58:36<75:51:12,  7.00s/it] 
Outer training loop:  22%|##2       | 11010/50000 [5:58:38<59:24:32,  5.49s/it]
Outer training loop:  22%|##2       | 11011/50000 [5:58:40<47:03:19,  4.34s/it]
Outer training loop:  22%|##2       | 11012/50000 [5:58:42<38:46:36,  3.58s/it]
Outer training loop:  22%|##2       | 11013/50000 [5:58:43<32:34:20,  3.01s/it]
Outer training loop:  22%|##2       | 11014/50000 [5:58:45<28:59:58,  2.68s/it]
Outer training loop:  22%|##2       | 11015/50000 [5:58:47<25:54:36,  2.39s/it]
Outer training loop:  22%|##2       | 11016/50000 [5:58:49<24:20:08,  2.25s/it]
Outer training loop:  22%|##2       | 11017/50000 [5:58:51<23:25:17,  2.16s/it]
Outer training loop:  22%|##2       | 11018/50000 [5:58:53<21:47:53,  2.01s/it]
Outer training loop:  22%|##2       | 11019/50000 [5:58:54<20:44:46,  1.92s/it]
Outer training loop:  22%|##2       | 11020/50000 [5:58:56<20:43:59,  1.91s/it]
Outer training loop:  22%|##2       | 11021/50000 [5:58:58<20:07:53,  1.86s/it]
Outer training loop:  22%|##2       | 11022/50000 [5:59:00<19:34:38,  1.81s/it]
Outer training loop:  22%|##2       | 11023/50000 [5:59:02<19:50:58,  1.83s/it]
Outer training loop:  22%|##2       | 11024/50000 [5:59:03<19:37:28,  1.81s/it]
Outer training loop:  22%|##2       | 11025/50000 [5:59:05<19:06:23,  1.76s/it]
Outer training loop:  22%|##2       | 11026/50000 [5:59:07<19:41:17,  1.82s/it]
Outer training loop:  22%|##2       | 11027/50000 [5:59:09<19:40:32,  1.82s/it]
Outer training loop:  22%|##2       | 11028/50000 [5:59:11<19:45:11,  1.82s/it]
Outer training loop:  22%|##2       | 11029/50000 [5:59:12<19:51:00,  1.83s/it]
Outer training loop:  22%|##2       | 11030/50000 [5:59:14<19:21:38,  1.79s/it]
Outer training loop:  22%|##2       | 11031/50000 [5:59:16<19:08:19,  1.77s/it]
Outer training loop:  22%|##2       | 11032/50000 [5:59:18<18:55:24,  1.75s/it]
Outer training loop:  22%|##2       | 11033/50000 [5:59:20<19:49:11,  1.83s/it]
Outer training loop:  22%|##2       | 11034/50000 [5:59:21<19:36:04,  1.81s/it]
Outer training loop:  22%|##2       | 11035/50000 [5:59:23<19:06:00,  1.76s/it]
Outer training loop:  22%|##2       | 11036/50000 [5:59:25<19:44:40,  1.82s/it]
Outer training loop:  22%|##2       | 11037/50000 [5:59:27<19:39:46,  1.82s/it]
Outer training loop:  22%|##2       | 11038/50000 [5:59:28<19:12:28,  1.77s/it]
Outer training loop:  22%|##2       | 11039/50000 [5:59:30<19:43:04,  1.82s/it]
Outer training loop:  22%|##2       | 11040/50000 [5:59:32<19:25:30,  1.79s/it]
Outer training loop:  22%|##2       | 11041/50000 [5:59:34<19:12:09,  1.77s/it]
Outer training loop:  22%|##2       | 11042/50000 [5:59:36<19:47:10,  1.83s/it]
Outer training loop:  22%|##2       | 11043/50000 [5:59:37<19:23:48,  1.79s/it]
Outer training loop:  22%|##2       | 11044/50000 [5:59:39<19:00:14,  1.76s/it]
Outer training loop:  22%|##2       | 11045/50000 [5:59:41<18:43:15,  1.73s/it]
Outer training loop:  22%|##2       | 11046/50000 [5:59:43<19:08:47,  1.77s/it]
Outer training loop:  22%|##2       | 11047/50000 [5:59:44<19:07:11,  1.77s/it]
Outer training loop:  22%|##2       | 11048/50000 [5:59:46<19:16:12,  1.78s/it]
Outer training loop:  22%|##2       | 11049/50000 [5:59:48<19:32:13,  1.81s/it]
Outer training loop:  22%|##2       | 11050/50000 [5:59:50<19:19:34,  1.79s/it]
Outer training loop:  22%|##2       | 11051/50000 [5:59:51<18:37:03,  1.72s/it]
Outer training loop:  22%|##2       | 11052/50000 [5:59:53<19:06:31,  1.77s/it]
Outer training loop:  22%|##2       | 11053/50000 [5:59:55<18:57:06,  1.75s/it]
Outer training loop:  22%|##2       | 11054/50000 [5:59:57<18:41:07,  1.73s/it]
Outer training loop:  22%|##2       | 11055/50000 [5:59:59<19:14:00,  1.78s/it]
Outer training loop:  22%|##2       | 11056/50000 [6:00:00<18:50:20,  1.74s/it]
Outer training loop:  22%|##2       | 11057/50000 [6:00:02<18:45:10,  1.73s/it]
Outer training loop:  22%|##2       | 11058/50000 [6:00:04<18:55:48,  1.75s/it]
Outer training loop:  22%|##2       | 11059/50000 [6:00:06<19:45:23,  1.83s/it]
Outer training loop:  22%|##2       | 11060/50000 [6:00:08<19:56:50,  1.84s/it]
Outer training loop:  22%|##2       | 11061/50000 [6:00:10<20:08:57,  1.86s/it]
Outer training loop:  22%|##2       | 11062/50000 [6:00:11<19:37:49,  1.81s/it]
Outer training loop:  22%|##2       | 11063/50000 [6:00:13<19:16:56,  1.78s/it]
Outer training loop:  22%|##2       | 11064/50000 [6:00:15<18:53:50,  1.75s/it]
Outer training loop:  22%|##2       | 11065/50000 [6:00:17<19:32:23,  1.81s/it]
Outer training loop:  22%|##2       | 11066/50000 [6:00:18<19:31:05,  1.80s/it]
Outer training loop:  22%|##2       | 11067/50000 [6:00:20<19:27:17,  1.80s/it]
Outer training loop:  22%|##2       | 11068/50000 [6:00:22<19:50:36,  1.83s/it]
Outer training loop:  22%|##2       | 11069/50000 [6:00:24<19:19:46,  1.79s/it]
Outer training loop:  22%|##2       | 11070/50000 [6:00:26<19:13:34,  1.78s/it]
Outer training loop:  22%|##2       | 11071/50000 [6:00:27<19:19:47,  1.79s/it]
Outer training loop:  22%|##2       | 11072/50000 [6:00:29<19:01:58,  1.76s/it]
Outer training loop:  22%|##2       | 11073/50000 [6:00:31<18:45:47,  1.74s/it]
Outer training loop:  22%|##2       | 11074/50000 [6:00:32<18:18:12,  1.69s/it]
Outer training loop:  22%|##2       | 11075/50000 [6:00:34<18:50:14,  1.74s/it]
Outer training loop:  22%|##2       | 11076/50000 [6:00:36<19:01:35,  1.76s/it]
Outer training loop:  22%|##2       | 11077/50000 [6:00:38<18:39:59,  1.73s/it]
Outer training loop:  22%|##2       | 11078/50000 [6:00:39<19:06:19,  1.77s/it]
Outer training loop:  22%|##2       | 11079/50000 [6:00:41<18:51:08,  1.74s/it]
Outer training loop:  22%|##2       | 11080/50000 [6:00:43<18:49:54,  1.74s/it]
Outer training loop:  22%|##2       | 11081/50000 [6:00:44<18:26:33,  1.71s/it]
Outer training loop:  22%|##2       | 11082/50000 [6:00:46<19:01:29,  1.76s/it]
Outer training loop:  22%|##2       | 11083/50000 [6:00:48<18:37:15,  1.72s/it]
Outer training loop:  22%|##2       | 11084/50000 [6:00:50<18:54:49,  1.75s/it]
Outer training loop:  22%|##2       | 11085/50000 [6:00:52<19:17:34,  1.78s/it]
Outer training loop:  22%|##2       | 11086/50000 [6:00:53<19:00:44,  1.76s/it]
Outer training loop:  22%|##2       | 11087/50000 [6:00:55<18:54:33,  1.75s/it]
Outer training loop:  22%|##2       | 11088/50000 [6:00:57<19:29:32,  1.80s/it]
Outer training loop:  22%|##2       | 11089/50000 [6:00:59<19:13:55,  1.78s/it]
Outer training loop:  22%|##2       | 11090/50000 [6:01:00<19:01:57,  1.76s/it]
Outer training loop:  22%|##2       | 11091/50000 [6:01:02<19:27:37,  1.80s/it]
Outer training loop:  22%|##2       | 11092/50000 [6:01:04<19:10:33,  1.77s/it]
Outer training loop:  22%|##2       | 11093/50000 [6:01:06<19:09:43,  1.77s/it]
Outer training loop:  22%|##2       | 11094/50000 [6:01:07<18:25:38,  1.71s/it]
Outer training loop:  22%|##2       | 11095/50000 [6:01:09<18:51:19,  1.74s/it]
Outer training loop:  22%|##2       | 11096/50000 [6:01:11<18:54:15,  1.75s/it]
Outer training loop:  22%|##2       | 11097/50000 [6:01:13<18:46:28,  1.74s/it]
Outer training loop:  22%|##2       | 11098/50000 [6:01:15<19:20:21,  1.79s/it]
Outer training loop:  22%|##2       | 11099/50000 [6:01:16<19:11:16,  1.78s/it]
Outer training loop:  22%|##2       | 11100/50000 [6:01:18<18:58:54,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.70s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  22%|##2       | 11101/50000 [6:01:25<36:42:24,  3.40s/it]
Outer training loop:  22%|##2       | 11102/50000 [6:01:27<31:02:05,  2.87s/it]
Outer training loop:  22%|##2       | 11103/50000 [6:01:29<26:59:28,  2.50s/it]
Outer training loop:  22%|##2       | 11104/50000 [6:01:30<24:08:30,  2.23s/it]
Outer training loop:  22%|##2       | 11105/50000 [6:01:32<22:57:31,  2.13s/it]
Outer training loop:  22%|##2       | 11106/50000 [6:01:34<21:45:06,  2.01s/it]
Outer training loop:  22%|##2       | 11107/50000 [6:01:36<21:02:09,  1.95s/it]
Outer training loop:  22%|##2       | 11108/50000 [6:01:37<20:20:49,  1.88s/it]
Outer training loop:  22%|##2       | 11109/50000 [6:01:39<20:12:03,  1.87s/it]
Outer training loop:  22%|##2       | 11110/50000 [6:01:41<19:45:10,  1.83s/it]
Outer training loop:  22%|##2       | 11111/50000 [6:01:43<19:24:26,  1.80s/it]
Outer training loop:  22%|##2       | 11112/50000 [6:01:45<19:42:48,  1.82s/it]
Outer training loop:  22%|##2       | 11113/50000 [6:01:46<19:43:06,  1.83s/it]
Outer training loop:  22%|##2       | 11114/50000 [6:01:48<19:28:19,  1.80s/it]
Outer training loop:  22%|##2       | 11115/50000 [6:01:50<19:26:15,  1.80s/it]
Outer training loop:  22%|##2       | 11116/50000 [6:01:52<19:40:08,  1.82s/it]
Outer training loop:  22%|##2       | 11117/50000 [6:01:54<19:21:54,  1.79s/it]
Outer training loop:  22%|##2       | 11118/50000 [6:01:55<18:52:08,  1.75s/it]
Outer training loop:  22%|##2       | 11119/50000 [6:01:57<19:27:28,  1.80s/it]
Outer training loop:  22%|##2       | 11120/50000 [6:01:59<19:29:29,  1.80s/it]
Outer training loop:  22%|##2       | 11121/50000 [6:02:01<19:20:02,  1.79s/it]
Outer training loop:  22%|##2       | 11122/50000 [6:02:02<19:01:54,  1.76s/it]
Outer training loop:  22%|##2       | 11123/50000 [6:02:04<19:31:46,  1.81s/it]
Outer training loop:  22%|##2       | 11124/50000 [6:02:06<19:40:38,  1.82s/it]
Outer training loop:  22%|##2       | 11125/50000 [6:02:08<19:18:05,  1.79s/it]
Outer training loop:  22%|##2       | 11126/50000 [6:02:10<19:43:32,  1.83s/it]
Outer training loop:  22%|##2       | 11127/50000 [6:02:11<19:15:10,  1.78s/it]
Outer training loop:  22%|##2       | 11128/50000 [6:02:13<19:18:01,  1.79s/it]
Outer training loop:  22%|##2       | 11129/50000 [6:02:15<19:12:56,  1.78s/it]
Outer training loop:  22%|##2       | 11130/50000 [6:02:17<18:45:53,  1.74s/it]
Outer training loop:  22%|##2       | 11131/50000 [6:02:18<18:24:50,  1.71s/it]
Outer training loop:  22%|##2       | 11132/50000 [6:02:20<18:57:35,  1.76s/it]
Outer training loop:  22%|##2       | 11133/50000 [6:02:22<18:43:45,  1.73s/it]
Outer training loop:  22%|##2       | 11134/50000 [6:02:23<18:25:20,  1.71s/it]
Outer training loop:  22%|##2       | 11135/50000 [6:02:25<18:32:46,  1.72s/it]
Outer training loop:  22%|##2       | 11136/50000 [6:02:27<19:00:48,  1.76s/it]
Outer training loop:  22%|##2       | 11137/50000 [6:02:29<18:31:20,  1.72s/it]
Outer training loop:  22%|##2       | 11138/50000 [6:02:30<18:34:42,  1.72s/it]
Outer training loop:  22%|##2       | 11139/50000 [6:02:32<19:11:37,  1.78s/it]
Outer training loop:  22%|##2       | 11140/50000 [6:02:34<18:56:43,  1.76s/it]
Outer training loop:  22%|##2       | 11141/50000 [6:02:36<18:35:36,  1.72s/it]
Outer training loop:  22%|##2       | 11142/50000 [6:02:38<19:05:39,  1.77s/it]
Outer training loop:  22%|##2       | 11143/50000 [6:02:39<18:47:43,  1.74s/it]
Outer training loop:  22%|##2       | 11144/50000 [6:02:41<18:44:30,  1.74s/it]
Outer training loop:  22%|##2       | 11145/50000 [6:02:43<18:40:00,  1.73s/it]
Outer training loop:  22%|##2       | 11146/50000 [6:02:45<19:24:48,  1.80s/it]
Outer training loop:  22%|##2       | 11147/50000 [6:02:46<19:14:58,  1.78s/it]
Outer training loop:  22%|##2       | 11148/50000 [6:02:48<18:50:33,  1.75s/it]
Outer training loop:  22%|##2       | 11149/50000 [6:02:50<19:15:37,  1.78s/it]
Outer training loop:  22%|##2       | 11150/50000 [6:02:52<18:49:45,  1.74s/it]
Outer training loop:  22%|##2       | 11151/50000 [6:02:53<18:54:52,  1.75s/it]
Outer training loop:  22%|##2       | 11152/50000 [6:02:55<18:50:52,  1.75s/it]
Outer training loop:  22%|##2       | 11153/50000 [6:02:57<19:18:58,  1.79s/it]
Outer training loop:  22%|##2       | 11154/50000 [6:02:59<19:01:53,  1.76s/it]
Outer training loop:  22%|##2       | 11155/50000 [6:03:00<18:40:21,  1.73s/it]
Outer training loop:  22%|##2       | 11156/50000 [6:03:02<19:11:31,  1.78s/it]
Outer training loop:  22%|##2       | 11157/50000 [6:03:04<18:54:32,  1.75s/it]
Outer training loop:  22%|##2       | 11158/50000 [6:03:05<18:27:38,  1.71s/it]
Outer training loop:  22%|##2       | 11159/50000 [6:03:07<18:49:45,  1.75s/it]
Outer training loop:  22%|##2       | 11160/50000 [6:03:09<18:42:48,  1.73s/it]
Outer training loop:  22%|##2       | 11161/50000 [6:03:11<18:50:41,  1.75s/it]
Outer training loop:  22%|##2       | 11162/50000 [6:03:13<19:23:08,  1.80s/it]
Outer training loop:  22%|##2       | 11163/50000 [6:03:15<19:38:19,  1.82s/it]
Outer training loop:  22%|##2       | 11164/50000 [6:03:16<19:08:20,  1.77s/it]
Outer training loop:  22%|##2       | 11165/50000 [6:03:18<19:39:42,  1.82s/it]
Outer training loop:  22%|##2       | 11166/50000 [6:03:20<19:19:39,  1.79s/it]
Outer training loop:  22%|##2       | 11167/50000 [6:03:22<19:12:38,  1.78s/it]
Outer training loop:  22%|##2       | 11168/50000 [6:03:23<19:21:36,  1.79s/it]
Outer training loop:  22%|##2       | 11169/50000 [6:03:25<19:56:57,  1.85s/it]
Outer training loop:  22%|##2       | 11170/50000 [6:03:27<19:23:56,  1.80s/it]
Outer training loop:  22%|##2       | 11171/50000 [6:03:29<18:55:04,  1.75s/it]
Outer training loop:  22%|##2       | 11172/50000 [6:03:31<19:48:33,  1.84s/it]
Outer training loop:  22%|##2       | 11173/50000 [6:03:33<19:22:12,  1.80s/it]
Outer training loop:  22%|##2       | 11174/50000 [6:03:34<19:02:52,  1.77s/it]
Outer training loop:  22%|##2       | 11175/50000 [6:03:36<19:36:32,  1.82s/it]
Outer training loop:  22%|##2       | 11176/50000 [6:03:38<19:17:20,  1.79s/it]
Outer training loop:  22%|##2       | 11177/50000 [6:03:40<19:06:29,  1.77s/it]
Outer training loop:  22%|##2       | 11178/50000 [6:03:41<19:06:31,  1.77s/it]
Outer training loop:  22%|##2       | 11179/50000 [6:03:43<19:13:29,  1.78s/it]
Outer training loop:  22%|##2       | 11180/50000 [6:03:45<18:49:55,  1.75s/it]
Outer training loop:  22%|##2       | 11181/50000 [6:03:46<18:28:04,  1.71s/it]
Outer training loop:  22%|##2       | 11182/50000 [6:03:48<19:05:34,  1.77s/it]
Outer training loop:  22%|##2       | 11183/50000 [6:03:50<18:50:37,  1.75s/it]
Outer training loop:  22%|##2       | 11184/50000 [6:03:52<18:53:24,  1.75s/it]
Outer training loop:  22%|##2       | 11185/50000 [6:03:54<18:41:28,  1.73s/it]
Outer training loop:  22%|##2       | 11186/50000 [6:03:55<19:02:36,  1.77s/it]
Outer training loop:  22%|##2       | 11187/50000 [6:03:57<19:13:42,  1.78s/it]
Outer training loop:  22%|##2       | 11188/50000 [6:03:59<19:43:31,  1.83s/it]
Outer training loop:  22%|##2       | 11189/50000 [6:04:01<19:18:05,  1.79s/it]
Outer training loop:  22%|##2       | 11190/50000 [6:04:03<19:03:22,  1.77s/it]
Outer training loop:  22%|##2       | 11191/50000 [6:04:04<19:35:36,  1.82s/it]
Outer training loop:  22%|##2       | 11192/50000 [6:04:06<18:52:33,  1.75s/it]
Outer training loop:  22%|##2       | 11193/50000 [6:04:08<18:47:47,  1.74s/it]
Outer training loop:  22%|##2       | 11194/50000 [6:04:10<18:58:38,  1.76s/it]
Outer training loop:  22%|##2       | 11195/50000 [6:04:12<19:28:27,  1.81s/it]
Outer training loop:  22%|##2       | 11196/50000 [6:04:13<19:34:48,  1.82s/it]
Outer training loop:  22%|##2       | 11197/50000 [6:04:15<19:16:52,  1.79s/it]
Outer training loop:  22%|##2       | 11198/50000 [6:04:17<19:48:15,  1.84s/it]
Outer training loop:  22%|##2       | 11199/50000 [6:04:19<19:23:30,  1.80s/it]
Outer training loop:  22%|##2       | 11200/50000 [6:04:20<18:53:38,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.65s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.68s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.63s/it]

Outer training loop:  22%|##2       | 11201/50000 [6:04:27<35:17:02,  3.27s/it]
Outer training loop:  22%|##2       | 11202/50000 [6:04:29<30:51:17,  2.86s/it]
Outer training loop:  22%|##2       | 11203/50000 [6:04:31<27:07:56,  2.52s/it]
Outer training loop:  22%|##2       | 11204/50000 [6:04:32<24:20:30,  2.26s/it]
Outer training loop:  22%|##2       | 11205/50000 [6:04:34<22:57:03,  2.13s/it]
Outer training loop:  22%|##2       | 11206/50000 [6:04:36<21:46:54,  2.02s/it]
Outer training loop:  22%|##2       | 11207/50000 [6:04:38<20:52:25,  1.94s/it]
Outer training loop:  22%|##2       | 11208/50000 [6:04:40<20:47:17,  1.93s/it]
Outer training loop:  22%|##2       | 11209/50000 [6:04:41<20:02:43,  1.86s/it]
Outer training loop:  22%|##2       | 11210/50000 [6:04:43<20:12:01,  1.87s/it]
Outer training loop:  22%|##2       | 11211/50000 [6:04:45<19:50:00,  1.84s/it]
Outer training loop:  22%|##2       | 11212/50000 [6:04:47<20:12:05,  1.87s/it]
Outer training loop:  22%|##2       | 11213/50000 [6:04:49<19:37:53,  1.82s/it]
Outer training loop:  22%|##2       | 11214/50000 [6:04:51<19:40:10,  1.83s/it]
Outer training loop:  22%|##2       | 11215/50000 [6:04:52<19:46:08,  1.83s/it]
Outer training loop:  22%|##2       | 11216/50000 [6:04:54<19:22:07,  1.80s/it]
Outer training loop:  22%|##2       | 11217/50000 [6:04:56<19:05:56,  1.77s/it]
Outer training loop:  22%|##2       | 11218/50000 [6:04:58<19:32:58,  1.81s/it]
Outer training loop:  22%|##2       | 11219/50000 [6:04:59<19:01:47,  1.77s/it]
Outer training loop:  22%|##2       | 11220/50000 [6:05:01<19:03:24,  1.77s/it]
Outer training loop:  22%|##2       | 11221/50000 [6:05:03<18:47:40,  1.74s/it]
Outer training loop:  22%|##2       | 11222/50000 [6:05:05<19:21:50,  1.80s/it]
Outer training loop:  22%|##2       | 11223/50000 [6:05:07<19:14:20,  1.79s/it]
Outer training loop:  22%|##2       | 11224/50000 [6:05:08<19:14:44,  1.79s/it]
Outer training loop:  22%|##2       | 11225/50000 [6:05:10<19:47:55,  1.84s/it]
Outer training loop:  22%|##2       | 11226/50000 [6:05:12<19:35:12,  1.82s/it]
Outer training loop:  22%|##2       | 11227/50000 [6:05:14<19:11:22,  1.78s/it]
Outer training loop:  22%|##2       | 11228/50000 [6:05:16<19:18:39,  1.79s/it]
Outer training loop:  22%|##2       | 11229/50000 [6:05:17<18:57:37,  1.76s/it]
Outer training loop:  22%|##2       | 11230/50000 [6:05:19<18:41:29,  1.74s/it]
Outer training loop:  22%|##2       | 11231/50000 [6:05:21<19:19:06,  1.79s/it]
Outer training loop:  22%|##2       | 11232/50000 [6:05:23<19:17:49,  1.79s/it]
Outer training loop:  22%|##2       | 11233/50000 [6:05:25<19:30:04,  1.81s/it]
Outer training loop:  22%|##2       | 11234/50000 [6:05:26<19:21:37,  1.80s/it]
Outer training loop:  22%|##2       | 11235/50000 [6:05:28<20:08:36,  1.87s/it]
Outer training loop:  22%|##2       | 11236/50000 [6:05:30<20:12:21,  1.88s/it]
Outer training loop:  22%|##2       | 11237/50000 [6:05:32<20:32:26,  1.91s/it]
Outer training loop:  22%|##2       | 11238/50000 [6:05:34<20:01:37,  1.86s/it]
Outer training loop:  22%|##2       | 11239/50000 [6:05:36<19:35:11,  1.82s/it]
Outer training loop:  22%|##2       | 11240/50000 [6:05:37<19:07:44,  1.78s/it]
Outer training loop:  22%|##2       | 11241/50000 [6:05:39<19:21:15,  1.80s/it]
Outer training loop:  22%|##2       | 11242/50000 [6:05:41<19:17:57,  1.79s/it]
Outer training loop:  22%|##2       | 11243/50000 [6:05:43<18:57:09,  1.76s/it]
Outer training loop:  22%|##2       | 11244/50000 [6:05:45<19:14:20,  1.79s/it]
Outer training loop:  22%|##2       | 11245/50000 [6:05:46<18:56:36,  1.76s/it]
Outer training loop:  22%|##2       | 11246/50000 [6:05:48<18:57:40,  1.76s/it]
Outer training loop:  22%|##2       | 11247/50000 [6:05:50<19:27:06,  1.81s/it]
Outer training loop:  22%|##2       | 11248/50000 [6:05:52<18:55:36,  1.76s/it]
Outer training loop:  22%|##2       | 11249/50000 [6:05:53<18:57:47,  1.76s/it]
Outer training loop:  22%|##2       | 11250/50000 [6:05:55<19:50:58,  1.84s/it]
Outer training loop:  23%|##2       | 11251/50000 [6:05:57<19:14:56,  1.79s/it]
Outer training loop:  23%|##2       | 11252/50000 [6:05:59<18:59:38,  1.76s/it]
Outer training loop:  23%|##2       | 11253/50000 [6:06:01<18:59:13,  1.76s/it]
Outer training loop:  23%|##2       | 11254/50000 [6:06:02<19:24:21,  1.80s/it]
Outer training loop:  23%|##2       | 11255/50000 [6:06:04<19:13:19,  1.79s/it]
Outer training loop:  23%|##2       | 11256/50000 [6:06:06<18:44:02,  1.74s/it]
Outer training loop:  23%|##2       | 11257/50000 [6:06:08<19:11:48,  1.78s/it]
Outer training loop:  23%|##2       | 11258/50000 [6:06:09<18:48:30,  1.75s/it]
Outer training loop:  23%|##2       | 11259/50000 [6:06:11<19:00:26,  1.77s/it]
Outer training loop:  23%|##2       | 11260/50000 [6:06:13<19:31:00,  1.81s/it]
Outer training loop:  23%|##2       | 11261/50000 [6:06:15<20:03:36,  1.86s/it]
Outer training loop:  23%|##2       | 11262/50000 [6:06:17<19:37:29,  1.82s/it]
Outer training loop:  23%|##2       | 11263/50000 [6:06:19<19:51:49,  1.85s/it]
Outer training loop:  23%|##2       | 11264/50000 [6:06:21<21:11:01,  1.97s/it]
Outer training loop:  23%|##2       | 11265/50000 [6:06:23<20:59:54,  1.95s/it]
Outer training loop:  23%|##2       | 11266/50000 [6:06:25<21:28:30,  2.00s/it]
Outer training loop:  23%|##2       | 11267/50000 [6:06:27<21:40:30,  2.01s/it]
Outer training loop:  23%|##2       | 11268/50000 [6:06:29<21:43:25,  2.02s/it]
Outer training loop:  23%|##2       | 11269/50000 [6:06:31<21:03:26,  1.96s/it]
Outer training loop:  23%|##2       | 11270/50000 [6:06:33<20:50:17,  1.94s/it]
Outer training loop:  23%|##2       | 11271/50000 [6:06:35<21:38:12,  2.01s/it]
Outer training loop:  23%|##2       | 11272/50000 [6:06:37<21:02:03,  1.96s/it]
Outer training loop:  23%|##2       | 11273/50000 [6:06:39<21:04:19,  1.96s/it]
Outer training loop:  23%|##2       | 11274/50000 [6:06:41<21:06:28,  1.96s/it]
Outer training loop:  23%|##2       | 11275/50000 [6:06:42<20:18:07,  1.89s/it]
Outer training loop:  23%|##2       | 11276/50000 [6:06:44<19:51:29,  1.85s/it]
Outer training loop:  23%|##2       | 11277/50000 [6:06:46<19:48:39,  1.84s/it]
Outer training loop:  23%|##2       | 11278/50000 [6:06:48<20:26:29,  1.90s/it]
Outer training loop:  23%|##2       | 11279/50000 [6:06:50<20:15:58,  1.88s/it]
Outer training loop:  23%|##2       | 11280/50000 [6:06:52<19:57:24,  1.86s/it]
Outer training loop:  23%|##2       | 11281/50000 [6:06:53<19:20:31,  1.80s/it]
Outer training loop:  23%|##2       | 11282/50000 [6:06:56<20:53:06,  1.94s/it]
Outer training loop:  23%|##2       | 11283/50000 [6:06:58<21:01:12,  1.95s/it]
Outer training loop:  23%|##2       | 11284/50000 [6:06:59<20:52:49,  1.94s/it]
Outer training loop:  23%|##2       | 11285/50000 [6:07:01<20:55:56,  1.95s/it]
Outer training loop:  23%|##2       | 11286/50000 [6:07:03<20:24:57,  1.90s/it]
Outer training loop:  23%|##2       | 11287/50000 [6:07:05<19:50:38,  1.85s/it]
Outer training loop:  23%|##2       | 11288/50000 [6:07:07<19:25:47,  1.81s/it]
Outer training loop:  23%|##2       | 11289/50000 [6:07:09<20:03:16,  1.87s/it]
Outer training loop:  23%|##2       | 11290/50000 [6:07:10<19:30:21,  1.81s/it]
Outer training loop:  23%|##2       | 11291/50000 [6:07:12<19:33:38,  1.82s/it]
Outer training loop:  23%|##2       | 11292/50000 [6:07:14<19:59:43,  1.86s/it]
Outer training loop:  23%|##2       | 11293/50000 [6:07:16<19:32:33,  1.82s/it]
Outer training loop:  23%|##2       | 11294/50000 [6:07:18<19:04:16,  1.77s/it]
Outer training loop:  23%|##2       | 11295/50000 [6:07:20<19:48:57,  1.84s/it]
Outer training loop:  23%|##2       | 11296/50000 [6:07:21<19:13:40,  1.79s/it]
Outer training loop:  23%|##2       | 11297/50000 [6:07:23<18:51:41,  1.75s/it]
Outer training loop:  23%|##2       | 11298/50000 [6:07:25<18:41:36,  1.74s/it]
Outer training loop:  23%|##2       | 11299/50000 [6:07:26<19:09:12,  1.78s/it]
Outer training loop:  23%|##2       | 11300/50000 [6:07:28<19:02:02,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  23%|##2       | 11301/50000 [6:07:35<36:12:37,  3.37s/it]
Outer training loop:  23%|##2       | 11302/50000 [6:07:37<31:32:07,  2.93s/it]
Outer training loop:  23%|##2       | 11303/50000 [6:07:39<27:35:52,  2.57s/it]
Outer training loop:  23%|##2       | 11304/50000 [6:07:41<24:50:44,  2.31s/it]
Outer training loop:  23%|##2       | 11305/50000 [6:07:43<23:24:06,  2.18s/it]
Outer training loop:  23%|##2       | 11306/50000 [6:07:44<22:29:42,  2.09s/it]
Outer training loop:  23%|##2       | 11307/50000 [6:07:46<22:07:20,  2.06s/it]
Outer training loop:  23%|##2       | 11308/50000 [6:07:48<21:00:46,  1.96s/it]
Outer training loop:  23%|##2       | 11309/50000 [6:07:50<21:21:36,  1.99s/it]
Outer training loop:  23%|##2       | 11310/50000 [6:07:52<20:27:13,  1.90s/it]
Outer training loop:  23%|##2       | 11311/50000 [6:07:54<20:29:39,  1.91s/it]
Outer training loop:  23%|##2       | 11312/50000 [6:07:56<20:53:27,  1.94s/it]
Outer training loop:  23%|##2       | 11313/50000 [6:07:58<20:35:13,  1.92s/it]
Outer training loop:  23%|##2       | 11314/50000 [6:07:59<19:50:34,  1.85s/it]
Outer training loop:  23%|##2       | 11315/50000 [6:08:01<19:46:17,  1.84s/it]
Outer training loop:  23%|##2       | 11316/50000 [6:08:03<20:19:34,  1.89s/it]
Outer training loop:  23%|##2       | 11317/50000 [6:08:05<20:04:46,  1.87s/it]
Outer training loop:  23%|##2       | 11318/50000 [6:08:07<19:40:10,  1.83s/it]
Outer training loop:  23%|##2       | 11319/50000 [6:08:09<20:52:27,  1.94s/it]
Outer training loop:  23%|##2       | 11320/50000 [6:08:11<20:31:15,  1.91s/it]
Outer training loop:  23%|##2       | 11321/50000 [6:08:13<19:58:24,  1.86s/it]
Outer training loop:  23%|##2       | 11322/50000 [6:08:15<20:33:05,  1.91s/it]
Outer training loop:  23%|##2       | 11323/50000 [6:08:16<20:02:41,  1.87s/it]
Outer training loop:  23%|##2       | 11324/50000 [6:08:18<19:27:23,  1.81s/it]
Outer training loop:  23%|##2       | 11325/50000 [6:08:20<20:15:55,  1.89s/it]
Outer training loop:  23%|##2       | 11326/50000 [6:08:22<20:03:24,  1.87s/it]
Outer training loop:  23%|##2       | 11327/50000 [6:08:24<20:11:09,  1.88s/it]
Outer training loop:  23%|##2       | 11328/50000 [6:08:26<19:51:31,  1.85s/it]
Outer training loop:  23%|##2       | 11329/50000 [6:08:28<20:10:53,  1.88s/it]
Outer training loop:  23%|##2       | 11330/50000 [6:08:29<19:50:20,  1.85s/it]
Outer training loop:  23%|##2       | 11331/50000 [6:08:31<20:01:17,  1.86s/it]
Outer training loop:  23%|##2       | 11332/50000 [6:08:33<20:33:33,  1.91s/it]
Outer training loop:  23%|##2       | 11333/50000 [6:08:35<20:00:26,  1.86s/it]
Outer training loop:  23%|##2       | 11334/50000 [6:08:37<19:37:44,  1.83s/it]
Outer training loop:  23%|##2       | 11335/50000 [6:08:39<20:29:37,  1.91s/it]
Outer training loop:  23%|##2       | 11336/50000 [6:08:41<20:11:14,  1.88s/it]
Outer training loop:  23%|##2       | 11337/50000 [6:08:42<19:56:52,  1.86s/it]
Outer training loop:  23%|##2       | 11338/50000 [6:08:44<19:09:25,  1.78s/it]
Outer training loop:  23%|##2       | 11339/50000 [6:08:46<19:36:41,  1.83s/it]
Outer training loop:  23%|##2       | 11340/50000 [6:08:48<19:19:35,  1.80s/it]
Outer training loop:  23%|##2       | 11341/50000 [6:08:49<19:03:11,  1.77s/it]
Outer training loop:  23%|##2       | 11342/50000 [6:08:51<19:54:56,  1.85s/it]
Outer training loop:  23%|##2       | 11343/50000 [6:08:53<19:53:44,  1.85s/it]
Outer training loop:  23%|##2       | 11344/50000 [6:08:55<19:29:52,  1.82s/it]
Outer training loop:  23%|##2       | 11345/50000 [6:08:57<19:57:43,  1.86s/it]
Outer training loop:  23%|##2       | 11346/50000 [6:08:59<19:31:56,  1.82s/it]
Outer training loop:  23%|##2       | 11347/50000 [6:09:00<19:10:44,  1.79s/it]
Outer training loop:  23%|##2       | 11348/50000 [6:09:02<19:40:29,  1.83s/it]
Outer training loop:  23%|##2       | 11349/50000 [6:09:04<19:43:31,  1.84s/it]
Outer training loop:  23%|##2       | 11350/50000 [6:09:06<19:30:16,  1.82s/it]
Outer training loop:  23%|##2       | 11351/50000 [6:09:08<20:01:52,  1.87s/it]
Outer training loop:  23%|##2       | 11352/50000 [6:09:10<19:31:03,  1.82s/it]
Outer training loop:  23%|##2       | 11353/50000 [6:09:11<19:29:20,  1.82s/it]
Outer training loop:  23%|##2       | 11354/50000 [6:09:13<19:18:59,  1.80s/it]
Outer training loop:  23%|##2       | 11355/50000 [6:09:15<19:37:56,  1.83s/it]
Outer training loop:  23%|##2       | 11356/50000 [6:09:17<19:17:52,  1.80s/it]
Outer training loop:  23%|##2       | 11357/50000 [6:09:19<19:35:05,  1.82s/it]
Outer training loop:  23%|##2       | 11358/50000 [6:09:21<19:57:32,  1.86s/it]
Outer training loop:  23%|##2       | 11359/50000 [6:09:22<19:38:12,  1.83s/it]
Outer training loop:  23%|##2       | 11360/50000 [6:09:24<19:15:53,  1.79s/it]
Outer training loop:  23%|##2       | 11361/50000 [6:09:26<19:39:24,  1.83s/it]
Outer training loop:  23%|##2       | 11362/50000 [6:09:28<19:10:09,  1.79s/it]
Outer training loop:  23%|##2       | 11363/50000 [6:09:30<19:05:13,  1.78s/it]
Outer training loop:  23%|##2       | 11364/50000 [6:09:31<19:00:19,  1.77s/it]
Outer training loop:  23%|##2       | 11365/50000 [6:09:33<19:41:26,  1.83s/it]
Outer training loop:  23%|##2       | 11366/50000 [6:09:35<19:55:46,  1.86s/it]
Outer training loop:  23%|##2       | 11367/50000 [6:09:37<19:34:06,  1.82s/it]
Outer training loop:  23%|##2       | 11368/50000 [6:09:39<19:39:24,  1.83s/it]
Outer training loop:  23%|##2       | 11369/50000 [6:09:41<19:21:49,  1.80s/it]
Outer training loop:  23%|##2       | 11370/50000 [6:09:42<19:33:07,  1.82s/it]
Outer training loop:  23%|##2       | 11371/50000 [6:09:44<19:35:45,  1.83s/it]
Outer training loop:  23%|##2       | 11372/50000 [6:09:46<19:28:05,  1.81s/it]
Outer training loop:  23%|##2       | 11373/50000 [6:09:48<19:07:27,  1.78s/it]
Outer training loop:  23%|##2       | 11374/50000 [6:09:49<18:58:21,  1.77s/it]
Outer training loop:  23%|##2       | 11375/50000 [6:09:51<19:30:44,  1.82s/it]
Outer training loop:  23%|##2       | 11376/50000 [6:09:53<19:29:05,  1.82s/it]
Outer training loop:  23%|##2       | 11377/50000 [6:09:55<19:21:13,  1.80s/it]
Outer training loop:  23%|##2       | 11378/50000 [6:09:57<19:40:22,  1.83s/it]
Outer training loop:  23%|##2       | 11379/50000 [6:09:59<19:21:03,  1.80s/it]
Outer training loop:  23%|##2       | 11380/50000 [6:10:00<19:12:19,  1.79s/it]
Outer training loop:  23%|##2       | 11381/50000 [6:10:02<19:40:44,  1.83s/it]
Outer training loop:  23%|##2       | 11382/50000 [6:10:04<19:33:24,  1.82s/it]
Outer training loop:  23%|##2       | 11383/50000 [6:10:06<19:29:53,  1.82s/it]
Outer training loop:  23%|##2       | 11384/50000 [6:10:08<19:52:48,  1.85s/it]
Outer training loop:  23%|##2       | 11385/50000 [6:10:10<19:54:02,  1.86s/it]
Outer training loop:  23%|##2       | 11386/50000 [6:10:11<19:25:14,  1.81s/it]
Outer training loop:  23%|##2       | 11387/50000 [6:10:13<19:38:51,  1.83s/it]
Outer training loop:  23%|##2       | 11388/50000 [6:10:15<19:06:35,  1.78s/it]
Outer training loop:  23%|##2       | 11389/50000 [6:10:17<18:48:11,  1.75s/it]
Outer training loop:  23%|##2       | 11390/50000 [6:10:18<18:59:52,  1.77s/it]
Outer training loop:  23%|##2       | 11391/50000 [6:10:20<19:30:22,  1.82s/it]
Outer training loop:  23%|##2       | 11392/50000 [6:10:22<19:22:30,  1.81s/it]
Outer training loop:  23%|##2       | 11393/50000 [6:10:24<19:14:43,  1.79s/it]
Outer training loop:  23%|##2       | 11394/50000 [6:10:26<19:40:12,  1.83s/it]
Outer training loop:  23%|##2       | 11395/50000 [6:10:28<19:10:56,  1.79s/it]
Outer training loop:  23%|##2       | 11396/50000 [6:10:29<18:48:26,  1.75s/it]
Outer training loop:  23%|##2       | 11397/50000 [6:10:31<18:35:38,  1.73s/it]
Outer training loop:  23%|##2       | 11398/50000 [6:10:33<19:10:51,  1.79s/it]
Outer training loop:  23%|##2       | 11399/50000 [6:10:35<18:55:00,  1.76s/it]
Outer training loop:  23%|##2       | 11400/50000 [6:10:36<18:43:55,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  23%|##2       | 11401/50000 [6:10:43<35:50:42,  3.34s/it]
Outer training loop:  23%|##2       | 11402/50000 [6:10:45<31:27:24,  2.93s/it]
Outer training loop:  23%|##2       | 11403/50000 [6:10:47<27:55:18,  2.60s/it]
Outer training loop:  23%|##2       | 11404/50000 [6:10:49<25:59:30,  2.42s/it]
Outer training loop:  23%|##2       | 11405/50000 [6:10:51<24:11:27,  2.26s/it]
Outer training loop:  23%|##2       | 11406/50000 [6:10:53<22:30:51,  2.10s/it]
Outer training loop:  23%|##2       | 11407/50000 [6:10:54<21:25:15,  2.00s/it]
Outer training loop:  23%|##2       | 11408/50000 [6:10:56<20:55:34,  1.95s/it]
Outer training loop:  23%|##2       | 11409/50000 [6:10:58<20:51:09,  1.95s/it]
Outer training loop:  23%|##2       | 11410/50000 [6:11:00<20:18:47,  1.89s/it]
Outer training loop:  23%|##2       | 11411/50000 [6:11:02<19:58:57,  1.86s/it]
Outer training loop:  23%|##2       | 11412/50000 [6:11:04<20:12:07,  1.88s/it]
Outer training loop:  23%|##2       | 11413/50000 [6:11:05<19:40:13,  1.84s/it]
Outer training loop:  23%|##2       | 11414/50000 [6:11:07<19:19:58,  1.80s/it]
Outer training loop:  23%|##2       | 11415/50000 [6:11:09<19:25:22,  1.81s/it]
Outer training loop:  23%|##2       | 11416/50000 [6:11:11<19:13:40,  1.79s/it]
Outer training loop:  23%|##2       | 11417/50000 [6:11:13<19:05:39,  1.78s/it]
Outer training loop:  23%|##2       | 11418/50000 [6:11:15<19:50:42,  1.85s/it]
Outer training loop:  23%|##2       | 11419/50000 [6:11:16<19:17:21,  1.80s/it]
Outer training loop:  23%|##2       | 11420/50000 [6:11:18<19:00:34,  1.77s/it]
Outer training loop:  23%|##2       | 11421/50000 [6:11:20<18:49:49,  1.76s/it]
Outer training loop:  23%|##2       | 11422/50000 [6:11:22<19:51:24,  1.85s/it]
Outer training loop:  23%|##2       | 11423/50000 [6:11:23<19:14:57,  1.80s/it]
Outer training loop:  23%|##2       | 11424/50000 [6:11:25<18:59:42,  1.77s/it]
Outer training loop:  23%|##2       | 11425/50000 [6:11:27<19:23:54,  1.81s/it]
Outer training loop:  23%|##2       | 11426/50000 [6:11:29<19:14:51,  1.80s/it]
Outer training loop:  23%|##2       | 11427/50000 [6:11:31<19:28:39,  1.82s/it]
Outer training loop:  23%|##2       | 11428/50000 [6:11:32<19:12:35,  1.79s/it]
Outer training loop:  23%|##2       | 11429/50000 [6:11:34<19:49:32,  1.85s/it]
Outer training loop:  23%|##2       | 11430/50000 [6:11:36<19:28:21,  1.82s/it]
Outer training loop:  23%|##2       | 11431/50000 [6:11:38<19:32:34,  1.82s/it]
Outer training loop:  23%|##2       | 11432/50000 [6:11:40<20:04:15,  1.87s/it]
Outer training loop:  23%|##2       | 11433/50000 [6:11:42<20:09:18,  1.88s/it]
Outer training loop:  23%|##2       | 11434/50000 [6:11:44<19:56:10,  1.86s/it]
Outer training loop:  23%|##2       | 11435/50000 [6:11:46<20:09:42,  1.88s/it]
Outer training loop:  23%|##2       | 11436/50000 [6:11:48<20:27:38,  1.91s/it]
Outer training loop:  23%|##2       | 11437/50000 [6:11:49<19:56:13,  1.86s/it]
Outer training loop:  23%|##2       | 11438/50000 [6:11:51<19:33:46,  1.83s/it]
Outer training loop:  23%|##2       | 11439/50000 [6:11:53<20:19:29,  1.90s/it]
Outer training loop:  23%|##2       | 11440/50000 [6:11:55<20:10:39,  1.88s/it]
Outer training loop:  23%|##2       | 11441/50000 [6:11:57<19:55:49,  1.86s/it]
Outer training loop:  23%|##2       | 11442/50000 [6:11:59<19:59:32,  1.87s/it]
Outer training loop:  23%|##2       | 11443/50000 [6:12:00<19:41:04,  1.84s/it]
Outer training loop:  23%|##2       | 11444/50000 [6:12:02<19:04:16,  1.78s/it]
Outer training loop:  23%|##2       | 11445/50000 [6:12:04<19:13:50,  1.80s/it]
Outer training loop:  23%|##2       | 11446/50000 [6:12:06<19:32:17,  1.82s/it]
Outer training loop:  23%|##2       | 11447/50000 [6:12:07<19:06:52,  1.78s/it]
Outer training loop:  23%|##2       | 11448/50000 [6:12:09<18:37:38,  1.74s/it]
Outer training loop:  23%|##2       | 11449/50000 [6:12:11<19:16:35,  1.80s/it]
Outer training loop:  23%|##2       | 11450/50000 [6:12:13<19:16:43,  1.80s/it]
Outer training loop:  23%|##2       | 11451/50000 [6:12:15<19:14:45,  1.80s/it]
Outer training loop:  23%|##2       | 11452/50000 [6:12:16<18:56:30,  1.77s/it]
Outer training loop:  23%|##2       | 11453/50000 [6:12:18<19:15:15,  1.80s/it]
Outer training loop:  23%|##2       | 11454/50000 [6:12:20<19:48:37,  1.85s/it]
Outer training loop:  23%|##2       | 11455/50000 [6:12:22<19:19:28,  1.80s/it]
Outer training loop:  23%|##2       | 11456/50000 [6:12:24<19:38:57,  1.84s/it]
Outer training loop:  23%|##2       | 11457/50000 [6:12:26<19:15:10,  1.80s/it]
Outer training loop:  23%|##2       | 11458/50000 [6:12:27<18:34:29,  1.73s/it]
Outer training loop:  23%|##2       | 11459/50000 [6:12:29<18:36:51,  1.74s/it]
Outer training loop:  23%|##2       | 11460/50000 [6:12:31<18:58:38,  1.77s/it]
Outer training loop:  23%|##2       | 11461/50000 [6:12:32<18:49:47,  1.76s/it]
Outer training loop:  23%|##2       | 11462/50000 [6:12:34<18:51:44,  1.76s/it]
Outer training loop:  23%|##2       | 11463/50000 [6:12:36<19:33:01,  1.83s/it]
Outer training loop:  23%|##2       | 11464/50000 [6:12:38<19:10:07,  1.79s/it]
Outer training loop:  23%|##2       | 11465/50000 [6:12:40<19:16:27,  1.80s/it]
Outer training loop:  23%|##2       | 11466/50000 [6:12:42<20:00:21,  1.87s/it]
Outer training loop:  23%|##2       | 11467/50000 [6:12:44<20:06:18,  1.88s/it]
Outer training loop:  23%|##2       | 11468/50000 [6:12:45<19:52:27,  1.86s/it]
Outer training loop:  23%|##2       | 11469/50000 [6:12:47<20:02:38,  1.87s/it]
Outer training loop:  23%|##2       | 11470/50000 [6:12:49<19:58:33,  1.87s/it]
Outer training loop:  23%|##2       | 11471/50000 [6:12:51<19:32:15,  1.83s/it]
Outer training loop:  23%|##2       | 11472/50000 [6:12:53<19:13:43,  1.80s/it]
Outer training loop:  23%|##2       | 11473/50000 [6:12:55<19:31:51,  1.82s/it]
Outer training loop:  23%|##2       | 11474/50000 [6:12:56<19:31:28,  1.82s/it]
Outer training loop:  23%|##2       | 11475/50000 [6:12:58<19:34:57,  1.83s/it]
Outer training loop:  23%|##2       | 11476/50000 [6:13:00<19:52:34,  1.86s/it]
Outer training loop:  23%|##2       | 11477/50000 [6:13:02<19:31:27,  1.82s/it]
Outer training loop:  23%|##2       | 11478/50000 [6:13:04<19:13:52,  1.80s/it]
Outer training loop:  23%|##2       | 11479/50000 [6:13:05<19:23:47,  1.81s/it]
Outer training loop:  23%|##2       | 11480/50000 [6:13:07<19:40:35,  1.84s/it]
Outer training loop:  23%|##2       | 11481/50000 [6:13:09<19:02:44,  1.78s/it]
Outer training loop:  23%|##2       | 11482/50000 [6:13:11<19:14:18,  1.80s/it]
Outer training loop:  23%|##2       | 11483/50000 [6:13:13<19:38:14,  1.84s/it]
Outer training loop:  23%|##2       | 11484/50000 [6:13:15<19:26:05,  1.82s/it]
Outer training loop:  23%|##2       | 11485/50000 [6:13:16<19:27:14,  1.82s/it]
Outer training loop:  23%|##2       | 11486/50000 [6:13:18<19:42:21,  1.84s/it]
Outer training loop:  23%|##2       | 11487/50000 [6:13:20<19:28:18,  1.82s/it]
Outer training loop:  23%|##2       | 11488/50000 [6:13:22<19:14:40,  1.80s/it]
Outer training loop:  23%|##2       | 11489/50000 [6:13:24<19:30:39,  1.82s/it]
Outer training loop:  23%|##2       | 11490/50000 [6:13:25<19:31:15,  1.82s/it]
Outer training loop:  23%|##2       | 11491/50000 [6:13:27<19:22:15,  1.81s/it]
Outer training loop:  23%|##2       | 11492/50000 [6:13:29<19:45:21,  1.85s/it]
Outer training loop:  23%|##2       | 11493/50000 [6:13:31<19:49:16,  1.85s/it]
Outer training loop:  23%|##2       | 11494/50000 [6:13:33<19:40:08,  1.84s/it]
Outer training loop:  23%|##2       | 11495/50000 [6:13:35<20:05:16,  1.88s/it]
Outer training loop:  23%|##2       | 11496/50000 [6:13:37<19:36:45,  1.83s/it]
Outer training loop:  23%|##2       | 11497/50000 [6:13:38<19:16:14,  1.80s/it]
Outer training loop:  23%|##2       | 11498/50000 [6:13:40<19:53:17,  1.86s/it]
Outer training loop:  23%|##2       | 11499/50000 [6:13:42<19:59:02,  1.87s/it]
Outer training loop:  23%|##3       | 11500/50000 [6:13:44<19:54:21,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.65s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.70s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:31,  1.85s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.82s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.79s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:26,  1.85s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:22,  1.83s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:18,  1.80s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:17,  1.81s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:19,  1.85s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:17,  1.86s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:12,  1.82s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:11,  1.83s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.88s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:10,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:06,  1.84s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:03,  1.82s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:01,  1.84s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:54,  1.79s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:53,  1.81s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:40<01:52,  1.84s/it]

Evaluating policy:  28%|##7       | 23/83 [00:42<01:50,  1.83s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:45,  1.82s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:49<01:46,  1.89s/it]

Evaluating policy:  34%|###3      | 28/83 [00:51<01:41,  1.84s/it]

Evaluating policy:  35%|###4      | 29/83 [00:53<01:36,  1.79s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:34,  1.78s/it]

Evaluating policy:  37%|###7      | 31/83 [00:56<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [00:58<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:00<01:30,  1.81s/it]

Evaluating policy:  41%|####      | 34/83 [01:02<01:26,  1.76s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:24,  1.76s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:22,  1.76s/it]

Evaluating policy:  45%|####4     | 37/83 [01:07<01:25,  1.86s/it]

Evaluating policy:  46%|####5     | 38/83 [01:09<01:21,  1.81s/it]

Evaluating policy:  47%|####6     | 39/83 [01:11<01:19,  1.81s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:16,  1.77s/it]

Evaluating policy:  49%|####9     | 41/83 [01:14<01:14,  1.77s/it]

Evaluating policy:  51%|#####     | 42/83 [01:16<01:13,  1.79s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:18<01:10,  1.77s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:20<01:12,  1.85s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:23<01:05,  1.77s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:25<01:03,  1.76s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:27<01:00,  1.74s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<00:58,  1.72s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:58,  1.77s/it]

Evaluating policy:  61%|######1   | 51/83 [01:32<00:56,  1.78s/it]

Evaluating policy:  63%|######2   | 52/83 [01:34<00:59,  1.91s/it]

Evaluating policy:  64%|######3   | 53/83 [01:36<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:38<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:39<00:50,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:41<00:48,  1.81s/it]

Evaluating policy:  69%|######8   | 57/83 [01:43<00:47,  1.83s/it]

Evaluating policy:  70%|######9   | 58/83 [01:45<00:45,  1.80s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:47<00:42,  1.78s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:49<00:44,  1.92s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:51<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:52<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:54<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:56<00:33,  1.79s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:58<00:31,  1.77s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:59<00:30,  1.78s/it]

Evaluating policy:  81%|########  | 67/83 [02:01<00:28,  1.76s/it]

Evaluating policy:  82%|########1 | 68/83 [02:03<00:28,  1.93s/it]

Evaluating policy:  83%|########3 | 69/83 [02:05<00:25,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:07<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:09<00:21,  1.80s/it]

Evaluating policy:  87%|########6 | 72/83 [02:10<00:19,  1.79s/it]

Evaluating policy:  88%|########7 | 73/83 [02:12<00:17,  1.79s/it]

Evaluating policy:  89%|########9 | 74/83 [02:14<00:16,  1.79s/it]

Evaluating policy:  90%|######### | 75/83 [02:16<00:14,  1.79s/it]

Evaluating policy:  92%|#########1| 76/83 [02:18<00:13,  1.96s/it]

Evaluating policy:  93%|#########2| 77/83 [02:20<00:11,  1.90s/it]

Evaluating policy:  94%|#########3| 78/83 [02:22<00:09,  1.86s/it]

Evaluating policy:  95%|#########5| 79/83 [02:24<00:07,  1.87s/it]

Evaluating policy:  96%|#########6| 80/83 [02:25<00:05,  1.85s/it]

Evaluating policy:  98%|#########7| 81/83 [02:27<00:03,  1.81s/it]

Evaluating policy:  99%|#########8| 82/83 [02:29<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 83/83 [02:31<00:00,  1.75s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:16,  1.66s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:16,  1.69s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:27,  1.85s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:26,  1.85s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:20,  1.80s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.77s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.84s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:14,  1.79s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:10,  1.76s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:12,  1.81s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:11,  1.83s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.79s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:03,  1.76s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:06,  1.83s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:03,  1.82s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:00,  1.80s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<02:00,  1.82s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:57,  1.81s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:58,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:54,  1.82s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:49,  1.77s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:46,  1.75s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:46,  1.77s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:47,  1.83s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:43,  1.79s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.77s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:38,  1.75s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:37,  1.78s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:36,  1.82s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:33,  1.79s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:30,  1.78s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:28,  1.77s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:27,  1.81s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:25,  1.83s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:22,  1.80s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:19,  1.76s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:16,  1.74s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:20,  1.87s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:16,  1.82s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:14,  1.81s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:11,  1.79s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:09,  1.77s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:08,  1.79s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:04,  1.74s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:06,  1.86s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<01:01,  1.80s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:58,  1.77s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:56,  1.77s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:54,  1.77s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:53,  1.77s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:39<00:50,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:48,  1.81s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:46,  1.77s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:44,  1.77s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:41,  1.74s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:39,  1.73s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:38,  1.73s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:53<00:36,  1.80s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:55<00:34,  1.79s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:31,  1.77s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:29,  1.75s/it]

Evaluating policy:  81%|########  | 67/83 [02:00<00:28,  1.81s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:26,  1.78s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:25,  1.80s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:23,  1.78s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:23,  1.92s/it]

Evaluating policy:  87%|########6 | 72/83 [02:09<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:11<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:13<00:16,  1.80s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:14,  1.80s/it]

Evaluating policy:  92%|#########1| 76/83 [02:16<00:12,  1.77s/it]

Evaluating policy:  93%|#########2| 77/83 [02:18<00:10,  1.77s/it]

Evaluating policy:  94%|#########3| 78/83 [02:20<00:08,  1.74s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.78s/it]

Evaluating policy:  96%|#########6| 80/83 [02:24<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:25<00:03,  1.85s/it]

Evaluating policy:  99%|#########8| 82/83 [02:27<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 83/83 [02:29<00:00,  1.81s/it]

Outer training loop:  23%|##3       | 11501/50000 [6:18:52<1001:11:14, 93.62s/it]
Outer training loop:  23%|##3       | 11502/50000 [6:18:53<706:18:14, 66.05s/it] 
Outer training loop:  23%|##3       | 11503/50000 [6:18:55<500:01:46, 46.76s/it]
Outer training loop:  23%|##3       | 11504/50000 [6:18:57<355:45:42, 33.27s/it]
Outer training loop:  23%|##3       | 11505/50000 [6:18:59<254:29:05, 23.80s/it]
Outer training loop:  23%|##3       | 11506/50000 [6:19:00<183:45:17, 17.18s/it]
Outer training loop:  23%|##3       | 11507/50000 [6:19:02<134:02:25, 12.54s/it]
Outer training loop:  23%|##3       | 11508/50000 [6:19:05<101:40:33,  9.51s/it]
Outer training loop:  23%|##3       | 11509/50000 [6:19:06<76:45:33,  7.18s/it] 
Outer training loop:  23%|##3       | 11510/50000 [6:19:08<59:30:34,  5.57s/it]
Outer training loop:  23%|##3       | 11511/50000 [6:19:10<47:44:16,  4.47s/it]
Outer training loop:  23%|##3       | 11512/50000 [6:19:12<38:46:34,  3.63s/it]
Outer training loop:  23%|##3       | 11513/50000 [6:19:13<32:52:16,  3.07s/it]
Outer training loop:  23%|##3       | 11514/50000 [6:19:15<28:58:17,  2.71s/it]
Outer training loop:  23%|##3       | 11515/50000 [6:19:17<26:06:14,  2.44s/it]
Outer training loop:  23%|##3       | 11516/50000 [6:19:19<23:46:31,  2.22s/it]
Outer training loop:  23%|##3       | 11517/50000 [6:19:21<22:40:40,  2.12s/it]
Outer training loop:  23%|##3       | 11518/50000 [6:19:23<21:26:02,  2.01s/it]
Outer training loop:  23%|##3       | 11519/50000 [6:19:24<20:44:56,  1.94s/it]
Outer training loop:  23%|##3       | 11520/50000 [6:19:26<20:32:03,  1.92s/it]
Outer training loop:  23%|##3       | 11521/50000 [6:19:28<20:04:54,  1.88s/it]
Outer training loop:  23%|##3       | 11522/50000 [6:19:30<19:28:08,  1.82s/it]
Outer training loop:  23%|##3       | 11523/50000 [6:19:31<19:25:57,  1.82s/it]
Outer training loop:  23%|##3       | 11524/50000 [6:19:33<20:00:08,  1.87s/it]
Outer training loop:  23%|##3       | 11525/50000 [6:19:35<19:26:45,  1.82s/it]
Outer training loop:  23%|##3       | 11526/50000 [6:19:37<18:53:34,  1.77s/it]
Outer training loop:  23%|##3       | 11527/50000 [6:19:39<19:26:28,  1.82s/it]
Outer training loop:  23%|##3       | 11528/50000 [6:19:40<19:05:14,  1.79s/it]
Outer training loop:  23%|##3       | 11529/50000 [6:19:42<18:44:00,  1.75s/it]
Outer training loop:  23%|##3       | 11530/50000 [6:19:44<19:04:14,  1.78s/it]
Outer training loop:  23%|##3       | 11531/50000 [6:19:46<18:42:05,  1.75s/it]
Outer training loop:  23%|##3       | 11532/50000 [6:19:47<18:47:51,  1.76s/it]
Outer training loop:  23%|##3       | 11533/50000 [6:19:49<18:57:21,  1.77s/it]
Outer training loop:  23%|##3       | 11534/50000 [6:19:51<19:39:36,  1.84s/it]
Outer training loop:  23%|##3       | 11535/50000 [6:19:53<19:34:50,  1.83s/it]
Outer training loop:  23%|##3       | 11536/50000 [6:19:55<19:46:14,  1.85s/it]
Outer training loop:  23%|##3       | 11537/50000 [6:19:57<19:13:02,  1.80s/it]
Outer training loop:  23%|##3       | 11538/50000 [6:19:58<18:41:53,  1.75s/it]
Outer training loop:  23%|##3       | 11539/50000 [6:20:00<18:23:49,  1.72s/it]
Outer training loop:  23%|##3       | 11540/50000 [6:20:02<19:14:46,  1.80s/it]
Outer training loop:  23%|##3       | 11541/50000 [6:20:04<18:45:36,  1.76s/it]
Outer training loop:  23%|##3       | 11542/50000 [6:20:05<18:26:32,  1.73s/it]
Outer training loop:  23%|##3       | 11543/50000 [6:20:07<19:16:57,  1.81s/it]
Outer training loop:  23%|##3       | 11544/50000 [6:20:09<19:12:45,  1.80s/it]
Outer training loop:  23%|##3       | 11545/50000 [6:20:11<19:08:10,  1.79s/it]
Outer training loop:  23%|##3       | 11546/50000 [6:20:13<19:33:32,  1.83s/it]
Outer training loop:  23%|##3       | 11547/50000 [6:20:14<19:20:42,  1.81s/it]
Outer training loop:  23%|##3       | 11548/50000 [6:20:16<19:02:43,  1.78s/it]
Outer training loop:  23%|##3       | 11549/50000 [6:20:18<19:18:54,  1.81s/it]
Outer training loop:  23%|##3       | 11550/50000 [6:20:20<19:17:15,  1.81s/it]
Outer training loop:  23%|##3       | 11551/50000 [6:20:22<19:26:28,  1.82s/it]
Outer training loop:  23%|##3       | 11552/50000 [6:20:24<19:33:29,  1.83s/it]
Outer training loop:  23%|##3       | 11553/50000 [6:20:25<19:27:06,  1.82s/it]
Outer training loop:  23%|##3       | 11554/50000 [6:20:27<19:23:27,  1.82s/it]
Outer training loop:  23%|##3       | 11555/50000 [6:20:29<19:44:47,  1.85s/it]
Outer training loop:  23%|##3       | 11556/50000 [6:20:31<19:23:07,  1.82s/it]
Outer training loop:  23%|##3       | 11557/50000 [6:20:33<19:13:30,  1.80s/it]
Outer training loop:  23%|##3       | 11558/50000 [6:20:34<19:00:04,  1.78s/it]
Outer training loop:  23%|##3       | 11559/50000 [6:20:36<19:38:42,  1.84s/it]
Outer training loop:  23%|##3       | 11560/50000 [6:20:38<19:19:24,  1.81s/it]
Outer training loop:  23%|##3       | 11561/50000 [6:20:40<19:00:11,  1.78s/it]
Outer training loop:  23%|##3       | 11562/50000 [6:20:42<19:32:06,  1.83s/it]
Outer training loop:  23%|##3       | 11563/50000 [6:20:43<19:23:42,  1.82s/it]
Outer training loop:  23%|##3       | 11564/50000 [6:20:45<19:30:15,  1.83s/it]
Outer training loop:  23%|##3       | 11565/50000 [6:20:47<19:58:43,  1.87s/it]
Outer training loop:  23%|##3       | 11566/50000 [6:20:49<19:50:51,  1.86s/it]
Outer training loop:  23%|##3       | 11567/50000 [6:20:51<19:47:47,  1.85s/it]
Outer training loop:  23%|##3       | 11568/50000 [6:20:53<19:40:20,  1.84s/it]
Outer training loop:  23%|##3       | 11569/50000 [6:20:55<19:23:49,  1.82s/it]
Outer training loop:  23%|##3       | 11570/50000 [6:20:56<19:08:52,  1.79s/it]
Outer training loop:  23%|##3       | 11571/50000 [6:20:58<19:21:28,  1.81s/it]
Outer training loop:  23%|##3       | 11572/50000 [6:21:00<20:03:16,  1.88s/it]
Outer training loop:  23%|##3       | 11573/50000 [6:21:02<19:56:22,  1.87s/it]
Outer training loop:  23%|##3       | 11574/50000 [6:21:04<19:35:37,  1.84s/it]
Outer training loop:  23%|##3       | 11575/50000 [6:21:06<20:11:07,  1.89s/it]
Outer training loop:  23%|##3       | 11576/50000 [6:21:08<19:39:02,  1.84s/it]
Outer training loop:  23%|##3       | 11577/50000 [6:21:09<19:21:17,  1.81s/it]
Outer training loop:  23%|##3       | 11578/50000 [6:21:11<19:17:17,  1.81s/it]
Outer training loop:  23%|##3       | 11579/50000 [6:21:13<19:39:25,  1.84s/it]
Outer training loop:  23%|##3       | 11580/50000 [6:21:15<19:29:23,  1.83s/it]
Outer training loop:  23%|##3       | 11581/50000 [6:21:16<19:00:39,  1.78s/it]
Outer training loop:  23%|##3       | 11582/50000 [6:21:18<19:33:15,  1.83s/it]
Outer training loop:  23%|##3       | 11583/50000 [6:21:20<19:30:45,  1.83s/it]
Outer training loop:  23%|##3       | 11584/50000 [6:21:22<19:14:01,  1.80s/it]
Outer training loop:  23%|##3       | 11585/50000 [6:21:24<20:37:19,  1.93s/it]
Outer training loop:  23%|##3       | 11586/50000 [6:21:26<19:53:40,  1.86s/it]
Outer training loop:  23%|##3       | 11587/50000 [6:21:28<19:44:18,  1.85s/it]
Outer training loop:  23%|##3       | 11588/50000 [6:21:30<19:34:48,  1.84s/it]
Outer training loop:  23%|##3       | 11589/50000 [6:21:31<20:01:21,  1.88s/it]
Outer training loop:  23%|##3       | 11590/50000 [6:21:33<19:28:43,  1.83s/it]
Outer training loop:  23%|##3       | 11591/50000 [6:21:35<19:12:41,  1.80s/it]
Outer training loop:  23%|##3       | 11592/50000 [6:21:37<19:34:12,  1.83s/it]
Outer training loop:  23%|##3       | 11593/50000 [6:21:39<19:17:38,  1.81s/it]
Outer training loop:  23%|##3       | 11594/50000 [6:21:40<19:06:54,  1.79s/it]
Outer training loop:  23%|##3       | 11595/50000 [6:21:42<19:51:43,  1.86s/it]
Outer training loop:  23%|##3       | 11596/50000 [6:21:44<19:44:14,  1.85s/it]
Outer training loop:  23%|##3       | 11597/50000 [6:21:46<19:27:12,  1.82s/it]
Outer training loop:  23%|##3       | 11598/50000 [6:21:48<19:44:30,  1.85s/it]
Outer training loop:  23%|##3       | 11599/50000 [6:21:50<19:35:32,  1.84s/it]
Outer training loop:  23%|##3       | 11600/50000 [6:21:51<19:14:32,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.86s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  23%|##3       | 11601/50000 [6:21:58<35:32:24,  3.33s/it]
Outer training loop:  23%|##3       | 11602/50000 [6:22:00<31:04:40,  2.91s/it]
Outer training loop:  23%|##3       | 11603/50000 [6:22:02<27:17:20,  2.56s/it]
Outer training loop:  23%|##3       | 11604/50000 [6:22:04<24:11:13,  2.27s/it]
Outer training loop:  23%|##3       | 11605/50000 [6:22:05<22:49:14,  2.14s/it]
Outer training loop:  23%|##3       | 11606/50000 [6:22:07<22:05:03,  2.07s/it]
Outer training loop:  23%|##3       | 11607/50000 [6:22:09<20:37:31,  1.93s/it]
Outer training loop:  23%|##3       | 11608/50000 [6:22:11<20:27:22,  1.92s/it]
Outer training loop:  23%|##3       | 11609/50000 [6:22:13<20:12:26,  1.89s/it]
Outer training loop:  23%|##3       | 11610/50000 [6:22:14<19:38:51,  1.84s/it]
Outer training loop:  23%|##3       | 11611/50000 [6:22:16<19:21:54,  1.82s/it]
Outer training loop:  23%|##3       | 11612/50000 [6:22:18<19:33:18,  1.83s/it]
Outer training loop:  23%|##3       | 11613/50000 [6:22:20<19:00:41,  1.78s/it]
Outer training loop:  23%|##3       | 11614/50000 [6:22:21<18:38:34,  1.75s/it]
Outer training loop:  23%|##3       | 11615/50000 [6:22:23<18:47:48,  1.76s/it]
Outer training loop:  23%|##3       | 11616/50000 [6:22:25<19:08:28,  1.80s/it]
Outer training loop:  23%|##3       | 11617/50000 [6:22:27<18:58:34,  1.78s/it]
Outer training loop:  23%|##3       | 11618/50000 [6:22:28<18:31:02,  1.74s/it]
Outer training loop:  23%|##3       | 11619/50000 [6:22:30<18:46:56,  1.76s/it]
Outer training loop:  23%|##3       | 11620/50000 [6:22:32<18:42:11,  1.75s/it]
Outer training loop:  23%|##3       | 11621/50000 [6:22:34<18:45:00,  1.76s/it]
Outer training loop:  23%|##3       | 11622/50000 [6:22:35<18:36:10,  1.75s/it]
Outer training loop:  23%|##3       | 11623/50000 [6:22:37<18:57:56,  1.78s/it]
Outer training loop:  23%|##3       | 11624/50000 [6:22:39<18:25:35,  1.73s/it]
Outer training loop:  23%|##3       | 11625/50000 [6:22:41<18:15:46,  1.71s/it]
Outer training loop:  23%|##3       | 11626/50000 [6:22:42<18:52:48,  1.77s/it]
Outer training loop:  23%|##3       | 11627/50000 [6:22:44<18:38:13,  1.75s/it]
Outer training loop:  23%|##3       | 11628/50000 [6:22:46<18:25:46,  1.73s/it]
Outer training loop:  23%|##3       | 11629/50000 [6:22:48<18:46:50,  1.76s/it]
Outer training loop:  23%|##3       | 11630/50000 [6:22:49<18:50:22,  1.77s/it]
Outer training loop:  23%|##3       | 11631/50000 [6:22:51<18:46:01,  1.76s/it]
Outer training loop:  23%|##3       | 11632/50000 [6:22:53<19:37:45,  1.84s/it]
Outer training loop:  23%|##3       | 11633/50000 [6:22:55<19:14:33,  1.81s/it]
Outer training loop:  23%|##3       | 11634/50000 [6:22:57<19:04:59,  1.79s/it]
Outer training loop:  23%|##3       | 11635/50000 [6:22:59<19:06:24,  1.79s/it]
Outer training loop:  23%|##3       | 11636/50000 [6:23:01<19:57:05,  1.87s/it]
Outer training loop:  23%|##3       | 11637/50000 [6:23:02<19:29:09,  1.83s/it]
Outer training loop:  23%|##3       | 11638/50000 [6:23:04<19:35:14,  1.84s/it]
Outer training loop:  23%|##3       | 11639/50000 [6:23:06<19:45:55,  1.85s/it]
Outer training loop:  23%|##3       | 11640/50000 [6:23:08<19:12:53,  1.80s/it]
Outer training loop:  23%|##3       | 11641/50000 [6:23:10<19:06:54,  1.79s/it]
Outer training loop:  23%|##3       | 11642/50000 [6:23:11<18:43:58,  1.76s/it]
Outer training loop:  23%|##3       | 11643/50000 [6:23:13<19:03:18,  1.79s/it]
Outer training loop:  23%|##3       | 11644/50000 [6:23:15<18:49:38,  1.77s/it]
Outer training loop:  23%|##3       | 11645/50000 [6:23:17<18:46:22,  1.76s/it]
Outer training loop:  23%|##3       | 11646/50000 [6:23:18<19:07:18,  1.79s/it]
Outer training loop:  23%|##3       | 11647/50000 [6:23:20<18:55:38,  1.78s/it]
Outer training loop:  23%|##3       | 11648/50000 [6:23:22<18:26:51,  1.73s/it]
Outer training loop:  23%|##3       | 11649/50000 [6:23:23<18:03:44,  1.70s/it]
Outer training loop:  23%|##3       | 11650/50000 [6:23:25<18:39:05,  1.75s/it]
Outer training loop:  23%|##3       | 11651/50000 [6:23:27<18:38:04,  1.75s/it]
Outer training loop:  23%|##3       | 11652/50000 [6:23:29<18:39:21,  1.75s/it]
Outer training loop:  23%|##3       | 11653/50000 [6:23:31<19:46:16,  1.86s/it]
Outer training loop:  23%|##3       | 11654/50000 [6:23:32<19:10:25,  1.80s/it]
Outer training loop:  23%|##3       | 11655/50000 [6:23:34<19:22:04,  1.82s/it]
Outer training loop:  23%|##3       | 11656/50000 [6:23:36<19:41:32,  1.85s/it]
Outer training loop:  23%|##3       | 11657/50000 [6:23:38<19:12:11,  1.80s/it]
Outer training loop:  23%|##3       | 11658/50000 [6:23:40<19:09:01,  1.80s/it]
Outer training loop:  23%|##3       | 11659/50000 [6:23:41<18:41:42,  1.76s/it]
Outer training loop:  23%|##3       | 11660/50000 [6:23:43<19:13:58,  1.81s/it]
Outer training loop:  23%|##3       | 11661/50000 [6:23:45<18:58:38,  1.78s/it]
Outer training loop:  23%|##3       | 11662/50000 [6:23:47<18:51:00,  1.77s/it]
Outer training loop:  23%|##3       | 11663/50000 [6:23:49<19:31:43,  1.83s/it]
Outer training loop:  23%|##3       | 11664/50000 [6:23:51<19:11:57,  1.80s/it]
Outer training loop:  23%|##3       | 11665/50000 [6:23:52<18:39:33,  1.75s/it]
Outer training loop:  23%|##3       | 11666/50000 [6:23:54<19:10:21,  1.80s/it]
Outer training loop:  23%|##3       | 11667/50000 [6:23:56<19:04:37,  1.79s/it]
Outer training loop:  23%|##3       | 11668/50000 [6:23:58<18:57:54,  1.78s/it]
Outer training loop:  23%|##3       | 11669/50000 [6:24:00<19:23:43,  1.82s/it]
Outer training loop:  23%|##3       | 11670/50000 [6:24:01<18:51:58,  1.77s/it]
Outer training loop:  23%|##3       | 11671/50000 [6:24:03<18:41:51,  1.76s/it]
Outer training loop:  23%|##3       | 11672/50000 [6:24:05<18:58:39,  1.78s/it]
Outer training loop:  23%|##3       | 11673/50000 [6:24:06<18:53:58,  1.78s/it]
Outer training loop:  23%|##3       | 11674/50000 [6:24:08<18:50:58,  1.77s/it]
Outer training loop:  23%|##3       | 11675/50000 [6:24:10<18:50:04,  1.77s/it]
Outer training loop:  23%|##3       | 11676/50000 [6:24:12<19:20:12,  1.82s/it]
Outer training loop:  23%|##3       | 11677/50000 [6:24:14<18:49:45,  1.77s/it]
Outer training loop:  23%|##3       | 11678/50000 [6:24:15<18:38:58,  1.75s/it]
Outer training loop:  23%|##3       | 11679/50000 [6:24:17<18:22:23,  1.73s/it]
Outer training loop:  23%|##3       | 11680/50000 [6:24:19<18:46:38,  1.76s/it]
Outer training loop:  23%|##3       | 11681/50000 [6:24:21<18:41:54,  1.76s/it]
Outer training loop:  23%|##3       | 11682/50000 [6:24:22<19:04:53,  1.79s/it]
Outer training loop:  23%|##3       | 11683/50000 [6:24:24<19:41:00,  1.85s/it]
Outer training loop:  23%|##3       | 11684/50000 [6:24:26<19:10:38,  1.80s/it]
Outer training loop:  23%|##3       | 11685/50000 [6:24:28<18:51:08,  1.77s/it]
Outer training loop:  23%|##3       | 11686/50000 [6:24:30<19:05:40,  1.79s/it]
Outer training loop:  23%|##3       | 11687/50000 [6:24:31<18:59:41,  1.78s/it]
Outer training loop:  23%|##3       | 11688/50000 [6:24:33<18:40:51,  1.76s/it]
Outer training loop:  23%|##3       | 11689/50000 [6:24:35<19:15:59,  1.81s/it]
Outer training loop:  23%|##3       | 11690/50000 [6:24:37<18:56:33,  1.78s/it]
Outer training loop:  23%|##3       | 11691/50000 [6:24:38<18:46:47,  1.76s/it]
Outer training loop:  23%|##3       | 11692/50000 [6:24:41<19:33:16,  1.84s/it]
Outer training loop:  23%|##3       | 11693/50000 [6:24:42<19:16:26,  1.81s/it]
Outer training loop:  23%|##3       | 11694/50000 [6:24:44<18:51:12,  1.77s/it]
Outer training loop:  23%|##3       | 11695/50000 [6:24:46<18:36:52,  1.75s/it]
Outer training loop:  23%|##3       | 11696/50000 [6:24:47<18:59:46,  1.79s/it]
Outer training loop:  23%|##3       | 11697/50000 [6:24:49<18:31:05,  1.74s/it]
Outer training loop:  23%|##3       | 11698/50000 [6:24:51<18:23:37,  1.73s/it]
Outer training loop:  23%|##3       | 11699/50000 [6:24:53<18:27:01,  1.73s/it]
Outer training loop:  23%|##3       | 11700/50000 [6:24:55<19:04:17,  1.79s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  23%|##3       | 11701/50000 [6:25:01<35:31:43,  3.34s/it]
Outer training loop:  23%|##3       | 11702/50000 [6:25:03<30:27:19,  2.86s/it]
Outer training loop:  23%|##3       | 11703/50000 [6:25:05<26:55:18,  2.53s/it]
Outer training loop:  23%|##3       | 11704/50000 [6:25:07<24:51:10,  2.34s/it]
Outer training loop:  23%|##3       | 11705/50000 [6:25:09<23:12:31,  2.18s/it]
Outer training loop:  23%|##3       | 11706/50000 [6:25:10<21:50:40,  2.05s/it]
Outer training loop:  23%|##3       | 11707/50000 [6:25:12<21:17:19,  2.00s/it]
Outer training loop:  23%|##3       | 11708/50000 [6:25:14<20:27:31,  1.92s/it]
Outer training loop:  23%|##3       | 11709/50000 [6:25:16<19:48:45,  1.86s/it]
Outer training loop:  23%|##3       | 11710/50000 [6:25:18<19:54:55,  1.87s/it]
Outer training loop:  23%|##3       | 11711/50000 [6:25:19<19:18:07,  1.81s/it]
Outer training loop:  23%|##3       | 11712/50000 [6:25:21<18:53:13,  1.78s/it]
Outer training loop:  23%|##3       | 11713/50000 [6:25:23<18:58:58,  1.78s/it]
Outer training loop:  23%|##3       | 11714/50000 [6:25:25<19:24:55,  1.83s/it]
Outer training loop:  23%|##3       | 11715/50000 [6:25:26<18:58:03,  1.78s/it]
Outer training loop:  23%|##3       | 11716/50000 [6:25:28<18:56:54,  1.78s/it]
Outer training loop:  23%|##3       | 11717/50000 [6:25:30<19:19:31,  1.82s/it]
Outer training loop:  23%|##3       | 11718/50000 [6:25:32<19:32:56,  1.84s/it]
Outer training loop:  23%|##3       | 11719/50000 [6:25:34<19:25:38,  1.83s/it]
Outer training loop:  23%|##3       | 11720/50000 [6:25:36<19:32:16,  1.84s/it]
Outer training loop:  23%|##3       | 11721/50000 [6:25:37<19:10:01,  1.80s/it]
Outer training loop:  23%|##3       | 11722/50000 [6:25:39<18:55:58,  1.78s/it]
Outer training loop:  23%|##3       | 11723/50000 [6:25:41<18:38:05,  1.75s/it]
Outer training loop:  23%|##3       | 11724/50000 [6:25:43<19:03:33,  1.79s/it]
Outer training loop:  23%|##3       | 11725/50000 [6:25:44<18:46:51,  1.77s/it]
Outer training loop:  23%|##3       | 11726/50000 [6:25:46<18:28:24,  1.74s/it]
Outer training loop:  23%|##3       | 11727/50000 [6:25:48<19:00:14,  1.79s/it]
Outer training loop:  23%|##3       | 11728/50000 [6:25:50<18:47:20,  1.77s/it]
Outer training loop:  23%|##3       | 11729/50000 [6:25:52<19:20:13,  1.82s/it]
Outer training loop:  23%|##3       | 11730/50000 [6:25:53<19:01:49,  1.79s/it]
Outer training loop:  23%|##3       | 11731/50000 [6:25:55<19:25:53,  1.83s/it]
Outer training loop:  23%|##3       | 11732/50000 [6:25:57<19:15:07,  1.81s/it]
Outer training loop:  23%|##3       | 11733/50000 [6:25:59<18:56:38,  1.78s/it]
Outer training loop:  23%|##3       | 11734/50000 [6:26:01<19:17:58,  1.82s/it]
Outer training loop:  23%|##3       | 11735/50000 [6:26:02<19:15:24,  1.81s/it]
Outer training loop:  23%|##3       | 11736/50000 [6:26:04<19:07:37,  1.80s/it]
Outer training loop:  23%|##3       | 11737/50000 [6:26:06<19:34:57,  1.84s/it]
Outer training loop:  23%|##3       | 11738/50000 [6:26:08<19:08:55,  1.80s/it]
Outer training loop:  23%|##3       | 11739/50000 [6:26:10<18:47:48,  1.77s/it]
Outer training loop:  23%|##3       | 11740/50000 [6:26:11<19:07:21,  1.80s/it]
Outer training loop:  23%|##3       | 11741/50000 [6:26:13<19:03:23,  1.79s/it]
Outer training loop:  23%|##3       | 11742/50000 [6:26:15<18:59:23,  1.79s/it]
Outer training loop:  23%|##3       | 11743/50000 [6:26:17<19:01:06,  1.79s/it]
Outer training loop:  23%|##3       | 11744/50000 [6:26:19<19:25:32,  1.83s/it]
Outer training loop:  23%|##3       | 11745/50000 [6:26:20<19:18:36,  1.82s/it]
Outer training loop:  23%|##3       | 11746/50000 [6:26:22<18:55:53,  1.78s/it]
Outer training loop:  23%|##3       | 11747/50000 [6:26:24<19:34:25,  1.84s/it]
Outer training loop:  23%|##3       | 11748/50000 [6:26:26<19:33:36,  1.84s/it]
Outer training loop:  23%|##3       | 11749/50000 [6:26:28<19:13:10,  1.81s/it]
Outer training loop:  24%|##3       | 11750/50000 [6:26:30<19:18:04,  1.82s/it]
Outer training loop:  24%|##3       | 11751/50000 [6:26:31<19:34:27,  1.84s/it]
Outer training loop:  24%|##3       | 11752/50000 [6:26:33<19:20:40,  1.82s/it]
Outer training loop:  24%|##3       | 11753/50000 [6:26:35<19:08:57,  1.80s/it]
Outer training loop:  24%|##3       | 11754/50000 [6:26:37<19:43:29,  1.86s/it]
Outer training loop:  24%|##3       | 11755/50000 [6:26:39<19:14:32,  1.81s/it]
Outer training loop:  24%|##3       | 11756/50000 [6:26:40<18:46:11,  1.77s/it]
Outer training loop:  24%|##3       | 11757/50000 [6:26:42<19:05:59,  1.80s/it]
Outer training loop:  24%|##3       | 11758/50000 [6:26:44<19:39:18,  1.85s/it]
Outer training loop:  24%|##3       | 11759/50000 [6:26:46<19:04:57,  1.80s/it]
Outer training loop:  24%|##3       | 11760/50000 [6:26:48<18:58:52,  1.79s/it]
Outer training loop:  24%|##3       | 11761/50000 [6:26:50<19:13:48,  1.81s/it]
Outer training loop:  24%|##3       | 11762/50000 [6:26:51<19:13:38,  1.81s/it]
Outer training loop:  24%|##3       | 11763/50000 [6:26:53<18:54:40,  1.78s/it]
Outer training loop:  24%|##3       | 11764/50000 [6:26:55<18:26:41,  1.74s/it]
Outer training loop:  24%|##3       | 11765/50000 [6:26:57<19:10:59,  1.81s/it]
Outer training loop:  24%|##3       | 11766/50000 [6:26:58<19:03:45,  1.79s/it]
Outer training loop:  24%|##3       | 11767/50000 [6:27:00<19:08:12,  1.80s/it]
Outer training loop:  24%|##3       | 11768/50000 [6:27:02<19:31:52,  1.84s/it]
Outer training loop:  24%|##3       | 11769/50000 [6:27:04<19:32:18,  1.84s/it]
Outer training loop:  24%|##3       | 11770/50000 [6:27:06<19:09:39,  1.80s/it]
Outer training loop:  24%|##3       | 11771/50000 [6:27:07<18:53:38,  1.78s/it]
Outer training loop:  24%|##3       | 11772/50000 [6:27:09<19:36:32,  1.85s/it]
Outer training loop:  24%|##3       | 11773/50000 [6:27:11<19:29:53,  1.84s/it]
Outer training loop:  24%|##3       | 11774/50000 [6:27:13<19:39:34,  1.85s/it]
Outer training loop:  24%|##3       | 11775/50000 [6:27:15<20:00:21,  1.88s/it]
Outer training loop:  24%|##3       | 11776/50000 [6:27:17<19:18:55,  1.82s/it]
Outer training loop:  24%|##3       | 11777/50000 [6:27:19<19:07:29,  1.80s/it]
Outer training loop:  24%|##3       | 11778/50000 [6:27:20<19:35:51,  1.85s/it]
Outer training loop:  24%|##3       | 11779/50000 [6:27:22<19:27:14,  1.83s/it]
Outer training loop:  24%|##3       | 11780/50000 [6:27:24<19:26:36,  1.83s/it]
Outer training loop:  24%|##3       | 11781/50000 [6:27:26<19:19:10,  1.82s/it]
Outer training loop:  24%|##3       | 11782/50000 [6:27:28<19:47:35,  1.86s/it]
Outer training loop:  24%|##3       | 11783/50000 [6:27:30<19:26:11,  1.83s/it]
Outer training loop:  24%|##3       | 11784/50000 [6:27:31<19:20:02,  1.82s/it]
Outer training loop:  24%|##3       | 11785/50000 [6:27:33<19:37:38,  1.85s/it]
Outer training loop:  24%|##3       | 11786/50000 [6:27:35<19:25:24,  1.83s/it]
Outer training loop:  24%|##3       | 11787/50000 [6:27:37<19:02:17,  1.79s/it]
Outer training loop:  24%|##3       | 11788/50000 [6:27:39<19:07:29,  1.80s/it]
Outer training loop:  24%|##3       | 11789/50000 [6:27:40<18:44:28,  1.77s/it]
Outer training loop:  24%|##3       | 11790/50000 [6:27:42<18:53:39,  1.78s/it]
Outer training loop:  24%|##3       | 11791/50000 [6:27:44<18:58:49,  1.79s/it]
Outer training loop:  24%|##3       | 11792/50000 [6:27:46<19:51:20,  1.87s/it]
Outer training loop:  24%|##3       | 11793/50000 [6:27:48<19:12:31,  1.81s/it]
Outer training loop:  24%|##3       | 11794/50000 [6:27:49<19:02:04,  1.79s/it]
Outer training loop:  24%|##3       | 11795/50000 [6:27:51<19:33:21,  1.84s/it]
Outer training loop:  24%|##3       | 11796/50000 [6:27:53<19:09:44,  1.81s/it]
Outer training loop:  24%|##3       | 11797/50000 [6:27:55<19:00:57,  1.79s/it]
Outer training loop:  24%|##3       | 11798/50000 [6:27:57<19:09:18,  1.81s/it]
Outer training loop:  24%|##3       | 11799/50000 [6:27:58<18:54:41,  1.78s/it]
Outer training loop:  24%|##3       | 11800/50000 [6:28:00<18:43:34,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.93s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  24%|##3       | 11801/50000 [6:28:07<35:57:33,  3.39s/it]
Outer training loop:  24%|##3       | 11802/50000 [6:28:09<30:54:01,  2.91s/it]
Outer training loop:  24%|##3       | 11803/50000 [6:28:11<27:38:07,  2.60s/it]
Outer training loop:  24%|##3       | 11804/50000 [6:28:13<25:02:24,  2.36s/it]
Outer training loop:  24%|##3       | 11805/50000 [6:28:15<23:26:30,  2.21s/it]
Outer training loop:  24%|##3       | 11806/50000 [6:28:17<22:35:01,  2.13s/it]
Outer training loop:  24%|##3       | 11807/50000 [6:28:18<21:19:51,  2.01s/it]
Outer training loop:  24%|##3       | 11808/50000 [6:28:20<20:26:13,  1.93s/it]
Outer training loop:  24%|##3       | 11809/50000 [6:28:22<20:34:20,  1.94s/it]
Outer training loop:  24%|##3       | 11810/50000 [6:28:24<19:51:15,  1.87s/it]
Outer training loop:  24%|##3       | 11811/50000 [6:28:26<19:45:33,  1.86s/it]
Outer training loop:  24%|##3       | 11812/50000 [6:28:28<20:01:36,  1.89s/it]
Outer training loop:  24%|##3       | 11813/50000 [6:28:29<19:43:20,  1.86s/it]
Outer training loop:  24%|##3       | 11814/50000 [6:28:31<19:34:34,  1.85s/it]
Outer training loop:  24%|##3       | 11815/50000 [6:28:33<19:41:18,  1.86s/it]
Outer training loop:  24%|##3       | 11816/50000 [6:28:35<19:34:59,  1.85s/it]
Outer training loop:  24%|##3       | 11817/50000 [6:28:37<19:10:14,  1.81s/it]
Outer training loop:  24%|##3       | 11818/50000 [6:28:38<18:45:30,  1.77s/it]
Outer training loop:  24%|##3       | 11819/50000 [6:28:40<19:39:59,  1.85s/it]
Outer training loop:  24%|##3       | 11820/50000 [6:28:42<19:46:12,  1.86s/it]
Outer training loop:  24%|##3       | 11821/50000 [6:28:44<19:35:03,  1.85s/it]
Outer training loop:  24%|##3       | 11822/50000 [6:28:46<20:05:59,  1.90s/it]
Outer training loop:  24%|##3       | 11823/50000 [6:28:48<19:27:11,  1.83s/it]
Outer training loop:  24%|##3       | 11824/50000 [6:28:49<19:09:51,  1.81s/it]
Outer training loop:  24%|##3       | 11825/50000 [6:28:51<19:41:10,  1.86s/it]
Outer training loop:  24%|##3       | 11826/50000 [6:28:53<19:50:12,  1.87s/it]
Outer training loop:  24%|##3       | 11827/50000 [6:28:55<19:18:02,  1.82s/it]
Outer training loop:  24%|##3       | 11828/50000 [6:28:57<19:14:51,  1.82s/it]
Outer training loop:  24%|##3       | 11829/50000 [6:28:59<19:23:47,  1.83s/it]
Outer training loop:  24%|##3       | 11830/50000 [6:29:01<19:26:10,  1.83s/it]
Outer training loop:  24%|##3       | 11831/50000 [6:29:02<19:12:38,  1.81s/it]
Outer training loop:  24%|##3       | 11832/50000 [6:29:04<19:36:55,  1.85s/it]
Outer training loop:  24%|##3       | 11833/50000 [6:29:06<19:39:27,  1.85s/it]
Outer training loop:  24%|##3       | 11834/50000 [6:29:08<19:30:59,  1.84s/it]
Outer training loop:  24%|##3       | 11835/50000 [6:29:10<20:03:59,  1.89s/it]
Outer training loop:  24%|##3       | 11836/50000 [6:29:12<19:55:41,  1.88s/it]
Outer training loop:  24%|##3       | 11837/50000 [6:29:14<20:11:26,  1.90s/it]
Outer training loop:  24%|##3       | 11838/50000 [6:29:15<19:33:18,  1.84s/it]
Outer training loop:  24%|##3       | 11839/50000 [6:29:17<19:54:56,  1.88s/it]
Outer training loop:  24%|##3       | 11840/50000 [6:29:19<19:19:01,  1.82s/it]
Outer training loop:  24%|##3       | 11841/50000 [6:29:21<19:13:27,  1.81s/it]
Outer training loop:  24%|##3       | 11842/50000 [6:29:23<19:55:47,  1.88s/it]
Outer training loop:  24%|##3       | 11843/50000 [6:29:25<19:28:23,  1.84s/it]
Outer training loop:  24%|##3       | 11844/50000 [6:29:27<19:37:59,  1.85s/it]
Outer training loop:  24%|##3       | 11845/50000 [6:29:28<19:55:26,  1.88s/it]
Outer training loop:  24%|##3       | 11846/50000 [6:29:30<19:40:21,  1.86s/it]
Outer training loop:  24%|##3       | 11847/50000 [6:29:32<19:32:42,  1.84s/it]
Outer training loop:  24%|##3       | 11848/50000 [6:29:34<19:18:56,  1.82s/it]
Outer training loop:  24%|##3       | 11849/50000 [6:29:36<19:44:55,  1.86s/it]
Outer training loop:  24%|##3       | 11850/50000 [6:29:38<19:15:41,  1.82s/it]
Outer training loop:  24%|##3       | 11851/50000 [6:29:39<19:22:15,  1.83s/it]
Outer training loop:  24%|##3       | 11852/50000 [6:29:41<19:51:17,  1.87s/it]
Outer training loop:  24%|##3       | 11853/50000 [6:29:43<20:04:56,  1.90s/it]
Outer training loop:  24%|##3       | 11854/50000 [6:29:45<20:33:03,  1.94s/it]
Outer training loop:  24%|##3       | 11855/50000 [6:29:47<20:16:30,  1.91s/it]
Outer training loop:  24%|##3       | 11856/50000 [6:29:49<19:43:15,  1.86s/it]
Outer training loop:  24%|##3       | 11857/50000 [6:29:51<19:53:26,  1.88s/it]
Outer training loop:  24%|##3       | 11858/50000 [6:29:53<19:42:59,  1.86s/it]
Outer training loop:  24%|##3       | 11859/50000 [6:29:54<19:23:54,  1.83s/it]
Outer training loop:  24%|##3       | 11860/50000 [6:29:56<19:02:40,  1.80s/it]
Outer training loop:  24%|##3       | 11861/50000 [6:29:58<19:36:03,  1.85s/it]
Outer training loop:  24%|##3       | 11862/50000 [6:30:00<19:03:58,  1.80s/it]
Outer training loop:  24%|##3       | 11863/50000 [6:30:02<18:53:30,  1.78s/it]
Outer training loop:  24%|##3       | 11864/50000 [6:30:04<19:29:46,  1.84s/it]
Outer training loop:  24%|##3       | 11865/50000 [6:30:05<19:15:02,  1.82s/it]
Outer training loop:  24%|##3       | 11866/50000 [6:30:07<19:14:24,  1.82s/it]
Outer training loop:  24%|##3       | 11867/50000 [6:30:09<19:29:09,  1.84s/it]
Outer training loop:  24%|##3       | 11868/50000 [6:30:11<19:16:13,  1.82s/it]
Outer training loop:  24%|##3       | 11869/50000 [6:30:13<19:07:13,  1.81s/it]
Outer training loop:  24%|##3       | 11870/50000 [6:30:14<19:14:17,  1.82s/it]
Outer training loop:  24%|##3       | 11871/50000 [6:30:16<19:06:57,  1.80s/it]
Outer training loop:  24%|##3       | 11872/50000 [6:30:18<19:00:41,  1.80s/it]
Outer training loop:  24%|##3       | 11873/50000 [6:30:20<18:51:57,  1.78s/it]
Outer training loop:  24%|##3       | 11874/50000 [6:30:22<19:32:37,  1.85s/it]
Outer training loop:  24%|##3       | 11875/50000 [6:30:23<19:05:14,  1.80s/it]
Outer training loop:  24%|##3       | 11876/50000 [6:30:25<19:32:45,  1.85s/it]
Outer training loop:  24%|##3       | 11877/50000 [6:30:27<20:13:46,  1.91s/it]
Outer training loop:  24%|##3       | 11878/50000 [6:30:29<19:45:01,  1.87s/it]
Outer training loop:  24%|##3       | 11879/50000 [6:30:31<19:32:55,  1.85s/it]
Outer training loop:  24%|##3       | 11880/50000 [6:30:33<19:39:19,  1.86s/it]
Outer training loop:  24%|##3       | 11881/50000 [6:30:35<19:27:02,  1.84s/it]
Outer training loop:  24%|##3       | 11882/50000 [6:30:36<19:16:18,  1.82s/it]
Outer training loop:  24%|##3       | 11883/50000 [6:30:38<19:08:24,  1.81s/it]
Outer training loop:  24%|##3       | 11884/50000 [6:30:40<19:22:37,  1.83s/it]
Outer training loop:  24%|##3       | 11885/50000 [6:30:42<19:39:47,  1.86s/it]
Outer training loop:  24%|##3       | 11886/50000 [6:30:44<19:00:56,  1.80s/it]
Outer training loop:  24%|##3       | 11887/50000 [6:30:46<19:27:16,  1.84s/it]
Outer training loop:  24%|##3       | 11888/50000 [6:30:47<19:10:35,  1.81s/it]
Outer training loop:  24%|##3       | 11889/50000 [6:30:49<19:17:30,  1.82s/it]
Outer training loop:  24%|##3       | 11890/50000 [6:30:51<20:04:56,  1.90s/it]
Outer training loop:  24%|##3       | 11891/50000 [6:30:53<19:31:58,  1.85s/it]
Outer training loop:  24%|##3       | 11892/50000 [6:30:55<19:13:00,  1.82s/it]
Outer training loop:  24%|##3       | 11893/50000 [6:30:57<19:51:49,  1.88s/it]
Outer training loop:  24%|##3       | 11894/50000 [6:30:59<19:53:02,  1.88s/it]
Outer training loop:  24%|##3       | 11895/50000 [6:31:01<19:54:26,  1.88s/it]
Outer training loop:  24%|##3       | 11896/50000 [6:31:02<20:10:59,  1.91s/it]
Outer training loop:  24%|##3       | 11897/50000 [6:31:04<19:47:57,  1.87s/it]
Outer training loop:  24%|##3       | 11898/50000 [6:31:06<19:32:28,  1.85s/it]
Outer training loop:  24%|##3       | 11899/50000 [6:31:08<19:25:05,  1.83s/it]
Outer training loop:  24%|##3       | 11900/50000 [6:31:10<19:40:10,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.66s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.68s/it]

Outer training loop:  24%|##3       | 11901/50000 [6:31:17<35:37:24,  3.37s/it]
Outer training loop:  24%|##3       | 11902/50000 [6:31:18<30:23:04,  2.87s/it]
Outer training loop:  24%|##3       | 11903/50000 [6:31:20<26:51:45,  2.54s/it]
Outer training loop:  24%|##3       | 11904/50000 [6:31:22<24:57:34,  2.36s/it]
Outer training loop:  24%|##3       | 11905/50000 [6:31:24<22:47:26,  2.15s/it]
Outer training loop:  24%|##3       | 11906/50000 [6:31:26<21:44:47,  2.06s/it]
Outer training loop:  24%|##3       | 11907/50000 [6:31:27<21:14:39,  2.01s/it]
Outer training loop:  24%|##3       | 11908/50000 [6:31:29<20:44:58,  1.96s/it]
Outer training loop:  24%|##3       | 11909/50000 [6:31:31<20:00:23,  1.89s/it]
Outer training loop:  24%|##3       | 11910/50000 [6:31:33<19:19:55,  1.83s/it]
Outer training loop:  24%|##3       | 11911/50000 [6:31:35<19:55:02,  1.88s/it]
Outer training loop:  24%|##3       | 11912/50000 [6:31:36<19:18:15,  1.82s/it]
Outer training loop:  24%|##3       | 11913/50000 [6:31:38<18:35:40,  1.76s/it]
Outer training loop:  24%|##3       | 11914/50000 [6:31:40<19:07:55,  1.81s/it]
Outer training loop:  24%|##3       | 11915/50000 [6:31:42<18:54:32,  1.79s/it]
Outer training loop:  24%|##3       | 11916/50000 [6:31:43<18:50:15,  1.78s/it]
Outer training loop:  24%|##3       | 11917/50000 [6:31:45<18:53:11,  1.79s/it]
Outer training loop:  24%|##3       | 11918/50000 [6:31:47<19:19:38,  1.83s/it]
Outer training loop:  24%|##3       | 11919/50000 [6:31:49<18:51:20,  1.78s/it]
Outer training loop:  24%|##3       | 11920/50000 [6:31:51<18:33:10,  1.75s/it]
Outer training loop:  24%|##3       | 11921/50000 [6:31:52<18:47:30,  1.78s/it]
Outer training loop:  24%|##3       | 11922/50000 [6:31:54<18:48:13,  1.78s/it]
Outer training loop:  24%|##3       | 11923/50000 [6:31:56<19:07:17,  1.81s/it]
Outer training loop:  24%|##3       | 11924/50000 [6:31:58<19:22:21,  1.83s/it]
Outer training loop:  24%|##3       | 11925/50000 [6:32:00<18:42:33,  1.77s/it]
Outer training loop:  24%|##3       | 11926/50000 [6:32:01<18:24:12,  1.74s/it]
Outer training loop:  24%|##3       | 11927/50000 [6:32:03<18:44:43,  1.77s/it]
Outer training loop:  24%|##3       | 11928/50000 [6:32:05<18:41:00,  1.77s/it]
Outer training loop:  24%|##3       | 11929/50000 [6:32:07<18:27:48,  1.75s/it]
Outer training loop:  24%|##3       | 11930/50000 [6:32:08<18:31:11,  1.75s/it]
Outer training loop:  24%|##3       | 11931/50000 [6:32:10<18:58:33,  1.79s/it]
Outer training loop:  24%|##3       | 11932/50000 [6:32:12<19:09:58,  1.81s/it]
Outer training loop:  24%|##3       | 11933/50000 [6:32:14<19:17:47,  1.82s/it]
Outer training loop:  24%|##3       | 11934/50000 [6:32:16<19:39:15,  1.86s/it]
Outer training loop:  24%|##3       | 11935/50000 [6:32:18<19:15:39,  1.82s/it]
Outer training loop:  24%|##3       | 11936/50000 [6:32:19<19:08:03,  1.81s/it]
Outer training loop:  24%|##3       | 11937/50000 [6:32:21<19:47:26,  1.87s/it]
Outer training loop:  24%|##3       | 11938/50000 [6:32:23<19:20:34,  1.83s/it]
Outer training loop:  24%|##3       | 11939/50000 [6:32:25<19:07:50,  1.81s/it]
Outer training loop:  24%|##3       | 11940/50000 [6:32:27<18:58:25,  1.79s/it]
Outer training loop:  24%|##3       | 11941/50000 [6:32:29<19:30:08,  1.84s/it]
Outer training loop:  24%|##3       | 11942/50000 [6:32:30<19:05:43,  1.81s/it]
Outer training loop:  24%|##3       | 11943/50000 [6:32:32<18:36:07,  1.76s/it]
Outer training loop:  24%|##3       | 11944/50000 [6:32:34<19:18:57,  1.83s/it]
Outer training loop:  24%|##3       | 11945/50000 [6:32:36<19:03:12,  1.80s/it]
Outer training loop:  24%|##3       | 11946/50000 [6:32:37<19:00:27,  1.80s/it]
Outer training loop:  24%|##3       | 11947/50000 [6:32:39<19:27:18,  1.84s/it]
Outer training loop:  24%|##3       | 11948/50000 [6:32:41<19:12:52,  1.82s/it]
Outer training loop:  24%|##3       | 11949/50000 [6:32:43<19:01:58,  1.80s/it]
Outer training loop:  24%|##3       | 11950/50000 [6:32:45<18:42:52,  1.77s/it]
Outer training loop:  24%|##3       | 11951/50000 [6:32:47<19:07:24,  1.81s/it]
Outer training loop:  24%|##3       | 11952/50000 [6:32:48<19:04:10,  1.80s/it]
Outer training loop:  24%|##3       | 11953/50000 [6:32:50<18:35:12,  1.76s/it]
Outer training loop:  24%|##3       | 11954/50000 [6:32:52<18:53:27,  1.79s/it]
Outer training loop:  24%|##3       | 11955/50000 [6:32:54<18:49:56,  1.78s/it]
Outer training loop:  24%|##3       | 11956/50000 [6:32:55<19:05:27,  1.81s/it]
Outer training loop:  24%|##3       | 11957/50000 [6:32:57<19:27:48,  1.84s/it]
Outer training loop:  24%|##3       | 11958/50000 [6:32:59<19:00:23,  1.80s/it]
Outer training loop:  24%|##3       | 11959/50000 [6:33:01<18:37:08,  1.76s/it]
Outer training loop:  24%|##3       | 11960/50000 [6:33:03<18:33:13,  1.76s/it]
Outer training loop:  24%|##3       | 11961/50000 [6:33:04<19:00:15,  1.80s/it]
Outer training loop:  24%|##3       | 11962/50000 [6:33:06<18:42:52,  1.77s/it]
Outer training loop:  24%|##3       | 11963/50000 [6:33:08<18:19:19,  1.73s/it]
Outer training loop:  24%|##3       | 11964/50000 [6:33:10<19:06:37,  1.81s/it]
Outer training loop:  24%|##3       | 11965/50000 [6:33:12<19:01:00,  1.80s/it]
Outer training loop:  24%|##3       | 11966/50000 [6:33:13<18:55:53,  1.79s/it]
Outer training loop:  24%|##3       | 11967/50000 [6:33:15<19:14:44,  1.82s/it]
Outer training loop:  24%|##3       | 11968/50000 [6:33:17<19:10:10,  1.81s/it]
Outer training loop:  24%|##3       | 11969/50000 [6:33:19<18:35:43,  1.76s/it]
Outer training loop:  24%|##3       | 11970/50000 [6:33:20<18:08:56,  1.72s/it]
Outer training loop:  24%|##3       | 11971/50000 [6:33:22<18:18:02,  1.73s/it]
Outer training loop:  24%|##3       | 11972/50000 [6:33:24<18:20:06,  1.74s/it]
Outer training loop:  24%|##3       | 11973/50000 [6:33:26<18:45:14,  1.78s/it]
Outer training loop:  24%|##3       | 11974/50000 [6:33:28<19:13:26,  1.82s/it]
Outer training loop:  24%|##3       | 11975/50000 [6:33:29<18:57:45,  1.80s/it]
Outer training loop:  24%|##3       | 11976/50000 [6:33:31<18:57:07,  1.79s/it]
Outer training loop:  24%|##3       | 11977/50000 [6:33:33<19:16:36,  1.83s/it]
Outer training loop:  24%|##3       | 11978/50000 [6:33:35<19:03:40,  1.80s/it]
Outer training loop:  24%|##3       | 11979/50000 [6:33:36<18:55:51,  1.79s/it]
Outer training loop:  24%|##3       | 11980/50000 [6:33:38<19:08:07,  1.81s/it]
Outer training loop:  24%|##3       | 11981/50000 [6:33:40<18:57:33,  1.80s/it]
Outer training loop:  24%|##3       | 11982/50000 [6:33:42<18:40:50,  1.77s/it]
Outer training loop:  24%|##3       | 11983/50000 [6:33:44<18:47:25,  1.78s/it]
Outer training loop:  24%|##3       | 11984/50000 [6:33:46<19:17:15,  1.83s/it]
Outer training loop:  24%|##3       | 11985/50000 [6:33:47<19:01:10,  1.80s/it]
Outer training loop:  24%|##3       | 11986/50000 [6:33:49<18:57:27,  1.80s/it]
Outer training loop:  24%|##3       | 11987/50000 [6:33:51<19:14:32,  1.82s/it]
Outer training loop:  24%|##3       | 11988/50000 [6:33:53<19:04:03,  1.81s/it]
Outer training loop:  24%|##3       | 11989/50000 [6:33:54<18:52:25,  1.79s/it]
Outer training loop:  24%|##3       | 11990/50000 [6:33:56<19:01:10,  1.80s/it]
Outer training loop:  24%|##3       | 11991/50000 [6:33:58<19:30:10,  1.85s/it]
Outer training loop:  24%|##3       | 11992/50000 [6:34:00<19:02:29,  1.80s/it]
Outer training loop:  24%|##3       | 11993/50000 [6:34:02<18:55:35,  1.79s/it]
Outer training loop:  24%|##3       | 11994/50000 [6:34:04<19:17:52,  1.83s/it]
Outer training loop:  24%|##3       | 11995/50000 [6:34:05<19:15:28,  1.82s/it]
Outer training loop:  24%|##3       | 11996/50000 [6:34:07<19:04:17,  1.81s/it]
Outer training loop:  24%|##3       | 11997/50000 [6:34:09<19:19:15,  1.83s/it]
Outer training loop:  24%|##3       | 11998/50000 [6:34:11<18:56:49,  1.79s/it]
Outer training loop:  24%|##3       | 11999/50000 [6:34:12<18:31:17,  1.75s/it]
Outer training loop:  24%|##4       | 12000/50000 [6:34:14<19:07:59,  1.81s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.72s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:19,  1.70s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:19,  1.72s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:19,  1.75s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:22,  1.80s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.79s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.77s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.84s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:15,  1.83s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:11,  1.80s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:14,  1.87s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:11,  1.85s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:06,  1.80s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:02,  1.78s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:04,  1.83s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:59,  1.78s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:55,  1.75s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:52,  1.73s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:53,  1.83s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:49,  1.80s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:48,  1.80s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:47,  1.85s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:43,  1.81s/it]

Evaluating policy:  33%|###2      | 27/83 [00:49<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:38,  1.80s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:56<01:35,  1.83s/it]

Evaluating policy:  39%|###8      | 32/83 [00:58<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:00<01:31,  1.83s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:27,  1.78s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:29,  1.86s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:26,  1.84s/it]

Evaluating policy:  45%|####4     | 37/83 [01:07<01:22,  1.80s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:19,  1.76s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:16,  1.75s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:13,  1.72s/it]

Evaluating policy:  49%|####9     | 41/83 [01:14<01:16,  1.81s/it]

Evaluating policy:  51%|#####     | 42/83 [01:16<01:15,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:18<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:09,  1.79s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:07,  1.77s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:23<01:04,  1.74s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:02,  1.75s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<01:01,  1.80s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:58,  1.77s/it]

Evaluating policy:  61%|######1   | 51/83 [01:32<00:56,  1.78s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:55,  1.79s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:53,  1.77s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:39<00:52,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:41<00:48,  1.81s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:46,  1.80s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:44,  1.80s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:42,  1.77s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:48<00:41,  1.79s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:38,  1.76s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:36,  1.75s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:53<00:36,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:55<00:34,  1.79s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:57<00:31,  1.75s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:29,  1.73s/it]

Evaluating policy:  81%|########  | 67/83 [02:00<00:27,  1.72s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:25,  1.73s/it]

Evaluating policy:  83%|########3 | 69/83 [02:04<00:24,  1.77s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:23,  1.78s/it]

Evaluating policy:  86%|########5 | 71/83 [02:08<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:09<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:11<00:18,  1.80s/it]

Evaluating policy:  89%|########9 | 74/83 [02:13<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:15<00:14,  1.84s/it]

Evaluating policy:  92%|#########1| 76/83 [02:17<00:12,  1.83s/it]

Evaluating policy:  93%|#########2| 77/83 [02:18<00:10,  1.82s/it]

Evaluating policy:  94%|#########3| 78/83 [02:20<00:09,  1.80s/it]

Evaluating policy:  95%|#########5| 79/83 [02:22<00:07,  1.84s/it]

Evaluating policy:  96%|#########6| 80/83 [02:24<00:05,  1.99s/it]

Evaluating policy:  98%|#########7| 81/83 [02:26<00:03,  1.92s/it]

Evaluating policy:  99%|#########8| 82/83 [02:28<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:30<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:16,  1.67s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:16,  1.68s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:16,  1.71s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:17,  1.74s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:14,  1.72s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:13,  1.74s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:28,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:17,  1.85s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:11,  1.80s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:14,  1.87s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:10,  1.83s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.79s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:02,  1.77s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:05,  1.84s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:02,  1.82s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.80s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:55,  1.78s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:51,  1.75s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:54,  1.81s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:49,  1.77s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:46,  1.74s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:44,  1.74s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:48,  1.83s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:45,  1.82s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.78s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:37,  1.74s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:36,  1.76s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:38,  1.82s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:35,  1.79s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.76s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:29,  1.75s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:27,  1.75s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:25,  1.75s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:31,  1.94s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:28,  1.91s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:19,  1.81s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:18,  1.82s/it]

Evaluating policy:  49%|####9     | 41/83 [01:14<01:21,  1.94s/it]

Evaluating policy:  51%|#####     | 42/83 [01:16<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:18<01:15,  1.88s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:13,  1.89s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:23<01:07,  1.83s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:25<01:08,  1.91s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:27<01:05,  1.86s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:29<01:03,  1.87s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:32<00:57,  1.80s/it]

Evaluating policy:  63%|######2   | 52/83 [01:34<00:55,  1.78s/it]

Evaluating policy:  64%|######3   | 53/83 [01:36<00:52,  1.76s/it]

Evaluating policy:  65%|######5   | 54/83 [01:38<00:53,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:40<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:41<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:43<00:47,  1.82s/it]

Evaluating policy:  70%|######9   | 58/83 [01:45<00:45,  1.82s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:47<00:43,  1.80s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:49<00:41,  1.79s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:50<00:38,  1.76s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:52<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:54<00:36,  1.82s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:56<00:33,  1.77s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:57<00:31,  1.75s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:59<00:29,  1.75s/it]

Evaluating policy:  81%|########  | 67/83 [02:01<00:27,  1.73s/it]

Evaluating policy:  82%|########1 | 68/83 [02:03<00:25,  1.73s/it]

Evaluating policy:  83%|########3 | 69/83 [02:04<00:24,  1.74s/it]

Evaluating policy:  84%|########4 | 70/83 [02:07<00:24,  1.91s/it]

Evaluating policy:  86%|########5 | 71/83 [02:08<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:10<00:19,  1.79s/it]

Evaluating policy:  88%|########7 | 73/83 [02:12<00:17,  1.79s/it]

Evaluating policy:  89%|########9 | 74/83 [02:14<00:16,  1.80s/it]

Evaluating policy:  90%|######### | 75/83 [02:15<00:14,  1.77s/it]

Evaluating policy:  92%|#########1| 76/83 [02:17<00:12,  1.76s/it]

Evaluating policy:  93%|#########2| 77/83 [02:19<00:10,  1.75s/it]

Evaluating policy:  94%|#########3| 78/83 [02:20<00:08,  1.73s/it]

Evaluating policy:  95%|#########5| 79/83 [02:22<00:07,  1.75s/it]

Evaluating policy:  96%|#########6| 80/83 [02:25<00:05,  1.91s/it]

Evaluating policy:  98%|#########7| 81/83 [02:26<00:03,  1.82s/it]

Evaluating policy:  99%|#########8| 82/83 [02:28<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 83/83 [02:30<00:00,  1.80s/it]

Outer training loop:  24%|##4       | 12001/50000 [6:39:22<987:13:32, 93.53s/it]
Outer training loop:  24%|##4       | 12002/50000 [6:39:24<696:46:20, 66.01s/it]
Outer training loop:  24%|##4       | 12003/50000 [6:39:26<493:47:05, 46.78s/it]
Outer training loop:  24%|##4       | 12004/50000 [6:39:28<352:07:12, 33.36s/it]
Outer training loop:  24%|##4       | 12005/50000 [6:39:29<251:38:15, 23.84s/it]
Outer training loop:  24%|##4       | 12006/50000 [6:39:31<181:40:10, 17.21s/it]
Outer training loop:  24%|##4       | 12007/50000 [6:39:33<132:49:36, 12.59s/it]
Outer training loop:  24%|##4       | 12008/50000 [6:39:35<100:36:30,  9.53s/it]
Outer training loop:  24%|##4       | 12009/50000 [6:39:37<75:48:23,  7.18s/it] 
Outer training loop:  24%|##4       | 12010/50000 [6:39:39<58:38:29,  5.56s/it]
Outer training loop:  24%|##4       | 12011/50000 [6:39:41<47:04:40,  4.46s/it]
Outer training loop:  24%|##4       | 12012/50000 [6:39:42<38:36:20,  3.66s/it]
Outer training loop:  24%|##4       | 12013/50000 [6:39:44<32:51:59,  3.11s/it]
Outer training loop:  24%|##4       | 12014/50000 [6:39:46<29:12:06,  2.77s/it]
Outer training loop:  24%|##4       | 12015/50000 [6:39:48<26:10:10,  2.48s/it]
Outer training loop:  24%|##4       | 12016/50000 [6:39:50<24:14:00,  2.30s/it]
Outer training loop:  24%|##4       | 12017/50000 [6:39:52<22:19:32,  2.12s/it]
Outer training loop:  24%|##4       | 12018/50000 [6:39:53<21:29:11,  2.04s/it]
Outer training loop:  24%|##4       | 12019/50000 [6:39:55<21:02:47,  1.99s/it]
Outer training loop:  24%|##4       | 12020/50000 [6:39:57<19:57:39,  1.89s/it]
Outer training loop:  24%|##4       | 12021/50000 [6:39:59<20:03:28,  1.90s/it]
Outer training loop:  24%|##4       | 12022/50000 [6:40:01<19:41:02,  1.87s/it]
Outer training loop:  24%|##4       | 12023/50000 [6:40:03<19:25:54,  1.84s/it]
Outer training loop:  24%|##4       | 12024/50000 [6:40:04<19:38:19,  1.86s/it]
Outer training loop:  24%|##4       | 12025/50000 [6:40:06<19:18:11,  1.83s/it]
Outer training loop:  24%|##4       | 12026/50000 [6:40:08<19:01:24,  1.80s/it]
Outer training loop:  24%|##4       | 12027/50000 [6:40:10<18:51:53,  1.79s/it]
Outer training loop:  24%|##4       | 12028/50000 [6:40:12<19:23:48,  1.84s/it]
Outer training loop:  24%|##4       | 12029/50000 [6:40:13<19:07:04,  1.81s/it]
Outer training loop:  24%|##4       | 12030/50000 [6:40:15<19:04:43,  1.81s/it]
Outer training loop:  24%|##4       | 12031/50000 [6:40:17<19:24:06,  1.84s/it]
Outer training loop:  24%|##4       | 12032/50000 [6:40:19<19:12:48,  1.82s/it]
Outer training loop:  24%|##4       | 12033/50000 [6:40:21<19:10:58,  1.82s/it]
Outer training loop:  24%|##4       | 12034/50000 [6:40:23<19:38:09,  1.86s/it]
Outer training loop:  24%|##4       | 12035/50000 [6:40:24<19:34:30,  1.86s/it]
Outer training loop:  24%|##4       | 12036/50000 [6:40:26<19:15:30,  1.83s/it]
Outer training loop:  24%|##4       | 12037/50000 [6:40:28<19:49:01,  1.88s/it]
Outer training loop:  24%|##4       | 12038/50000 [6:40:30<19:28:46,  1.85s/it]
Outer training loop:  24%|##4       | 12039/50000 [6:40:32<19:03:23,  1.81s/it]
Outer training loop:  24%|##4       | 12040/50000 [6:40:34<19:21:29,  1.84s/it]
Outer training loop:  24%|##4       | 12041/50000 [6:40:35<18:45:06,  1.78s/it]
Outer training loop:  24%|##4       | 12042/50000 [6:40:37<18:34:05,  1.76s/it]
Outer training loop:  24%|##4       | 12043/50000 [6:40:39<18:37:16,  1.77s/it]
Outer training loop:  24%|##4       | 12044/50000 [6:40:41<19:11:25,  1.82s/it]
Outer training loop:  24%|##4       | 12045/50000 [6:40:43<19:01:03,  1.80s/it]
Outer training loop:  24%|##4       | 12046/50000 [6:40:44<18:33:37,  1.76s/it]
Outer training loop:  24%|##4       | 12047/50000 [6:40:46<19:08:10,  1.82s/it]
Outer training loop:  24%|##4       | 12048/50000 [6:40:48<18:58:30,  1.80s/it]
Outer training loop:  24%|##4       | 12049/50000 [6:40:50<18:55:01,  1.79s/it]
Outer training loop:  24%|##4       | 12050/50000 [6:40:52<19:10:12,  1.82s/it]
Outer training loop:  24%|##4       | 12051/50000 [6:40:53<19:30:34,  1.85s/it]
Outer training loop:  24%|##4       | 12052/50000 [6:40:55<19:17:44,  1.83s/it]
Outer training loop:  24%|##4       | 12053/50000 [6:40:57<19:35:31,  1.86s/it]
Outer training loop:  24%|##4       | 12054/50000 [6:40:59<19:51:00,  1.88s/it]
Outer training loop:  24%|##4       | 12055/50000 [6:41:01<19:12:56,  1.82s/it]
Outer training loop:  24%|##4       | 12056/50000 [6:41:03<19:36:50,  1.86s/it]
Outer training loop:  24%|##4       | 12057/50000 [6:41:04<19:13:12,  1.82s/it]
Outer training loop:  24%|##4       | 12058/50000 [6:41:06<19:07:32,  1.81s/it]
Outer training loop:  24%|##4       | 12059/50000 [6:41:08<19:37:07,  1.86s/it]
Outer training loop:  24%|##4       | 12060/50000 [6:41:10<19:18:22,  1.83s/it]
Outer training loop:  24%|##4       | 12061/50000 [6:41:12<19:10:04,  1.82s/it]
Outer training loop:  24%|##4       | 12062/50000 [6:41:13<18:39:18,  1.77s/it]
Outer training loop:  24%|##4       | 12063/50000 [6:41:15<19:01:45,  1.81s/it]
Outer training loop:  24%|##4       | 12064/50000 [6:41:17<18:54:04,  1.79s/it]
Outer training loop:  24%|##4       | 12065/50000 [6:41:19<19:21:34,  1.84s/it]
Outer training loop:  24%|##4       | 12066/50000 [6:41:21<19:09:01,  1.82s/it]
Outer training loop:  24%|##4       | 12067/50000 [6:41:23<18:53:02,  1.79s/it]
Outer training loop:  24%|##4       | 12068/50000 [6:41:24<18:40:43,  1.77s/it]
Outer training loop:  24%|##4       | 12069/50000 [6:41:26<19:00:34,  1.80s/it]
Outer training loop:  24%|##4       | 12070/50000 [6:41:28<18:59:04,  1.80s/it]
Outer training loop:  24%|##4       | 12071/50000 [6:41:30<18:33:50,  1.76s/it]
Outer training loop:  24%|##4       | 12072/50000 [6:41:32<19:00:15,  1.80s/it]
Outer training loop:  24%|##4       | 12073/50000 [6:41:33<18:58:10,  1.80s/it]
Outer training loop:  24%|##4       | 12074/50000 [6:41:35<18:56:15,  1.80s/it]
Outer training loop:  24%|##4       | 12075/50000 [6:41:37<18:49:00,  1.79s/it]
Outer training loop:  24%|##4       | 12076/50000 [6:41:39<19:17:20,  1.83s/it]
Outer training loop:  24%|##4       | 12077/50000 [6:41:41<19:09:12,  1.82s/it]
Outer training loop:  24%|##4       | 12078/50000 [6:41:42<18:41:53,  1.78s/it]
Outer training loop:  24%|##4       | 12079/50000 [6:41:44<19:12:54,  1.82s/it]
Outer training loop:  24%|##4       | 12080/50000 [6:41:46<18:46:28,  1.78s/it]
Outer training loop:  24%|##4       | 12081/50000 [6:41:48<19:03:26,  1.81s/it]
Outer training loop:  24%|##4       | 12082/50000 [6:41:50<18:57:20,  1.80s/it]
Outer training loop:  24%|##4       | 12083/50000 [6:41:52<19:33:48,  1.86s/it]
Outer training loop:  24%|##4       | 12084/50000 [6:41:53<19:19:19,  1.83s/it]
Outer training loop:  24%|##4       | 12085/50000 [6:41:55<19:02:27,  1.81s/it]
Outer training loop:  24%|##4       | 12086/50000 [6:41:57<19:30:00,  1.85s/it]
Outer training loop:  24%|##4       | 12087/50000 [6:41:59<18:54:30,  1.80s/it]
Outer training loop:  24%|##4       | 12088/50000 [6:42:00<19:02:23,  1.81s/it]
Outer training loop:  24%|##4       | 12089/50000 [6:42:02<19:26:45,  1.85s/it]
Outer training loop:  24%|##4       | 12090/50000 [6:42:04<18:57:17,  1.80s/it]
Outer training loop:  24%|##4       | 12091/50000 [6:42:06<18:50:18,  1.79s/it]
Outer training loop:  24%|##4       | 12092/50000 [6:42:08<19:05:50,  1.81s/it]
Outer training loop:  24%|##4       | 12093/50000 [6:42:10<19:10:48,  1.82s/it]
Outer training loop:  24%|##4       | 12094/50000 [6:42:11<19:06:51,  1.82s/it]
Outer training loop:  24%|##4       | 12095/50000 [6:42:13<19:29:53,  1.85s/it]
Outer training loop:  24%|##4       | 12096/50000 [6:42:15<19:23:51,  1.84s/it]
Outer training loop:  24%|##4       | 12097/50000 [6:42:17<19:17:58,  1.83s/it]
Outer training loop:  24%|##4       | 12098/50000 [6:42:19<18:37:44,  1.77s/it]
Outer training loop:  24%|##4       | 12099/50000 [6:42:20<19:02:53,  1.81s/it]
Outer training loop:  24%|##4       | 12100/50000 [6:42:22<19:12:21,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  24%|##4       | 12101/50000 [6:42:29<35:43:45,  3.39s/it]
Outer training loop:  24%|##4       | 12102/50000 [6:42:31<30:36:25,  2.91s/it]
Outer training loop:  24%|##4       | 12103/50000 [6:42:33<27:23:02,  2.60s/it]
Outer training loop:  24%|##4       | 12104/50000 [6:42:35<24:45:32,  2.35s/it]
Outer training loop:  24%|##4       | 12105/50000 [6:42:37<23:11:23,  2.20s/it]
Outer training loop:  24%|##4       | 12106/50000 [6:42:39<22:42:33,  2.16s/it]
Outer training loop:  24%|##4       | 12107/50000 [6:42:41<21:27:44,  2.04s/it]
Outer training loop:  24%|##4       | 12108/50000 [6:42:42<20:34:11,  1.95s/it]
Outer training loop:  24%|##4       | 12109/50000 [6:42:44<20:42:25,  1.97s/it]
Outer training loop:  24%|##4       | 12110/50000 [6:42:46<20:01:59,  1.90s/it]
Outer training loop:  24%|##4       | 12111/50000 [6:42:48<19:23:06,  1.84s/it]
Outer training loop:  24%|##4       | 12112/50000 [6:42:50<19:22:12,  1.84s/it]
Outer training loop:  24%|##4       | 12113/50000 [6:42:52<19:59:22,  1.90s/it]
Outer training loop:  24%|##4       | 12114/50000 [6:42:53<19:29:01,  1.85s/it]
Outer training loop:  24%|##4       | 12115/50000 [6:42:55<19:34:17,  1.86s/it]
Outer training loop:  24%|##4       | 12116/50000 [6:42:57<20:04:03,  1.91s/it]
Outer training loop:  24%|##4       | 12117/50000 [6:42:59<19:40:10,  1.87s/it]
Outer training loop:  24%|##4       | 12118/50000 [6:43:01<19:18:01,  1.83s/it]
Outer training loop:  24%|##4       | 12119/50000 [6:43:03<19:31:18,  1.86s/it]
Outer training loop:  24%|##4       | 12120/50000 [6:43:04<19:11:06,  1.82s/it]
Outer training loop:  24%|##4       | 12121/50000 [6:43:06<18:53:45,  1.80s/it]
Outer training loop:  24%|##4       | 12122/50000 [6:43:08<18:41:06,  1.78s/it]
Outer training loop:  24%|##4       | 12123/50000 [6:43:10<19:15:24,  1.83s/it]
Outer training loop:  24%|##4       | 12124/50000 [6:43:11<18:41:56,  1.78s/it]
Outer training loop:  24%|##4       | 12125/50000 [6:43:13<18:33:30,  1.76s/it]
Outer training loop:  24%|##4       | 12126/50000 [6:43:15<18:50:58,  1.79s/it]
Outer training loop:  24%|##4       | 12127/50000 [6:43:17<18:21:21,  1.74s/it]
Outer training loop:  24%|##4       | 12128/50000 [6:43:19<18:37:01,  1.77s/it]
Outer training loop:  24%|##4       | 12129/50000 [6:43:20<19:08:43,  1.82s/it]
Outer training loop:  24%|##4       | 12130/50000 [6:43:22<18:56:59,  1.80s/it]
Outer training loop:  24%|##4       | 12131/50000 [6:43:24<19:01:56,  1.81s/it]
Outer training loop:  24%|##4       | 12132/50000 [6:43:26<19:07:47,  1.82s/it]
Outer training loop:  24%|##4       | 12133/50000 [6:43:28<19:02:06,  1.81s/it]
Outer training loop:  24%|##4       | 12134/50000 [6:43:29<18:48:36,  1.79s/it]
Outer training loop:  24%|##4       | 12135/50000 [6:43:31<19:19:59,  1.84s/it]
Outer training loop:  24%|##4       | 12136/50000 [6:43:33<18:52:34,  1.79s/it]
Outer training loop:  24%|##4       | 12137/50000 [6:43:35<18:21:38,  1.75s/it]
Outer training loop:  24%|##4       | 12138/50000 [6:43:36<18:15:23,  1.74s/it]
Outer training loop:  24%|##4       | 12139/50000 [6:43:38<18:52:06,  1.79s/it]
Outer training loop:  24%|##4       | 12140/50000 [6:43:40<18:54:38,  1.80s/it]
Outer training loop:  24%|##4       | 12141/50000 [6:43:42<18:42:03,  1.78s/it]
Outer training loop:  24%|##4       | 12142/50000 [6:43:44<18:38:35,  1.77s/it]
Outer training loop:  24%|##4       | 12143/50000 [6:43:46<18:59:27,  1.81s/it]
Outer training loop:  24%|##4       | 12144/50000 [6:43:47<18:49:57,  1.79s/it]
Outer training loop:  24%|##4       | 12145/50000 [6:43:49<18:39:03,  1.77s/it]
Outer training loop:  24%|##4       | 12146/50000 [6:43:51<18:51:37,  1.79s/it]
Outer training loop:  24%|##4       | 12147/50000 [6:43:53<18:24:07,  1.75s/it]
Outer training loop:  24%|##4       | 12148/50000 [6:43:54<18:28:30,  1.76s/it]
Outer training loop:  24%|##4       | 12149/50000 [6:43:56<18:56:12,  1.80s/it]
Outer training loop:  24%|##4       | 12150/50000 [6:43:58<18:52:43,  1.80s/it]
Outer training loop:  24%|##4       | 12151/50000 [6:44:00<19:20:41,  1.84s/it]
Outer training loop:  24%|##4       | 12152/50000 [6:44:02<19:30:03,  1.85s/it]
Outer training loop:  24%|##4       | 12153/50000 [6:44:03<18:52:10,  1.79s/it]
Outer training loop:  24%|##4       | 12154/50000 [6:44:05<18:32:43,  1.76s/it]
Outer training loop:  24%|##4       | 12155/50000 [6:44:07<18:11:39,  1.73s/it]
Outer training loop:  24%|##4       | 12156/50000 [6:44:09<18:53:40,  1.80s/it]
Outer training loop:  24%|##4       | 12157/50000 [6:44:11<18:48:29,  1.79s/it]
Outer training loop:  24%|##4       | 12158/50000 [6:44:12<18:56:39,  1.80s/it]
Outer training loop:  24%|##4       | 12159/50000 [6:44:14<19:32:52,  1.86s/it]
Outer training loop:  24%|##4       | 12160/50000 [6:44:16<19:19:27,  1.84s/it]
Outer training loop:  24%|##4       | 12161/50000 [6:44:18<19:18:08,  1.84s/it]
Outer training loop:  24%|##4       | 12162/50000 [6:44:20<19:59:02,  1.90s/it]
Outer training loop:  24%|##4       | 12163/50000 [6:44:22<19:46:56,  1.88s/it]
Outer training loop:  24%|##4       | 12164/50000 [6:44:24<19:21:10,  1.84s/it]
Outer training loop:  24%|##4       | 12165/50000 [6:44:25<19:29:22,  1.85s/it]
Outer training loop:  24%|##4       | 12166/50000 [6:44:27<19:57:39,  1.90s/it]
Outer training loop:  24%|##4       | 12167/50000 [6:44:29<19:35:43,  1.86s/it]
Outer training loop:  24%|##4       | 12168/50000 [6:44:31<19:14:26,  1.83s/it]
Outer training loop:  24%|##4       | 12169/50000 [6:44:33<19:25:37,  1.85s/it]
Outer training loop:  24%|##4       | 12170/50000 [6:44:35<19:45:34,  1.88s/it]
Outer training loop:  24%|##4       | 12171/50000 [6:44:37<19:31:55,  1.86s/it]
Outer training loop:  24%|##4       | 12172/50000 [6:44:39<19:35:37,  1.86s/it]
Outer training loop:  24%|##4       | 12173/50000 [6:44:40<18:58:17,  1.81s/it]
Outer training loop:  24%|##4       | 12174/50000 [6:44:42<19:13:42,  1.83s/it]
Outer training loop:  24%|##4       | 12175/50000 [6:44:44<18:51:48,  1.80s/it]
Outer training loop:  24%|##4       | 12176/50000 [6:44:46<19:15:50,  1.83s/it]
Outer training loop:  24%|##4       | 12177/50000 [6:44:48<19:11:46,  1.83s/it]
Outer training loop:  24%|##4       | 12178/50000 [6:44:49<18:58:04,  1.81s/it]
Outer training loop:  24%|##4       | 12179/50000 [6:44:51<19:23:29,  1.85s/it]
Outer training loop:  24%|##4       | 12180/50000 [6:44:53<19:01:02,  1.81s/it]
Outer training loop:  24%|##4       | 12181/50000 [6:44:55<18:53:38,  1.80s/it]
Outer training loop:  24%|##4       | 12182/50000 [6:44:56<18:40:55,  1.78s/it]
Outer training loop:  24%|##4       | 12183/50000 [6:44:58<19:05:36,  1.82s/it]
Outer training loop:  24%|##4       | 12184/50000 [6:45:00<18:39:05,  1.78s/it]
Outer training loop:  24%|##4       | 12185/50000 [6:45:02<18:40:42,  1.78s/it]
Outer training loop:  24%|##4       | 12186/50000 [6:45:04<18:57:09,  1.80s/it]
Outer training loop:  24%|##4       | 12187/50000 [6:45:05<18:43:23,  1.78s/it]
Outer training loop:  24%|##4       | 12188/50000 [6:45:07<18:47:11,  1.79s/it]
Outer training loop:  24%|##4       | 12189/50000 [6:45:09<19:13:06,  1.83s/it]
Outer training loop:  24%|##4       | 12190/50000 [6:45:11<19:12:34,  1.83s/it]
Outer training loop:  24%|##4       | 12191/50000 [6:45:13<19:23:24,  1.85s/it]
Outer training loop:  24%|##4       | 12192/50000 [6:45:15<19:22:22,  1.84s/it]
Outer training loop:  24%|##4       | 12193/50000 [6:45:17<19:30:53,  1.86s/it]
Outer training loop:  24%|##4       | 12194/50000 [6:45:18<19:07:00,  1.82s/it]
Outer training loop:  24%|##4       | 12195/50000 [6:45:20<19:01:35,  1.81s/it]
Outer training loop:  24%|##4       | 12196/50000 [6:45:22<19:36:22,  1.87s/it]
Outer training loop:  24%|##4       | 12197/50000 [6:45:24<19:44:42,  1.88s/it]
Outer training loop:  24%|##4       | 12198/50000 [6:45:26<19:13:38,  1.83s/it]
Outer training loop:  24%|##4       | 12199/50000 [6:45:28<18:57:51,  1.81s/it]
Outer training loop:  24%|##4       | 12200/50000 [6:45:29<19:07:07,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  24%|##4       | 12201/50000 [6:45:37<36:30:14,  3.48s/it]
Outer training loop:  24%|##4       | 12202/50000 [6:45:39<31:19:35,  2.98s/it]
Outer training loop:  24%|##4       | 12203/50000 [6:45:40<27:31:48,  2.62s/it]
Outer training loop:  24%|##4       | 12204/50000 [6:45:42<25:39:25,  2.44s/it]
Outer training loop:  24%|##4       | 12205/50000 [6:45:44<23:12:36,  2.21s/it]
Outer training loop:  24%|##4       | 12206/50000 [6:45:46<21:57:56,  2.09s/it]
Outer training loop:  24%|##4       | 12207/50000 [6:45:48<21:40:50,  2.07s/it]
Outer training loop:  24%|##4       | 12208/50000 [6:45:50<21:03:40,  2.01s/it]
Outer training loop:  24%|##4       | 12209/50000 [6:45:51<20:12:09,  1.92s/it]
Outer training loop:  24%|##4       | 12210/50000 [6:45:53<19:58:47,  1.90s/it]
Outer training loop:  24%|##4       | 12211/50000 [6:45:55<19:32:19,  1.86s/it]
Outer training loop:  24%|##4       | 12212/50000 [6:45:57<19:12:12,  1.83s/it]
Outer training loop:  24%|##4       | 12213/50000 [6:45:59<18:59:55,  1.81s/it]
Outer training loop:  24%|##4       | 12214/50000 [6:46:01<19:20:41,  1.84s/it]
Outer training loop:  24%|##4       | 12215/50000 [6:46:02<18:58:18,  1.81s/it]
Outer training loop:  24%|##4       | 12216/50000 [6:46:04<18:43:36,  1.78s/it]
Outer training loop:  24%|##4       | 12217/50000 [6:46:06<19:11:34,  1.83s/it]
Outer training loop:  24%|##4       | 12218/50000 [6:46:08<18:54:18,  1.80s/it]
Outer training loop:  24%|##4       | 12219/50000 [6:46:09<18:47:24,  1.79s/it]
Outer training loop:  24%|##4       | 12220/50000 [6:46:11<19:19:05,  1.84s/it]
Outer training loop:  24%|##4       | 12221/50000 [6:46:13<19:10:50,  1.83s/it]
Outer training loop:  24%|##4       | 12222/50000 [6:46:15<19:12:56,  1.83s/it]
Outer training loop:  24%|##4       | 12223/50000 [6:46:17<19:35:58,  1.87s/it]
Outer training loop:  24%|##4       | 12224/50000 [6:46:19<19:28:02,  1.86s/it]
Outer training loop:  24%|##4       | 12225/50000 [6:46:21<19:21:12,  1.84s/it]
Outer training loop:  24%|##4       | 12226/50000 [6:46:23<19:51:03,  1.89s/it]
Outer training loop:  24%|##4       | 12227/50000 [6:46:24<19:38:19,  1.87s/it]
Outer training loop:  24%|##4       | 12228/50000 [6:46:26<19:11:08,  1.83s/it]
Outer training loop:  24%|##4       | 12229/50000 [6:46:28<19:18:02,  1.84s/it]
Outer training loop:  24%|##4       | 12230/50000 [6:46:30<19:43:53,  1.88s/it]
Outer training loop:  24%|##4       | 12231/50000 [6:46:32<19:31:42,  1.86s/it]
Outer training loop:  24%|##4       | 12232/50000 [6:46:33<18:59:07,  1.81s/it]
Outer training loop:  24%|##4       | 12233/50000 [6:46:35<18:47:31,  1.79s/it]
Outer training loop:  24%|##4       | 12234/50000 [6:46:37<19:09:52,  1.83s/it]
Outer training loop:  24%|##4       | 12235/50000 [6:46:39<18:36:33,  1.77s/it]
Outer training loop:  24%|##4       | 12236/50000 [6:46:41<18:30:36,  1.76s/it]
Outer training loop:  24%|##4       | 12237/50000 [6:46:42<18:52:20,  1.80s/it]
Outer training loop:  24%|##4       | 12238/50000 [6:46:44<18:39:44,  1.78s/it]
Outer training loop:  24%|##4       | 12239/50000 [6:46:46<18:58:57,  1.81s/it]
Outer training loop:  24%|##4       | 12240/50000 [6:46:48<19:07:26,  1.82s/it]
Outer training loop:  24%|##4       | 12241/50000 [6:46:50<19:58:51,  1.91s/it]
Outer training loop:  24%|##4       | 12242/50000 [6:46:52<19:18:09,  1.84s/it]
Outer training loop:  24%|##4       | 12243/50000 [6:46:54<19:14:36,  1.83s/it]
Outer training loop:  24%|##4       | 12244/50000 [6:46:56<19:48:06,  1.89s/it]
Outer training loop:  24%|##4       | 12245/50000 [6:46:57<19:40:29,  1.88s/it]
Outer training loop:  24%|##4       | 12246/50000 [6:46:59<19:19:00,  1.84s/it]
Outer training loop:  24%|##4       | 12247/50000 [6:47:01<19:27:44,  1.86s/it]
Outer training loop:  24%|##4       | 12248/50000 [6:47:03<19:42:10,  1.88s/it]
Outer training loop:  24%|##4       | 12249/50000 [6:47:05<19:31:39,  1.86s/it]
Outer training loop:  24%|##4       | 12250/50000 [6:47:07<19:09:15,  1.83s/it]
Outer training loop:  25%|##4       | 12251/50000 [6:47:08<19:14:04,  1.83s/it]
Outer training loop:  25%|##4       | 12252/50000 [6:47:10<18:53:18,  1.80s/it]
Outer training loop:  25%|##4       | 12253/50000 [6:47:12<19:25:44,  1.85s/it]
Outer training loop:  25%|##4       | 12254/50000 [6:47:14<19:43:25,  1.88s/it]
Outer training loop:  25%|##4       | 12255/50000 [6:47:16<19:20:23,  1.84s/it]
Outer training loop:  25%|##4       | 12256/50000 [6:47:17<18:53:34,  1.80s/it]
Outer training loop:  25%|##4       | 12257/50000 [6:47:19<19:11:41,  1.83s/it]
Outer training loop:  25%|##4       | 12258/50000 [6:47:21<19:54:51,  1.90s/it]
Outer training loop:  25%|##4       | 12259/50000 [6:47:23<19:45:52,  1.89s/it]
Outer training loop:  25%|##4       | 12260/50000 [6:47:25<19:29:21,  1.86s/it]
Outer training loop:  25%|##4       | 12261/50000 [6:47:27<19:27:32,  1.86s/it]
Outer training loop:  25%|##4       | 12262/50000 [6:47:29<18:56:18,  1.81s/it]
Outer training loop:  25%|##4       | 12263/50000 [6:47:30<19:01:47,  1.82s/it]
Outer training loop:  25%|##4       | 12264/50000 [6:47:32<19:01:10,  1.81s/it]
Outer training loop:  25%|##4       | 12265/50000 [6:47:34<19:26:07,  1.85s/it]
Outer training loop:  25%|##4       | 12266/50000 [6:47:36<19:37:45,  1.87s/it]
Outer training loop:  25%|##4       | 12267/50000 [6:47:38<19:06:04,  1.82s/it]
Outer training loop:  25%|##4       | 12268/50000 [6:47:40<19:36:59,  1.87s/it]
Outer training loop:  25%|##4       | 12269/50000 [6:47:42<19:31:50,  1.86s/it]
Outer training loop:  25%|##4       | 12270/50000 [6:47:44<19:32:40,  1.86s/it]
Outer training loop:  25%|##4       | 12271/50000 [6:47:45<19:20:09,  1.84s/it]
Outer training loop:  25%|##4       | 12272/50000 [6:47:47<20:02:50,  1.91s/it]
Outer training loop:  25%|##4       | 12273/50000 [6:47:49<19:56:33,  1.90s/it]
Outer training loop:  25%|##4       | 12274/50000 [6:47:51<20:42:10,  1.98s/it]
Outer training loop:  25%|##4       | 12275/50000 [6:47:53<20:10:08,  1.92s/it]
Outer training loop:  25%|##4       | 12276/50000 [6:47:55<20:02:08,  1.91s/it]
Outer training loop:  25%|##4       | 12277/50000 [6:47:57<19:40:21,  1.88s/it]
Outer training loop:  25%|##4       | 12278/50000 [6:47:59<20:08:42,  1.92s/it]
Outer training loop:  25%|##4       | 12279/50000 [6:48:01<19:40:03,  1.88s/it]
Outer training loop:  25%|##4       | 12280/50000 [6:48:03<19:23:55,  1.85s/it]
Outer training loop:  25%|##4       | 12281/50000 [6:48:04<19:27:43,  1.86s/it]
Outer training loop:  25%|##4       | 12282/50000 [6:48:06<19:20:28,  1.85s/it]
Outer training loop:  25%|##4       | 12283/50000 [6:48:08<18:57:39,  1.81s/it]
Outer training loop:  25%|##4       | 12284/50000 [6:48:10<19:27:41,  1.86s/it]
Outer training loop:  25%|##4       | 12285/50000 [6:48:12<19:04:42,  1.82s/it]
Outer training loop:  25%|##4       | 12286/50000 [6:48:13<18:40:58,  1.78s/it]
Outer training loop:  25%|##4       | 12287/50000 [6:48:15<18:19:48,  1.75s/it]
Outer training loop:  25%|##4       | 12288/50000 [6:48:17<18:37:25,  1.78s/it]
Outer training loop:  25%|##4       | 12289/50000 [6:48:19<18:50:53,  1.80s/it]
Outer training loop:  25%|##4       | 12290/50000 [6:48:21<19:03:04,  1.82s/it]
Outer training loop:  25%|##4       | 12291/50000 [6:48:23<19:30:06,  1.86s/it]
Outer training loop:  25%|##4       | 12292/50000 [6:48:24<18:49:52,  1.80s/it]
Outer training loop:  25%|##4       | 12293/50000 [6:48:26<19:12:41,  1.83s/it]
Outer training loop:  25%|##4       | 12294/50000 [6:48:28<18:52:40,  1.80s/it]
Outer training loop:  25%|##4       | 12295/50000 [6:48:30<19:18:17,  1.84s/it]
Outer training loop:  25%|##4       | 12296/50000 [6:48:31<18:54:17,  1.81s/it]
Outer training loop:  25%|##4       | 12297/50000 [6:48:33<18:45:25,  1.79s/it]
Outer training loop:  25%|##4       | 12298/50000 [6:48:35<19:45:39,  1.89s/it]
Outer training loop:  25%|##4       | 12299/50000 [6:48:37<19:19:30,  1.85s/it]
Outer training loop:  25%|##4       | 12300/50000 [6:48:39<19:02:06,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.79s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  25%|##4       | 12301/50000 [6:48:46<36:24:03,  3.48s/it]
Outer training loop:  25%|##4       | 12302/50000 [6:48:48<31:07:14,  2.97s/it]
Outer training loop:  25%|##4       | 12303/50000 [6:48:50<27:18:30,  2.61s/it]
Outer training loop:  25%|##4       | 12304/50000 [6:48:51<24:24:26,  2.33s/it]
Outer training loop:  25%|##4       | 12305/50000 [6:48:53<23:26:23,  2.24s/it]
Outer training loop:  25%|##4       | 12306/50000 [6:48:55<21:57:12,  2.10s/it]
Outer training loop:  25%|##4       | 12307/50000 [6:48:57<20:50:53,  1.99s/it]
Outer training loop:  25%|##4       | 12308/50000 [6:48:59<20:42:50,  1.98s/it]
Outer training loop:  25%|##4       | 12309/50000 [6:49:01<19:42:38,  1.88s/it]
Outer training loop:  25%|##4       | 12310/50000 [6:49:02<19:23:14,  1.85s/it]
Outer training loop:  25%|##4       | 12311/50000 [6:49:04<19:29:10,  1.86s/it]
Outer training loop:  25%|##4       | 12312/50000 [6:49:06<19:06:49,  1.83s/it]
Outer training loop:  25%|##4       | 12313/50000 [6:49:08<18:45:33,  1.79s/it]
Outer training loop:  25%|##4       | 12314/50000 [6:49:10<19:19:29,  1.85s/it]
Outer training loop:  25%|##4       | 12315/50000 [6:49:11<18:51:47,  1.80s/it]
Outer training loop:  25%|##4       | 12316/50000 [6:49:13<18:52:55,  1.80s/it]
Outer training loop:  25%|##4       | 12317/50000 [6:49:15<18:34:46,  1.77s/it]
Outer training loop:  25%|##4       | 12318/50000 [6:49:17<19:08:09,  1.83s/it]
Outer training loop:  25%|##4       | 12319/50000 [6:49:19<19:01:04,  1.82s/it]
Outer training loop:  25%|##4       | 12320/50000 [6:49:20<18:56:36,  1.81s/it]
Outer training loop:  25%|##4       | 12321/50000 [6:49:22<19:16:09,  1.84s/it]
Outer training loop:  25%|##4       | 12322/50000 [6:49:24<18:49:56,  1.80s/it]
Outer training loop:  25%|##4       | 12323/50000 [6:49:26<18:32:49,  1.77s/it]
Outer training loop:  25%|##4       | 12324/50000 [6:49:28<19:07:08,  1.83s/it]
Outer training loop:  25%|##4       | 12325/50000 [6:49:29<18:29:47,  1.77s/it]
Outer training loop:  25%|##4       | 12326/50000 [6:49:31<18:15:19,  1.74s/it]
Outer training loop:  25%|##4       | 12327/50000 [6:49:33<18:01:30,  1.72s/it]
Outer training loop:  25%|##4       | 12328/50000 [6:49:35<18:42:25,  1.79s/it]
Outer training loop:  25%|##4       | 12329/50000 [6:49:36<18:36:53,  1.78s/it]
Outer training loop:  25%|##4       | 12330/50000 [6:49:38<18:38:20,  1.78s/it]
Outer training loop:  25%|##4       | 12331/50000 [6:49:40<18:55:50,  1.81s/it]
Outer training loop:  25%|##4       | 12332/50000 [6:49:42<18:42:34,  1.79s/it]
Outer training loop:  25%|##4       | 12333/50000 [6:49:43<18:26:14,  1.76s/it]
Outer training loop:  25%|##4       | 12334/50000 [6:49:45<18:42:58,  1.79s/it]
Outer training loop:  25%|##4       | 12335/50000 [6:49:47<18:13:04,  1.74s/it]
Outer training loop:  25%|##4       | 12336/50000 [6:49:49<18:03:25,  1.73s/it]
Outer training loop:  25%|##4       | 12337/50000 [6:49:50<18:10:32,  1.74s/it]
Outer training loop:  25%|##4       | 12338/50000 [6:49:52<18:45:34,  1.79s/it]
Outer training loop:  25%|##4       | 12339/50000 [6:49:54<18:46:53,  1.80s/it]
Outer training loop:  25%|##4       | 12340/50000 [6:49:56<18:29:03,  1.77s/it]
Outer training loop:  25%|##4       | 12341/50000 [6:49:58<18:34:32,  1.78s/it]
Outer training loop:  25%|##4       | 12342/50000 [6:49:59<18:23:26,  1.76s/it]
Outer training loop:  25%|##4       | 12343/50000 [6:50:01<18:22:25,  1.76s/it]
Outer training loop:  25%|##4       | 12344/50000 [6:50:03<18:55:37,  1.81s/it]
Outer training loop:  25%|##4       | 12345/50000 [6:50:05<19:28:54,  1.86s/it]
Outer training loop:  25%|##4       | 12346/50000 [6:50:07<19:06:46,  1.83s/it]
Outer training loop:  25%|##4       | 12347/50000 [6:50:08<18:49:58,  1.80s/it]
Outer training loop:  25%|##4       | 12348/50000 [6:50:10<19:06:33,  1.83s/it]
Outer training loop:  25%|##4       | 12349/50000 [6:50:12<18:45:54,  1.79s/it]
Outer training loop:  25%|##4       | 12350/50000 [6:50:14<19:04:14,  1.82s/it]
Outer training loop:  25%|##4       | 12351/50000 [6:50:16<19:17:32,  1.84s/it]
Outer training loop:  25%|##4       | 12352/50000 [6:50:18<19:00:44,  1.82s/it]
Outer training loop:  25%|##4       | 12353/50000 [6:50:19<19:00:16,  1.82s/it]
Outer training loop:  25%|##4       | 12354/50000 [6:50:21<19:14:17,  1.84s/it]
Outer training loop:  25%|##4       | 12355/50000 [6:50:23<19:32:03,  1.87s/it]
Outer training loop:  25%|##4       | 12356/50000 [6:50:25<19:30:13,  1.87s/it]
Outer training loop:  25%|##4       | 12357/50000 [6:50:27<19:32:41,  1.87s/it]
Outer training loop:  25%|##4       | 12358/50000 [6:50:29<20:25:43,  1.95s/it]
Outer training loop:  25%|##4       | 12359/50000 [6:50:31<19:35:34,  1.87s/it]
Outer training loop:  25%|##4       | 12360/50000 [6:50:33<19:35:45,  1.87s/it]
Outer training loop:  25%|##4       | 12361/50000 [6:50:35<20:21:40,  1.95s/it]
Outer training loop:  25%|##4       | 12362/50000 [6:50:37<19:48:48,  1.90s/it]
Outer training loop:  25%|##4       | 12363/50000 [6:50:39<19:52:55,  1.90s/it]
Outer training loop:  25%|##4       | 12364/50000 [6:50:40<19:09:34,  1.83s/it]
Outer training loop:  25%|##4       | 12365/50000 [6:50:42<20:04:38,  1.92s/it]
Outer training loop:  25%|##4       | 12366/50000 [6:50:44<19:36:04,  1.88s/it]
Outer training loop:  25%|##4       | 12367/50000 [6:50:46<19:24:25,  1.86s/it]
Outer training loop:  25%|##4       | 12368/50000 [6:50:48<19:30:56,  1.87s/it]
Outer training loop:  25%|##4       | 12369/50000 [6:50:50<19:51:49,  1.90s/it]
Outer training loop:  25%|##4       | 12370/50000 [6:50:51<19:13:53,  1.84s/it]
Outer training loop:  25%|##4       | 12371/50000 [6:50:53<19:17:12,  1.85s/it]
Outer training loop:  25%|##4       | 12372/50000 [6:50:55<19:34:06,  1.87s/it]
Outer training loop:  25%|##4       | 12373/50000 [6:50:57<19:29:47,  1.87s/it]
Outer training loop:  25%|##4       | 12374/50000 [6:50:59<19:06:16,  1.83s/it]
Outer training loop:  25%|##4       | 12375/50000 [6:51:01<19:25:10,  1.86s/it]
Outer training loop:  25%|##4       | 12376/50000 [6:51:03<19:16:22,  1.84s/it]
Outer training loop:  25%|##4       | 12377/50000 [6:51:04<19:14:27,  1.84s/it]
Outer training loop:  25%|##4       | 12378/50000 [6:51:06<19:29:49,  1.87s/it]
Outer training loop:  25%|##4       | 12379/50000 [6:51:08<18:53:45,  1.81s/it]
Outer training loop:  25%|##4       | 12380/50000 [6:51:10<18:44:36,  1.79s/it]
Outer training loop:  25%|##4       | 12381/50000 [6:51:12<19:00:50,  1.82s/it]
Outer training loop:  25%|##4       | 12382/50000 [6:51:13<18:47:59,  1.80s/it]
Outer training loop:  25%|##4       | 12383/50000 [6:51:15<18:37:41,  1.78s/it]
Outer training loop:  25%|##4       | 12384/50000 [6:51:17<18:39:06,  1.79s/it]
Outer training loop:  25%|##4       | 12385/50000 [6:51:19<19:05:34,  1.83s/it]
Outer training loop:  25%|##4       | 12386/50000 [6:51:21<18:54:35,  1.81s/it]
Outer training loop:  25%|##4       | 12387/50000 [6:51:22<18:33:22,  1.78s/it]
Outer training loop:  25%|##4       | 12388/50000 [6:51:24<18:48:07,  1.80s/it]
Outer training loop:  25%|##4       | 12389/50000 [6:51:26<19:04:24,  1.83s/it]
Outer training loop:  25%|##4       | 12390/50000 [6:51:28<18:59:35,  1.82s/it]
Outer training loop:  25%|##4       | 12391/50000 [6:51:30<18:39:03,  1.79s/it]
Outer training loop:  25%|##4       | 12392/50000 [6:51:31<18:53:39,  1.81s/it]
Outer training loop:  25%|##4       | 12393/50000 [6:51:33<18:55:49,  1.81s/it]
Outer training loop:  25%|##4       | 12394/50000 [6:51:35<18:46:54,  1.80s/it]
Outer training loop:  25%|##4       | 12395/50000 [6:51:37<19:36:15,  1.88s/it]
Outer training loop:  25%|##4       | 12396/50000 [6:51:39<19:14:04,  1.84s/it]
Outer training loop:  25%|##4       | 12397/50000 [6:51:41<18:52:41,  1.81s/it]
Outer training loop:  25%|##4       | 12398/50000 [6:51:43<19:17:29,  1.85s/it]
Outer training loop:  25%|##4       | 12399/50000 [6:51:44<18:49:57,  1.80s/it]
Outer training loop:  25%|##4       | 12400/50000 [6:51:46<18:35:12,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.91s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  25%|##4       | 12401/50000 [6:51:53<34:52:11,  3.34s/it]
Outer training loop:  25%|##4       | 12402/50000 [6:51:55<30:23:11,  2.91s/it]
Outer training loop:  25%|##4       | 12403/50000 [6:51:57<26:52:34,  2.57s/it]
Outer training loop:  25%|##4       | 12404/50000 [6:51:59<24:37:23,  2.36s/it]
Outer training loop:  25%|##4       | 12405/50000 [6:52:00<23:16:37,  2.23s/it]
Outer training loop:  25%|##4       | 12406/50000 [6:52:02<22:01:55,  2.11s/it]
Outer training loop:  25%|##4       | 12407/50000 [6:52:04<20:58:03,  2.01s/it]
Outer training loop:  25%|##4       | 12408/50000 [6:52:06<20:44:32,  1.99s/it]
Outer training loop:  25%|##4       | 12409/50000 [6:52:08<20:01:32,  1.92s/it]
Outer training loop:  25%|##4       | 12410/50000 [6:52:10<19:46:38,  1.89s/it]
Outer training loop:  25%|##4       | 12411/50000 [6:52:11<19:10:08,  1.84s/it]
Outer training loop:  25%|##4       | 12412/50000 [6:52:13<19:20:46,  1.85s/it]
Outer training loop:  25%|##4       | 12413/50000 [6:52:15<18:55:30,  1.81s/it]
Outer training loop:  25%|##4       | 12414/50000 [6:52:17<18:28:12,  1.77s/it]
Outer training loop:  25%|##4       | 12415/50000 [6:52:19<19:06:38,  1.83s/it]
Outer training loop:  25%|##4       | 12416/50000 [6:52:20<19:12:56,  1.84s/it]
Outer training loop:  25%|##4       | 12417/50000 [6:52:22<19:26:22,  1.86s/it]
Outer training loop:  25%|##4       | 12418/50000 [6:52:24<19:45:13,  1.89s/it]
Outer training loop:  25%|##4       | 12419/50000 [6:52:26<19:15:22,  1.84s/it]
Outer training loop:  25%|##4       | 12420/50000 [6:52:28<19:14:30,  1.84s/it]
Outer training loop:  25%|##4       | 12421/50000 [6:52:30<19:41:12,  1.89s/it]
Outer training loop:  25%|##4       | 12422/50000 [6:52:31<18:59:44,  1.82s/it]
Outer training loop:  25%|##4       | 12423/50000 [6:52:33<18:34:39,  1.78s/it]
Outer training loop:  25%|##4       | 12424/50000 [6:52:35<19:29:15,  1.87s/it]
Outer training loop:  25%|##4       | 12425/50000 [6:52:37<18:55:04,  1.81s/it]
Outer training loop:  25%|##4       | 12426/50000 [6:52:39<18:43:00,  1.79s/it]
Outer training loop:  25%|##4       | 12427/50000 [6:52:40<18:27:34,  1.77s/it]
Outer training loop:  25%|##4       | 12428/50000 [6:52:42<18:56:26,  1.81s/it]
Outer training loop:  25%|##4       | 12429/50000 [6:52:44<18:48:51,  1.80s/it]
Outer training loop:  25%|##4       | 12430/50000 [6:52:46<18:31:23,  1.77s/it]
Outer training loop:  25%|##4       | 12431/50000 [6:52:48<19:00:00,  1.82s/it]
Outer training loop:  25%|##4       | 12432/50000 [6:52:49<18:43:40,  1.79s/it]
Outer training loop:  25%|##4       | 12433/50000 [6:52:51<18:54:56,  1.81s/it]
Outer training loop:  25%|##4       | 12434/50000 [6:52:53<18:48:32,  1.80s/it]
Outer training loop:  25%|##4       | 12435/50000 [6:52:55<19:15:42,  1.85s/it]
Outer training loop:  25%|##4       | 12436/50000 [6:52:57<18:31:20,  1.78s/it]
Outer training loop:  25%|##4       | 12437/50000 [6:52:58<18:17:43,  1.75s/it]
Outer training loop:  25%|##4       | 12438/50000 [6:53:00<18:54:22,  1.81s/it]
Outer training loop:  25%|##4       | 12439/50000 [6:53:02<18:38:18,  1.79s/it]
Outer training loop:  25%|##4       | 12440/50000 [6:53:04<18:34:10,  1.78s/it]
Outer training loop:  25%|##4       | 12441/50000 [6:53:06<19:07:43,  1.83s/it]
Outer training loop:  25%|##4       | 12442/50000 [6:53:08<19:35:25,  1.88s/it]
Outer training loop:  25%|##4       | 12443/50000 [6:53:09<19:06:31,  1.83s/it]
Outer training loop:  25%|##4       | 12444/50000 [6:53:11<19:17:24,  1.85s/it]
Outer training loop:  25%|##4       | 12445/50000 [6:53:13<19:14:05,  1.84s/it]
Outer training loop:  25%|##4       | 12446/50000 [6:53:15<18:55:25,  1.81s/it]
Outer training loop:  25%|##4       | 12447/50000 [6:53:17<18:40:33,  1.79s/it]
Outer training loop:  25%|##4       | 12448/50000 [6:53:19<19:20:26,  1.85s/it]
Outer training loop:  25%|##4       | 12449/50000 [6:53:20<18:53:37,  1.81s/it]
Outer training loop:  25%|##4       | 12450/50000 [6:53:22<18:34:23,  1.78s/it]
Outer training loop:  25%|##4       | 12451/50000 [6:53:24<18:59:24,  1.82s/it]
Outer training loop:  25%|##4       | 12452/50000 [6:53:26<18:44:41,  1.80s/it]
Outer training loop:  25%|##4       | 12453/50000 [6:53:27<18:25:43,  1.77s/it]
Outer training loop:  25%|##4       | 12454/50000 [6:53:29<18:51:07,  1.81s/it]
Outer training loop:  25%|##4       | 12455/50000 [6:53:31<18:56:52,  1.82s/it]
Outer training loop:  25%|##4       | 12456/50000 [6:53:33<18:46:18,  1.80s/it]
Outer training loop:  25%|##4       | 12457/50000 [6:53:35<18:17:59,  1.75s/it]
Outer training loop:  25%|##4       | 12458/50000 [6:53:37<18:49:27,  1.81s/it]
Outer training loop:  25%|##4       | 12459/50000 [6:53:38<18:14:50,  1.75s/it]
Outer training loop:  25%|##4       | 12460/50000 [6:53:40<18:34:48,  1.78s/it]
Outer training loop:  25%|##4       | 12461/50000 [6:53:42<19:08:21,  1.84s/it]
Outer training loop:  25%|##4       | 12462/50000 [6:53:44<18:39:12,  1.79s/it]
Outer training loop:  25%|##4       | 12463/50000 [6:53:45<18:20:33,  1.76s/it]
Outer training loop:  25%|##4       | 12464/50000 [6:53:47<18:56:37,  1.82s/it]
Outer training loop:  25%|##4       | 12465/50000 [6:53:49<18:50:00,  1.81s/it]
Outer training loop:  25%|##4       | 12466/50000 [6:53:51<18:52:12,  1.81s/it]
Outer training loop:  25%|##4       | 12467/50000 [6:53:53<19:09:54,  1.84s/it]
Outer training loop:  25%|##4       | 12468/50000 [6:53:55<18:52:28,  1.81s/it]
Outer training loop:  25%|##4       | 12469/50000 [6:53:56<18:23:54,  1.76s/it]
Outer training loop:  25%|##4       | 12470/50000 [6:53:58<18:05:49,  1.74s/it]
Outer training loop:  25%|##4       | 12471/50000 [6:54:00<18:45:07,  1.80s/it]
Outer training loop:  25%|##4       | 12472/50000 [6:54:02<18:46:52,  1.80s/it]
Outer training loop:  25%|##4       | 12473/50000 [6:54:03<18:15:54,  1.75s/it]
Outer training loop:  25%|##4       | 12474/50000 [6:54:05<18:44:12,  1.80s/it]
Outer training loop:  25%|##4       | 12475/50000 [6:54:07<18:39:12,  1.79s/it]
Outer training loop:  25%|##4       | 12476/50000 [6:54:09<18:39:48,  1.79s/it]
Outer training loop:  25%|##4       | 12477/50000 [6:54:11<19:30:13,  1.87s/it]
Outer training loop:  25%|##4       | 12478/50000 [6:54:13<19:12:08,  1.84s/it]
Outer training loop:  25%|##4       | 12479/50000 [6:54:14<19:07:49,  1.84s/it]
Outer training loop:  25%|##4       | 12480/50000 [6:54:16<18:26:31,  1.77s/it]
Outer training loop:  25%|##4       | 12481/50000 [6:54:18<18:50:02,  1.81s/it]
Outer training loop:  25%|##4       | 12482/50000 [6:54:20<18:59:15,  1.82s/it]
Outer training loop:  25%|##4       | 12483/50000 [6:54:22<18:52:34,  1.81s/it]
Outer training loop:  25%|##4       | 12484/50000 [6:54:23<18:54:19,  1.81s/it]
Outer training loop:  25%|##4       | 12485/50000 [6:54:25<18:27:38,  1.77s/it]
Outer training loop:  25%|##4       | 12486/50000 [6:54:27<18:15:23,  1.75s/it]
Outer training loop:  25%|##4       | 12487/50000 [6:54:29<18:45:32,  1.80s/it]
Outer training loop:  25%|##4       | 12488/50000 [6:54:30<18:09:39,  1.74s/it]
Outer training loop:  25%|##4       | 12489/50000 [6:54:32<17:53:09,  1.72s/it]
Outer training loop:  25%|##4       | 12490/50000 [6:54:33<17:31:01,  1.68s/it]
Outer training loop:  25%|##4       | 12491/50000 [6:54:35<18:06:24,  1.74s/it]
Outer training loop:  25%|##4       | 12492/50000 [6:54:37<18:25:00,  1.77s/it]
Outer training loop:  25%|##4       | 12493/50000 [6:54:39<18:19:37,  1.76s/it]
Outer training loop:  25%|##4       | 12494/50000 [6:54:41<18:40:42,  1.79s/it]
Outer training loop:  25%|##4       | 12495/50000 [6:54:43<18:36:52,  1.79s/it]
Outer training loop:  25%|##4       | 12496/50000 [6:54:44<18:09:37,  1.74s/it]
Outer training loop:  25%|##4       | 12497/50000 [6:54:46<18:09:58,  1.74s/it]
Outer training loop:  25%|##4       | 12498/50000 [6:54:48<18:40:26,  1.79s/it]
Outer training loop:  25%|##4       | 12499/50000 [6:54:50<18:41:25,  1.79s/it]
Outer training loop:  25%|##5       | 12500/50000 [6:54:51<18:25:03,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.63s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:20,  1.74s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:19,  1.74s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.79s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.79s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:17,  1.79s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.84s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:13,  1.80s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:10,  1.78s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:12,  1.85s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.83s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.79s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:02,  1.78s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:05,  1.85s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:01,  1.81s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.79s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:54,  1.76s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:51,  1.75s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:56,  1.85s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:52,  1.81s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:48,  1.77s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:46,  1.78s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:42,  1.80s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:41,  1.81s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:38,  1.78s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:37,  1.81s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:41,  1.91s/it]

Evaluating policy:  37%|###7      | 31/83 [00:56<01:37,  1.87s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:32,  1.82s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:29,  1.78s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:25,  1.73s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:22,  1.72s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:26,  1.83s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:21,  1.78s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:21,  1.80s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:19,  1.80s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:15,  1.75s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:13,  1.75s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:14,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:10,  1.81s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:07,  1.77s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:03,  1.73s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:02,  1.73s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:01,  1.75s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<01:02,  1.84s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:59,  1.79s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:56,  1.77s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:54,  1.77s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:53,  1.78s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:50,  1.75s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:48,  1.74s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:46,  1.74s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:47,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:43,  1.82s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:48<00:41,  1.81s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:39,  1.80s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:37,  1.80s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:53<00:35,  1.78s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:33,  1.74s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:57<00:33,  1.87s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:30,  1.82s/it]

Evaluating policy:  81%|########  | 67/83 [02:00<00:28,  1.79s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:26,  1.79s/it]

Evaluating policy:  83%|########3 | 69/83 [02:04<00:24,  1.75s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:22,  1.76s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:20,  1.75s/it]

Evaluating policy:  87%|########6 | 72/83 [02:09<00:19,  1.73s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.72s/it]

Evaluating policy:  89%|########9 | 74/83 [02:13<00:16,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:14,  1.86s/it]

Evaluating policy:  92%|#########1| 76/83 [02:16<00:12,  1.81s/it]

Evaluating policy:  93%|#########2| 77/83 [02:18<00:10,  1.78s/it]

Evaluating policy:  94%|#########3| 78/83 [02:20<00:08,  1.79s/it]

Evaluating policy:  95%|#########5| 79/83 [02:22<00:07,  1.84s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.81s/it]

Evaluating policy:  98%|#########7| 81/83 [02:25<00:03,  1.78s/it]

Evaluating policy:  99%|#########8| 82/83 [02:27<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 83/83 [02:29<00:00,  1.90s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:18,  1.69s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:18,  1.71s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:16,  1.71s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:20,  1.78s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:21,  1.82s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:20,  1.83s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.84s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:16,  1.82s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:14,  1.82s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:24,  1.98s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:20,  1.95s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:15,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:08,  1.84s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:09,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:03,  1.82s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:01,  1.81s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<01:57,  1.78s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:57,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:54,  1.82s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:51,  1.79s/it]

Evaluating policy:  27%|##6       | 22/83 [00:40<01:47,  1.77s/it]

Evaluating policy:  28%|##7       | 23/83 [00:42<01:52,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:44,  1.80s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:41,  1.78s/it]

Evaluating policy:  33%|###2      | 27/83 [00:49<01:37,  1.74s/it]

Evaluating policy:  34%|###3      | 28/83 [00:51<01:41,  1.84s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:37,  1.80s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:34,  1.78s/it]

Evaluating policy:  37%|###7      | 31/83 [00:56<01:32,  1.77s/it]

Evaluating policy:  39%|###8      | 32/83 [00:58<01:30,  1.77s/it]

Evaluating policy:  40%|###9      | 33/83 [01:00<01:32,  1.84s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:28,  1.80s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:26,  1.80s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:24,  1.81s/it]

Evaluating policy:  45%|####4     | 37/83 [01:07<01:22,  1.79s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:19,  1.78s/it]

Evaluating policy:  47%|####6     | 39/83 [01:11<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:19,  1.85s/it]

Evaluating policy:  49%|####9     | 41/83 [01:14<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:16<01:14,  1.81s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:18<01:11,  1.79s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:10,  1.80s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:23<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:25<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:27<01:02,  1.78s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<01:00,  1.78s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:57,  1.74s/it]

Evaluating policy:  61%|######1   | 51/83 [01:32<00:55,  1.74s/it]

Evaluating policy:  63%|######2   | 52/83 [01:34<00:53,  1.73s/it]

Evaluating policy:  64%|######3   | 53/83 [01:36<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:51,  1.79s/it]

Evaluating policy:  66%|######6   | 55/83 [01:39<00:51,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:41<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:43<00:46,  1.79s/it]

Evaluating policy:  70%|######9   | 58/83 [01:45<00:44,  1.80s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:42,  1.79s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:48<00:41,  1.79s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:50<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:52<00:38,  1.84s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:54<00:35,  1.79s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:55<00:33,  1.78s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:57<00:32,  1.78s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:59<00:30,  1.76s/it]

Evaluating policy:  81%|########  | 67/83 [02:01<00:28,  1.76s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:26,  1.75s/it]

Evaluating policy:  83%|########3 | 69/83 [02:04<00:24,  1.76s/it]

Evaluating policy:  84%|########4 | 70/83 [02:06<00:24,  1.90s/it]

Evaluating policy:  86%|########5 | 71/83 [02:08<00:21,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:10<00:19,  1.77s/it]

Evaluating policy:  88%|########7 | 73/83 [02:11<00:17,  1.75s/it]

Evaluating policy:  89%|########9 | 74/83 [02:13<00:15,  1.75s/it]

Evaluating policy:  90%|######### | 75/83 [02:15<00:14,  1.76s/it]

Evaluating policy:  92%|#########1| 76/83 [02:17<00:12,  1.76s/it]

Evaluating policy:  93%|#########2| 77/83 [02:18<00:10,  1.76s/it]

Evaluating policy:  94%|#########3| 78/83 [02:20<00:08,  1.74s/it]

Evaluating policy:  95%|#########5| 79/83 [02:23<00:07,  1.92s/it]

Evaluating policy:  96%|#########6| 80/83 [02:24<00:05,  1.85s/it]

Evaluating policy:  98%|#########7| 81/83 [02:26<00:03,  1.79s/it]

Evaluating policy:  99%|#########8| 82/83 [02:28<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 83/83 [02:29<00:00,  1.78s/it]

Outer training loop:  25%|##5       | 12501/50000 [6:59:58<971:44:41, 93.29s/it]
Outer training loop:  25%|##5       | 12502/50000 [7:00:00<685:17:11, 65.79s/it]
Outer training loop:  25%|##5       | 12503/50000 [7:00:02<485:01:49, 46.57s/it]
Outer training loop:  25%|##5       | 12504/50000 [7:00:03<345:12:41, 33.14s/it]
Outer training loop:  25%|##5       | 12505/50000 [7:00:05<247:21:29, 23.75s/it]
Outer training loop:  25%|##5       | 12506/50000 [7:00:07<178:21:26, 17.13s/it]
Outer training loop:  25%|##5       | 12507/50000 [7:00:09<130:11:12, 12.50s/it]
Outer training loop:  25%|##5       | 12508/50000 [7:00:11<98:44:26,  9.48s/it] 
Outer training loop:  25%|##5       | 12509/50000 [7:00:13<74:18:24,  7.14s/it]
Outer training loop:  25%|##5       | 12510/50000 [7:00:14<57:36:22,  5.53s/it]
Outer training loop:  25%|##5       | 12511/50000 [7:00:16<46:11:54,  4.44s/it]
Outer training loop:  25%|##5       | 12512/50000 [7:00:18<37:51:42,  3.64s/it]
Outer training loop:  25%|##5       | 12513/50000 [7:00:20<32:11:56,  3.09s/it]
Outer training loop:  25%|##5       | 12514/50000 [7:00:22<28:19:51,  2.72s/it]
Outer training loop:  25%|##5       | 12515/50000 [7:00:23<24:48:54,  2.38s/it]
Outer training loop:  25%|##5       | 12516/50000 [7:00:25<22:42:05,  2.18s/it]
Outer training loop:  25%|##5       | 12517/50000 [7:00:27<21:52:36,  2.10s/it]
Outer training loop:  25%|##5       | 12518/50000 [7:00:29<20:33:00,  1.97s/it]
Outer training loop:  25%|##5       | 12519/50000 [7:00:30<19:43:18,  1.89s/it]
Outer training loop:  25%|##5       | 12520/50000 [7:00:32<18:57:07,  1.82s/it]
Outer training loop:  25%|##5       | 12521/50000 [7:00:34<19:22:14,  1.86s/it]
Outer training loop:  25%|##5       | 12522/50000 [7:00:36<19:07:48,  1.84s/it]
Outer training loop:  25%|##5       | 12523/50000 [7:00:38<19:23:15,  1.86s/it]
Outer training loop:  25%|##5       | 12524/50000 [7:00:40<19:37:12,  1.88s/it]
Outer training loop:  25%|##5       | 12525/50000 [7:00:41<19:01:48,  1.83s/it]
Outer training loop:  25%|##5       | 12526/50000 [7:00:43<19:14:00,  1.85s/it]
Outer training loop:  25%|##5       | 12527/50000 [7:00:45<18:38:12,  1.79s/it]
Outer training loop:  25%|##5       | 12528/50000 [7:00:47<19:13:51,  1.85s/it]
Outer training loop:  25%|##5       | 12529/50000 [7:00:49<18:54:52,  1.82s/it]
Outer training loop:  25%|##5       | 12530/50000 [7:00:50<19:05:27,  1.83s/it]
Outer training loop:  25%|##5       | 12531/50000 [7:00:52<19:32:08,  1.88s/it]
Outer training loop:  25%|##5       | 12532/50000 [7:00:54<19:01:38,  1.83s/it]
Outer training loop:  25%|##5       | 12533/50000 [7:00:56<18:55:30,  1.82s/it]
Outer training loop:  25%|##5       | 12534/50000 [7:00:58<19:08:39,  1.84s/it]
Outer training loop:  25%|##5       | 12535/50000 [7:01:00<18:36:03,  1.79s/it]
Outer training loop:  25%|##5       | 12536/50000 [7:01:01<18:54:22,  1.82s/it]
Outer training loop:  25%|##5       | 12537/50000 [7:01:03<18:15:35,  1.75s/it]
Outer training loop:  25%|##5       | 12538/50000 [7:01:05<18:56:06,  1.82s/it]
Outer training loop:  25%|##5       | 12539/50000 [7:01:07<18:31:21,  1.78s/it]
Outer training loop:  25%|##5       | 12540/50000 [7:01:08<18:14:17,  1.75s/it]
Outer training loop:  25%|##5       | 12541/50000 [7:01:10<19:03:36,  1.83s/it]
Outer training loop:  25%|##5       | 12542/50000 [7:01:12<18:59:07,  1.82s/it]
Outer training loop:  25%|##5       | 12543/50000 [7:01:14<18:39:17,  1.79s/it]
Outer training loop:  25%|##5       | 12544/50000 [7:01:16<19:04:17,  1.83s/it]
Outer training loop:  25%|##5       | 12545/50000 [7:01:18<18:42:28,  1.80s/it]
Outer training loop:  25%|##5       | 12546/50000 [7:01:19<18:26:50,  1.77s/it]
Outer training loop:  25%|##5       | 12547/50000 [7:01:21<18:18:35,  1.76s/it]
Outer training loop:  25%|##5       | 12548/50000 [7:01:23<18:52:41,  1.81s/it]
Outer training loop:  25%|##5       | 12549/50000 [7:01:25<18:31:57,  1.78s/it]
Outer training loop:  25%|##5       | 12550/50000 [7:01:26<18:33:21,  1.78s/it]
Outer training loop:  25%|##5       | 12551/50000 [7:01:28<19:08:23,  1.84s/it]
Outer training loop:  25%|##5       | 12552/50000 [7:01:30<18:40:41,  1.80s/it]
Outer training loop:  25%|##5       | 12553/50000 [7:01:32<18:26:22,  1.77s/it]
Outer training loop:  25%|##5       | 12554/50000 [7:01:34<18:16:06,  1.76s/it]
Outer training loop:  25%|##5       | 12555/50000 [7:01:35<18:50:32,  1.81s/it]
Outer training loop:  25%|##5       | 12556/50000 [7:01:37<18:12:05,  1.75s/it]
Outer training loop:  25%|##5       | 12557/50000 [7:01:39<18:11:16,  1.75s/it]
Outer training loop:  25%|##5       | 12558/50000 [7:01:41<18:46:43,  1.81s/it]
Outer training loop:  25%|##5       | 12559/50000 [7:01:42<18:24:39,  1.77s/it]
Outer training loop:  25%|##5       | 12560/50000 [7:01:44<18:03:51,  1.74s/it]
Outer training loop:  25%|##5       | 12561/50000 [7:01:46<18:04:43,  1.74s/it]
Outer training loop:  25%|##5       | 12562/50000 [7:01:48<18:24:39,  1.77s/it]
Outer training loop:  25%|##5       | 12563/50000 [7:01:50<18:34:25,  1.79s/it]
Outer training loop:  25%|##5       | 12564/50000 [7:01:51<18:25:59,  1.77s/it]
Outer training loop:  25%|##5       | 12565/50000 [7:01:53<18:21:55,  1.77s/it]
Outer training loop:  25%|##5       | 12566/50000 [7:01:55<18:22:06,  1.77s/it]
Outer training loop:  25%|##5       | 12567/50000 [7:01:57<18:21:34,  1.77s/it]
Outer training loop:  25%|##5       | 12568/50000 [7:01:59<19:05:22,  1.84s/it]
Outer training loop:  25%|##5       | 12569/50000 [7:02:00<18:36:42,  1.79s/it]
Outer training loop:  25%|##5       | 12570/50000 [7:02:02<18:35:19,  1.79s/it]
Outer training loop:  25%|##5       | 12571/50000 [7:02:04<18:55:32,  1.82s/it]
Outer training loop:  25%|##5       | 12572/50000 [7:02:06<18:29:33,  1.78s/it]
Outer training loop:  25%|##5       | 12573/50000 [7:02:07<18:44:30,  1.80s/it]
Outer training loop:  25%|##5       | 12574/50000 [7:02:09<18:58:49,  1.83s/it]
Outer training loop:  25%|##5       | 12575/50000 [7:02:11<18:45:31,  1.80s/it]
Outer training loop:  25%|##5       | 12576/50000 [7:02:13<18:35:39,  1.79s/it]
Outer training loop:  25%|##5       | 12577/50000 [7:02:15<18:39:26,  1.79s/it]
Outer training loop:  25%|##5       | 12578/50000 [7:02:17<19:09:16,  1.84s/it]
Outer training loop:  25%|##5       | 12579/50000 [7:02:18<18:43:09,  1.80s/it]
Outer training loop:  25%|##5       | 12580/50000 [7:02:20<18:12:29,  1.75s/it]
Outer training loop:  25%|##5       | 12581/50000 [7:02:22<18:17:58,  1.76s/it]
Outer training loop:  25%|##5       | 12582/50000 [7:02:23<18:11:43,  1.75s/it]
Outer training loop:  25%|##5       | 12583/50000 [7:02:25<18:03:06,  1.74s/it]
Outer training loop:  25%|##5       | 12584/50000 [7:02:27<18:14:28,  1.76s/it]
Outer training loop:  25%|##5       | 12585/50000 [7:02:29<18:54:04,  1.82s/it]
Outer training loop:  25%|##5       | 12586/50000 [7:02:31<18:36:25,  1.79s/it]
Outer training loop:  25%|##5       | 12587/50000 [7:02:32<18:15:21,  1.76s/it]
Outer training loop:  25%|##5       | 12588/50000 [7:02:34<18:30:07,  1.78s/it]
Outer training loop:  25%|##5       | 12589/50000 [7:02:36<18:45:04,  1.80s/it]
Outer training loop:  25%|##5       | 12590/50000 [7:02:38<18:22:40,  1.77s/it]
Outer training loop:  25%|##5       | 12591/50000 [7:02:40<18:44:52,  1.80s/it]
Outer training loop:  25%|##5       | 12592/50000 [7:02:42<19:07:11,  1.84s/it]
Outer training loop:  25%|##5       | 12593/50000 [7:02:43<18:45:56,  1.81s/it]
Outer training loop:  25%|##5       | 12594/50000 [7:02:45<18:59:37,  1.83s/it]
Outer training loop:  25%|##5       | 12595/50000 [7:02:47<19:33:25,  1.88s/it]
Outer training loop:  25%|##5       | 12596/50000 [7:02:49<19:05:52,  1.84s/it]
Outer training loop:  25%|##5       | 12597/50000 [7:02:51<18:55:06,  1.82s/it]
Outer training loop:  25%|##5       | 12598/50000 [7:02:53<19:17:03,  1.86s/it]
Outer training loop:  25%|##5       | 12599/50000 [7:02:54<18:56:22,  1.82s/it]
Outer training loop:  25%|##5       | 12600/50000 [7:02:56<18:40:12,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  25%|##5       | 12601/50000 [7:03:03<34:40:36,  3.34s/it]
Outer training loop:  25%|##5       | 12602/50000 [7:03:05<30:34:07,  2.94s/it]
Outer training loop:  25%|##5       | 12603/50000 [7:03:07<26:51:19,  2.59s/it]
Outer training loop:  25%|##5       | 12604/50000 [7:03:09<24:34:44,  2.37s/it]
Outer training loop:  25%|##5       | 12605/50000 [7:03:10<22:27:40,  2.16s/it]
Outer training loop:  25%|##5       | 12606/50000 [7:03:12<21:53:10,  2.11s/it]
Outer training loop:  25%|##5       | 12607/50000 [7:03:14<21:05:52,  2.03s/it]
Outer training loop:  25%|##5       | 12608/50000 [7:03:16<20:03:16,  1.93s/it]
Outer training loop:  25%|##5       | 12609/50000 [7:03:18<19:53:25,  1.92s/it]
Outer training loop:  25%|##5       | 12610/50000 [7:03:19<19:15:09,  1.85s/it]
Outer training loop:  25%|##5       | 12611/50000 [7:03:21<19:01:27,  1.83s/it]
Outer training loop:  25%|##5       | 12612/50000 [7:03:23<19:08:03,  1.84s/it]
Outer training loop:  25%|##5       | 12613/50000 [7:03:25<18:41:01,  1.80s/it]
Outer training loop:  25%|##5       | 12614/50000 [7:03:26<18:18:52,  1.76s/it]
Outer training loop:  25%|##5       | 12615/50000 [7:03:28<18:55:37,  1.82s/it]
Outer training loop:  25%|##5       | 12616/50000 [7:03:30<18:26:35,  1.78s/it]
Outer training loop:  25%|##5       | 12617/50000 [7:03:32<18:23:38,  1.77s/it]
Outer training loop:  25%|##5       | 12618/50000 [7:03:34<18:02:54,  1.74s/it]
Outer training loop:  25%|##5       | 12619/50000 [7:03:36<18:52:09,  1.82s/it]
Outer training loop:  25%|##5       | 12620/50000 [7:03:37<19:16:19,  1.86s/it]
Outer training loop:  25%|##5       | 12621/50000 [7:03:39<18:55:11,  1.82s/it]
Outer training loop:  25%|##5       | 12622/50000 [7:03:41<19:29:56,  1.88s/it]
Outer training loop:  25%|##5       | 12623/50000 [7:03:43<19:05:28,  1.84s/it]
Outer training loop:  25%|##5       | 12624/50000 [7:03:45<18:55:43,  1.82s/it]
Outer training loop:  25%|##5       | 12625/50000 [7:03:47<18:46:40,  1.81s/it]
Outer training loop:  25%|##5       | 12626/50000 [7:03:48<18:55:27,  1.82s/it]
Outer training loop:  25%|##5       | 12627/50000 [7:03:50<18:43:41,  1.80s/it]
Outer training loop:  25%|##5       | 12628/50000 [7:03:52<18:31:42,  1.78s/it]
Outer training loop:  25%|##5       | 12629/50000 [7:03:54<18:56:50,  1.83s/it]
Outer training loop:  25%|##5       | 12630/50000 [7:03:55<18:28:12,  1.78s/it]
Outer training loop:  25%|##5       | 12631/50000 [7:03:57<18:25:47,  1.78s/it]
Outer training loop:  25%|##5       | 12632/50000 [7:03:59<18:27:35,  1.78s/it]
Outer training loop:  25%|##5       | 12633/50000 [7:04:01<18:23:23,  1.77s/it]
Outer training loop:  25%|##5       | 12634/50000 [7:04:02<18:14:37,  1.76s/it]
Outer training loop:  25%|##5       | 12635/50000 [7:04:04<18:45:58,  1.81s/it]
Outer training loop:  25%|##5       | 12636/50000 [7:04:06<18:55:08,  1.82s/it]
Outer training loop:  25%|##5       | 12637/50000 [7:04:08<18:29:38,  1.78s/it]
Outer training loop:  25%|##5       | 12638/50000 [7:04:10<18:37:54,  1.80s/it]
Outer training loop:  25%|##5       | 12639/50000 [7:04:12<18:35:12,  1.79s/it]
Outer training loop:  25%|##5       | 12640/50000 [7:04:13<18:26:25,  1.78s/it]
Outer training loop:  25%|##5       | 12641/50000 [7:04:15<18:35:18,  1.79s/it]
Outer training loop:  25%|##5       | 12642/50000 [7:04:17<19:04:04,  1.84s/it]
Outer training loop:  25%|##5       | 12643/50000 [7:04:19<18:53:08,  1.82s/it]
Outer training loop:  25%|##5       | 12644/50000 [7:04:21<18:26:30,  1.78s/it]
Outer training loop:  25%|##5       | 12645/50000 [7:04:22<18:39:14,  1.80s/it]
Outer training loop:  25%|##5       | 12646/50000 [7:04:24<18:41:32,  1.80s/it]
Outer training loop:  25%|##5       | 12647/50000 [7:04:26<18:16:16,  1.76s/it]
Outer training loop:  25%|##5       | 12648/50000 [7:04:28<18:21:26,  1.77s/it]
Outer training loop:  25%|##5       | 12649/50000 [7:04:29<18:35:05,  1.79s/it]
Outer training loop:  25%|##5       | 12650/50000 [7:04:31<18:07:01,  1.75s/it]
Outer training loop:  25%|##5       | 12651/50000 [7:04:33<17:54:50,  1.73s/it]
Outer training loop:  25%|##5       | 12652/50000 [7:04:35<18:40:57,  1.80s/it]
Outer training loop:  25%|##5       | 12653/50000 [7:04:37<18:37:38,  1.80s/it]
Outer training loop:  25%|##5       | 12654/50000 [7:04:38<18:21:16,  1.77s/it]
Outer training loop:  25%|##5       | 12655/50000 [7:04:40<18:41:58,  1.80s/it]
Outer training loop:  25%|##5       | 12656/50000 [7:04:42<18:36:09,  1.79s/it]
Outer training loop:  25%|##5       | 12657/50000 [7:04:44<18:15:18,  1.76s/it]
Outer training loop:  25%|##5       | 12658/50000 [7:04:45<18:34:21,  1.79s/it]
Outer training loop:  25%|##5       | 12659/50000 [7:04:47<18:16:38,  1.76s/it]
Outer training loop:  25%|##5       | 12660/50000 [7:04:49<18:22:58,  1.77s/it]
Outer training loop:  25%|##5       | 12661/50000 [7:04:51<18:53:20,  1.82s/it]
Outer training loop:  25%|##5       | 12662/50000 [7:04:53<18:49:30,  1.82s/it]
Outer training loop:  25%|##5       | 12663/50000 [7:04:54<18:43:49,  1.81s/it]
Outer training loop:  25%|##5       | 12664/50000 [7:04:56<18:34:18,  1.79s/it]
Outer training loop:  25%|##5       | 12665/50000 [7:04:58<18:48:08,  1.81s/it]
Outer training loop:  25%|##5       | 12666/50000 [7:05:00<18:27:42,  1.78s/it]
Outer training loop:  25%|##5       | 12667/50000 [7:05:01<18:10:06,  1.75s/it]
Outer training loop:  25%|##5       | 12668/50000 [7:05:03<18:39:12,  1.80s/it]
Outer training loop:  25%|##5       | 12669/50000 [7:05:05<18:33:57,  1.79s/it]
Outer training loop:  25%|##5       | 12670/50000 [7:05:07<18:47:33,  1.81s/it]
Outer training loop:  25%|##5       | 12671/50000 [7:05:09<19:02:46,  1.84s/it]
Outer training loop:  25%|##5       | 12672/50000 [7:05:11<18:45:39,  1.81s/it]
Outer training loop:  25%|##5       | 12673/50000 [7:05:12<18:42:46,  1.80s/it]
Outer training loop:  25%|##5       | 12674/50000 [7:05:14<18:35:11,  1.79s/it]
Outer training loop:  25%|##5       | 12675/50000 [7:05:16<19:15:32,  1.86s/it]
Outer training loop:  25%|##5       | 12676/50000 [7:05:18<18:33:37,  1.79s/it]
Outer training loop:  25%|##5       | 12677/50000 [7:05:20<18:31:23,  1.79s/it]
Outer training loop:  25%|##5       | 12678/50000 [7:05:22<18:45:58,  1.81s/it]
Outer training loop:  25%|##5       | 12679/50000 [7:05:23<18:42:52,  1.81s/it]
Outer training loop:  25%|##5       | 12680/50000 [7:05:25<18:38:33,  1.80s/it]
Outer training loop:  25%|##5       | 12681/50000 [7:05:27<19:03:17,  1.84s/it]
Outer training loop:  25%|##5       | 12682/50000 [7:05:29<18:35:43,  1.79s/it]
Outer training loop:  25%|##5       | 12683/50000 [7:05:30<18:16:04,  1.76s/it]
Outer training loop:  25%|##5       | 12684/50000 [7:05:32<18:47:24,  1.81s/it]
Outer training loop:  25%|##5       | 12685/50000 [7:05:34<18:29:57,  1.78s/it]
Outer training loop:  25%|##5       | 12686/50000 [7:05:36<17:59:22,  1.74s/it]
Outer training loop:  25%|##5       | 12687/50000 [7:05:38<18:21:25,  1.77s/it]
Outer training loop:  25%|##5       | 12688/50000 [7:05:39<18:44:08,  1.81s/it]
Outer training loop:  25%|##5       | 12689/50000 [7:05:41<18:33:21,  1.79s/it]
Outer training loop:  25%|##5       | 12690/50000 [7:05:43<19:00:51,  1.83s/it]
Outer training loop:  25%|##5       | 12691/50000 [7:05:45<18:33:45,  1.79s/it]
Outer training loop:  25%|##5       | 12692/50000 [7:05:47<18:20:20,  1.77s/it]
Outer training loop:  25%|##5       | 12693/50000 [7:05:48<18:54:01,  1.82s/it]
Outer training loop:  25%|##5       | 12694/50000 [7:05:50<18:46:45,  1.81s/it]
Outer training loop:  25%|##5       | 12695/50000 [7:05:52<18:31:24,  1.79s/it]
Outer training loop:  25%|##5       | 12696/50000 [7:05:54<18:19:05,  1.77s/it]
Outer training loop:  25%|##5       | 12697/50000 [7:05:56<18:58:27,  1.83s/it]
Outer training loop:  25%|##5       | 12698/50000 [7:05:57<18:52:08,  1.82s/it]
Outer training loop:  25%|##5       | 12699/50000 [7:05:59<18:45:25,  1.81s/it]
Outer training loop:  25%|##5       | 12700/50000 [7:06:01<18:53:28,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.67s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.70s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.71s/it]

Outer training loop:  25%|##5       | 12701/50000 [7:06:08<34:41:24,  3.35s/it]
Outer training loop:  25%|##5       | 12702/50000 [7:06:10<29:46:31,  2.87s/it]
Outer training loop:  25%|##5       | 12703/50000 [7:06:12<26:08:53,  2.52s/it]
Outer training loop:  25%|##5       | 12704/50000 [7:06:13<24:21:29,  2.35s/it]
Outer training loop:  25%|##5       | 12705/50000 [7:06:15<22:25:46,  2.17s/it]
Outer training loop:  25%|##5       | 12706/50000 [7:06:17<20:56:14,  2.02s/it]
Outer training loop:  25%|##5       | 12707/50000 [7:06:19<20:03:32,  1.94s/it]
Outer training loop:  25%|##5       | 12708/50000 [7:06:21<20:15:33,  1.96s/it]
Outer training loop:  25%|##5       | 12709/50000 [7:06:22<19:44:21,  1.91s/it]
Outer training loop:  25%|##5       | 12710/50000 [7:06:24<19:20:40,  1.87s/it]
Outer training loop:  25%|##5       | 12711/50000 [7:06:26<19:25:58,  1.88s/it]
Outer training loop:  25%|##5       | 12712/50000 [7:06:28<18:59:17,  1.83s/it]
Outer training loop:  25%|##5       | 12713/50000 [7:06:30<18:41:34,  1.80s/it]
Outer training loop:  25%|##5       | 12714/50000 [7:06:31<18:29:36,  1.79s/it]
Outer training loop:  25%|##5       | 12715/50000 [7:06:33<18:51:16,  1.82s/it]
Outer training loop:  25%|##5       | 12716/50000 [7:06:35<18:45:21,  1.81s/it]
Outer training loop:  25%|##5       | 12717/50000 [7:06:37<18:54:48,  1.83s/it]
Outer training loop:  25%|##5       | 12718/50000 [7:06:39<19:29:00,  1.88s/it]
Outer training loop:  25%|##5       | 12719/50000 [7:06:41<19:16:17,  1.86s/it]
Outer training loop:  25%|##5       | 12720/50000 [7:06:42<18:55:13,  1.83s/it]
Outer training loop:  25%|##5       | 12721/50000 [7:06:44<19:19:07,  1.87s/it]
Outer training loop:  25%|##5       | 12722/50000 [7:06:46<18:51:29,  1.82s/it]
Outer training loop:  25%|##5       | 12723/50000 [7:06:48<18:18:12,  1.77s/it]
Outer training loop:  25%|##5       | 12724/50000 [7:06:50<18:27:19,  1.78s/it]
Outer training loop:  25%|##5       | 12725/50000 [7:06:52<19:08:03,  1.85s/it]
Outer training loop:  25%|##5       | 12726/50000 [7:06:53<18:38:43,  1.80s/it]
Outer training loop:  25%|##5       | 12727/50000 [7:06:55<18:37:59,  1.80s/it]
Outer training loop:  25%|##5       | 12728/50000 [7:06:57<18:59:30,  1.83s/it]
Outer training loop:  25%|##5       | 12729/50000 [7:06:59<18:20:30,  1.77s/it]
Outer training loop:  25%|##5       | 12730/50000 [7:07:00<18:18:52,  1.77s/it]
Outer training loop:  25%|##5       | 12731/50000 [7:07:02<18:18:44,  1.77s/it]
Outer training loop:  25%|##5       | 12732/50000 [7:07:04<18:30:03,  1.79s/it]
Outer training loop:  25%|##5       | 12733/50000 [7:07:06<18:08:40,  1.75s/it]
Outer training loop:  25%|##5       | 12734/50000 [7:07:07<18:10:31,  1.76s/it]
Outer training loop:  25%|##5       | 12735/50000 [7:07:09<18:49:29,  1.82s/it]
Outer training loop:  25%|##5       | 12736/50000 [7:07:11<18:22:16,  1.77s/it]
Outer training loop:  25%|##5       | 12737/50000 [7:07:13<18:26:40,  1.78s/it]
Outer training loop:  25%|##5       | 12738/50000 [7:07:14<17:53:55,  1.73s/it]
Outer training loop:  25%|##5       | 12739/50000 [7:07:16<18:24:04,  1.78s/it]
Outer training loop:  25%|##5       | 12740/50000 [7:07:18<18:06:41,  1.75s/it]
Outer training loop:  25%|##5       | 12741/50000 [7:07:20<17:47:00,  1.72s/it]
Outer training loop:  25%|##5       | 12742/50000 [7:07:21<17:40:30,  1.71s/it]
Outer training loop:  25%|##5       | 12743/50000 [7:07:23<18:19:36,  1.77s/it]
Outer training loop:  25%|##5       | 12744/50000 [7:07:25<18:38:12,  1.80s/it]
Outer training loop:  25%|##5       | 12745/50000 [7:07:27<18:29:36,  1.79s/it]
Outer training loop:  25%|##5       | 12746/50000 [7:07:29<18:56:20,  1.83s/it]
Outer training loop:  25%|##5       | 12747/50000 [7:07:30<18:28:11,  1.78s/it]
Outer training loop:  25%|##5       | 12748/50000 [7:07:32<18:46:39,  1.81s/it]
Outer training loop:  25%|##5       | 12749/50000 [7:07:34<18:35:19,  1.80s/it]
Outer training loop:  26%|##5       | 12750/50000 [7:07:36<18:44:19,  1.81s/it]
Outer training loop:  26%|##5       | 12751/50000 [7:07:38<19:07:08,  1.85s/it]
Outer training loop:  26%|##5       | 12752/50000 [7:07:40<18:48:34,  1.82s/it]
Outer training loop:  26%|##5       | 12753/50000 [7:07:42<19:25:32,  1.88s/it]
Outer training loop:  26%|##5       | 12754/50000 [7:07:43<19:03:23,  1.84s/it]
Outer training loop:  26%|##5       | 12755/50000 [7:07:45<19:04:56,  1.84s/it]
Outer training loop:  26%|##5       | 12756/50000 [7:07:47<19:23:47,  1.87s/it]
Outer training loop:  26%|##5       | 12757/50000 [7:07:49<19:04:13,  1.84s/it]
Outer training loop:  26%|##5       | 12758/50000 [7:07:51<18:34:28,  1.80s/it]
Outer training loop:  26%|##5       | 12759/50000 [7:07:52<18:35:37,  1.80s/it]
Outer training loop:  26%|##5       | 12760/50000 [7:07:54<18:48:58,  1.82s/it]
Outer training loop:  26%|##5       | 12761/50000 [7:07:56<18:40:39,  1.81s/it]
Outer training loop:  26%|##5       | 12762/50000 [7:07:58<18:49:17,  1.82s/it]
Outer training loop:  26%|##5       | 12763/50000 [7:08:00<18:52:51,  1.83s/it]
Outer training loop:  26%|##5       | 12764/50000 [7:08:02<18:38:02,  1.80s/it]
Outer training loop:  26%|##5       | 12765/50000 [7:08:03<18:27:09,  1.78s/it]
Outer training loop:  26%|##5       | 12766/50000 [7:08:05<18:53:54,  1.83s/it]
Outer training loop:  26%|##5       | 12767/50000 [7:08:07<19:12:19,  1.86s/it]
Outer training loop:  26%|##5       | 12768/50000 [7:08:09<18:49:56,  1.82s/it]
Outer training loop:  26%|##5       | 12769/50000 [7:08:11<19:26:37,  1.88s/it]
Outer training loop:  26%|##5       | 12770/50000 [7:08:13<19:16:56,  1.86s/it]
Outer training loop:  26%|##5       | 12771/50000 [7:08:14<18:44:54,  1.81s/it]
Outer training loop:  26%|##5       | 12772/50000 [7:08:16<18:20:35,  1.77s/it]
Outer training loop:  26%|##5       | 12773/50000 [7:08:18<18:59:05,  1.84s/it]
Outer training loop:  26%|##5       | 12774/50000 [7:08:20<18:44:11,  1.81s/it]
Outer training loop:  26%|##5       | 12775/50000 [7:08:22<18:32:37,  1.79s/it]
Outer training loop:  26%|##5       | 12776/50000 [7:08:24<19:10:55,  1.86s/it]
Outer training loop:  26%|##5       | 12777/50000 [7:08:25<18:50:39,  1.82s/it]
Outer training loop:  26%|##5       | 12778/50000 [7:08:27<18:50:58,  1.82s/it]
Outer training loop:  26%|##5       | 12779/50000 [7:08:29<18:33:54,  1.80s/it]
Outer training loop:  26%|##5       | 12780/50000 [7:08:31<19:03:43,  1.84s/it]
Outer training loop:  26%|##5       | 12781/50000 [7:08:33<18:52:01,  1.82s/it]
Outer training loop:  26%|##5       | 12782/50000 [7:08:34<18:37:42,  1.80s/it]
Outer training loop:  26%|##5       | 12783/50000 [7:08:36<18:55:00,  1.83s/it]
Outer training loop:  26%|##5       | 12784/50000 [7:08:38<19:07:43,  1.85s/it]
Outer training loop:  26%|##5       | 12785/50000 [7:08:40<19:10:06,  1.85s/it]
Outer training loop:  26%|##5       | 12786/50000 [7:08:42<19:37:10,  1.90s/it]
Outer training loop:  26%|##5       | 12787/50000 [7:08:44<19:01:32,  1.84s/it]
Outer training loop:  26%|##5       | 12788/50000 [7:08:45<18:27:25,  1.79s/it]
Outer training loop:  26%|##5       | 12789/50000 [7:08:47<18:18:38,  1.77s/it]
Outer training loop:  26%|##5       | 12790/50000 [7:08:49<19:11:23,  1.86s/it]
Outer training loop:  26%|##5       | 12791/50000 [7:08:51<19:01:30,  1.84s/it]
Outer training loop:  26%|##5       | 12792/50000 [7:08:53<18:40:24,  1.81s/it]
Outer training loop:  26%|##5       | 12793/50000 [7:08:55<19:08:04,  1.85s/it]
Outer training loop:  26%|##5       | 12794/50000 [7:08:56<18:47:39,  1.82s/it]
Outer training loop:  26%|##5       | 12795/50000 [7:08:58<18:37:27,  1.80s/it]
Outer training loop:  26%|##5       | 12796/50000 [7:09:00<18:38:48,  1.80s/it]
Outer training loop:  26%|##5       | 12797/50000 [7:09:02<19:04:25,  1.85s/it]
Outer training loop:  26%|##5       | 12798/50000 [7:09:04<19:01:36,  1.84s/it]
Outer training loop:  26%|##5       | 12799/50000 [7:09:06<18:54:41,  1.83s/it]
Outer training loop:  26%|##5       | 12800/50000 [7:09:07<18:56:28,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  26%|##5       | 12801/50000 [7:09:14<35:07:25,  3.40s/it]
Outer training loop:  26%|##5       | 12802/50000 [7:09:16<29:42:52,  2.88s/it]
Outer training loop:  26%|##5       | 12803/50000 [7:09:18<26:50:26,  2.60s/it]
Outer training loop:  26%|##5       | 12804/50000 [7:09:20<24:22:43,  2.36s/it]
Outer training loop:  26%|##5       | 12805/50000 [7:09:22<22:21:29,  2.16s/it]
Outer training loop:  26%|##5       | 12806/50000 [7:09:23<20:51:45,  2.02s/it]
Outer training loop:  26%|##5       | 12807/50000 [7:09:25<20:38:01,  2.00s/it]
Outer training loop:  26%|##5       | 12808/50000 [7:09:27<19:51:21,  1.92s/it]
Outer training loop:  26%|##5       | 12809/50000 [7:09:29<19:28:48,  1.89s/it]
Outer training loop:  26%|##5       | 12810/50000 [7:09:30<18:58:16,  1.84s/it]
Outer training loop:  26%|##5       | 12811/50000 [7:09:32<19:33:29,  1.89s/it]
Outer training loop:  26%|##5       | 12812/50000 [7:09:34<19:18:31,  1.87s/it]
Outer training loop:  26%|##5       | 12813/50000 [7:09:36<19:25:33,  1.88s/it]
Outer training loop:  26%|##5       | 12814/50000 [7:09:38<20:06:41,  1.95s/it]
Outer training loop:  26%|##5       | 12815/50000 [7:09:40<19:56:16,  1.93s/it]
Outer training loop:  26%|##5       | 12816/50000 [7:09:42<19:22:59,  1.88s/it]
Outer training loop:  26%|##5       | 12817/50000 [7:09:44<19:28:20,  1.89s/it]
Outer training loop:  26%|##5       | 12818/50000 [7:09:46<18:46:03,  1.82s/it]
Outer training loop:  26%|##5       | 12819/50000 [7:09:47<18:27:06,  1.79s/it]
Outer training loop:  26%|##5       | 12820/50000 [7:09:49<18:27:31,  1.79s/it]
Outer training loop:  26%|##5       | 12821/50000 [7:09:51<18:54:18,  1.83s/it]
Outer training loop:  26%|##5       | 12822/50000 [7:09:53<18:55:17,  1.83s/it]
Outer training loop:  26%|##5       | 12823/50000 [7:09:54<18:27:59,  1.79s/it]
Outer training loop:  26%|##5       | 12824/50000 [7:09:56<18:40:12,  1.81s/it]
Outer training loop:  26%|##5       | 12825/50000 [7:09:58<18:41:20,  1.81s/it]
Outer training loop:  26%|##5       | 12826/50000 [7:10:00<18:36:28,  1.80s/it]
Outer training loop:  26%|##5       | 12827/50000 [7:10:02<18:51:10,  1.83s/it]
Outer training loop:  26%|##5       | 12828/50000 [7:10:04<19:30:29,  1.89s/it]
Outer training loop:  26%|##5       | 12829/50000 [7:10:06<18:58:27,  1.84s/it]
Outer training loop:  26%|##5       | 12830/50000 [7:10:07<18:57:29,  1.84s/it]
Outer training loop:  26%|##5       | 12831/50000 [7:10:09<18:28:23,  1.79s/it]
Outer training loop:  26%|##5       | 12832/50000 [7:10:11<19:15:05,  1.86s/it]
Outer training loop:  26%|##5       | 12833/50000 [7:10:13<18:46:42,  1.82s/it]
Outer training loop:  26%|##5       | 12834/50000 [7:10:14<18:15:41,  1.77s/it]
Outer training loop:  26%|##5       | 12835/50000 [7:10:16<18:52:24,  1.83s/it]
Outer training loop:  26%|##5       | 12836/50000 [7:10:18<19:00:28,  1.84s/it]
Outer training loop:  26%|##5       | 12837/50000 [7:10:20<19:11:46,  1.86s/it]
Outer training loop:  26%|##5       | 12838/50000 [7:10:22<18:40:25,  1.81s/it]
Outer training loop:  26%|##5       | 12839/50000 [7:10:24<18:50:19,  1.83s/it]
Outer training loop:  26%|##5       | 12840/50000 [7:10:26<18:35:17,  1.80s/it]
Outer training loop:  26%|##5       | 12841/50000 [7:10:27<18:22:41,  1.78s/it]
Outer training loop:  26%|##5       | 12842/50000 [7:10:29<19:06:29,  1.85s/it]
Outer training loop:  26%|##5       | 12843/50000 [7:10:31<18:46:18,  1.82s/it]
Outer training loop:  26%|##5       | 12844/50000 [7:10:33<18:44:53,  1.82s/it]
Outer training loop:  26%|##5       | 12845/50000 [7:10:35<19:10:51,  1.86s/it]
Outer training loop:  26%|##5       | 12846/50000 [7:10:37<19:05:48,  1.85s/it]
Outer training loop:  26%|##5       | 12847/50000 [7:10:38<18:44:56,  1.82s/it]
Outer training loop:  26%|##5       | 12848/50000 [7:10:40<18:16:57,  1.77s/it]
Outer training loop:  26%|##5       | 12849/50000 [7:10:42<19:02:10,  1.84s/it]
Outer training loop:  26%|##5       | 12850/50000 [7:10:44<18:52:28,  1.83s/it]
Outer training loop:  26%|##5       | 12851/50000 [7:10:45<18:11:19,  1.76s/it]
Outer training loop:  26%|##5       | 12852/50000 [7:10:47<18:41:01,  1.81s/it]
Outer training loop:  26%|##5       | 12853/50000 [7:10:49<18:27:28,  1.79s/it]
Outer training loop:  26%|##5       | 12854/50000 [7:10:51<18:27:03,  1.79s/it]
Outer training loop:  26%|##5       | 12855/50000 [7:10:53<18:53:44,  1.83s/it]
Outer training loop:  26%|##5       | 12856/50000 [7:10:55<18:50:40,  1.83s/it]
Outer training loop:  26%|##5       | 12857/50000 [7:10:56<18:24:39,  1.78s/it]
Outer training loop:  26%|##5       | 12858/50000 [7:10:58<18:18:51,  1.78s/it]
Outer training loop:  26%|##5       | 12859/50000 [7:11:00<18:51:28,  1.83s/it]
Outer training loop:  26%|##5       | 12860/50000 [7:11:02<18:23:31,  1.78s/it]
Outer training loop:  26%|##5       | 12861/50000 [7:11:04<18:39:39,  1.81s/it]
Outer training loop:  26%|##5       | 12862/50000 [7:11:06<19:08:28,  1.86s/it]
Outer training loop:  26%|##5       | 12863/50000 [7:11:07<18:39:35,  1.81s/it]
Outer training loop:  26%|##5       | 12864/50000 [7:11:09<19:08:07,  1.86s/it]
Outer training loop:  26%|##5       | 12865/50000 [7:11:11<19:28:05,  1.89s/it]
Outer training loop:  26%|##5       | 12866/50000 [7:11:13<19:11:42,  1.86s/it]
Outer training loop:  26%|##5       | 12867/50000 [7:11:15<18:59:10,  1.84s/it]
Outer training loop:  26%|##5       | 12868/50000 [7:11:17<18:53:18,  1.83s/it]
Outer training loop:  26%|##5       | 12869/50000 [7:11:18<18:43:13,  1.82s/it]
Outer training loop:  26%|##5       | 12870/50000 [7:11:20<18:41:28,  1.81s/it]
Outer training loop:  26%|##5       | 12871/50000 [7:11:22<18:32:09,  1.80s/it]
Outer training loop:  26%|##5       | 12872/50000 [7:11:24<18:52:59,  1.83s/it]
Outer training loop:  26%|##5       | 12873/50000 [7:11:26<18:45:15,  1.82s/it]
Outer training loop:  26%|##5       | 12874/50000 [7:11:27<18:43:44,  1.82s/it]
Outer training loop:  26%|##5       | 12875/50000 [7:11:29<18:39:33,  1.81s/it]
Outer training loop:  26%|##5       | 12876/50000 [7:11:31<18:45:34,  1.82s/it]
Outer training loop:  26%|##5       | 12877/50000 [7:11:33<18:34:56,  1.80s/it]
Outer training loop:  26%|##5       | 12878/50000 [7:11:35<18:31:43,  1.80s/it]
Outer training loop:  26%|##5       | 12879/50000 [7:11:36<18:47:08,  1.82s/it]
Outer training loop:  26%|##5       | 12880/50000 [7:11:38<18:36:06,  1.80s/it]
Outer training loop:  26%|##5       | 12881/50000 [7:11:40<18:22:14,  1.78s/it]
Outer training loop:  26%|##5       | 12882/50000 [7:11:42<19:21:02,  1.88s/it]
Outer training loop:  26%|##5       | 12883/50000 [7:11:44<18:47:38,  1.82s/it]
Outer training loop:  26%|##5       | 12884/50000 [7:11:46<18:35:35,  1.80s/it]
Outer training loop:  26%|##5       | 12885/50000 [7:11:47<18:09:30,  1.76s/it]
Outer training loop:  26%|##5       | 12886/50000 [7:11:49<19:00:46,  1.84s/it]
Outer training loop:  26%|##5       | 12887/50000 [7:11:51<19:11:46,  1.86s/it]
Outer training loop:  26%|##5       | 12888/50000 [7:11:53<19:12:06,  1.86s/it]
Outer training loop:  26%|##5       | 12889/50000 [7:11:55<19:42:10,  1.91s/it]
Outer training loop:  26%|##5       | 12890/50000 [7:11:57<19:16:04,  1.87s/it]
Outer training loop:  26%|##5       | 12891/50000 [7:11:59<18:48:17,  1.82s/it]
Outer training loop:  26%|##5       | 12892/50000 [7:12:01<19:21:45,  1.88s/it]
Outer training loop:  26%|##5       | 12893/50000 [7:12:02<18:51:59,  1.83s/it]
Outer training loop:  26%|##5       | 12894/50000 [7:12:04<18:59:06,  1.84s/it]
Outer training loop:  26%|##5       | 12895/50000 [7:12:06<18:37:20,  1.81s/it]
Outer training loop:  26%|##5       | 12896/50000 [7:12:08<18:50:28,  1.83s/it]
Outer training loop:  26%|##5       | 12897/50000 [7:12:09<18:41:50,  1.81s/it]
Outer training loop:  26%|##5       | 12898/50000 [7:12:11<18:11:41,  1.77s/it]
Outer training loop:  26%|##5       | 12899/50000 [7:12:13<18:44:20,  1.82s/it]
Outer training loop:  26%|##5       | 12900/50000 [7:12:15<18:09:36,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.71s/it]

Outer training loop:  26%|##5       | 12901/50000 [7:12:22<34:21:51,  3.33s/it]
Outer training loop:  26%|##5       | 12902/50000 [7:12:24<29:36:43,  2.87s/it]
Outer training loop:  26%|##5       | 12903/50000 [7:12:25<26:38:22,  2.59s/it]
Outer training loop:  26%|##5       | 12904/50000 [7:12:27<23:56:46,  2.32s/it]
Outer training loop:  26%|##5       | 12905/50000 [7:12:29<22:24:50,  2.18s/it]
Outer training loop:  26%|##5       | 12906/50000 [7:12:31<21:38:16,  2.10s/it]
Outer training loop:  26%|##5       | 12907/50000 [7:12:33<20:28:46,  1.99s/it]
Outer training loop:  26%|##5       | 12908/50000 [7:12:34<20:05:19,  1.95s/it]
Outer training loop:  26%|##5       | 12909/50000 [7:12:36<19:14:32,  1.87s/it]
Outer training loop:  26%|##5       | 12910/50000 [7:12:38<19:20:45,  1.88s/it]
Outer training loop:  26%|##5       | 12911/50000 [7:12:40<19:32:27,  1.90s/it]
Outer training loop:  26%|##5       | 12912/50000 [7:12:42<19:23:39,  1.88s/it]
Outer training loop:  26%|##5       | 12913/50000 [7:12:44<19:34:49,  1.90s/it]
Outer training loop:  26%|##5       | 12914/50000 [7:12:45<18:56:16,  1.84s/it]
Outer training loop:  26%|##5       | 12915/50000 [7:12:47<18:34:26,  1.80s/it]
Outer training loop:  26%|##5       | 12916/50000 [7:12:49<18:48:19,  1.83s/it]
Outer training loop:  26%|##5       | 12917/50000 [7:12:51<18:49:13,  1.83s/it]
Outer training loop:  26%|##5       | 12918/50000 [7:12:53<18:34:41,  1.80s/it]
Outer training loop:  26%|##5       | 12919/50000 [7:12:54<18:37:22,  1.81s/it]
Outer training loop:  26%|##5       | 12920/50000 [7:12:56<18:56:40,  1.84s/it]
Outer training loop:  26%|##5       | 12921/50000 [7:12:58<18:39:17,  1.81s/it]
Outer training loop:  26%|##5       | 12922/50000 [7:13:00<18:15:42,  1.77s/it]
Outer training loop:  26%|##5       | 12923/50000 [7:13:02<18:40:32,  1.81s/it]
Outer training loop:  26%|##5       | 12924/50000 [7:13:03<18:08:41,  1.76s/it]
Outer training loop:  26%|##5       | 12925/50000 [7:13:05<18:01:31,  1.75s/it]
Outer training loop:  26%|##5       | 12926/50000 [7:13:07<18:21:11,  1.78s/it]
Outer training loop:  26%|##5       | 12927/50000 [7:13:09<18:13:49,  1.77s/it]
Outer training loop:  26%|##5       | 12928/50000 [7:13:10<18:01:47,  1.75s/it]
Outer training loop:  26%|##5       | 12929/50000 [7:13:12<17:54:39,  1.74s/it]
Outer training loop:  26%|##5       | 12930/50000 [7:13:14<18:31:54,  1.80s/it]
Outer training loop:  26%|##5       | 12931/50000 [7:13:16<18:29:43,  1.80s/it]
Outer training loop:  26%|##5       | 12932/50000 [7:13:18<18:08:03,  1.76s/it]
Outer training loop:  26%|##5       | 12933/50000 [7:13:19<18:32:26,  1.80s/it]
Outer training loop:  26%|##5       | 12934/50000 [7:13:21<18:13:15,  1.77s/it]
Outer training loop:  26%|##5       | 12935/50000 [7:13:23<18:04:41,  1.76s/it]
Outer training loop:  26%|##5       | 12936/50000 [7:13:25<18:02:49,  1.75s/it]
Outer training loop:  26%|##5       | 12937/50000 [7:13:26<18:19:41,  1.78s/it]
Outer training loop:  26%|##5       | 12938/50000 [7:13:28<18:19:51,  1.78s/it]
Outer training loop:  26%|##5       | 12939/50000 [7:13:30<17:58:37,  1.75s/it]
Outer training loop:  26%|##5       | 12940/50000 [7:13:32<18:35:27,  1.81s/it]
Outer training loop:  26%|##5       | 12941/50000 [7:13:34<18:36:43,  1.81s/it]
Outer training loop:  26%|##5       | 12942/50000 [7:13:35<18:31:47,  1.80s/it]
Outer training loop:  26%|##5       | 12943/50000 [7:13:37<18:32:07,  1.80s/it]
Outer training loop:  26%|##5       | 12944/50000 [7:13:39<19:14:36,  1.87s/it]
Outer training loop:  26%|##5       | 12945/50000 [7:13:41<18:51:31,  1.83s/it]
Outer training loop:  26%|##5       | 12946/50000 [7:13:43<18:59:02,  1.84s/it]
Outer training loop:  26%|##5       | 12947/50000 [7:13:45<19:25:41,  1.89s/it]
Outer training loop:  26%|##5       | 12948/50000 [7:13:47<18:58:02,  1.84s/it]
Outer training loop:  26%|##5       | 12949/50000 [7:13:48<18:31:51,  1.80s/it]
Outer training loop:  26%|##5       | 12950/50000 [7:13:50<19:29:19,  1.89s/it]
Outer training loop:  26%|##5       | 12951/50000 [7:13:52<19:03:13,  1.85s/it]
Outer training loop:  26%|##5       | 12952/50000 [7:13:54<19:02:17,  1.85s/it]
Outer training loop:  26%|##5       | 12953/50000 [7:13:56<19:18:49,  1.88s/it]
Outer training loop:  26%|##5       | 12954/50000 [7:13:58<19:24:49,  1.89s/it]
Outer training loop:  26%|##5       | 12955/50000 [7:14:00<18:57:29,  1.84s/it]
Outer training loop:  26%|##5       | 12956/50000 [7:14:01<18:50:10,  1.83s/it]
Outer training loop:  26%|##5       | 12957/50000 [7:14:03<19:19:42,  1.88s/it]
Outer training loop:  26%|##5       | 12958/50000 [7:14:05<19:17:44,  1.88s/it]
Outer training loop:  26%|##5       | 12959/50000 [7:14:07<18:59:23,  1.85s/it]
Outer training loop:  26%|##5       | 12960/50000 [7:14:09<19:37:49,  1.91s/it]
Outer training loop:  26%|##5       | 12961/50000 [7:14:11<19:59:49,  1.94s/it]
Outer training loop:  26%|##5       | 12962/50000 [7:14:13<20:02:20,  1.95s/it]
Outer training loop:  26%|##5       | 12963/50000 [7:14:15<20:24:15,  1.98s/it]
Outer training loop:  26%|##5       | 12964/50000 [7:14:17<19:44:27,  1.92s/it]
Outer training loop:  26%|##5       | 12965/50000 [7:14:19<19:42:49,  1.92s/it]
Outer training loop:  26%|##5       | 12966/50000 [7:14:21<20:18:20,  1.97s/it]
Outer training loop:  26%|##5       | 12967/50000 [7:14:23<19:50:31,  1.93s/it]
Outer training loop:  26%|##5       | 12968/50000 [7:14:25<19:21:15,  1.88s/it]
Outer training loop:  26%|##5       | 12969/50000 [7:14:26<19:13:10,  1.87s/it]
Outer training loop:  26%|##5       | 12970/50000 [7:14:28<19:47:24,  1.92s/it]
Outer training loop:  26%|##5       | 12971/50000 [7:14:30<19:34:29,  1.90s/it]
Outer training loop:  26%|##5       | 12972/50000 [7:14:32<19:39:34,  1.91s/it]
Outer training loop:  26%|##5       | 12973/50000 [7:14:34<20:05:59,  1.95s/it]
Outer training loop:  26%|##5       | 12974/50000 [7:14:36<20:02:06,  1.95s/it]
Outer training loop:  26%|##5       | 12975/50000 [7:14:38<19:26:18,  1.89s/it]
Outer training loop:  26%|##5       | 12976/50000 [7:14:40<19:52:04,  1.93s/it]
Outer training loop:  26%|##5       | 12977/50000 [7:14:42<19:39:40,  1.91s/it]
Outer training loop:  26%|##5       | 12978/50000 [7:14:44<19:22:57,  1.88s/it]
Outer training loop:  26%|##5       | 12979/50000 [7:14:46<19:45:11,  1.92s/it]
Outer training loop:  26%|##5       | 12980/50000 [7:14:47<19:21:55,  1.88s/it]
Outer training loop:  26%|##5       | 12981/50000 [7:14:49<19:11:09,  1.87s/it]
Outer training loop:  26%|##5       | 12982/50000 [7:14:51<18:58:12,  1.84s/it]
Outer training loop:  26%|##5       | 12983/50000 [7:14:53<19:34:17,  1.90s/it]
Outer training loop:  26%|##5       | 12984/50000 [7:14:55<19:18:05,  1.88s/it]
Outer training loop:  26%|##5       | 12985/50000 [7:14:57<18:46:32,  1.83s/it]
Outer training loop:  26%|##5       | 12986/50000 [7:14:59<19:17:34,  1.88s/it]
Outer training loop:  26%|##5       | 12987/50000 [7:15:00<18:57:58,  1.84s/it]
Outer training loop:  26%|##5       | 12988/50000 [7:15:02<19:16:38,  1.88s/it]
Outer training loop:  26%|##5       | 12989/50000 [7:15:04<18:51:12,  1.83s/it]
Outer training loop:  26%|##5       | 12990/50000 [7:15:06<19:18:43,  1.88s/it]
Outer training loop:  26%|##5       | 12991/50000 [7:15:08<18:52:35,  1.84s/it]
Outer training loop:  26%|##5       | 12992/50000 [7:15:10<18:50:50,  1.83s/it]
Outer training loop:  26%|##5       | 12993/50000 [7:15:11<18:57:54,  1.84s/it]
Outer training loop:  26%|##5       | 12994/50000 [7:15:13<18:45:47,  1.83s/it]
Outer training loop:  26%|##5       | 12995/50000 [7:15:15<18:22:08,  1.79s/it]
Outer training loop:  26%|##5       | 12996/50000 [7:15:17<19:10:02,  1.86s/it]
Outer training loop:  26%|##5       | 12997/50000 [7:15:19<19:10:10,  1.86s/it]
Outer training loop:  26%|##5       | 12998/50000 [7:15:21<19:05:39,  1.86s/it]
Outer training loop:  26%|##5       | 12999/50000 [7:15:23<19:28:02,  1.89s/it]
Outer training loop:  26%|##6       | 13000/50000 [7:15:25<19:32:54,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:31,  1.85s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:31,  1.87s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:34,  1.93s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.89s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.89s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.93s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:28,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:25,  1.96s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:20,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:17,  1.93s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:18,  1.97s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:14,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.90s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:54,  1.92s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:44,  1.84s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:38,  1.86s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:38,  1.97s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:32,  1.89s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:27,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:21,  1.81s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:25,  1.94s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.86s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.87s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:13,  1.84s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:11,  1.83s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.91s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:07,  1.88s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<00:59,  1.81s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:57,  1.81s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:54,  1.81s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:55,  1.90s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:52,  1.89s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.93s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.90s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:40,  1.84s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:40,  1.94s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.88s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:28,  1.80s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.81s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:26,  2.06s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:23,  1.97s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:21,  1.92s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:12,  1.85s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:10,  1.83s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.82s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:08,  2.02s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.93s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.81s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:18,  1.68s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:17,  1.70s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:18,  1.73s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:19,  1.77s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:16,  1.75s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:29,  1.94s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:17,  1.83s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:13,  1.80s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:10,  1.79s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:14,  1.87s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.83s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.79s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:08,  1.86s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:04,  1.83s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:02,  1.82s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.80s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:56,  1.85s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:51,  1.79s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:47,  1.76s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:47,  1.78s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:54,  1.94s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:50,  1.91s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:49<01:44,  1.87s/it]

Evaluating policy:  34%|###3      | 28/83 [00:51<01:43,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:53<01:44,  1.94s/it]

Evaluating policy:  36%|###6      | 30/83 [00:55<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [00:57<01:40,  1.93s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:34,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:02<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:34,  1.97s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:32,  1.97s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:30,  1.96s/it]

Evaluating policy:  46%|####5     | 38/83 [01:10<01:25,  1.91s/it]

Evaluating policy:  47%|####6     | 39/83 [01:12<01:22,  1.88s/it]

Evaluating policy:  48%|####8     | 40/83 [01:14<01:21,  1.89s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:23,  2.00s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:22,  2.00s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:17,  1.94s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:13,  1.93s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:10,  1.90s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:08,  1.92s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:09,  2.00s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:06,  1.96s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.92s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:56,  1.83s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:54,  1.82s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.81s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:50,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:52,  1.96s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.93s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:48,  1.95s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:45,  1.90s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:37,  1.96s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:34,  1.90s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:32,  1.90s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:21,  1.82s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:21,  1.94s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:19,  1.91s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:17,  1.94s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.91s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.87s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.83s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.85s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.85s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.99s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.95s/it]

Outer training loop:  26%|##6       | 13001/50000 [7:20:45<1000:26:53, 97.34s/it]
Outer training loop:  26%|##6       | 13002/50000 [7:20:46<705:53:30, 68.69s/it] 
Outer training loop:  26%|##6       | 13003/50000 [7:20:48<499:27:16, 48.60s/it]
Outer training loop:  26%|##6       | 13004/50000 [7:20:50<355:19:44, 34.58s/it]
Outer training loop:  26%|##6       | 13005/50000 [7:20:52<254:18:25, 24.75s/it]
Outer training loop:  26%|##6       | 13006/50000 [7:20:54<183:47:11, 17.88s/it]
Outer training loop:  26%|##6       | 13007/50000 [7:20:56<134:03:05, 13.05s/it]
Outer training loop:  26%|##6       | 13008/50000 [7:20:57<99:39:40,  9.70s/it] 
Outer training loop:  26%|##6       | 13009/50000 [7:20:59<75:24:43,  7.34s/it]
Outer training loop:  26%|##6       | 13010/50000 [7:21:02<60:22:29,  5.88s/it]
Outer training loop:  26%|##6       | 13011/50000 [7:21:04<48:03:43,  4.68s/it]
Outer training loop:  26%|##6       | 13012/50000 [7:21:05<39:10:59,  3.81s/it]
Outer training loop:  26%|##6       | 13013/50000 [7:21:07<33:28:47,  3.26s/it]
Outer training loop:  26%|##6       | 13014/50000 [7:21:09<28:47:55,  2.80s/it]
Outer training loop:  26%|##6       | 13015/50000 [7:21:11<25:44:15,  2.51s/it]
Outer training loop:  26%|##6       | 13016/50000 [7:21:13<23:35:56,  2.30s/it]
Outer training loop:  26%|##6       | 13017/50000 [7:21:15<22:33:05,  2.20s/it]
Outer training loop:  26%|##6       | 13018/50000 [7:21:17<21:46:16,  2.12s/it]
Outer training loop:  26%|##6       | 13019/50000 [7:21:18<20:49:30,  2.03s/it]
Outer training loop:  26%|##6       | 13020/50000 [7:21:21<21:13:55,  2.07s/it]
Outer training loop:  26%|##6       | 13021/50000 [7:21:22<20:25:25,  1.99s/it]
Outer training loop:  26%|##6       | 13022/50000 [7:21:24<19:45:09,  1.92s/it]
Outer training loop:  26%|##6       | 13023/50000 [7:21:26<19:52:09,  1.93s/it]
Outer training loop:  26%|##6       | 13024/50000 [7:21:28<19:32:34,  1.90s/it]
Outer training loop:  26%|##6       | 13025/50000 [7:21:30<19:33:18,  1.90s/it]
Outer training loop:  26%|##6       | 13026/50000 [7:21:32<19:19:56,  1.88s/it]
Outer training loop:  26%|##6       | 13027/50000 [7:21:34<19:47:06,  1.93s/it]
Outer training loop:  26%|##6       | 13028/50000 [7:21:36<19:29:19,  1.90s/it]
Outer training loop:  26%|##6       | 13029/50000 [7:21:37<19:08:04,  1.86s/it]
Outer training loop:  26%|##6       | 13030/50000 [7:21:39<19:23:45,  1.89s/it]
Outer training loop:  26%|##6       | 13031/50000 [7:21:41<19:01:00,  1.85s/it]
Outer training loop:  26%|##6       | 13032/50000 [7:21:43<19:05:32,  1.86s/it]
Outer training loop:  26%|##6       | 13033/50000 [7:21:45<19:16:24,  1.88s/it]
Outer training loop:  26%|##6       | 13034/50000 [7:21:47<18:52:08,  1.84s/it]
Outer training loop:  26%|##6       | 13035/50000 [7:21:48<18:52:53,  1.84s/it]
Outer training loop:  26%|##6       | 13036/50000 [7:21:51<19:52:28,  1.94s/it]
Outer training loop:  26%|##6       | 13037/50000 [7:21:52<19:17:19,  1.88s/it]
Outer training loop:  26%|##6       | 13038/50000 [7:21:54<19:24:51,  1.89s/it]
Outer training loop:  26%|##6       | 13039/50000 [7:21:56<19:06:39,  1.86s/it]
Outer training loop:  26%|##6       | 13040/50000 [7:21:58<19:31:44,  1.90s/it]
Outer training loop:  26%|##6       | 13041/50000 [7:22:00<19:29:08,  1.90s/it]
Outer training loop:  26%|##6       | 13042/50000 [7:22:02<20:04:10,  1.95s/it]
Outer training loop:  26%|##6       | 13043/50000 [7:22:04<20:33:16,  2.00s/it]
Outer training loop:  26%|##6       | 13044/50000 [7:22:06<20:31:13,  2.00s/it]
Outer training loop:  26%|##6       | 13045/50000 [7:22:08<19:55:35,  1.94s/it]
Outer training loop:  26%|##6       | 13046/50000 [7:22:10<20:13:05,  1.97s/it]
Outer training loop:  26%|##6       | 13047/50000 [7:22:12<19:55:21,  1.94s/it]
Outer training loop:  26%|##6       | 13048/50000 [7:22:14<19:32:02,  1.90s/it]
Outer training loop:  26%|##6       | 13049/50000 [7:22:15<19:12:27,  1.87s/it]
Outer training loop:  26%|##6       | 13050/50000 [7:22:17<19:35:58,  1.91s/it]
Outer training loop:  26%|##6       | 13051/50000 [7:22:19<19:34:35,  1.91s/it]
Outer training loop:  26%|##6       | 13052/50000 [7:22:21<19:14:47,  1.88s/it]
Outer training loop:  26%|##6       | 13053/50000 [7:22:23<19:50:57,  1.93s/it]
Outer training loop:  26%|##6       | 13054/50000 [7:22:25<19:28:09,  1.90s/it]
Outer training loop:  26%|##6       | 13055/50000 [7:22:27<19:15:40,  1.88s/it]
Outer training loop:  26%|##6       | 13056/50000 [7:22:29<19:09:04,  1.87s/it]
Outer training loop:  26%|##6       | 13057/50000 [7:22:31<19:44:35,  1.92s/it]
Outer training loop:  26%|##6       | 13058/50000 [7:22:33<19:16:31,  1.88s/it]
Outer training loop:  26%|##6       | 13059/50000 [7:22:34<18:53:50,  1.84s/it]
Outer training loop:  26%|##6       | 13060/50000 [7:22:36<19:27:16,  1.90s/it]
Outer training loop:  26%|##6       | 13061/50000 [7:22:38<19:18:55,  1.88s/it]
Outer training loop:  26%|##6       | 13062/50000 [7:22:40<19:25:03,  1.89s/it]
Outer training loop:  26%|##6       | 13063/50000 [7:22:42<19:50:40,  1.93s/it]
Outer training loop:  26%|##6       | 13064/50000 [7:22:44<19:45:16,  1.93s/it]
Outer training loop:  26%|##6       | 13065/50000 [7:22:46<19:35:07,  1.91s/it]
Outer training loop:  26%|##6       | 13066/50000 [7:22:48<19:18:53,  1.88s/it]
Outer training loop:  26%|##6       | 13067/50000 [7:22:50<19:49:01,  1.93s/it]
Outer training loop:  26%|##6       | 13068/50000 [7:22:52<19:23:51,  1.89s/it]
Outer training loop:  26%|##6       | 13069/50000 [7:22:53<19:13:04,  1.87s/it]
Outer training loop:  26%|##6       | 13070/50000 [7:22:55<18:53:33,  1.84s/it]
Outer training loop:  26%|##6       | 13071/50000 [7:22:57<19:27:05,  1.90s/it]
Outer training loop:  26%|##6       | 13072/50000 [7:22:59<19:08:08,  1.87s/it]
Outer training loop:  26%|##6       | 13073/50000 [7:23:01<18:42:35,  1.82s/it]
Outer training loop:  26%|##6       | 13074/50000 [7:23:03<18:48:23,  1.83s/it]
Outer training loop:  26%|##6       | 13075/50000 [7:23:04<18:47:06,  1.83s/it]
Outer training loop:  26%|##6       | 13076/50000 [7:23:06<18:40:49,  1.82s/it]
Outer training loop:  26%|##6       | 13077/50000 [7:23:08<19:22:57,  1.89s/it]
Outer training loop:  26%|##6       | 13078/50000 [7:23:10<19:02:42,  1.86s/it]
Outer training loop:  26%|##6       | 13079/50000 [7:23:12<18:54:45,  1.84s/it]
Outer training loop:  26%|##6       | 13080/50000 [7:23:14<19:12:40,  1.87s/it]
Outer training loop:  26%|##6       | 13081/50000 [7:23:16<18:57:48,  1.85s/it]
Outer training loop:  26%|##6       | 13082/50000 [7:23:17<18:37:58,  1.82s/it]
Outer training loop:  26%|##6       | 13083/50000 [7:23:19<18:54:27,  1.84s/it]
Outer training loop:  26%|##6       | 13084/50000 [7:23:21<19:21:57,  1.89s/it]
Outer training loop:  26%|##6       | 13085/50000 [7:23:23<18:43:40,  1.83s/it]
Outer training loop:  26%|##6       | 13086/50000 [7:23:25<19:08:57,  1.87s/it]
Outer training loop:  26%|##6       | 13087/50000 [7:23:27<19:40:01,  1.92s/it]
Outer training loop:  26%|##6       | 13088/50000 [7:23:29<18:56:12,  1.85s/it]
Outer training loop:  26%|##6       | 13089/50000 [7:23:30<18:56:35,  1.85s/it]
Outer training loop:  26%|##6       | 13090/50000 [7:23:32<18:43:39,  1.83s/it]
Outer training loop:  26%|##6       | 13091/50000 [7:23:34<19:24:06,  1.89s/it]
Outer training loop:  26%|##6       | 13092/50000 [7:23:36<19:53:14,  1.94s/it]
Outer training loop:  26%|##6       | 13093/50000 [7:23:38<19:12:07,  1.87s/it]
Outer training loop:  26%|##6       | 13094/50000 [7:23:40<18:53:58,  1.84s/it]
Outer training loop:  26%|##6       | 13095/50000 [7:23:42<19:28:20,  1.90s/it]
Outer training loop:  26%|##6       | 13096/50000 [7:23:44<19:22:54,  1.89s/it]
Outer training loop:  26%|##6       | 13097/50000 [7:23:46<19:16:07,  1.88s/it]
Outer training loop:  26%|##6       | 13098/50000 [7:23:48<19:55:04,  1.94s/it]
Outer training loop:  26%|##6       | 13099/50000 [7:23:49<19:39:11,  1.92s/it]
Outer training loop:  26%|##6       | 13100/50000 [7:23:51<19:24:36,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  26%|##6       | 13101/50000 [7:23:59<36:24:28,  3.55s/it]
Outer training loop:  26%|##6       | 13102/50000 [7:24:01<32:20:29,  3.16s/it]
Outer training loop:  26%|##6       | 13103/50000 [7:24:03<28:09:48,  2.75s/it]
Outer training loop:  26%|##6       | 13104/50000 [7:24:05<25:30:31,  2.49s/it]
Outer training loop:  26%|##6       | 13105/50000 [7:24:07<23:49:44,  2.33s/it]
Outer training loop:  26%|##6       | 13106/50000 [7:24:08<22:34:26,  2.20s/it]
Outer training loop:  26%|##6       | 13107/50000 [7:24:10<21:27:25,  2.09s/it]
Outer training loop:  26%|##6       | 13108/50000 [7:24:12<20:44:48,  2.02s/it]
Outer training loop:  26%|##6       | 13109/50000 [7:24:14<20:17:50,  1.98s/it]
Outer training loop:  26%|##6       | 13110/50000 [7:24:16<19:44:31,  1.93s/it]
Outer training loop:  26%|##6       | 13111/50000 [7:24:18<19:36:35,  1.91s/it]
Outer training loop:  26%|##6       | 13112/50000 [7:24:20<20:07:35,  1.96s/it]
Outer training loop:  26%|##6       | 13113/50000 [7:24:22<19:35:22,  1.91s/it]
Outer training loop:  26%|##6       | 13114/50000 [7:24:23<19:20:34,  1.89s/it]
Outer training loop:  26%|##6       | 13115/50000 [7:24:25<19:27:10,  1.90s/it]
Outer training loop:  26%|##6       | 13116/50000 [7:24:27<19:23:13,  1.89s/it]
Outer training loop:  26%|##6       | 13117/50000 [7:24:29<18:47:23,  1.83s/it]
Outer training loop:  26%|##6       | 13118/50000 [7:24:31<18:51:26,  1.84s/it]
Outer training loop:  26%|##6       | 13119/50000 [7:24:33<19:13:31,  1.88s/it]
Outer training loop:  26%|##6       | 13120/50000 [7:24:35<19:22:10,  1.89s/it]
Outer training loop:  26%|##6       | 13121/50000 [7:24:37<19:16:50,  1.88s/it]
Outer training loop:  26%|##6       | 13122/50000 [7:24:39<19:32:41,  1.91s/it]
Outer training loop:  26%|##6       | 13123/50000 [7:24:41<19:44:10,  1.93s/it]
Outer training loop:  26%|##6       | 13124/50000 [7:24:42<19:24:01,  1.89s/it]
Outer training loop:  26%|##6       | 13125/50000 [7:24:44<19:44:10,  1.93s/it]
Outer training loop:  26%|##6       | 13126/50000 [7:24:46<19:26:37,  1.90s/it]
Outer training loop:  26%|##6       | 13127/50000 [7:24:48<19:05:53,  1.86s/it]
Outer training loop:  26%|##6       | 13128/50000 [7:24:50<18:58:46,  1.85s/it]
Outer training loop:  26%|##6       | 13129/50000 [7:24:52<19:30:58,  1.91s/it]
Outer training loop:  26%|##6       | 13130/50000 [7:24:54<19:21:04,  1.89s/it]
Outer training loop:  26%|##6       | 13131/50000 [7:24:55<19:03:04,  1.86s/it]
Outer training loop:  26%|##6       | 13132/50000 [7:24:57<19:37:24,  1.92s/it]
Outer training loop:  26%|##6       | 13133/50000 [7:24:59<19:33:39,  1.91s/it]
Outer training loop:  26%|##6       | 13134/50000 [7:25:01<19:00:18,  1.86s/it]
Outer training loop:  26%|##6       | 13135/50000 [7:25:03<19:30:46,  1.91s/it]
Outer training loop:  26%|##6       | 13136/50000 [7:25:05<19:07:07,  1.87s/it]
Outer training loop:  26%|##6       | 13137/50000 [7:25:07<19:16:01,  1.88s/it]
Outer training loop:  26%|##6       | 13138/50000 [7:25:09<19:19:48,  1.89s/it]
Outer training loop:  26%|##6       | 13139/50000 [7:25:11<20:00:20,  1.95s/it]
Outer training loop:  26%|##6       | 13140/50000 [7:25:13<19:20:49,  1.89s/it]
Outer training loop:  26%|##6       | 13141/50000 [7:25:14<19:07:40,  1.87s/it]
Outer training loop:  26%|##6       | 13142/50000 [7:25:16<18:39:58,  1.82s/it]
Outer training loop:  26%|##6       | 13143/50000 [7:25:18<19:17:09,  1.88s/it]
Outer training loop:  26%|##6       | 13144/50000 [7:25:20<19:10:27,  1.87s/it]
Outer training loop:  26%|##6       | 13145/50000 [7:25:22<18:55:34,  1.85s/it]
Outer training loop:  26%|##6       | 13146/50000 [7:25:24<19:23:18,  1.89s/it]
Outer training loop:  26%|##6       | 13147/50000 [7:25:26<19:07:42,  1.87s/it]
Outer training loop:  26%|##6       | 13148/50000 [7:25:27<19:01:43,  1.86s/it]
Outer training loop:  26%|##6       | 13149/50000 [7:25:29<19:09:47,  1.87s/it]
Outer training loop:  26%|##6       | 13150/50000 [7:25:31<19:04:20,  1.86s/it]
Outer training loop:  26%|##6       | 13151/50000 [7:25:33<18:44:23,  1.83s/it]
Outer training loop:  26%|##6       | 13152/50000 [7:25:35<18:52:22,  1.84s/it]
Outer training loop:  26%|##6       | 13153/50000 [7:25:37<19:37:16,  1.92s/it]
Outer training loop:  26%|##6       | 13154/50000 [7:25:39<19:19:03,  1.89s/it]
Outer training loop:  26%|##6       | 13155/50000 [7:25:40<19:00:10,  1.86s/it]
Outer training loop:  26%|##6       | 13156/50000 [7:25:42<19:00:49,  1.86s/it]
Outer training loop:  26%|##6       | 13157/50000 [7:25:44<19:10:25,  1.87s/it]
Outer training loop:  26%|##6       | 13158/50000 [7:25:46<18:59:27,  1.86s/it]
Outer training loop:  26%|##6       | 13159/50000 [7:25:48<18:37:37,  1.82s/it]
Outer training loop:  26%|##6       | 13160/50000 [7:25:50<19:16:01,  1.88s/it]
Outer training loop:  26%|##6       | 13161/50000 [7:25:52<18:55:56,  1.85s/it]
Outer training loop:  26%|##6       | 13162/50000 [7:25:53<18:42:16,  1.83s/it]
Outer training loop:  26%|##6       | 13163/50000 [7:25:55<19:29:06,  1.90s/it]
Outer training loop:  26%|##6       | 13164/50000 [7:25:57<19:00:42,  1.86s/it]
Outer training loop:  26%|##6       | 13165/50000 [7:25:59<18:40:43,  1.83s/it]
Outer training loop:  26%|##6       | 13166/50000 [7:26:01<18:04:36,  1.77s/it]
Outer training loop:  26%|##6       | 13167/50000 [7:26:02<18:26:43,  1.80s/it]
Outer training loop:  26%|##6       | 13168/50000 [7:26:04<18:28:25,  1.81s/it]
Outer training loop:  26%|##6       | 13169/50000 [7:26:06<18:28:19,  1.81s/it]
Outer training loop:  26%|##6       | 13170/50000 [7:26:08<19:00:18,  1.86s/it]
Outer training loop:  26%|##6       | 13171/50000 [7:26:10<19:05:05,  1.87s/it]
Outer training loop:  26%|##6       | 13172/50000 [7:26:12<18:43:39,  1.83s/it]
Outer training loop:  26%|##6       | 13173/50000 [7:26:14<19:18:41,  1.89s/it]
Outer training loop:  26%|##6       | 13174/50000 [7:26:15<18:50:59,  1.84s/it]
Outer training loop:  26%|##6       | 13175/50000 [7:26:17<18:47:03,  1.84s/it]
Outer training loop:  26%|##6       | 13176/50000 [7:26:19<18:50:12,  1.84s/it]
Outer training loop:  26%|##6       | 13177/50000 [7:26:21<18:28:44,  1.81s/it]
Outer training loop:  26%|##6       | 13178/50000 [7:26:23<18:14:06,  1.78s/it]
Outer training loop:  26%|##6       | 13179/50000 [7:26:24<18:05:58,  1.77s/it]
Outer training loop:  26%|##6       | 13180/50000 [7:26:26<18:21:59,  1.80s/it]
Outer training loop:  26%|##6       | 13181/50000 [7:26:28<18:07:50,  1.77s/it]
Outer training loop:  26%|##6       | 13182/50000 [7:26:30<17:54:51,  1.75s/it]
Outer training loop:  26%|##6       | 13183/50000 [7:26:32<18:26:46,  1.80s/it]
Outer training loop:  26%|##6       | 13184/50000 [7:26:33<18:06:44,  1.77s/it]
Outer training loop:  26%|##6       | 13185/50000 [7:26:35<18:25:09,  1.80s/it]
Outer training loop:  26%|##6       | 13186/50000 [7:26:37<18:10:01,  1.78s/it]
Outer training loop:  26%|##6       | 13187/50000 [7:26:39<18:34:30,  1.82s/it]
Outer training loop:  26%|##6       | 13188/50000 [7:26:40<18:03:55,  1.77s/it]
Outer training loop:  26%|##6       | 13189/50000 [7:26:42<18:33:08,  1.81s/it]
Outer training loop:  26%|##6       | 13190/50000 [7:26:44<18:58:35,  1.86s/it]
Outer training loop:  26%|##6       | 13191/50000 [7:26:46<18:46:36,  1.84s/it]
Outer training loop:  26%|##6       | 13192/50000 [7:26:48<18:37:49,  1.82s/it]
Outer training loop:  26%|##6       | 13193/50000 [7:26:50<18:29:01,  1.81s/it]
Outer training loop:  26%|##6       | 13194/50000 [7:26:52<18:57:24,  1.85s/it]
Outer training loop:  26%|##6       | 13195/50000 [7:26:53<18:42:06,  1.83s/it]
Outer training loop:  26%|##6       | 13196/50000 [7:26:55<18:49:37,  1.84s/it]
Outer training loop:  26%|##6       | 13197/50000 [7:26:57<18:36:47,  1.82s/it]
Outer training loop:  26%|##6       | 13198/50000 [7:26:59<18:42:12,  1.83s/it]
Outer training loop:  26%|##6       | 13199/50000 [7:27:01<18:47:09,  1.84s/it]
Outer training loop:  26%|##6       | 13200/50000 [7:27:02<18:36:53,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.73s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  26%|##6       | 13201/50000 [7:27:10<35:19:18,  3.46s/it]
Outer training loop:  26%|##6       | 13202/50000 [7:27:11<30:01:44,  2.94s/it]
Outer training loop:  26%|##6       | 13203/50000 [7:27:13<26:32:16,  2.60s/it]
Outer training loop:  26%|##6       | 13204/50000 [7:27:15<24:37:52,  2.41s/it]
Outer training loop:  26%|##6       | 13205/50000 [7:27:17<22:22:20,  2.19s/it]
Outer training loop:  26%|##6       | 13206/50000 [7:27:19<20:46:42,  2.03s/it]
Outer training loop:  26%|##6       | 13207/50000 [7:27:20<20:02:51,  1.96s/it]
Outer training loop:  26%|##6       | 13208/50000 [7:27:22<19:50:12,  1.94s/it]
Outer training loop:  26%|##6       | 13209/50000 [7:27:24<19:05:58,  1.87s/it]
Outer training loop:  26%|##6       | 13210/50000 [7:27:26<18:56:22,  1.85s/it]
Outer training loop:  26%|##6       | 13211/50000 [7:27:28<19:04:36,  1.87s/it]
Outer training loop:  26%|##6       | 13212/50000 [7:27:30<20:01:13,  1.96s/it]
Outer training loop:  26%|##6       | 13213/50000 [7:27:32<19:23:24,  1.90s/it]
Outer training loop:  26%|##6       | 13214/50000 [7:27:33<18:56:46,  1.85s/it]
Outer training loop:  26%|##6       | 13215/50000 [7:27:35<19:19:42,  1.89s/it]
Outer training loop:  26%|##6       | 13216/50000 [7:27:37<18:47:06,  1.84s/it]
Outer training loop:  26%|##6       | 13217/50000 [7:27:39<18:15:05,  1.79s/it]
Outer training loop:  26%|##6       | 13218/50000 [7:27:41<18:43:39,  1.83s/it]
Outer training loop:  26%|##6       | 13219/50000 [7:27:43<18:57:13,  1.86s/it]
Outer training loop:  26%|##6       | 13220/50000 [7:27:44<18:48:31,  1.84s/it]
Outer training loop:  26%|##6       | 13221/50000 [7:27:46<18:20:52,  1.80s/it]
Outer training loop:  26%|##6       | 13222/50000 [7:27:48<18:42:22,  1.83s/it]
Outer training loop:  26%|##6       | 13223/50000 [7:27:50<18:26:16,  1.80s/it]
Outer training loop:  26%|##6       | 13224/50000 [7:27:51<18:14:11,  1.79s/it]
Outer training loop:  26%|##6       | 13225/50000 [7:27:53<18:42:42,  1.83s/it]
Outer training loop:  26%|##6       | 13226/50000 [7:27:55<18:39:59,  1.83s/it]
Outer training loop:  26%|##6       | 13227/50000 [7:27:57<18:12:44,  1.78s/it]
Outer training loop:  26%|##6       | 13228/50000 [7:27:59<18:37:49,  1.82s/it]
Outer training loop:  26%|##6       | 13229/50000 [7:28:00<18:03:23,  1.77s/it]
Outer training loop:  26%|##6       | 13230/50000 [7:28:02<18:09:14,  1.78s/it]
Outer training loop:  26%|##6       | 13231/50000 [7:28:04<18:52:33,  1.85s/it]
Outer training loop:  26%|##6       | 13232/50000 [7:28:06<18:24:05,  1.80s/it]
Outer training loop:  26%|##6       | 13233/50000 [7:28:08<18:14:51,  1.79s/it]
Outer training loop:  26%|##6       | 13234/50000 [7:28:09<17:59:51,  1.76s/it]
Outer training loop:  26%|##6       | 13235/50000 [7:28:11<18:28:25,  1.81s/it]
Outer training loop:  26%|##6       | 13236/50000 [7:28:13<18:47:01,  1.84s/it]
Outer training loop:  26%|##6       | 13237/50000 [7:28:15<18:21:44,  1.80s/it]
Outer training loop:  26%|##6       | 13238/50000 [7:28:17<18:50:29,  1.85s/it]
Outer training loop:  26%|##6       | 13239/50000 [7:28:19<18:31:37,  1.81s/it]
Outer training loop:  26%|##6       | 13240/50000 [7:28:21<18:41:24,  1.83s/it]
Outer training loop:  26%|##6       | 13241/50000 [7:28:22<18:22:52,  1.80s/it]
Outer training loop:  26%|##6       | 13242/50000 [7:28:24<19:08:24,  1.87s/it]
Outer training loop:  26%|##6       | 13243/50000 [7:28:26<18:50:08,  1.84s/it]
Outer training loop:  26%|##6       | 13244/50000 [7:28:28<18:24:37,  1.80s/it]
Outer training loop:  26%|##6       | 13245/50000 [7:28:30<18:44:14,  1.84s/it]
Outer training loop:  26%|##6       | 13246/50000 [7:28:31<18:31:05,  1.81s/it]
Outer training loop:  26%|##6       | 13247/50000 [7:28:33<18:28:26,  1.81s/it]
Outer training loop:  26%|##6       | 13248/50000 [7:28:35<18:56:55,  1.86s/it]
Outer training loop:  26%|##6       | 13249/50000 [7:28:37<18:24:23,  1.80s/it]
Outer training loop:  26%|##6       | 13250/50000 [7:28:39<18:08:25,  1.78s/it]
Outer training loop:  27%|##6       | 13251/50000 [7:28:41<18:47:01,  1.84s/it]
Outer training loop:  27%|##6       | 13252/50000 [7:28:42<18:17:16,  1.79s/it]
Outer training loop:  27%|##6       | 13253/50000 [7:28:44<17:47:11,  1.74s/it]
Outer training loop:  27%|##6       | 13254/50000 [7:28:46<17:55:22,  1.76s/it]
Outer training loop:  27%|##6       | 13255/50000 [7:28:48<18:26:40,  1.81s/it]
Outer training loop:  27%|##6       | 13256/50000 [7:28:49<18:31:03,  1.81s/it]
Outer training loop:  27%|##6       | 13257/50000 [7:28:51<18:30:19,  1.81s/it]
Outer training loop:  27%|##6       | 13258/50000 [7:28:53<18:43:46,  1.84s/it]
Outer training loop:  27%|##6       | 13259/50000 [7:28:55<18:27:16,  1.81s/it]
Outer training loop:  27%|##6       | 13260/50000 [7:28:57<18:31:09,  1.81s/it]
Outer training loop:  27%|##6       | 13261/50000 [7:28:59<18:59:12,  1.86s/it]
Outer training loop:  27%|##6       | 13262/50000 [7:29:00<18:27:37,  1.81s/it]
Outer training loop:  27%|##6       | 13263/50000 [7:29:02<18:17:38,  1.79s/it]
Outer training loop:  27%|##6       | 13264/50000 [7:29:04<17:52:57,  1.75s/it]
Outer training loop:  27%|##6       | 13265/50000 [7:29:06<18:26:28,  1.81s/it]
Outer training loop:  27%|##6       | 13266/50000 [7:29:08<18:20:11,  1.80s/it]
Outer training loop:  27%|##6       | 13267/50000 [7:29:09<18:21:01,  1.80s/it]
Outer training loop:  27%|##6       | 13268/50000 [7:29:11<18:36:56,  1.82s/it]
Outer training loop:  27%|##6       | 13269/50000 [7:29:13<18:20:56,  1.80s/it]
Outer training loop:  27%|##6       | 13270/50000 [7:29:15<18:09:15,  1.78s/it]
Outer training loop:  27%|##6       | 13271/50000 [7:29:16<17:50:17,  1.75s/it]
Outer training loop:  27%|##6       | 13272/50000 [7:29:18<18:13:54,  1.79s/it]
Outer training loop:  27%|##6       | 13273/50000 [7:29:20<18:03:26,  1.77s/it]
Outer training loop:  27%|##6       | 13274/50000 [7:29:22<17:49:45,  1.75s/it]
Outer training loop:  27%|##6       | 13275/50000 [7:29:24<18:23:24,  1.80s/it]
Outer training loop:  27%|##6       | 13276/50000 [7:29:25<18:22:36,  1.80s/it]
Outer training loop:  27%|##6       | 13277/50000 [7:29:27<18:36:17,  1.82s/it]
Outer training loop:  27%|##6       | 13278/50000 [7:29:29<18:43:52,  1.84s/it]
Outer training loop:  27%|##6       | 13279/50000 [7:29:31<18:16:26,  1.79s/it]
Outer training loop:  27%|##6       | 13280/50000 [7:29:33<18:04:06,  1.77s/it]
Outer training loop:  27%|##6       | 13281/50000 [7:29:34<17:52:08,  1.75s/it]
Outer training loop:  27%|##6       | 13282/50000 [7:29:36<18:24:41,  1.81s/it]
Outer training loop:  27%|##6       | 13283/50000 [7:29:38<18:02:10,  1.77s/it]
Outer training loop:  27%|##6       | 13284/50000 [7:29:40<17:55:22,  1.76s/it]
Outer training loop:  27%|##6       | 13285/50000 [7:29:41<18:15:40,  1.79s/it]
Outer training loop:  27%|##6       | 13286/50000 [7:29:43<18:42:30,  1.83s/it]
Outer training loop:  27%|##6       | 13287/50000 [7:29:45<18:36:25,  1.82s/it]
Outer training loop:  27%|##6       | 13288/50000 [7:29:47<18:17:19,  1.79s/it]
Outer training loop:  27%|##6       | 13289/50000 [7:29:49<18:56:26,  1.86s/it]
Outer training loop:  27%|##6       | 13290/50000 [7:29:51<18:36:03,  1.82s/it]
Outer training loop:  27%|##6       | 13291/50000 [7:29:52<18:10:56,  1.78s/it]
Outer training loop:  27%|##6       | 13292/50000 [7:29:54<18:51:07,  1.85s/it]
Outer training loop:  27%|##6       | 13293/50000 [7:29:56<18:12:44,  1.79s/it]
Outer training loop:  27%|##6       | 13294/50000 [7:29:58<18:17:48,  1.79s/it]
Outer training loop:  27%|##6       | 13295/50000 [7:30:00<18:50:00,  1.85s/it]
Outer training loop:  27%|##6       | 13296/50000 [7:30:02<18:45:59,  1.84s/it]
Outer training loop:  27%|##6       | 13297/50000 [7:30:03<18:28:17,  1.81s/it]
Outer training loop:  27%|##6       | 13298/50000 [7:30:05<18:25:18,  1.81s/it]
Outer training loop:  27%|##6       | 13299/50000 [7:30:07<18:32:16,  1.82s/it]
Outer training loop:  27%|##6       | 13300/50000 [7:30:09<18:31:02,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  27%|##6       | 13301/50000 [7:30:16<34:48:23,  3.41s/it]
Outer training loop:  27%|##6       | 13302/50000 [7:30:18<30:04:17,  2.95s/it]
Outer training loop:  27%|##6       | 13303/50000 [7:30:20<27:11:41,  2.67s/it]
Outer training loop:  27%|##6       | 13304/50000 [7:30:22<24:16:32,  2.38s/it]
Outer training loop:  27%|##6       | 13305/50000 [7:30:23<22:19:00,  2.19s/it]
Outer training loop:  27%|##6       | 13306/50000 [7:30:25<21:05:08,  2.07s/it]
Outer training loop:  27%|##6       | 13307/50000 [7:30:27<21:01:32,  2.06s/it]
Outer training loop:  27%|##6       | 13308/50000 [7:30:29<20:04:48,  1.97s/it]
Outer training loop:  27%|##6       | 13309/50000 [7:30:31<19:33:33,  1.92s/it]
Outer training loop:  27%|##6       | 13310/50000 [7:30:33<19:37:38,  1.93s/it]
Outer training loop:  27%|##6       | 13311/50000 [7:30:34<18:55:47,  1.86s/it]
Outer training loop:  27%|##6       | 13312/50000 [7:30:36<18:51:18,  1.85s/it]
Outer training loop:  27%|##6       | 13313/50000 [7:30:38<18:23:17,  1.80s/it]
Outer training loop:  27%|##6       | 13314/50000 [7:30:40<18:45:53,  1.84s/it]
Outer training loop:  27%|##6       | 13315/50000 [7:30:42<18:41:04,  1.83s/it]
Outer training loop:  27%|##6       | 13316/50000 [7:30:43<18:38:42,  1.83s/it]
Outer training loop:  27%|##6       | 13317/50000 [7:30:46<19:27:48,  1.91s/it]
Outer training loop:  27%|##6       | 13318/50000 [7:30:47<19:02:26,  1.87s/it]
Outer training loop:  27%|##6       | 13319/50000 [7:30:49<19:02:24,  1.87s/it]
Outer training loop:  27%|##6       | 13320/50000 [7:30:51<19:14:44,  1.89s/it]
Outer training loop:  27%|##6       | 13321/50000 [7:30:53<18:39:28,  1.83s/it]
Outer training loop:  27%|##6       | 13322/50000 [7:30:55<18:33:14,  1.82s/it]
Outer training loop:  27%|##6       | 13323/50000 [7:30:56<18:22:57,  1.80s/it]
Outer training loop:  27%|##6       | 13324/50000 [7:30:58<18:56:57,  1.86s/it]
Outer training loop:  27%|##6       | 13325/50000 [7:31:00<18:15:22,  1.79s/it]
Outer training loop:  27%|##6       | 13326/50000 [7:31:02<17:51:15,  1.75s/it]
Outer training loop:  27%|##6       | 13327/50000 [7:31:04<18:33:56,  1.82s/it]
Outer training loop:  27%|##6       | 13328/50000 [7:31:05<18:22:25,  1.80s/it]
Outer training loop:  27%|##6       | 13329/50000 [7:31:07<17:59:02,  1.77s/it]
Outer training loop:  27%|##6       | 13330/50000 [7:31:09<17:37:28,  1.73s/it]
Outer training loop:  27%|##6       | 13331/50000 [7:31:10<17:49:23,  1.75s/it]
Outer training loop:  27%|##6       | 13332/50000 [7:31:12<17:54:47,  1.76s/it]
Outer training loop:  27%|##6       | 13333/50000 [7:31:14<17:54:54,  1.76s/it]
Outer training loop:  27%|##6       | 13334/50000 [7:31:16<18:30:18,  1.82s/it]
Outer training loop:  27%|##6       | 13335/50000 [7:31:18<18:25:12,  1.81s/it]
Outer training loop:  27%|##6       | 13336/50000 [7:31:20<18:16:52,  1.80s/it]
Outer training loop:  27%|##6       | 13337/50000 [7:31:21<18:43:02,  1.84s/it]
Outer training loop:  27%|##6       | 13338/50000 [7:31:23<18:26:22,  1.81s/it]
Outer training loop:  27%|##6       | 13339/50000 [7:31:25<18:35:56,  1.83s/it]
Outer training loop:  27%|##6       | 13340/50000 [7:31:27<18:22:34,  1.80s/it]
Outer training loop:  27%|##6       | 13341/50000 [7:31:29<18:43:09,  1.84s/it]
Outer training loop:  27%|##6       | 13342/50000 [7:31:31<18:40:17,  1.83s/it]
Outer training loop:  27%|##6       | 13343/50000 [7:31:32<18:11:53,  1.79s/it]
Outer training loop:  27%|##6       | 13344/50000 [7:31:34<18:43:12,  1.84s/it]
Outer training loop:  27%|##6       | 13345/50000 [7:31:36<18:45:12,  1.84s/it]
Outer training loop:  27%|##6       | 13346/50000 [7:31:38<18:49:28,  1.85s/it]
Outer training loop:  27%|##6       | 13347/50000 [7:31:40<19:22:48,  1.90s/it]
Outer training loop:  27%|##6       | 13348/50000 [7:31:42<18:57:04,  1.86s/it]
Outer training loop:  27%|##6       | 13349/50000 [7:31:43<18:39:55,  1.83s/it]
Outer training loop:  27%|##6       | 13350/50000 [7:31:45<18:19:52,  1.80s/it]
Outer training loop:  27%|##6       | 13351/50000 [7:31:47<18:37:29,  1.83s/it]
Outer training loop:  27%|##6       | 13352/50000 [7:31:49<18:18:41,  1.80s/it]
Outer training loop:  27%|##6       | 13353/50000 [7:31:51<18:19:49,  1.80s/it]
Outer training loop:  27%|##6       | 13354/50000 [7:31:53<18:56:20,  1.86s/it]
Outer training loop:  27%|##6       | 13355/50000 [7:31:54<18:44:21,  1.84s/it]
Outer training loop:  27%|##6       | 13356/50000 [7:31:56<18:08:12,  1.78s/it]
Outer training loop:  27%|##6       | 13357/50000 [7:31:58<18:26:51,  1.81s/it]
Outer training loop:  27%|##6       | 13358/50000 [7:32:00<17:59:01,  1.77s/it]
Outer training loop:  27%|##6       | 13359/50000 [7:32:01<17:57:47,  1.76s/it]
Outer training loop:  27%|##6       | 13360/50000 [7:32:03<18:03:39,  1.77s/it]
Outer training loop:  27%|##6       | 13361/50000 [7:32:05<18:36:27,  1.83s/it]
Outer training loop:  27%|##6       | 13362/50000 [7:32:07<18:06:46,  1.78s/it]
Outer training loop:  27%|##6       | 13363/50000 [7:32:09<17:55:12,  1.76s/it]
Outer training loop:  27%|##6       | 13364/50000 [7:32:10<18:32:43,  1.82s/it]
Outer training loop:  27%|##6       | 13365/50000 [7:32:12<18:13:48,  1.79s/it]
Outer training loop:  27%|##6       | 13366/50000 [7:32:14<18:31:17,  1.82s/it]
Outer training loop:  27%|##6       | 13367/50000 [7:32:16<18:44:03,  1.84s/it]
Outer training loop:  27%|##6       | 13368/50000 [7:32:18<18:14:08,  1.79s/it]
Outer training loop:  27%|##6       | 13369/50000 [7:32:19<18:12:09,  1.79s/it]
Outer training loop:  27%|##6       | 13370/50000 [7:32:21<18:40:35,  1.84s/it]
Outer training loop:  27%|##6       | 13371/50000 [7:32:23<18:07:58,  1.78s/it]
Outer training loop:  27%|##6       | 13372/50000 [7:32:25<17:37:06,  1.73s/it]
Outer training loop:  27%|##6       | 13373/50000 [7:32:27<18:20:30,  1.80s/it]
Outer training loop:  27%|##6       | 13374/50000 [7:32:28<18:19:12,  1.80s/it]
Outer training loop:  27%|##6       | 13375/50000 [7:32:30<17:54:00,  1.76s/it]
Outer training loop:  27%|##6       | 13376/50000 [7:32:32<17:43:15,  1.74s/it]
Outer training loop:  27%|##6       | 13377/50000 [7:32:34<17:51:11,  1.75s/it]
Outer training loop:  27%|##6       | 13378/50000 [7:32:35<17:55:22,  1.76s/it]
Outer training loop:  27%|##6       | 13379/50000 [7:32:37<17:51:54,  1.76s/it]
Outer training loop:  27%|##6       | 13380/50000 [7:32:39<18:30:10,  1.82s/it]
Outer training loop:  27%|##6       | 13381/50000 [7:32:41<18:10:01,  1.79s/it]
Outer training loop:  27%|##6       | 13382/50000 [7:32:43<18:05:31,  1.78s/it]
Outer training loop:  27%|##6       | 13383/50000 [7:32:44<18:34:43,  1.83s/it]
Outer training loop:  27%|##6       | 13384/50000 [7:32:46<19:04:46,  1.88s/it]
Outer training loop:  27%|##6       | 13385/50000 [7:32:48<18:56:35,  1.86s/it]
Outer training loop:  27%|##6       | 13386/50000 [7:32:50<18:30:30,  1.82s/it]
Outer training loop:  27%|##6       | 13387/50000 [7:32:52<18:42:28,  1.84s/it]
Outer training loop:  27%|##6       | 13388/50000 [7:32:54<18:18:37,  1.80s/it]
Outer training loop:  27%|##6       | 13389/50000 [7:32:55<18:22:34,  1.81s/it]
Outer training loop:  27%|##6       | 13390/50000 [7:32:57<19:01:18,  1.87s/it]
Outer training loop:  27%|##6       | 13391/50000 [7:32:59<18:29:59,  1.82s/it]
Outer training loop:  27%|##6       | 13392/50000 [7:33:01<18:14:49,  1.79s/it]
Outer training loop:  27%|##6       | 13393/50000 [7:33:03<17:51:59,  1.76s/it]
Outer training loop:  27%|##6       | 13394/50000 [7:33:05<18:45:54,  1.85s/it]
Outer training loop:  27%|##6       | 13395/50000 [7:33:06<18:37:59,  1.83s/it]
Outer training loop:  27%|##6       | 13396/50000 [7:33:08<18:26:13,  1.81s/it]
Outer training loop:  27%|##6       | 13397/50000 [7:33:10<18:30:58,  1.82s/it]
Outer training loop:  27%|##6       | 13398/50000 [7:33:12<18:33:23,  1.83s/it]
Outer training loop:  27%|##6       | 13399/50000 [7:33:14<18:14:09,  1.79s/it]
Outer training loop:  27%|##6       | 13400/50000 [7:33:15<18:08:26,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  27%|##6       | 13401/50000 [7:33:23<35:05:03,  3.45s/it]
Outer training loop:  27%|##6       | 13402/50000 [7:33:25<30:09:35,  2.97s/it]
Outer training loop:  27%|##6       | 13403/50000 [7:33:26<26:41:05,  2.62s/it]
Outer training loop:  27%|##6       | 13404/50000 [7:33:28<24:46:02,  2.44s/it]
Outer training loop:  27%|##6       | 13405/50000 [7:33:30<22:38:32,  2.23s/it]
Outer training loop:  27%|##6       | 13406/50000 [7:33:32<21:34:33,  2.12s/it]
Outer training loop:  27%|##6       | 13407/50000 [7:33:34<20:26:54,  2.01s/it]
Outer training loop:  27%|##6       | 13408/50000 [7:33:36<20:30:56,  2.02s/it]
Outer training loop:  27%|##6       | 13409/50000 [7:33:37<19:41:46,  1.94s/it]
Outer training loop:  27%|##6       | 13410/50000 [7:33:39<18:58:11,  1.87s/it]
Outer training loop:  27%|##6       | 13411/50000 [7:33:41<19:03:13,  1.87s/it]
Outer training loop:  27%|##6       | 13412/50000 [7:33:43<18:42:24,  1.84s/it]
Outer training loop:  27%|##6       | 13413/50000 [7:33:45<18:27:42,  1.82s/it]
Outer training loop:  27%|##6       | 13414/50000 [7:33:46<18:14:19,  1.79s/it]
Outer training loop:  27%|##6       | 13415/50000 [7:33:48<18:50:59,  1.85s/it]
Outer training loop:  27%|##6       | 13416/50000 [7:33:50<18:28:52,  1.82s/it]
Outer training loop:  27%|##6       | 13417/50000 [7:33:52<18:12:17,  1.79s/it]
Outer training loop:  27%|##6       | 13418/50000 [7:33:54<18:43:03,  1.84s/it]
Outer training loop:  27%|##6       | 13419/50000 [7:33:56<18:29:24,  1.82s/it]
Outer training loop:  27%|##6       | 13420/50000 [7:33:57<18:40:32,  1.84s/it]
Outer training loop:  27%|##6       | 13421/50000 [7:33:59<18:38:12,  1.83s/it]
Outer training loop:  27%|##6       | 13422/50000 [7:34:01<18:54:01,  1.86s/it]
Outer training loop:  27%|##6       | 13423/50000 [7:34:03<18:27:44,  1.82s/it]
Outer training loop:  27%|##6       | 13424/50000 [7:34:05<18:13:49,  1.79s/it]
Outer training loop:  27%|##6       | 13425/50000 [7:34:07<18:49:38,  1.85s/it]
Outer training loop:  27%|##6       | 13426/50000 [7:34:08<18:19:34,  1.80s/it]
Outer training loop:  27%|##6       | 13427/50000 [7:34:10<18:31:56,  1.82s/it]
Outer training loop:  27%|##6       | 13428/50000 [7:34:12<18:15:03,  1.80s/it]
Outer training loop:  27%|##6       | 13429/50000 [7:34:14<18:37:13,  1.83s/it]
Outer training loop:  27%|##6       | 13430/50000 [7:34:16<18:34:13,  1.83s/it]
Outer training loop:  27%|##6       | 13431/50000 [7:34:17<18:29:37,  1.82s/it]
Outer training loop:  27%|##6       | 13432/50000 [7:34:19<18:36:45,  1.83s/it]
Outer training loop:  27%|##6       | 13433/50000 [7:34:21<18:25:25,  1.81s/it]
Outer training loop:  27%|##6       | 13434/50000 [7:34:23<18:05:20,  1.78s/it]
Outer training loop:  27%|##6       | 13435/50000 [7:34:25<18:36:46,  1.83s/it]
Outer training loop:  27%|##6       | 13436/50000 [7:34:26<18:25:57,  1.81s/it]
Outer training loop:  27%|##6       | 13437/50000 [7:34:28<17:57:50,  1.77s/it]
Outer training loop:  27%|##6       | 13438/50000 [7:34:30<17:48:16,  1.75s/it]
Outer training loop:  27%|##6       | 13439/50000 [7:34:32<18:38:37,  1.84s/it]
Outer training loop:  27%|##6       | 13440/50000 [7:34:34<18:23:26,  1.81s/it]
Outer training loop:  27%|##6       | 13441/50000 [7:34:35<18:07:38,  1.79s/it]
Outer training loop:  27%|##6       | 13442/50000 [7:34:37<18:39:35,  1.84s/it]
Outer training loop:  27%|##6       | 13443/50000 [7:34:39<18:25:44,  1.81s/it]
Outer training loop:  27%|##6       | 13444/50000 [7:34:41<18:29:37,  1.82s/it]
Outer training loop:  27%|##6       | 13445/50000 [7:34:43<18:37:17,  1.83s/it]
Outer training loop:  27%|##6       | 13446/50000 [7:34:45<18:33:27,  1.83s/it]
Outer training loop:  27%|##6       | 13447/50000 [7:34:46<18:03:43,  1.78s/it]
Outer training loop:  27%|##6       | 13448/50000 [7:34:48<17:43:36,  1.75s/it]
Outer training loop:  27%|##6       | 13449/50000 [7:34:50<18:20:19,  1.81s/it]
Outer training loop:  27%|##6       | 13450/50000 [7:34:52<18:19:23,  1.80s/it]
Outer training loop:  27%|##6       | 13451/50000 [7:34:53<18:00:21,  1.77s/it]
Outer training loop:  27%|##6       | 13452/50000 [7:34:55<18:30:26,  1.82s/it]
Outer training loop:  27%|##6       | 13453/50000 [7:34:57<18:24:28,  1.81s/it]
Outer training loop:  27%|##6       | 13454/50000 [7:34:59<18:04:48,  1.78s/it]
Outer training loop:  27%|##6       | 13455/50000 [7:35:01<17:45:57,  1.75s/it]
Outer training loop:  27%|##6       | 13456/50000 [7:35:02<18:20:38,  1.81s/it]
Outer training loop:  27%|##6       | 13457/50000 [7:35:04<18:01:59,  1.78s/it]
Outer training loop:  27%|##6       | 13458/50000 [7:35:06<17:49:36,  1.76s/it]
Outer training loop:  27%|##6       | 13459/50000 [7:35:08<18:08:35,  1.79s/it]
Outer training loop:  27%|##6       | 13460/50000 [7:35:10<18:25:55,  1.82s/it]
Outer training loop:  27%|##6       | 13461/50000 [7:35:11<18:07:25,  1.79s/it]
Outer training loop:  27%|##6       | 13462/50000 [7:35:13<18:02:33,  1.78s/it]
Outer training loop:  27%|##6       | 13463/50000 [7:35:15<18:27:02,  1.82s/it]
Outer training loop:  27%|##6       | 13464/50000 [7:35:17<18:16:30,  1.80s/it]
Outer training loop:  27%|##6       | 13465/50000 [7:35:19<18:11:44,  1.79s/it]
Outer training loop:  27%|##6       | 13466/50000 [7:35:20<18:42:07,  1.84s/it]
Outer training loop:  27%|##6       | 13467/50000 [7:35:22<18:28:06,  1.82s/it]
Outer training loop:  27%|##6       | 13468/50000 [7:35:24<18:10:05,  1.79s/it]
Outer training loop:  27%|##6       | 13469/50000 [7:35:26<18:44:13,  1.85s/it]
Outer training loop:  27%|##6       | 13470/50000 [7:35:28<18:25:34,  1.82s/it]
Outer training loop:  27%|##6       | 13471/50000 [7:35:30<18:43:38,  1.85s/it]
Outer training loop:  27%|##6       | 13472/50000 [7:35:32<18:53:38,  1.86s/it]
Outer training loop:  27%|##6       | 13473/50000 [7:35:33<19:16:18,  1.90s/it]
Outer training loop:  27%|##6       | 13474/50000 [7:35:35<18:58:55,  1.87s/it]
Outer training loop:  27%|##6       | 13475/50000 [7:35:37<18:32:02,  1.83s/it]
Outer training loop:  27%|##6       | 13476/50000 [7:35:39<18:43:12,  1.85s/it]
Outer training loop:  27%|##6       | 13477/50000 [7:35:41<18:30:13,  1.82s/it]
Outer training loop:  27%|##6       | 13478/50000 [7:35:42<18:16:28,  1.80s/it]
Outer training loop:  27%|##6       | 13479/50000 [7:35:44<17:57:29,  1.77s/it]
Outer training loop:  27%|##6       | 13480/50000 [7:35:46<18:39:13,  1.84s/it]
Outer training loop:  27%|##6       | 13481/50000 [7:35:48<18:22:21,  1.81s/it]
Outer training loop:  27%|##6       | 13482/50000 [7:35:50<17:57:47,  1.77s/it]
Outer training loop:  27%|##6       | 13483/50000 [7:35:52<18:43:39,  1.85s/it]
Outer training loop:  27%|##6       | 13484/50000 [7:35:53<18:24:01,  1.81s/it]
Outer training loop:  27%|##6       | 13485/50000 [7:35:55<18:06:26,  1.79s/it]
Outer training loop:  27%|##6       | 13486/50000 [7:35:57<18:42:07,  1.84s/it]
Outer training loop:  27%|##6       | 13487/50000 [7:35:59<18:41:35,  1.84s/it]
Outer training loop:  27%|##6       | 13488/50000 [7:36:01<18:13:46,  1.80s/it]
Outer training loop:  27%|##6       | 13489/50000 [7:36:02<18:14:13,  1.80s/it]
Outer training loop:  27%|##6       | 13490/50000 [7:36:04<18:36:08,  1.83s/it]
Outer training loop:  27%|##6       | 13491/50000 [7:36:06<18:36:48,  1.84s/it]
Outer training loop:  27%|##6       | 13492/50000 [7:36:08<18:30:44,  1.83s/it]
Outer training loop:  27%|##6       | 13493/50000 [7:36:10<19:24:07,  1.91s/it]
Outer training loop:  27%|##6       | 13494/50000 [7:36:12<18:58:45,  1.87s/it]
Outer training loop:  27%|##6       | 13495/50000 [7:36:14<18:35:59,  1.83s/it]
Outer training loop:  27%|##6       | 13496/50000 [7:36:16<18:58:10,  1.87s/it]
Outer training loop:  27%|##6       | 13497/50000 [7:36:17<18:40:00,  1.84s/it]
Outer training loop:  27%|##6       | 13498/50000 [7:36:19<18:28:38,  1.82s/it]
Outer training loop:  27%|##6       | 13499/50000 [7:36:21<18:20:57,  1.81s/it]
Outer training loop:  27%|##7       | 13500/50000 [7:36:23<18:52:17,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:26,  1.79s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:29,  1.87s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:24,  1.83s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:23,  1.84s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:21,  1.83s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:25,  1.92s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:17,  1.86s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:15,  1.85s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:12,  1.87s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:08,  1.83s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:05,  1.81s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:02,  1.84s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<01:59,  1.81s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:55,  1.77s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:56,  1.81s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:54,  1.85s/it]

Evaluating policy:  27%|##6       | 22/83 [00:40<01:52,  1.84s/it]

Evaluating policy:  28%|##7       | 23/83 [00:42<01:50,  1.84s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:49,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:45,  1.88s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:41,  1.85s/it]

Evaluating policy:  35%|###4      | 29/83 [00:53<01:37,  1.80s/it]

Evaluating policy:  36%|###6      | 30/83 [00:55<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [00:57<01:39,  1.91s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:36,  1.89s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:34,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:32,  1.92s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:28,  1.88s/it]

Evaluating policy:  45%|####4     | 37/83 [01:08<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:10<01:21,  1.80s/it]

Evaluating policy:  47%|####6     | 39/83 [01:12<01:20,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:14<01:16,  1.79s/it]

Evaluating policy:  49%|####9     | 41/83 [01:15<01:14,  1.77s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:19<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:21<01:12,  1.85s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:23<01:09,  1.83s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:25<01:07,  1.82s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:27<01:04,  1.80s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:28<01:01,  1.77s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:30<01:04,  1.89s/it]

Evaluating policy:  60%|######    | 50/83 [01:32<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:34<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:36<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:37<00:53,  1.80s/it]

Evaluating policy:  65%|######5   | 54/83 [01:39<00:51,  1.78s/it]

Evaluating policy:  66%|######6   | 55/83 [01:41<00:49,  1.76s/it]

Evaluating policy:  67%|######7   | 56/83 [01:43<00:47,  1.78s/it]

Evaluating policy:  69%|######8   | 57/83 [01:45<00:49,  1.91s/it]

Evaluating policy:  70%|######9   | 58/83 [01:47<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:48<00:44,  1.83s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:50<00:42,  1.86s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:52<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:54<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:56<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:58<00:37,  1.95s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:00<00:33,  1.87s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:01<00:30,  1.82s/it]

Evaluating policy:  81%|########  | 67/83 [02:03<00:28,  1.80s/it]

Evaluating policy:  82%|########1 | 68/83 [02:05<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:07<00:25,  1.79s/it]

Evaluating policy:  84%|########4 | 70/83 [02:09<00:23,  1.84s/it]

Evaluating policy:  86%|########5 | 71/83 [02:11<00:21,  1.81s/it]

Evaluating policy:  87%|########6 | 72/83 [02:12<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:15<00:19,  1.94s/it]

Evaluating policy:  89%|########9 | 74/83 [02:16<00:17,  1.90s/it]

Evaluating policy:  90%|######### | 75/83 [02:18<00:15,  1.89s/it]

Evaluating policy:  92%|#########1| 76/83 [02:20<00:13,  1.94s/it]

Evaluating policy:  93%|#########2| 77/83 [02:22<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:24<00:09,  1.83s/it]

Evaluating policy:  95%|#########5| 79/83 [02:26<00:07,  1.87s/it]

Evaluating policy:  96%|#########6| 80/83 [02:28<00:05,  1.88s/it]

Evaluating policy:  98%|#########7| 81/83 [02:29<00:03,  1.84s/it]

Evaluating policy:  99%|#########8| 82/83 [02:32<00:02,  2.03s/it]

Evaluating policy: 100%|##########| 83/83 [02:34<00:00,  1.93s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:28,  1.82s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.82s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:26,  1.83s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:26,  1.85s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:21,  1.81s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:21,  1.84s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.84s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:17,  1.83s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:15,  1.83s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:25,  2.00s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:21,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:14,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:04,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:54,  1.85s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:49,  1.79s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:47,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:45,  1.84s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:41,  1.81s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:57<01:35,  1.84s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:32,  1.81s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:30,  1.81s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:32,  1.90s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:27,  1.86s/it]

Evaluating policy:  45%|####4     | 37/83 [01:08<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:10<01:20,  1.79s/it]

Evaluating policy:  47%|####6     | 39/83 [01:12<01:18,  1.77s/it]

Evaluating policy:  48%|####8     | 40/83 [01:14<01:21,  1.88s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:18,  1.88s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:16,  1.86s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:16,  1.91s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:23<01:11,  1.87s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:12,  1.96s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:10,  1.96s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:29<01:06,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:03,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<01:00,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:58,  1.84s/it]

Evaluating policy:  63%|######2   | 52/83 [01:36<00:55,  1.80s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:57,  1.93s/it]

Evaluating policy:  65%|######5   | 54/83 [01:40<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:42<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:46,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:49<00:43,  1.83s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:51<00:41,  1.81s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:53<00:42,  1.93s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:55<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:57<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:34,  1.81s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:02<00:31,  1.82s/it]

Evaluating policy:  81%|########  | 67/83 [02:04<00:29,  1.82s/it]

Evaluating policy:  82%|########1 | 68/83 [02:06<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:08<00:26,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:10<00:25,  1.98s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:22,  1.91s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:17<00:16,  1.81s/it]

Evaluating policy:  90%|######### | 75/83 [02:19<00:14,  1.82s/it]

Evaluating policy:  92%|#########1| 76/83 [02:21<00:12,  1.81s/it]

Evaluating policy:  93%|#########2| 77/83 [02:23<00:10,  1.79s/it]

Evaluating policy:  94%|#########3| 78/83 [02:24<00:08,  1.75s/it]

Evaluating policy:  95%|#########5| 79/83 [02:27<00:07,  1.93s/it]

Evaluating policy:  96%|#########6| 80/83 [02:28<00:05,  1.88s/it]

Evaluating policy:  98%|#########7| 81/83 [02:30<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:32<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:34<00:00,  1.85s/it]

Outer training loop:  27%|##7       | 13501/50000 [7:41:39<974:20:58, 96.10s/it]
Outer training loop:  27%|##7       | 13502/50000 [7:41:41<687:12:58, 67.78s/it]
Outer training loop:  27%|##7       | 13503/50000 [7:41:42<486:22:32, 47.98s/it]
Outer training loop:  27%|##7       | 13504/50000 [7:41:44<345:40:37, 34.10s/it]
Outer training loop:  27%|##7       | 13505/50000 [7:41:46<247:22:53, 24.40s/it]
Outer training loop:  27%|##7       | 13506/50000 [7:41:48<180:31:26, 17.81s/it]
Outer training loop:  27%|##7       | 13507/50000 [7:41:50<132:16:34, 13.05s/it]
Outer training loop:  27%|##7       | 13508/50000 [7:41:52<98:10:52,  9.69s/it] 
Outer training loop:  27%|##7       | 13509/50000 [7:41:54<73:47:16,  7.28s/it]
Outer training loop:  27%|##7       | 13510/50000 [7:41:56<57:30:43,  5.67s/it]
Outer training loop:  27%|##7       | 13511/50000 [7:41:58<46:08:37,  4.55s/it]
Outer training loop:  27%|##7       | 13512/50000 [7:41:59<37:40:42,  3.72s/it]
Outer training loop:  27%|##7       | 13513/50000 [7:42:01<32:04:44,  3.17s/it]
Outer training loop:  27%|##7       | 13514/50000 [7:42:03<28:03:45,  2.77s/it]
Outer training loop:  27%|##7       | 13515/50000 [7:42:05<25:39:35,  2.53s/it]
Outer training loop:  27%|##7       | 13516/50000 [7:42:07<24:09:23,  2.38s/it]
Outer training loop:  27%|##7       | 13517/50000 [7:42:09<22:36:09,  2.23s/it]
Outer training loop:  27%|##7       | 13518/50000 [7:42:11<21:25:36,  2.11s/it]
Outer training loop:  27%|##7       | 13519/50000 [7:42:13<20:29:11,  2.02s/it]
Outer training loop:  27%|##7       | 13520/50000 [7:42:14<20:16:31,  2.00s/it]
Outer training loop:  27%|##7       | 13521/50000 [7:42:16<19:41:41,  1.94s/it]
Outer training loop:  27%|##7       | 13522/50000 [7:42:18<19:05:16,  1.88s/it]
Outer training loop:  27%|##7       | 13523/50000 [7:42:20<19:48:13,  1.95s/it]
Outer training loop:  27%|##7       | 13524/50000 [7:42:22<19:33:46,  1.93s/it]
Outer training loop:  27%|##7       | 13525/50000 [7:42:24<19:20:03,  1.91s/it]
Outer training loop:  27%|##7       | 13526/50000 [7:42:26<18:59:26,  1.87s/it]
Outer training loop:  27%|##7       | 13527/50000 [7:42:28<19:38:53,  1.94s/it]
Outer training loop:  27%|##7       | 13528/50000 [7:42:29<18:58:43,  1.87s/it]
Outer training loop:  27%|##7       | 13529/50000 [7:42:32<19:34:40,  1.93s/it]
Outer training loop:  27%|##7       | 13530/50000 [7:42:34<19:54:23,  1.97s/it]
Outer training loop:  27%|##7       | 13531/50000 [7:42:35<19:28:44,  1.92s/it]
Outer training loop:  27%|##7       | 13532/50000 [7:42:37<18:54:24,  1.87s/it]
Outer training loop:  27%|##7       | 13533/50000 [7:42:39<18:32:11,  1.83s/it]
Outer training loop:  27%|##7       | 13534/50000 [7:42:41<19:07:53,  1.89s/it]
Outer training loop:  27%|##7       | 13535/50000 [7:42:43<18:51:27,  1.86s/it]
Outer training loop:  27%|##7       | 13536/50000 [7:42:44<18:28:20,  1.82s/it]
Outer training loop:  27%|##7       | 13537/50000 [7:42:47<19:12:03,  1.90s/it]
Outer training loop:  27%|##7       | 13538/50000 [7:42:48<18:58:49,  1.87s/it]
Outer training loop:  27%|##7       | 13539/50000 [7:42:50<18:55:19,  1.87s/it]
Outer training loop:  27%|##7       | 13540/50000 [7:42:52<19:10:18,  1.89s/it]
Outer training loop:  27%|##7       | 13541/50000 [7:42:54<19:42:44,  1.95s/it]
Outer training loop:  27%|##7       | 13542/50000 [7:42:56<19:12:03,  1.90s/it]
Outer training loop:  27%|##7       | 13543/50000 [7:42:58<19:41:30,  1.94s/it]
Outer training loop:  27%|##7       | 13544/50000 [7:43:00<19:56:00,  1.97s/it]
Outer training loop:  27%|##7       | 13545/50000 [7:43:02<18:46:18,  1.85s/it]
Outer training loop:  27%|##7       | 13546/50000 [7:43:03<18:37:41,  1.84s/it]
Outer training loop:  27%|##7       | 13547/50000 [7:43:05<18:16:01,  1.80s/it]
Outer training loop:  27%|##7       | 13548/50000 [7:43:07<18:46:02,  1.85s/it]
Outer training loop:  27%|##7       | 13549/50000 [7:43:09<18:18:09,  1.81s/it]
Outer training loop:  27%|##7       | 13550/50000 [7:43:11<18:29:22,  1.83s/it]
Outer training loop:  27%|##7       | 13551/50000 [7:43:13<18:40:33,  1.84s/it]
Outer training loop:  27%|##7       | 13552/50000 [7:43:14<18:28:26,  1.82s/it]
Outer training loop:  27%|##7       | 13553/50000 [7:43:16<18:27:47,  1.82s/it]
Outer training loop:  27%|##7       | 13554/50000 [7:43:18<19:15:00,  1.90s/it]
Outer training loop:  27%|##7       | 13555/50000 [7:43:20<19:12:13,  1.90s/it]
Outer training loop:  27%|##7       | 13556/50000 [7:43:22<18:41:37,  1.85s/it]
Outer training loop:  27%|##7       | 13557/50000 [7:43:24<18:27:16,  1.82s/it]
Outer training loop:  27%|##7       | 13558/50000 [7:43:26<19:10:59,  1.90s/it]
Outer training loop:  27%|##7       | 13559/50000 [7:43:28<19:02:36,  1.88s/it]
Outer training loop:  27%|##7       | 13560/50000 [7:43:29<18:42:56,  1.85s/it]
Outer training loop:  27%|##7       | 13561/50000 [7:43:32<19:46:31,  1.95s/it]
Outer training loop:  27%|##7       | 13562/50000 [7:43:33<19:20:32,  1.91s/it]
Outer training loop:  27%|##7       | 13563/50000 [7:43:35<18:59:25,  1.88s/it]
Outer training loop:  27%|##7       | 13564/50000 [7:43:37<19:05:02,  1.89s/it]
Outer training loop:  27%|##7       | 13565/50000 [7:43:39<18:52:05,  1.86s/it]
Outer training loop:  27%|##7       | 13566/50000 [7:43:41<18:59:05,  1.88s/it]
Outer training loop:  27%|##7       | 13567/50000 [7:43:43<19:30:05,  1.93s/it]
Outer training loop:  27%|##7       | 13568/50000 [7:43:45<19:54:00,  1.97s/it]
Outer training loop:  27%|##7       | 13569/50000 [7:43:47<19:05:05,  1.89s/it]
Outer training loop:  27%|##7       | 13570/50000 [7:43:48<18:45:33,  1.85s/it]
Outer training loop:  27%|##7       | 13571/50000 [7:43:51<19:30:44,  1.93s/it]
Outer training loop:  27%|##7       | 13572/50000 [7:43:52<19:03:36,  1.88s/it]
Outer training loop:  27%|##7       | 13573/50000 [7:43:54<18:42:19,  1.85s/it]
Outer training loop:  27%|##7       | 13574/50000 [7:43:56<18:35:26,  1.84s/it]
Outer training loop:  27%|##7       | 13575/50000 [7:43:58<18:59:42,  1.88s/it]
Outer training loop:  27%|##7       | 13576/50000 [7:44:00<18:40:34,  1.85s/it]
Outer training loop:  27%|##7       | 13577/50000 [7:44:01<18:34:08,  1.84s/it]
Outer training loop:  27%|##7       | 13578/50000 [7:44:03<18:58:58,  1.88s/it]
Outer training loop:  27%|##7       | 13579/50000 [7:44:05<18:46:17,  1.86s/it]
Outer training loop:  27%|##7       | 13580/50000 [7:44:07<18:43:33,  1.85s/it]
Outer training loop:  27%|##7       | 13581/50000 [7:44:09<18:48:08,  1.86s/it]
Outer training loop:  27%|##7       | 13582/50000 [7:44:11<19:26:40,  1.92s/it]
Outer training loop:  27%|##7       | 13583/50000 [7:44:13<19:07:52,  1.89s/it]
Outer training loop:  27%|##7       | 13584/50000 [7:44:15<18:50:53,  1.86s/it]
Outer training loop:  27%|##7       | 13585/50000 [7:44:17<19:24:38,  1.92s/it]
Outer training loop:  27%|##7       | 13586/50000 [7:44:18<19:10:52,  1.90s/it]
Outer training loop:  27%|##7       | 13587/50000 [7:44:20<18:57:39,  1.87s/it]
Outer training loop:  27%|##7       | 13588/50000 [7:44:22<19:03:50,  1.88s/it]
Outer training loop:  27%|##7       | 13589/50000 [7:44:24<18:53:36,  1.87s/it]
Outer training loop:  27%|##7       | 13590/50000 [7:44:26<19:01:37,  1.88s/it]
Outer training loop:  27%|##7       | 13591/50000 [7:44:28<19:22:00,  1.91s/it]
Outer training loop:  27%|##7       | 13592/50000 [7:44:30<19:18:10,  1.91s/it]
Outer training loop:  27%|##7       | 13593/50000 [7:44:32<18:50:13,  1.86s/it]
Outer training loop:  27%|##7       | 13594/50000 [7:44:33<18:34:28,  1.84s/it]
Outer training loop:  27%|##7       | 13595/50000 [7:44:35<18:59:36,  1.88s/it]
Outer training loop:  27%|##7       | 13596/50000 [7:44:37<18:40:14,  1.85s/it]
Outer training loop:  27%|##7       | 13597/50000 [7:44:39<18:46:19,  1.86s/it]
Outer training loop:  27%|##7       | 13598/50000 [7:44:41<19:16:38,  1.91s/it]
Outer training loop:  27%|##7       | 13599/50000 [7:44:43<19:21:02,  1.91s/it]
Outer training loop:  27%|##7       | 13600/50000 [7:44:45<18:54:22,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  27%|##7       | 13601/50000 [7:44:52<35:54:36,  3.55s/it]
Outer training loop:  27%|##7       | 13602/50000 [7:44:54<31:43:06,  3.14s/it]
Outer training loop:  27%|##7       | 13603/50000 [7:44:56<27:15:48,  2.70s/it]
Outer training loop:  27%|##7       | 13604/50000 [7:44:58<24:38:45,  2.44s/it]
Outer training loop:  27%|##7       | 13605/50000 [7:45:00<22:56:43,  2.27s/it]
Outer training loop:  27%|##7       | 13606/50000 [7:45:02<21:27:49,  2.12s/it]
Outer training loop:  27%|##7       | 13607/50000 [7:45:03<20:25:40,  2.02s/it]
Outer training loop:  27%|##7       | 13608/50000 [7:45:05<20:38:09,  2.04s/it]
Outer training loop:  27%|##7       | 13609/50000 [7:45:07<20:24:19,  2.02s/it]
Outer training loop:  27%|##7       | 13610/50000 [7:45:09<19:48:35,  1.96s/it]
Outer training loop:  27%|##7       | 13611/50000 [7:45:11<19:37:04,  1.94s/it]
Outer training loop:  27%|##7       | 13612/50000 [7:45:13<19:27:03,  1.92s/it]
Outer training loop:  27%|##7       | 13613/50000 [7:45:15<19:03:12,  1.89s/it]
Outer training loop:  27%|##7       | 13614/50000 [7:45:17<18:45:54,  1.86s/it]
Outer training loop:  27%|##7       | 13615/50000 [7:45:18<18:50:24,  1.86s/it]
Outer training loop:  27%|##7       | 13616/50000 [7:45:21<19:26:54,  1.92s/it]
Outer training loop:  27%|##7       | 13617/50000 [7:45:22<19:09:53,  1.90s/it]
Outer training loop:  27%|##7       | 13618/50000 [7:45:24<18:59:03,  1.88s/it]
Outer training loop:  27%|##7       | 13619/50000 [7:45:26<19:37:14,  1.94s/it]
Outer training loop:  27%|##7       | 13620/50000 [7:45:28<19:23:39,  1.92s/it]
Outer training loop:  27%|##7       | 13621/50000 [7:45:30<19:07:10,  1.89s/it]
Outer training loop:  27%|##7       | 13622/50000 [7:45:32<19:33:55,  1.94s/it]
Outer training loop:  27%|##7       | 13623/50000 [7:45:34<19:13:49,  1.90s/it]
Outer training loop:  27%|##7       | 13624/50000 [7:45:36<18:52:04,  1.87s/it]
Outer training loop:  27%|##7       | 13625/50000 [7:45:38<19:24:54,  1.92s/it]
Outer training loop:  27%|##7       | 13626/50000 [7:45:40<19:27:39,  1.93s/it]
Outer training loop:  27%|##7       | 13627/50000 [7:45:42<19:35:35,  1.94s/it]
Outer training loop:  27%|##7       | 13628/50000 [7:45:43<19:30:30,  1.93s/it]
Outer training loop:  27%|##7       | 13629/50000 [7:45:45<19:41:55,  1.95s/it]
Outer training loop:  27%|##7       | 13630/50000 [7:45:47<19:36:40,  1.94s/it]
Outer training loop:  27%|##7       | 13631/50000 [7:45:49<19:23:26,  1.92s/it]
Outer training loop:  27%|##7       | 13632/50000 [7:45:51<19:45:25,  1.96s/it]
Outer training loop:  27%|##7       | 13633/50000 [7:45:53<19:27:55,  1.93s/it]
Outer training loop:  27%|##7       | 13634/50000 [7:45:55<18:56:15,  1.87s/it]
Outer training loop:  27%|##7       | 13635/50000 [7:45:57<18:15:57,  1.81s/it]
Outer training loop:  27%|##7       | 13636/50000 [7:45:59<18:44:42,  1.86s/it]
Outer training loop:  27%|##7       | 13637/50000 [7:46:00<18:12:20,  1.80s/it]
Outer training loop:  27%|##7       | 13638/50000 [7:46:02<17:48:23,  1.76s/it]
Outer training loop:  27%|##7       | 13639/50000 [7:46:04<18:20:06,  1.82s/it]
Outer training loop:  27%|##7       | 13640/50000 [7:46:06<18:25:33,  1.82s/it]
Outer training loop:  27%|##7       | 13641/50000 [7:46:08<18:32:51,  1.84s/it]
Outer training loop:  27%|##7       | 13642/50000 [7:46:09<18:26:35,  1.83s/it]
Outer training loop:  27%|##7       | 13643/50000 [7:46:11<18:56:13,  1.88s/it]
Outer training loop:  27%|##7       | 13644/50000 [7:46:13<18:49:28,  1.86s/it]
Outer training loop:  27%|##7       | 13645/50000 [7:46:15<18:54:35,  1.87s/it]
Outer training loop:  27%|##7       | 13646/50000 [7:46:17<19:26:19,  1.92s/it]
Outer training loop:  27%|##7       | 13647/50000 [7:46:19<19:06:51,  1.89s/it]
Outer training loop:  27%|##7       | 13648/50000 [7:46:21<18:59:54,  1.88s/it]
Outer training loop:  27%|##7       | 13649/50000 [7:46:23<19:31:53,  1.93s/it]
Outer training loop:  27%|##7       | 13650/50000 [7:46:25<19:28:28,  1.93s/it]
Outer training loop:  27%|##7       | 13651/50000 [7:46:27<19:13:51,  1.90s/it]
Outer training loop:  27%|##7       | 13652/50000 [7:46:28<18:59:47,  1.88s/it]
Outer training loop:  27%|##7       | 13653/50000 [7:46:31<19:51:47,  1.97s/it]
Outer training loop:  27%|##7       | 13654/50000 [7:46:32<19:31:20,  1.93s/it]
Outer training loop:  27%|##7       | 13655/50000 [7:46:34<19:01:19,  1.88s/it]
Outer training loop:  27%|##7       | 13656/50000 [7:46:36<19:20:48,  1.92s/it]
Outer training loop:  27%|##7       | 13657/50000 [7:46:38<18:59:39,  1.88s/it]
Outer training loop:  27%|##7       | 13658/50000 [7:46:40<19:04:16,  1.89s/it]
Outer training loop:  27%|##7       | 13659/50000 [7:46:42<19:33:54,  1.94s/it]
Outer training loop:  27%|##7       | 13660/50000 [7:46:44<19:23:23,  1.92s/it]
Outer training loop:  27%|##7       | 13661/50000 [7:46:46<19:11:11,  1.90s/it]
Outer training loop:  27%|##7       | 13662/50000 [7:46:47<18:40:03,  1.85s/it]
Outer training loop:  27%|##7       | 13663/50000 [7:46:49<19:18:40,  1.91s/it]
Outer training loop:  27%|##7       | 13664/50000 [7:46:51<19:21:51,  1.92s/it]
Outer training loop:  27%|##7       | 13665/50000 [7:46:53<19:03:59,  1.89s/it]
Outer training loop:  27%|##7       | 13666/50000 [7:46:55<19:09:25,  1.90s/it]
Outer training loop:  27%|##7       | 13667/50000 [7:46:57<19:58:30,  1.98s/it]
Outer training loop:  27%|##7       | 13668/50000 [7:46:59<19:20:00,  1.92s/it]
Outer training loop:  27%|##7       | 13669/50000 [7:47:01<19:17:40,  1.91s/it]
Outer training loop:  27%|##7       | 13670/50000 [7:47:03<19:44:13,  1.96s/it]
Outer training loop:  27%|##7       | 13671/50000 [7:47:05<19:32:19,  1.94s/it]
Outer training loop:  27%|##7       | 13672/50000 [7:47:07<19:15:10,  1.91s/it]
Outer training loop:  27%|##7       | 13673/50000 [7:47:09<18:58:47,  1.88s/it]
Outer training loop:  27%|##7       | 13674/50000 [7:47:11<19:27:43,  1.93s/it]
Outer training loop:  27%|##7       | 13675/50000 [7:47:13<19:20:31,  1.92s/it]
Outer training loop:  27%|##7       | 13676/50000 [7:47:14<19:07:34,  1.90s/it]
Outer training loop:  27%|##7       | 13677/50000 [7:47:16<19:32:04,  1.94s/it]
Outer training loop:  27%|##7       | 13678/50000 [7:47:18<19:05:28,  1.89s/it]
Outer training loop:  27%|##7       | 13679/50000 [7:47:20<19:12:39,  1.90s/it]
Outer training loop:  27%|##7       | 13680/50000 [7:47:22<19:26:04,  1.93s/it]
Outer training loop:  27%|##7       | 13681/50000 [7:47:24<18:52:18,  1.87s/it]
Outer training loop:  27%|##7       | 13682/50000 [7:47:26<18:47:16,  1.86s/it]
Outer training loop:  27%|##7       | 13683/50000 [7:47:28<18:37:28,  1.85s/it]
Outer training loop:  27%|##7       | 13684/50000 [7:47:30<19:10:30,  1.90s/it]
Outer training loop:  27%|##7       | 13685/50000 [7:47:31<19:01:35,  1.89s/it]
Outer training loop:  27%|##7       | 13686/50000 [7:47:33<18:32:21,  1.84s/it]
Outer training loop:  27%|##7       | 13687/50000 [7:47:35<18:55:47,  1.88s/it]
Outer training loop:  27%|##7       | 13688/50000 [7:47:37<18:29:53,  1.83s/it]
Outer training loop:  27%|##7       | 13689/50000 [7:47:39<18:36:56,  1.85s/it]
Outer training loop:  27%|##7       | 13690/50000 [7:47:41<19:46:18,  1.96s/it]
Outer training loop:  27%|##7       | 13691/50000 [7:47:43<19:30:48,  1.93s/it]
Outer training loop:  27%|##7       | 13692/50000 [7:47:45<18:58:10,  1.88s/it]
Outer training loop:  27%|##7       | 13693/50000 [7:47:46<18:30:27,  1.84s/it]
Outer training loop:  27%|##7       | 13694/50000 [7:47:48<18:54:23,  1.87s/it]
Outer training loop:  27%|##7       | 13695/50000 [7:47:50<18:57:29,  1.88s/it]
Outer training loop:  27%|##7       | 13696/50000 [7:47:52<18:28:16,  1.83s/it]
Outer training loop:  27%|##7       | 13697/50000 [7:47:54<19:04:45,  1.89s/it]
Outer training loop:  27%|##7       | 13698/50000 [7:47:56<18:37:58,  1.85s/it]
Outer training loop:  27%|##7       | 13699/50000 [7:47:58<18:53:02,  1.87s/it]
Outer training loop:  27%|##7       | 13700/50000 [7:48:00<19:13:56,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  27%|##7       | 13701/50000 [7:48:07<35:19:14,  3.50s/it]
Outer training loop:  27%|##7       | 13702/50000 [7:48:09<30:43:39,  3.05s/it]
Outer training loop:  27%|##7       | 13703/50000 [7:48:11<27:27:22,  2.72s/it]
Outer training loop:  27%|##7       | 13704/50000 [7:48:13<25:14:21,  2.50s/it]
Outer training loop:  27%|##7       | 13705/50000 [7:48:15<24:27:36,  2.43s/it]
Outer training loop:  27%|##7       | 13706/50000 [7:48:17<22:29:04,  2.23s/it]
Outer training loop:  27%|##7       | 13707/50000 [7:48:19<21:31:22,  2.13s/it]
Outer training loop:  27%|##7       | 13708/50000 [7:48:21<21:06:09,  2.09s/it]
Outer training loop:  27%|##7       | 13709/50000 [7:48:22<19:57:54,  1.98s/it]
Outer training loop:  27%|##7       | 13710/50000 [7:48:24<19:45:29,  1.96s/it]
Outer training loop:  27%|##7       | 13711/50000 [7:48:26<20:07:35,  2.00s/it]
Outer training loop:  27%|##7       | 13712/50000 [7:48:28<20:14:04,  2.01s/it]
Outer training loop:  27%|##7       | 13713/50000 [7:48:30<19:39:08,  1.95s/it]
Outer training loop:  27%|##7       | 13714/50000 [7:48:32<19:57:08,  1.98s/it]
Outer training loop:  27%|##7       | 13715/50000 [7:48:34<20:25:33,  2.03s/it]
Outer training loop:  27%|##7       | 13716/50000 [7:48:36<19:40:50,  1.95s/it]
Outer training loop:  27%|##7       | 13717/50000 [7:48:38<18:59:06,  1.88s/it]
Outer training loop:  27%|##7       | 13718/50000 [7:48:40<19:13:49,  1.91s/it]
Outer training loop:  27%|##7       | 13719/50000 [7:48:42<18:55:17,  1.88s/it]
Outer training loop:  27%|##7       | 13720/50000 [7:48:43<18:43:07,  1.86s/it]
Outer training loop:  27%|##7       | 13721/50000 [7:48:45<18:43:34,  1.86s/it]
Outer training loop:  27%|##7       | 13722/50000 [7:48:47<18:32:39,  1.84s/it]
Outer training loop:  27%|##7       | 13723/50000 [7:48:49<18:23:03,  1.82s/it]
Outer training loop:  27%|##7       | 13724/50000 [7:48:51<19:10:13,  1.90s/it]
Outer training loop:  27%|##7       | 13725/50000 [7:48:53<18:51:07,  1.87s/it]
Outer training loop:  27%|##7       | 13726/50000 [7:48:54<18:16:31,  1.81s/it]
Outer training loop:  27%|##7       | 13727/50000 [7:48:56<18:01:37,  1.79s/it]
Outer training loop:  27%|##7       | 13728/50000 [7:48:58<18:57:52,  1.88s/it]
Outer training loop:  27%|##7       | 13729/50000 [7:49:00<18:42:44,  1.86s/it]
Outer training loop:  27%|##7       | 13730/50000 [7:49:02<18:33:24,  1.84s/it]
Outer training loop:  27%|##7       | 13731/50000 [7:49:04<18:56:59,  1.88s/it]
Outer training loop:  27%|##7       | 13732/50000 [7:49:06<18:31:43,  1.84s/it]
Outer training loop:  27%|##7       | 13733/50000 [7:49:07<18:22:07,  1.82s/it]
Outer training loop:  27%|##7       | 13734/50000 [7:49:10<19:14:30,  1.91s/it]
Outer training loop:  27%|##7       | 13735/50000 [7:49:11<18:52:22,  1.87s/it]
Outer training loop:  27%|##7       | 13736/50000 [7:49:13<18:27:14,  1.83s/it]
Outer training loop:  27%|##7       | 13737/50000 [7:49:15<18:04:58,  1.80s/it]
Outer training loop:  27%|##7       | 13738/50000 [7:49:17<18:31:11,  1.84s/it]
Outer training loop:  27%|##7       | 13739/50000 [7:49:19<18:29:45,  1.84s/it]
Outer training loop:  27%|##7       | 13740/50000 [7:49:20<17:59:46,  1.79s/it]
Outer training loop:  27%|##7       | 13741/50000 [7:49:22<18:25:10,  1.83s/it]
Outer training loop:  27%|##7       | 13742/50000 [7:49:24<18:22:55,  1.83s/it]
Outer training loop:  27%|##7       | 13743/50000 [7:49:26<18:25:53,  1.83s/it]
Outer training loop:  27%|##7       | 13744/50000 [7:49:28<18:05:47,  1.80s/it]
Outer training loop:  27%|##7       | 13745/50000 [7:49:30<18:50:48,  1.87s/it]
Outer training loop:  27%|##7       | 13746/50000 [7:49:31<18:43:20,  1.86s/it]
Outer training loop:  27%|##7       | 13747/50000 [7:49:33<18:17:50,  1.82s/it]
Outer training loop:  27%|##7       | 13748/50000 [7:49:35<18:30:24,  1.84s/it]
Outer training loop:  27%|##7       | 13749/50000 [7:49:37<18:14:14,  1.81s/it]
Outer training loop:  28%|##7       | 13750/50000 [7:49:39<18:16:35,  1.82s/it]
Outer training loop:  28%|##7       | 13751/50000 [7:49:40<18:24:37,  1.83s/it]
Outer training loop:  28%|##7       | 13752/50000 [7:49:42<18:44:53,  1.86s/it]
Outer training loop:  28%|##7       | 13753/50000 [7:49:44<18:21:36,  1.82s/it]
Outer training loop:  28%|##7       | 13754/50000 [7:49:46<18:23:35,  1.83s/it]
Outer training loop:  28%|##7       | 13755/50000 [7:49:48<19:13:09,  1.91s/it]
Outer training loop:  28%|##7       | 13756/50000 [7:49:50<18:59:22,  1.89s/it]
Outer training loop:  28%|##7       | 13757/50000 [7:49:52<18:28:18,  1.83s/it]
Outer training loop:  28%|##7       | 13758/50000 [7:49:53<18:33:33,  1.84s/it]
Outer training loop:  28%|##7       | 13759/50000 [7:49:55<18:43:02,  1.86s/it]
Outer training loop:  28%|##7       | 13760/50000 [7:49:57<18:18:06,  1.82s/it]
Outer training loop:  28%|##7       | 13761/50000 [7:49:59<18:07:58,  1.80s/it]
Outer training loop:  28%|##7       | 13762/50000 [7:50:01<18:32:35,  1.84s/it]
Outer training loop:  28%|##7       | 13763/50000 [7:50:03<18:33:47,  1.84s/it]
Outer training loop:  28%|##7       | 13764/50000 [7:50:04<18:32:45,  1.84s/it]
Outer training loop:  28%|##7       | 13765/50000 [7:50:06<18:28:39,  1.84s/it]
Outer training loop:  28%|##7       | 13766/50000 [7:50:08<19:09:36,  1.90s/it]
Outer training loop:  28%|##7       | 13767/50000 [7:50:10<18:47:54,  1.87s/it]
Outer training loop:  28%|##7       | 13768/50000 [7:50:12<18:47:11,  1.87s/it]
Outer training loop:  28%|##7       | 13769/50000 [7:50:14<19:00:43,  1.89s/it]
Outer training loop:  28%|##7       | 13770/50000 [7:50:16<18:29:27,  1.84s/it]
Outer training loop:  28%|##7       | 13771/50000 [7:50:17<17:59:30,  1.79s/it]
Outer training loop:  28%|##7       | 13772/50000 [7:50:19<18:14:28,  1.81s/it]
Outer training loop:  28%|##7       | 13773/50000 [7:50:21<18:51:21,  1.87s/it]
Outer training loop:  28%|##7       | 13774/50000 [7:50:23<18:25:31,  1.83s/it]
Outer training loop:  28%|##7       | 13775/50000 [7:50:25<18:11:48,  1.81s/it]
Outer training loop:  28%|##7       | 13776/50000 [7:50:27<18:31:24,  1.84s/it]
Outer training loop:  28%|##7       | 13777/50000 [7:50:28<18:29:15,  1.84s/it]
Outer training loop:  28%|##7       | 13778/50000 [7:50:30<18:28:09,  1.84s/it]
Outer training loop:  28%|##7       | 13779/50000 [7:50:32<18:02:48,  1.79s/it]
Outer training loop:  28%|##7       | 13780/50000 [7:50:34<18:21:46,  1.83s/it]
Outer training loop:  28%|##7       | 13781/50000 [7:50:36<17:55:08,  1.78s/it]
Outer training loop:  28%|##7       | 13782/50000 [7:50:37<17:45:11,  1.76s/it]
Outer training loop:  28%|##7       | 13783/50000 [7:50:39<18:08:26,  1.80s/it]
Outer training loop:  28%|##7       | 13784/50000 [7:50:41<17:43:03,  1.76s/it]
Outer training loop:  28%|##7       | 13785/50000 [7:50:43<18:04:30,  1.80s/it]
Outer training loop:  28%|##7       | 13786/50000 [7:50:44<17:51:41,  1.78s/it]
Outer training loop:  28%|##7       | 13787/50000 [7:50:46<18:27:38,  1.84s/it]
Outer training loop:  28%|##7       | 13788/50000 [7:50:48<18:21:09,  1.82s/it]
Outer training loop:  28%|##7       | 13789/50000 [7:50:50<18:24:42,  1.83s/it]
Outer training loop:  28%|##7       | 13790/50000 [7:50:52<18:10:18,  1.81s/it]
Outer training loop:  28%|##7       | 13791/50000 [7:50:54<18:44:51,  1.86s/it]
Outer training loop:  28%|##7       | 13792/50000 [7:50:56<18:35:52,  1.85s/it]
Outer training loop:  28%|##7       | 13793/50000 [7:50:57<18:27:29,  1.84s/it]
Outer training loop:  28%|##7       | 13794/50000 [7:50:59<18:30:33,  1.84s/it]
Outer training loop:  28%|##7       | 13795/50000 [7:51:01<18:05:32,  1.80s/it]
Outer training loop:  28%|##7       | 13796/50000 [7:51:03<18:12:54,  1.81s/it]
Outer training loop:  28%|##7       | 13797/50000 [7:51:05<18:15:04,  1.81s/it]
Outer training loop:  28%|##7       | 13798/50000 [7:51:07<18:51:55,  1.88s/it]
Outer training loop:  28%|##7       | 13799/50000 [7:51:08<18:31:02,  1.84s/it]
Outer training loop:  28%|##7       | 13800/50000 [7:51:10<18:19:22,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.73s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  28%|##7       | 13801/50000 [7:51:17<34:47:08,  3.46s/it]
Outer training loop:  28%|##7       | 13802/50000 [7:51:19<30:29:49,  3.03s/it]
Outer training loop:  28%|##7       | 13803/50000 [7:51:21<26:42:29,  2.66s/it]
Outer training loop:  28%|##7       | 13804/50000 [7:51:23<24:00:59,  2.39s/it]
Outer training loop:  28%|##7       | 13805/50000 [7:51:25<22:34:42,  2.25s/it]
Outer training loop:  28%|##7       | 13806/50000 [7:51:27<20:56:34,  2.08s/it]
Outer training loop:  28%|##7       | 13807/50000 [7:51:28<19:58:01,  1.99s/it]
Outer training loop:  28%|##7       | 13808/50000 [7:51:30<19:40:34,  1.96s/it]
Outer training loop:  28%|##7       | 13809/50000 [7:51:32<19:09:58,  1.91s/it]
Outer training loop:  28%|##7       | 13810/50000 [7:51:34<18:44:34,  1.86s/it]
Outer training loop:  28%|##7       | 13811/50000 [7:51:36<18:37:28,  1.85s/it]
Outer training loop:  28%|##7       | 13812/50000 [7:51:38<18:43:45,  1.86s/it]
Outer training loop:  28%|##7       | 13813/50000 [7:51:39<17:56:08,  1.78s/it]
Outer training loop:  28%|##7       | 13814/50000 [7:51:41<17:55:51,  1.78s/it]
Outer training loop:  28%|##7       | 13815/50000 [7:51:43<18:36:02,  1.85s/it]
Outer training loop:  28%|##7       | 13816/50000 [7:51:45<18:38:03,  1.85s/it]
Outer training loop:  28%|##7       | 13817/50000 [7:51:47<18:12:52,  1.81s/it]
Outer training loop:  28%|##7       | 13818/50000 [7:51:48<18:23:19,  1.83s/it]
Outer training loop:  28%|##7       | 13819/50000 [7:51:50<17:59:38,  1.79s/it]
Outer training loop:  28%|##7       | 13820/50000 [7:51:52<17:51:05,  1.78s/it]
Outer training loop:  28%|##7       | 13821/50000 [7:51:54<17:44:12,  1.76s/it]
Outer training loop:  28%|##7       | 13822/50000 [7:51:55<18:00:10,  1.79s/it]
Outer training loop:  28%|##7       | 13823/50000 [7:51:57<17:39:50,  1.76s/it]
Outer training loop:  28%|##7       | 13824/50000 [7:51:59<17:37:16,  1.75s/it]
Outer training loop:  28%|##7       | 13825/50000 [7:52:01<17:30:02,  1.74s/it]
Outer training loop:  28%|##7       | 13826/50000 [7:52:03<18:07:59,  1.80s/it]
Outer training loop:  28%|##7       | 13827/50000 [7:52:04<18:21:07,  1.83s/it]
Outer training loop:  28%|##7       | 13828/50000 [7:52:06<18:00:35,  1.79s/it]
Outer training loop:  28%|##7       | 13829/50000 [7:52:08<18:29:44,  1.84s/it]
Outer training loop:  28%|##7       | 13830/50000 [7:52:10<18:24:26,  1.83s/it]
Outer training loop:  28%|##7       | 13831/50000 [7:52:12<18:16:32,  1.82s/it]
Outer training loop:  28%|##7       | 13832/50000 [7:52:13<17:45:37,  1.77s/it]
Outer training loop:  28%|##7       | 13833/50000 [7:52:15<18:05:41,  1.80s/it]
Outer training loop:  28%|##7       | 13834/50000 [7:52:17<18:07:44,  1.80s/it]
Outer training loop:  28%|##7       | 13835/50000 [7:52:19<18:00:33,  1.79s/it]
Outer training loop:  28%|##7       | 13836/50000 [7:52:21<18:20:11,  1.83s/it]
Outer training loop:  28%|##7       | 13837/50000 [7:52:22<18:07:16,  1.80s/it]
Outer training loop:  28%|##7       | 13838/50000 [7:52:24<17:56:07,  1.79s/it]
Outer training loop:  28%|##7       | 13839/50000 [7:52:26<17:44:45,  1.77s/it]
Outer training loop:  28%|##7       | 13840/50000 [7:52:28<18:39:46,  1.86s/it]
Outer training loop:  28%|##7       | 13841/50000 [7:52:30<18:27:16,  1.84s/it]
Outer training loop:  28%|##7       | 13842/50000 [7:52:32<18:19:16,  1.82s/it]
Outer training loop:  28%|##7       | 13843/50000 [7:52:33<18:35:42,  1.85s/it]
Outer training loop:  28%|##7       | 13844/50000 [7:52:35<18:41:00,  1.86s/it]
Outer training loop:  28%|##7       | 13845/50000 [7:52:37<18:09:07,  1.81s/it]
Outer training loop:  28%|##7       | 13846/50000 [7:52:39<18:31:29,  1.84s/it]
Outer training loop:  28%|##7       | 13847/50000 [7:52:41<18:04:14,  1.80s/it]
Outer training loop:  28%|##7       | 13848/50000 [7:52:43<18:32:20,  1.85s/it]
Outer training loop:  28%|##7       | 13849/50000 [7:52:45<18:56:44,  1.89s/it]
Outer training loop:  28%|##7       | 13850/50000 [7:52:46<18:33:58,  1.85s/it]
Outer training loop:  28%|##7       | 13851/50000 [7:52:48<18:14:53,  1.82s/it]
Outer training loop:  28%|##7       | 13852/50000 [7:52:50<19:35:39,  1.95s/it]
Outer training loop:  28%|##7       | 13853/50000 [7:52:52<18:48:24,  1.87s/it]
Outer training loop:  28%|##7       | 13854/50000 [7:52:54<18:36:58,  1.85s/it]
Outer training loop:  28%|##7       | 13855/50000 [7:52:56<18:35:10,  1.85s/it]
Outer training loop:  28%|##7       | 13856/50000 [7:52:58<19:03:24,  1.90s/it]
Outer training loop:  28%|##7       | 13857/50000 [7:52:59<18:40:59,  1.86s/it]
Outer training loop:  28%|##7       | 13858/50000 [7:53:01<18:37:18,  1.85s/it]
Outer training loop:  28%|##7       | 13859/50000 [7:53:03<18:47:13,  1.87s/it]
Outer training loop:  28%|##7       | 13860/50000 [7:53:05<18:27:36,  1.84s/it]
Outer training loop:  28%|##7       | 13861/50000 [7:53:07<18:23:21,  1.83s/it]
Outer training loop:  28%|##7       | 13862/50000 [7:53:09<18:56:14,  1.89s/it]
Outer training loop:  28%|##7       | 13863/50000 [7:53:11<18:18:12,  1.82s/it]
Outer training loop:  28%|##7       | 13864/50000 [7:53:12<18:00:52,  1.79s/it]
Outer training loop:  28%|##7       | 13865/50000 [7:53:14<17:52:20,  1.78s/it]
Outer training loop:  28%|##7       | 13866/50000 [7:53:16<18:09:14,  1.81s/it]
Outer training loop:  28%|##7       | 13867/50000 [7:53:18<17:54:02,  1.78s/it]
Outer training loop:  28%|##7       | 13868/50000 [7:53:19<17:42:19,  1.76s/it]
Outer training loop:  28%|##7       | 13869/50000 [7:53:21<18:04:25,  1.80s/it]
Outer training loop:  28%|##7       | 13870/50000 [7:53:23<17:47:39,  1.77s/it]
Outer training loop:  28%|##7       | 13871/50000 [7:53:25<17:32:28,  1.75s/it]
Outer training loop:  28%|##7       | 13872/50000 [7:53:26<17:24:59,  1.74s/it]
Outer training loop:  28%|##7       | 13873/50000 [7:53:28<18:08:55,  1.81s/it]
Outer training loop:  28%|##7       | 13874/50000 [7:53:30<18:03:40,  1.80s/it]
Outer training loop:  28%|##7       | 13875/50000 [7:53:32<17:56:34,  1.79s/it]
Outer training loop:  28%|##7       | 13876/50000 [7:53:34<18:55:11,  1.89s/it]
Outer training loop:  28%|##7       | 13877/50000 [7:53:36<18:44:58,  1.87s/it]
Outer training loop:  28%|##7       | 13878/50000 [7:53:38<18:44:14,  1.87s/it]
Outer training loop:  28%|##7       | 13879/50000 [7:53:39<18:09:38,  1.81s/it]
Outer training loop:  28%|##7       | 13880/50000 [7:53:41<18:24:21,  1.83s/it]
Outer training loop:  28%|##7       | 13881/50000 [7:53:43<18:05:37,  1.80s/it]
Outer training loop:  28%|##7       | 13882/50000 [7:53:45<17:57:07,  1.79s/it]
Outer training loop:  28%|##7       | 13883/50000 [7:53:47<18:04:50,  1.80s/it]
Outer training loop:  28%|##7       | 13884/50000 [7:53:48<17:51:22,  1.78s/it]
Outer training loop:  28%|##7       | 13885/50000 [7:53:50<17:57:17,  1.79s/it]
Outer training loop:  28%|##7       | 13886/50000 [7:53:52<18:08:56,  1.81s/it]
Outer training loop:  28%|##7       | 13887/50000 [7:53:54<18:08:23,  1.81s/it]
Outer training loop:  28%|##7       | 13888/50000 [7:53:55<17:54:10,  1.78s/it]
Outer training loop:  28%|##7       | 13889/50000 [7:53:57<18:00:31,  1.80s/it]
Outer training loop:  28%|##7       | 13890/50000 [7:53:59<18:13:52,  1.82s/it]
Outer training loop:  28%|##7       | 13891/50000 [7:54:01<17:43:58,  1.77s/it]
Outer training loop:  28%|##7       | 13892/50000 [7:54:03<17:37:41,  1.76s/it]
Outer training loop:  28%|##7       | 13893/50000 [7:54:05<18:28:43,  1.84s/it]
Outer training loop:  28%|##7       | 13894/50000 [7:54:06<18:08:21,  1.81s/it]
Outer training loop:  28%|##7       | 13895/50000 [7:54:08<17:48:30,  1.78s/it]
Outer training loop:  28%|##7       | 13896/50000 [7:54:10<18:31:29,  1.85s/it]
Outer training loop:  28%|##7       | 13897/50000 [7:54:12<18:23:08,  1.83s/it]
Outer training loop:  28%|##7       | 13898/50000 [7:54:13<17:59:38,  1.79s/it]
Outer training loop:  28%|##7       | 13899/50000 [7:54:15<17:42:19,  1.77s/it]
Outer training loop:  28%|##7       | 13900/50000 [7:54:17<18:08:19,  1.81s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.86s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  28%|##7       | 13901/50000 [7:54:24<34:11:32,  3.41s/it]
Outer training loop:  28%|##7       | 13902/50000 [7:54:26<29:12:09,  2.91s/it]
Outer training loop:  28%|##7       | 13903/50000 [7:54:28<26:08:29,  2.61s/it]
Outer training loop:  28%|##7       | 13904/50000 [7:54:30<23:26:11,  2.34s/it]
Outer training loop:  28%|##7       | 13905/50000 [7:54:31<21:42:34,  2.17s/it]
Outer training loop:  28%|##7       | 13906/50000 [7:54:33<20:49:16,  2.08s/it]
Outer training loop:  28%|##7       | 13907/50000 [7:54:35<19:50:01,  1.98s/it]
Outer training loop:  28%|##7       | 13908/50000 [7:54:37<19:15:25,  1.92s/it]
Outer training loop:  28%|##7       | 13909/50000 [7:54:39<19:13:01,  1.92s/it]
Outer training loop:  28%|##7       | 13910/50000 [7:54:40<18:50:54,  1.88s/it]
Outer training loop:  28%|##7       | 13911/50000 [7:54:42<18:35:26,  1.85s/it]
Outer training loop:  28%|##7       | 13912/50000 [7:54:44<18:21:54,  1.83s/it]
Outer training loop:  28%|##7       | 13913/50000 [7:54:46<18:27:55,  1.84s/it]
Outer training loop:  28%|##7       | 13914/50000 [7:54:48<17:52:42,  1.78s/it]
Outer training loop:  28%|##7       | 13915/50000 [7:54:49<18:01:20,  1.80s/it]
Outer training loop:  28%|##7       | 13916/50000 [7:54:51<18:19:36,  1.83s/it]
Outer training loop:  28%|##7       | 13917/50000 [7:54:53<18:07:28,  1.81s/it]
Outer training loop:  28%|##7       | 13918/50000 [7:54:55<18:03:40,  1.80s/it]
Outer training loop:  28%|##7       | 13919/50000 [7:54:57<17:50:33,  1.78s/it]
Outer training loop:  28%|##7       | 13920/50000 [7:54:58<18:07:15,  1.81s/it]
Outer training loop:  28%|##7       | 13921/50000 [7:55:00<18:12:48,  1.82s/it]
Outer training loop:  28%|##7       | 13922/50000 [7:55:02<18:08:34,  1.81s/it]
Outer training loop:  28%|##7       | 13923/50000 [7:55:04<18:26:11,  1.84s/it]
Outer training loop:  28%|##7       | 13924/50000 [7:55:06<17:57:49,  1.79s/it]
Outer training loop:  28%|##7       | 13925/50000 [7:55:07<18:02:23,  1.80s/it]
Outer training loop:  28%|##7       | 13926/50000 [7:55:09<18:36:54,  1.86s/it]
Outer training loop:  28%|##7       | 13927/50000 [7:55:11<17:58:27,  1.79s/it]
Outer training loop:  28%|##7       | 13928/50000 [7:55:13<17:41:35,  1.77s/it]
Outer training loop:  28%|##7       | 13929/50000 [7:55:15<18:35:16,  1.86s/it]
Outer training loop:  28%|##7       | 13930/50000 [7:55:17<18:18:40,  1.83s/it]
Outer training loop:  28%|##7       | 13931/50000 [7:55:18<18:13:39,  1.82s/it]
Outer training loop:  28%|##7       | 13932/50000 [7:55:20<18:13:22,  1.82s/it]
Outer training loop:  28%|##7       | 13933/50000 [7:55:22<18:24:35,  1.84s/it]
Outer training loop:  28%|##7       | 13934/50000 [7:55:24<17:49:47,  1.78s/it]
Outer training loop:  28%|##7       | 13935/50000 [7:55:25<17:31:52,  1.75s/it]
Outer training loop:  28%|##7       | 13936/50000 [7:55:27<18:08:47,  1.81s/it]
Outer training loop:  28%|##7       | 13937/50000 [7:55:29<17:43:22,  1.77s/it]
Outer training loop:  28%|##7       | 13938/50000 [7:55:31<17:21:30,  1.73s/it]
Outer training loop:  28%|##7       | 13939/50000 [7:55:33<18:06:46,  1.81s/it]
Outer training loop:  28%|##7       | 13940/50000 [7:55:35<18:00:40,  1.80s/it]
Outer training loop:  28%|##7       | 13941/50000 [7:55:36<17:48:25,  1.78s/it]
Outer training loop:  28%|##7       | 13942/50000 [7:55:38<18:11:06,  1.82s/it]
Outer training loop:  28%|##7       | 13943/50000 [7:55:40<18:00:16,  1.80s/it]
Outer training loop:  28%|##7       | 13944/50000 [7:55:42<17:49:56,  1.78s/it]
Outer training loop:  28%|##7       | 13945/50000 [7:55:43<17:35:19,  1.76s/it]
Outer training loop:  28%|##7       | 13946/50000 [7:55:45<17:53:04,  1.79s/it]
Outer training loop:  28%|##7       | 13947/50000 [7:55:47<17:32:06,  1.75s/it]
Outer training loop:  28%|##7       | 13948/50000 [7:55:49<17:13:41,  1.72s/it]
Outer training loop:  28%|##7       | 13949/50000 [7:55:51<18:53:09,  1.89s/it]
Outer training loop:  28%|##7       | 13950/50000 [7:55:53<18:35:16,  1.86s/it]
Outer training loop:  28%|##7       | 13951/50000 [7:55:54<18:19:23,  1.83s/it]
Outer training loop:  28%|##7       | 13952/50000 [7:55:56<18:20:28,  1.83s/it]
Outer training loop:  28%|##7       | 13953/50000 [7:55:58<18:33:01,  1.85s/it]
Outer training loop:  28%|##7       | 13954/50000 [7:56:00<18:01:38,  1.80s/it]
Outer training loop:  28%|##7       | 13955/50000 [7:56:02<17:56:05,  1.79s/it]
Outer training loop:  28%|##7       | 13956/50000 [7:56:03<18:05:34,  1.81s/it]
Outer training loop:  28%|##7       | 13957/50000 [7:56:05<17:55:23,  1.79s/it]
Outer training loop:  28%|##7       | 13958/50000 [7:56:07<17:43:42,  1.77s/it]
Outer training loop:  28%|##7       | 13959/50000 [7:56:09<18:11:50,  1.82s/it]
Outer training loop:  28%|##7       | 13960/50000 [7:56:11<18:18:12,  1.83s/it]
Outer training loop:  28%|##7       | 13961/50000 [7:56:12<18:24:06,  1.84s/it]
Outer training loop:  28%|##7       | 13962/50000 [7:56:14<18:24:06,  1.84s/it]
Outer training loop:  28%|##7       | 13963/50000 [7:56:16<18:53:44,  1.89s/it]
Outer training loop:  28%|##7       | 13964/50000 [7:56:18<18:31:47,  1.85s/it]
Outer training loop:  28%|##7       | 13965/50000 [7:56:20<18:06:21,  1.81s/it]
Outer training loop:  28%|##7       | 13966/50000 [7:56:22<18:11:43,  1.82s/it]
Outer training loop:  28%|##7       | 13967/50000 [7:56:23<18:00:57,  1.80s/it]
Outer training loop:  28%|##7       | 13968/50000 [7:56:25<18:06:28,  1.81s/it]
Outer training loop:  28%|##7       | 13969/50000 [7:56:27<18:18:27,  1.83s/it]
Outer training loop:  28%|##7       | 13970/50000 [7:56:29<18:27:19,  1.84s/it]
Outer training loop:  28%|##7       | 13971/50000 [7:56:31<18:21:54,  1.84s/it]
Outer training loop:  28%|##7       | 13972/50000 [7:56:33<18:14:31,  1.82s/it]
Outer training loop:  28%|##7       | 13973/50000 [7:56:35<18:33:00,  1.85s/it]
Outer training loop:  28%|##7       | 13974/50000 [7:56:36<18:24:53,  1.84s/it]
Outer training loop:  28%|##7       | 13975/50000 [7:56:38<18:14:31,  1.82s/it]
Outer training loop:  28%|##7       | 13976/50000 [7:56:40<18:39:30,  1.86s/it]
Outer training loop:  28%|##7       | 13977/50000 [7:56:42<18:34:08,  1.86s/it]
Outer training loop:  28%|##7       | 13978/50000 [7:56:44<18:32:43,  1.85s/it]
Outer training loop:  28%|##7       | 13979/50000 [7:56:46<18:30:51,  1.85s/it]
Outer training loop:  28%|##7       | 13980/50000 [7:56:47<18:18:43,  1.83s/it]
Outer training loop:  28%|##7       | 13981/50000 [7:56:49<17:54:45,  1.79s/it]
Outer training loop:  28%|##7       | 13982/50000 [7:56:51<17:53:29,  1.79s/it]
Outer training loop:  28%|##7       | 13983/50000 [7:56:53<18:17:14,  1.83s/it]
Outer training loop:  28%|##7       | 13984/50000 [7:56:54<17:49:41,  1.78s/it]
Outer training loop:  28%|##7       | 13985/50000 [7:56:56<17:39:10,  1.76s/it]
Outer training loop:  28%|##7       | 13986/50000 [7:56:58<18:01:13,  1.80s/it]
Outer training loop:  28%|##7       | 13987/50000 [7:57:00<17:47:15,  1.78s/it]
Outer training loop:  28%|##7       | 13988/50000 [7:57:02<17:44:51,  1.77s/it]
Outer training loop:  28%|##7       | 13989/50000 [7:57:03<17:52:20,  1.79s/it]
Outer training loop:  28%|##7       | 13990/50000 [7:57:05<18:15:42,  1.83s/it]
Outer training loop:  28%|##7       | 13991/50000 [7:57:07<18:03:44,  1.81s/it]
Outer training loop:  28%|##7       | 13992/50000 [7:57:09<17:58:35,  1.80s/it]
Outer training loop:  28%|##7       | 13993/50000 [7:57:11<18:45:14,  1.88s/it]
Outer training loop:  28%|##7       | 13994/50000 [7:57:13<18:36:47,  1.86s/it]
Outer training loop:  28%|##7       | 13995/50000 [7:57:15<18:23:53,  1.84s/it]
Outer training loop:  28%|##7       | 13996/50000 [7:57:16<18:25:13,  1.84s/it]
Outer training loop:  28%|##7       | 13997/50000 [7:57:18<18:07:02,  1.81s/it]
Outer training loop:  28%|##7       | 13998/50000 [7:57:20<18:08:00,  1.81s/it]
Outer training loop:  28%|##7       | 13999/50000 [7:57:22<18:59:22,  1.90s/it]
Outer training loop:  28%|##8       | 14000/50000 [7:57:24<18:38:38,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:21,  1.73s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:20,  1.73s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:24,  1.81s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:19,  1.77s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.78s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:23,  1.86s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.83s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:17,  1.84s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:15,  1.83s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:13,  1.86s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.83s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:10,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:08,  1.86s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:03,  1.81s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<01:57,  1.76s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:00,  1.83s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:57,  1.81s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:54,  1.80s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:53,  1.81s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:50,  1.78s/it]

Evaluating policy:  27%|##6       | 22/83 [00:40<01:53,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:42<01:50,  1.85s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:44,  1.81s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:42,  1.80s/it]

Evaluating policy:  33%|###2      | 27/83 [00:49<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:51<01:41,  1.85s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:36,  1.79s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:35,  1.80s/it]

Evaluating policy:  37%|###7      | 31/83 [00:56<01:33,  1.80s/it]

Evaluating policy:  39%|###8      | 32/83 [00:58<01:35,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:00<01:31,  1.83s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:26,  1.77s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:23,  1.74s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:21,  1.74s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:18,  1.71s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:20,  1.80s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:19,  1.80s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:15,  1.75s/it]

Evaluating policy:  49%|####9     | 41/83 [01:14<01:13,  1.74s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:11,  1.76s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:10,  1.75s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:11,  1.84s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:10,  1.85s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:23<01:06,  1.80s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:03,  1.76s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:01,  1.75s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<00:58,  1.73s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:56,  1.72s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:55,  1.74s/it]

Evaluating policy:  63%|######2   | 52/83 [01:34<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:54,  1.83s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:51,  1.79s/it]

Evaluating policy:  66%|######6   | 55/83 [01:39<00:49,  1.78s/it]

Evaluating policy:  67%|######7   | 56/83 [01:41<00:48,  1.79s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:46,  1.80s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:44,  1.79s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:45,  1.88s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:48<00:42,  1.85s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:50<00:39,  1.81s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:37,  1.78s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:53<00:35,  1.77s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:55<00:33,  1.74s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:57<00:31,  1.72s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:29,  1.72s/it]

Evaluating policy:  81%|########  | 67/83 [02:00<00:29,  1.86s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:04<00:24,  1.78s/it]

Evaluating policy:  84%|########4 | 70/83 [02:06<00:23,  1.77s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:20,  1.75s/it]

Evaluating policy:  87%|########6 | 72/83 [02:09<00:19,  1.75s/it]

Evaluating policy:  88%|########7 | 73/83 [02:11<00:17,  1.77s/it]

Evaluating policy:  89%|########9 | 74/83 [02:13<00:15,  1.77s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:14,  1.79s/it]

Evaluating policy:  92%|#########1| 76/83 [02:17<00:13,  1.93s/it]

Evaluating policy:  93%|#########2| 77/83 [02:19<00:11,  1.91s/it]

Evaluating policy:  94%|#########3| 78/83 [02:20<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:22<00:07,  1.86s/it]

Evaluating policy:  96%|#########6| 80/83 [02:24<00:05,  1.83s/it]

Evaluating policy:  98%|#########7| 81/83 [02:26<00:03,  1.80s/it]

Evaluating policy:  99%|#########8| 82/83 [02:27<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 83/83 [02:29<00:00,  1.73s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:12,  1.62s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.82s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.78s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.79s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:17,  1.76s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:20,  1.82s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:14,  1.79s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:11,  1.77s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:13,  1.82s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:10,  1.81s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:10,  1.84s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.80s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:09,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:05,  1.84s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:02,  1.83s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<01:59,  1.81s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:57,  1.81s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:59,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:54,  1.81s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:49,  1.77s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:47,  1.76s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:46,  1.78s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:46,  1.83s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:42,  1.80s/it]

Evaluating policy:  33%|###2      | 27/83 [00:49<01:39,  1.78s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:36,  1.76s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:38,  1.83s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:36,  1.82s/it]

Evaluating policy:  37%|###7      | 31/83 [00:56<01:32,  1.78s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:29,  1.75s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:27,  1.74s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:25,  1.75s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:26,  1.83s/it]

Evaluating policy:  45%|####4     | 37/83 [01:07<01:23,  1.80s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:19,  1.77s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:16,  1.75s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:15,  1.76s/it]

Evaluating policy:  49%|####9     | 41/83 [01:14<01:17,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:16<01:14,  1.82s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:11,  1.78s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:10,  1.80s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:08,  1.80s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:23<01:05,  1.78s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:25<01:08,  1.90s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<01:01,  1.81s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:57,  1.75s/it]

Evaluating policy:  61%|######1   | 51/83 [01:32<00:56,  1.77s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:54,  1.76s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:52,  1.75s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:52,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:39<00:50,  1.80s/it]

Evaluating policy:  67%|######7   | 56/83 [01:41<00:48,  1.81s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:46,  1.79s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:44,  1.77s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:42,  1.75s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:39,  1.73s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:38,  1.73s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:53<00:36,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:55<00:34,  1.81s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:57<00:31,  1.76s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:29,  1.74s/it]

Evaluating policy:  81%|########  | 67/83 [02:00<00:28,  1.75s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:26,  1.73s/it]

Evaluating policy:  83%|########3 | 69/83 [02:04<00:24,  1.77s/it]

Evaluating policy:  84%|########4 | 70/83 [02:06<00:24,  1.90s/it]

Evaluating policy:  86%|########5 | 71/83 [02:08<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:09<00:19,  1.80s/it]

Evaluating policy:  88%|########7 | 73/83 [02:11<00:18,  1.80s/it]

Evaluating policy:  89%|########9 | 74/83 [02:13<00:16,  1.79s/it]

Evaluating policy:  90%|######### | 75/83 [02:15<00:14,  1.78s/it]

Evaluating policy:  92%|#########1| 76/83 [02:16<00:12,  1.75s/it]

Evaluating policy:  93%|#########2| 77/83 [02:18<00:10,  1.75s/it]

Evaluating policy:  94%|#########3| 78/83 [02:20<00:08,  1.71s/it]

Evaluating policy:  95%|#########5| 79/83 [02:22<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:24<00:05,  1.86s/it]

Evaluating policy:  98%|#########7| 81/83 [02:25<00:03,  1.80s/it]

Evaluating policy:  99%|#########8| 82/83 [02:27<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 83/83 [02:29<00:00,  1.78s/it]

Outer training loop:  28%|##8       | 14001/50000 [8:02:30<932:05:58, 93.21s/it]
Outer training loop:  28%|##8       | 14002/50000 [8:02:32<657:58:39, 65.80s/it]
Outer training loop:  28%|##8       | 14003/50000 [8:02:34<465:48:15, 46.58s/it]
Outer training loop:  28%|##8       | 14004/50000 [8:02:36<331:26:01, 33.15s/it]
Outer training loop:  28%|##8       | 14005/50000 [8:02:37<236:54:44, 23.69s/it]
Outer training loop:  28%|##8       | 14006/50000 [8:02:39<170:58:38, 17.10s/it]
Outer training loop:  28%|##8       | 14007/50000 [8:02:41<126:29:19, 12.65s/it]
Outer training loop:  28%|##8       | 14008/50000 [8:02:43<94:18:07,  9.43s/it] 
Outer training loop:  28%|##8       | 14009/50000 [8:02:45<70:59:05,  7.10s/it]
Outer training loop:  28%|##8       | 14010/50000 [8:02:46<54:49:39,  5.48s/it]
Outer training loop:  28%|##8       | 14011/50000 [8:02:48<44:11:11,  4.42s/it]
Outer training loop:  28%|##8       | 14012/50000 [8:02:50<36:07:14,  3.61s/it]
Outer training loop:  28%|##8       | 14013/50000 [8:02:52<30:39:04,  3.07s/it]
Outer training loop:  28%|##8       | 14014/50000 [8:02:54<27:06:01,  2.71s/it]
Outer training loop:  28%|##8       | 14015/50000 [8:02:56<24:27:23,  2.45s/it]
Outer training loop:  28%|##8       | 14016/50000 [8:02:57<22:03:47,  2.21s/it]
Outer training loop:  28%|##8       | 14017/50000 [8:02:59<21:10:44,  2.12s/it]
Outer training loop:  28%|##8       | 14018/50000 [8:03:01<20:05:15,  2.01s/it]
Outer training loop:  28%|##8       | 14019/50000 [8:03:03<19:12:57,  1.92s/it]
Outer training loop:  28%|##8       | 14020/50000 [8:03:05<19:15:59,  1.93s/it]
Outer training loop:  28%|##8       | 14021/50000 [8:03:06<18:26:42,  1.85s/it]
Outer training loop:  28%|##8       | 14022/50000 [8:03:08<17:45:37,  1.78s/it]
Outer training loop:  28%|##8       | 14023/50000 [8:03:10<17:34:31,  1.76s/it]
Outer training loop:  28%|##8       | 14024/50000 [8:03:12<18:13:28,  1.82s/it]
Outer training loop:  28%|##8       | 14025/50000 [8:03:13<17:40:04,  1.77s/it]
Outer training loop:  28%|##8       | 14026/50000 [8:03:15<17:36:01,  1.76s/it]
Outer training loop:  28%|##8       | 14027/50000 [8:03:17<17:17:58,  1.73s/it]
Outer training loop:  28%|##8       | 14028/50000 [8:03:18<17:47:51,  1.78s/it]
Outer training loop:  28%|##8       | 14029/50000 [8:03:20<17:40:24,  1.77s/it]
Outer training loop:  28%|##8       | 14030/50000 [8:03:22<18:10:10,  1.82s/it]
Outer training loop:  28%|##8       | 14031/50000 [8:03:24<18:32:04,  1.86s/it]
Outer training loop:  28%|##8       | 14032/50000 [8:03:26<18:27:05,  1.85s/it]
Outer training loop:  28%|##8       | 14033/50000 [8:03:28<17:54:48,  1.79s/it]
Outer training loop:  28%|##8       | 14034/50000 [8:03:30<18:18:48,  1.83s/it]
Outer training loop:  28%|##8       | 14035/50000 [8:03:31<18:02:51,  1.81s/it]
Outer training loop:  28%|##8       | 14036/50000 [8:03:33<17:40:52,  1.77s/it]
Outer training loop:  28%|##8       | 14037/50000 [8:03:35<17:48:52,  1.78s/it]
Outer training loop:  28%|##8       | 14038/50000 [8:03:37<18:20:38,  1.84s/it]
Outer training loop:  28%|##8       | 14039/50000 [8:03:38<17:58:33,  1.80s/it]
Outer training loop:  28%|##8       | 14040/50000 [8:03:40<17:47:25,  1.78s/it]
Outer training loop:  28%|##8       | 14041/50000 [8:03:42<18:49:13,  1.88s/it]
Outer training loop:  28%|##8       | 14042/50000 [8:03:44<18:27:45,  1.85s/it]
Outer training loop:  28%|##8       | 14043/50000 [8:03:46<18:21:38,  1.84s/it]
Outer training loop:  28%|##8       | 14044/50000 [8:03:48<18:10:39,  1.82s/it]
Outer training loop:  28%|##8       | 14045/50000 [8:03:50<18:48:03,  1.88s/it]
Outer training loop:  28%|##8       | 14046/50000 [8:03:51<18:14:31,  1.83s/it]
Outer training loop:  28%|##8       | 14047/50000 [8:03:53<18:06:56,  1.81s/it]
Outer training loop:  28%|##8       | 14048/50000 [8:03:55<18:34:39,  1.86s/it]
Outer training loop:  28%|##8       | 14049/50000 [8:03:57<18:31:38,  1.86s/it]
Outer training loop:  28%|##8       | 14050/50000 [8:03:59<18:00:31,  1.80s/it]
Outer training loop:  28%|##8       | 14051/50000 [8:04:01<18:21:59,  1.84s/it]
Outer training loop:  28%|##8       | 14052/50000 [8:04:02<17:55:09,  1.79s/it]
Outer training loop:  28%|##8       | 14053/50000 [8:04:04<17:45:44,  1.78s/it]
Outer training loop:  28%|##8       | 14054/50000 [8:04:06<18:15:52,  1.83s/it]
Outer training loop:  28%|##8       | 14055/50000 [8:04:08<18:16:12,  1.83s/it]
Outer training loop:  28%|##8       | 14056/50000 [8:04:10<17:57:50,  1.80s/it]
Outer training loop:  28%|##8       | 14057/50000 [8:04:11<17:42:59,  1.77s/it]
Outer training loop:  28%|##8       | 14058/50000 [8:04:13<17:43:52,  1.78s/it]
Outer training loop:  28%|##8       | 14059/50000 [8:04:15<18:05:13,  1.81s/it]
Outer training loop:  28%|##8       | 14060/50000 [8:04:17<17:59:07,  1.80s/it]
Outer training loop:  28%|##8       | 14061/50000 [8:04:19<18:22:52,  1.84s/it]
Outer training loop:  28%|##8       | 14062/50000 [8:04:20<18:18:26,  1.83s/it]
Outer training loop:  28%|##8       | 14063/50000 [8:04:22<17:55:30,  1.80s/it]
Outer training loop:  28%|##8       | 14064/50000 [8:04:24<18:10:38,  1.82s/it]
Outer training loop:  28%|##8       | 14065/50000 [8:04:26<17:57:32,  1.80s/it]
Outer training loop:  28%|##8       | 14066/50000 [8:04:28<17:49:46,  1.79s/it]
Outer training loop:  28%|##8       | 14067/50000 [8:04:29<17:33:37,  1.76s/it]
Outer training loop:  28%|##8       | 14068/50000 [8:04:31<17:43:21,  1.78s/it]
Outer training loop:  28%|##8       | 14069/50000 [8:04:33<17:32:00,  1.76s/it]
Outer training loop:  28%|##8       | 14070/50000 [8:04:34<17:15:43,  1.73s/it]
Outer training loop:  28%|##8       | 14071/50000 [8:04:36<17:39:25,  1.77s/it]
Outer training loop:  28%|##8       | 14072/50000 [8:04:38<17:32:19,  1.76s/it]
Outer training loop:  28%|##8       | 14073/50000 [8:04:40<17:22:43,  1.74s/it]
Outer training loop:  28%|##8       | 14074/50000 [8:04:42<17:37:53,  1.77s/it]
Outer training loop:  28%|##8       | 14075/50000 [8:04:44<18:26:12,  1.85s/it]
Outer training loop:  28%|##8       | 14076/50000 [8:04:45<17:56:01,  1.80s/it]
Outer training loop:  28%|##8       | 14077/50000 [8:04:47<17:43:33,  1.78s/it]
Outer training loop:  28%|##8       | 14078/50000 [8:04:49<18:18:39,  1.84s/it]
Outer training loop:  28%|##8       | 14079/50000 [8:04:51<17:48:10,  1.78s/it]
Outer training loop:  28%|##8       | 14080/50000 [8:04:53<18:19:10,  1.84s/it]
Outer training loop:  28%|##8       | 14081/50000 [8:04:54<18:15:29,  1.83s/it]
Outer training loop:  28%|##8       | 14082/50000 [8:04:56<18:02:29,  1.81s/it]
Outer training loop:  28%|##8       | 14083/50000 [8:04:58<17:56:59,  1.80s/it]
Outer training loop:  28%|##8       | 14084/50000 [8:05:00<18:23:04,  1.84s/it]
Outer training loop:  28%|##8       | 14085/50000 [8:05:02<17:52:11,  1.79s/it]
Outer training loop:  28%|##8       | 14086/50000 [8:05:03<17:54:48,  1.80s/it]
Outer training loop:  28%|##8       | 14087/50000 [8:05:05<17:58:04,  1.80s/it]
Outer training loop:  28%|##8       | 14088/50000 [8:05:07<18:03:48,  1.81s/it]
Outer training loop:  28%|##8       | 14089/50000 [8:05:09<17:40:42,  1.77s/it]
Outer training loop:  28%|##8       | 14090/50000 [8:05:11<17:49:56,  1.79s/it]
Outer training loop:  28%|##8       | 14091/50000 [8:05:13<18:31:18,  1.86s/it]
Outer training loop:  28%|##8       | 14092/50000 [8:05:14<17:50:37,  1.79s/it]
Outer training loop:  28%|##8       | 14093/50000 [8:05:16<17:46:57,  1.78s/it]
Outer training loop:  28%|##8       | 14094/50000 [8:05:18<17:18:58,  1.74s/it]
Outer training loop:  28%|##8       | 14095/50000 [8:05:20<18:07:05,  1.82s/it]
Outer training loop:  28%|##8       | 14096/50000 [8:05:21<17:38:43,  1.77s/it]
Outer training loop:  28%|##8       | 14097/50000 [8:05:23<17:46:08,  1.78s/it]
Outer training loop:  28%|##8       | 14098/50000 [8:05:25<18:13:51,  1.83s/it]
Outer training loop:  28%|##8       | 14099/50000 [8:05:27<17:54:46,  1.80s/it]
Outer training loop:  28%|##8       | 14100/50000 [8:05:29<18:06:03,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  28%|##8       | 14101/50000 [8:05:36<33:39:58,  3.38s/it]
Outer training loop:  28%|##8       | 14102/50000 [8:05:37<28:52:01,  2.89s/it]
Outer training loop:  28%|##8       | 14103/50000 [8:05:39<26:12:43,  2.63s/it]
Outer training loop:  28%|##8       | 14104/50000 [8:05:41<23:22:26,  2.34s/it]
Outer training loop:  28%|##8       | 14105/50000 [8:05:43<21:40:57,  2.17s/it]
Outer training loop:  28%|##8       | 14106/50000 [8:05:45<20:57:04,  2.10s/it]
Outer training loop:  28%|##8       | 14107/50000 [8:05:46<19:39:50,  1.97s/it]
Outer training loop:  28%|##8       | 14108/50000 [8:05:48<19:11:22,  1.92s/it]
Outer training loop:  28%|##8       | 14109/50000 [8:05:50<19:15:04,  1.93s/it]
Outer training loop:  28%|##8       | 14110/50000 [8:05:52<18:29:46,  1.86s/it]
Outer training loop:  28%|##8       | 14111/50000 [8:05:54<17:59:03,  1.80s/it]
Outer training loop:  28%|##8       | 14112/50000 [8:05:55<17:42:40,  1.78s/it]
Outer training loop:  28%|##8       | 14113/50000 [8:05:57<18:16:21,  1.83s/it]
Outer training loop:  28%|##8       | 14114/50000 [8:05:59<17:52:42,  1.79s/it]
Outer training loop:  28%|##8       | 14115/50000 [8:06:01<17:45:57,  1.78s/it]
Outer training loop:  28%|##8       | 14116/50000 [8:06:03<18:04:51,  1.81s/it]
Outer training loop:  28%|##8       | 14117/50000 [8:06:04<17:51:01,  1.79s/it]
Outer training loop:  28%|##8       | 14118/50000 [8:06:06<17:40:24,  1.77s/it]
Outer training loop:  28%|##8       | 14119/50000 [8:06:08<17:21:52,  1.74s/it]
Outer training loop:  28%|##8       | 14120/50000 [8:06:10<17:47:09,  1.78s/it]
Outer training loop:  28%|##8       | 14121/50000 [8:06:11<17:32:45,  1.76s/it]
Outer training loop:  28%|##8       | 14122/50000 [8:06:13<17:40:50,  1.77s/it]
Outer training loop:  28%|##8       | 14123/50000 [8:06:15<18:01:15,  1.81s/it]
Outer training loop:  28%|##8       | 14124/50000 [8:06:17<17:27:14,  1.75s/it]
Outer training loop:  28%|##8       | 14125/50000 [8:06:18<17:20:36,  1.74s/it]
Outer training loop:  28%|##8       | 14126/50000 [8:06:20<17:19:06,  1.74s/it]
Outer training loop:  28%|##8       | 14127/50000 [8:06:22<18:13:39,  1.83s/it]
Outer training loop:  28%|##8       | 14128/50000 [8:06:24<17:58:27,  1.80s/it]
Outer training loop:  28%|##8       | 14129/50000 [8:06:26<17:58:04,  1.80s/it]
Outer training loop:  28%|##8       | 14130/50000 [8:06:28<18:25:25,  1.85s/it]
Outer training loop:  28%|##8       | 14131/50000 [8:06:29<17:54:03,  1.80s/it]
Outer training loop:  28%|##8       | 14132/50000 [8:06:31<18:00:01,  1.81s/it]
Outer training loop:  28%|##8       | 14133/50000 [8:06:33<18:19:27,  1.84s/it]
Outer training loop:  28%|##8       | 14134/50000 [8:06:35<17:53:58,  1.80s/it]
Outer training loop:  28%|##8       | 14135/50000 [8:06:36<17:37:56,  1.77s/it]
Outer training loop:  28%|##8       | 14136/50000 [8:06:38<17:40:14,  1.77s/it]
Outer training loop:  28%|##8       | 14137/50000 [8:06:40<18:11:53,  1.83s/it]
Outer training loop:  28%|##8       | 14138/50000 [8:06:42<18:04:57,  1.82s/it]
Outer training loop:  28%|##8       | 14139/50000 [8:06:44<17:47:24,  1.79s/it]
Outer training loop:  28%|##8       | 14140/50000 [8:06:45<17:40:52,  1.78s/it]
Outer training loop:  28%|##8       | 14141/50000 [8:06:47<18:20:51,  1.84s/it]
Outer training loop:  28%|##8       | 14142/50000 [8:06:49<18:11:13,  1.83s/it]
Outer training loop:  28%|##8       | 14143/50000 [8:06:51<17:54:35,  1.80s/it]
Outer training loop:  28%|##8       | 14144/50000 [8:06:53<18:09:28,  1.82s/it]
Outer training loop:  28%|##8       | 14145/50000 [8:06:55<17:50:51,  1.79s/it]
Outer training loop:  28%|##8       | 14146/50000 [8:06:56<17:51:07,  1.79s/it]
Outer training loop:  28%|##8       | 14147/50000 [8:06:58<18:13:45,  1.83s/it]
Outer training loop:  28%|##8       | 14148/50000 [8:07:00<17:48:36,  1.79s/it]
Outer training loop:  28%|##8       | 14149/50000 [8:07:02<17:50:52,  1.79s/it]
Outer training loop:  28%|##8       | 14150/50000 [8:07:04<17:55:47,  1.80s/it]
Outer training loop:  28%|##8       | 14151/50000 [8:07:05<18:14:15,  1.83s/it]
Outer training loop:  28%|##8       | 14152/50000 [8:07:07<17:51:25,  1.79s/it]
Outer training loop:  28%|##8       | 14153/50000 [8:07:09<17:34:16,  1.76s/it]
Outer training loop:  28%|##8       | 14154/50000 [8:07:11<17:54:25,  1.80s/it]
Outer training loop:  28%|##8       | 14155/50000 [8:07:12<17:43:01,  1.78s/it]
Outer training loop:  28%|##8       | 14156/50000 [8:07:14<17:27:12,  1.75s/it]
Outer training loop:  28%|##8       | 14157/50000 [8:07:16<17:42:32,  1.78s/it]
Outer training loop:  28%|##8       | 14158/50000 [8:07:18<17:22:12,  1.74s/it]
Outer training loop:  28%|##8       | 14159/50000 [8:07:19<17:24:49,  1.75s/it]
Outer training loop:  28%|##8       | 14160/50000 [8:07:21<17:39:36,  1.77s/it]
Outer training loop:  28%|##8       | 14161/50000 [8:07:23<17:45:59,  1.78s/it]
Outer training loop:  28%|##8       | 14162/50000 [8:07:25<17:34:17,  1.77s/it]
Outer training loop:  28%|##8       | 14163/50000 [8:07:26<17:17:08,  1.74s/it]
Outer training loop:  28%|##8       | 14164/50000 [8:07:28<17:22:02,  1.74s/it]
Outer training loop:  28%|##8       | 14165/50000 [8:07:30<17:40:50,  1.78s/it]
Outer training loop:  28%|##8       | 14166/50000 [8:07:32<17:25:30,  1.75s/it]
Outer training loop:  28%|##8       | 14167/50000 [8:07:33<17:00:10,  1.71s/it]
Outer training loop:  28%|##8       | 14168/50000 [8:07:35<17:41:22,  1.78s/it]
Outer training loop:  28%|##8       | 14169/50000 [8:07:37<17:44:32,  1.78s/it]
Outer training loop:  28%|##8       | 14170/50000 [8:07:39<17:22:23,  1.75s/it]
Outer training loop:  28%|##8       | 14171/50000 [8:07:41<17:56:50,  1.80s/it]
Outer training loop:  28%|##8       | 14172/50000 [8:07:42<17:49:13,  1.79s/it]
Outer training loop:  28%|##8       | 14173/50000 [8:07:44<17:25:52,  1.75s/it]
Outer training loop:  28%|##8       | 14174/50000 [8:07:46<17:37:54,  1.77s/it]
Outer training loop:  28%|##8       | 14175/50000 [8:07:48<17:27:34,  1.75s/it]
Outer training loop:  28%|##8       | 14176/50000 [8:07:49<17:13:22,  1.73s/it]
Outer training loop:  28%|##8       | 14177/50000 [8:07:51<17:31:13,  1.76s/it]
Outer training loop:  28%|##8       | 14178/50000 [8:07:53<18:00:05,  1.81s/it]
Outer training loop:  28%|##8       | 14179/50000 [8:07:55<17:52:58,  1.80s/it]
Outer training loop:  28%|##8       | 14180/50000 [8:07:57<17:41:10,  1.78s/it]
Outer training loop:  28%|##8       | 14181/50000 [8:07:58<17:55:32,  1.80s/it]
Outer training loop:  28%|##8       | 14182/50000 [8:08:00<17:53:18,  1.80s/it]
Outer training loop:  28%|##8       | 14183/50000 [8:08:02<17:37:44,  1.77s/it]
Outer training loop:  28%|##8       | 14184/50000 [8:08:04<18:12:10,  1.83s/it]
Outer training loop:  28%|##8       | 14185/50000 [8:08:06<17:58:16,  1.81s/it]
Outer training loop:  28%|##8       | 14186/50000 [8:08:07<17:53:13,  1.80s/it]
Outer training loop:  28%|##8       | 14187/50000 [8:08:09<17:37:53,  1.77s/it]
Outer training loop:  28%|##8       | 14188/50000 [8:08:11<18:05:57,  1.82s/it]
Outer training loop:  28%|##8       | 14189/50000 [8:08:13<17:54:58,  1.80s/it]
Outer training loop:  28%|##8       | 14190/50000 [8:08:14<17:29:41,  1.76s/it]
Outer training loop:  28%|##8       | 14191/50000 [8:08:16<18:09:22,  1.83s/it]
Outer training loop:  28%|##8       | 14192/50000 [8:08:18<17:41:57,  1.78s/it]
Outer training loop:  28%|##8       | 14193/50000 [8:08:20<17:59:49,  1.81s/it]
Outer training loop:  28%|##8       | 14194/50000 [8:08:22<18:01:54,  1.81s/it]
Outer training loop:  28%|##8       | 14195/50000 [8:08:24<17:44:14,  1.78s/it]
Outer training loop:  28%|##8       | 14196/50000 [8:08:25<17:57:27,  1.81s/it]
Outer training loop:  28%|##8       | 14197/50000 [8:08:27<18:25:12,  1.85s/it]
Outer training loop:  28%|##8       | 14198/50000 [8:08:29<18:06:00,  1.82s/it]
Outer training loop:  28%|##8       | 14199/50000 [8:08:31<17:55:48,  1.80s/it]
Outer training loop:  28%|##8       | 14200/50000 [8:08:33<17:33:38,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.70s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  28%|##8       | 14201/50000 [8:08:40<33:35:15,  3.38s/it]
Outer training loop:  28%|##8       | 14202/50000 [8:08:42<29:04:50,  2.92s/it]
Outer training loop:  28%|##8       | 14203/50000 [8:08:43<25:25:57,  2.56s/it]
Outer training loop:  28%|##8       | 14204/50000 [8:08:45<23:02:44,  2.32s/it]
Outer training loop:  28%|##8       | 14205/50000 [8:08:47<22:00:40,  2.21s/it]
Outer training loop:  28%|##8       | 14206/50000 [8:08:49<20:28:46,  2.06s/it]
Outer training loop:  28%|##8       | 14207/50000 [8:08:50<19:41:33,  1.98s/it]
Outer training loop:  28%|##8       | 14208/50000 [8:08:52<18:49:22,  1.89s/it]
Outer training loop:  28%|##8       | 14209/50000 [8:08:54<18:40:43,  1.88s/it]
Outer training loop:  28%|##8       | 14210/50000 [8:08:56<18:27:32,  1.86s/it]
Outer training loop:  28%|##8       | 14211/50000 [8:08:58<18:04:09,  1.82s/it]
Outer training loop:  28%|##8       | 14212/50000 [8:08:59<18:16:27,  1.84s/it]
Outer training loop:  28%|##8       | 14213/50000 [8:09:01<17:56:48,  1.81s/it]
Outer training loop:  28%|##8       | 14214/50000 [8:09:03<17:41:35,  1.78s/it]
Outer training loop:  28%|##8       | 14215/50000 [8:09:05<17:40:21,  1.78s/it]
Outer training loop:  28%|##8       | 14216/50000 [8:09:07<17:58:38,  1.81s/it]
Outer training loop:  28%|##8       | 14217/50000 [8:09:08<17:43:37,  1.78s/it]
Outer training loop:  28%|##8       | 14218/50000 [8:09:10<17:39:50,  1.78s/it]
Outer training loop:  28%|##8       | 14219/50000 [8:09:12<17:50:21,  1.79s/it]
Outer training loop:  28%|##8       | 14220/50000 [8:09:14<17:41:08,  1.78s/it]
Outer training loop:  28%|##8       | 14221/50000 [8:09:15<17:13:46,  1.73s/it]
Outer training loop:  28%|##8       | 14222/50000 [8:09:17<17:57:16,  1.81s/it]
Outer training loop:  28%|##8       | 14223/50000 [8:09:19<17:38:24,  1.78s/it]
Outer training loop:  28%|##8       | 14224/50000 [8:09:21<18:03:41,  1.82s/it]
Outer training loop:  28%|##8       | 14225/50000 [8:09:22<17:34:52,  1.77s/it]
Outer training loop:  28%|##8       | 14226/50000 [8:09:24<17:59:05,  1.81s/it]
Outer training loop:  28%|##8       | 14227/50000 [8:09:26<17:54:28,  1.80s/it]
Outer training loop:  28%|##8       | 14228/50000 [8:09:28<17:54:07,  1.80s/it]
Outer training loop:  28%|##8       | 14229/50000 [8:09:30<18:26:57,  1.86s/it]
Outer training loop:  28%|##8       | 14230/50000 [8:09:32<18:15:41,  1.84s/it]
Outer training loop:  28%|##8       | 14231/50000 [8:09:34<18:31:20,  1.86s/it]
Outer training loop:  28%|##8       | 14232/50000 [8:09:35<17:57:36,  1.81s/it]
Outer training loop:  28%|##8       | 14233/50000 [8:09:37<18:26:33,  1.86s/it]
Outer training loop:  28%|##8       | 14234/50000 [8:09:39<18:00:03,  1.81s/it]
Outer training loop:  28%|##8       | 14235/50000 [8:09:41<17:51:58,  1.80s/it]
Outer training loop:  28%|##8       | 14236/50000 [8:09:43<18:46:01,  1.89s/it]
Outer training loop:  28%|##8       | 14237/50000 [8:09:45<18:37:24,  1.87s/it]
Outer training loop:  28%|##8       | 14238/50000 [8:09:46<18:00:59,  1.81s/it]
Outer training loop:  28%|##8       | 14239/50000 [8:09:48<18:07:06,  1.82s/it]
Outer training loop:  28%|##8       | 14240/50000 [8:09:50<18:06:26,  1.82s/it]
Outer training loop:  28%|##8       | 14241/50000 [8:09:52<17:58:41,  1.81s/it]
Outer training loop:  28%|##8       | 14242/50000 [8:09:54<18:35:25,  1.87s/it]
Outer training loop:  28%|##8       | 14243/50000 [8:09:56<18:29:16,  1.86s/it]
Outer training loop:  28%|##8       | 14244/50000 [8:09:57<18:06:58,  1.82s/it]
Outer training loop:  28%|##8       | 14245/50000 [8:09:59<18:06:22,  1.82s/it]
Outer training loop:  28%|##8       | 14246/50000 [8:10:01<18:27:24,  1.86s/it]
Outer training loop:  28%|##8       | 14247/50000 [8:10:03<18:10:33,  1.83s/it]
Outer training loop:  28%|##8       | 14248/50000 [8:10:05<18:20:37,  1.85s/it]
Outer training loop:  28%|##8       | 14249/50000 [8:10:07<18:32:29,  1.87s/it]
Outer training loop:  28%|##8       | 14250/50000 [8:10:09<18:11:15,  1.83s/it]
Outer training loop:  29%|##8       | 14251/50000 [8:10:10<17:43:08,  1.78s/it]
Outer training loop:  29%|##8       | 14252/50000 [8:10:12<17:41:47,  1.78s/it]
Outer training loop:  29%|##8       | 14253/50000 [8:10:14<18:21:01,  1.85s/it]
Outer training loop:  29%|##8       | 14254/50000 [8:10:16<18:21:01,  1.85s/it]
Outer training loop:  29%|##8       | 14255/50000 [8:10:18<18:18:29,  1.84s/it]
Outer training loop:  29%|##8       | 14256/50000 [8:10:20<18:28:50,  1.86s/it]
Outer training loop:  29%|##8       | 14257/50000 [8:10:21<18:04:39,  1.82s/it]
Outer training loop:  29%|##8       | 14258/50000 [8:10:23<18:13:07,  1.84s/it]
Outer training loop:  29%|##8       | 14259/50000 [8:10:25<18:05:17,  1.82s/it]
Outer training loop:  29%|##8       | 14260/50000 [8:10:27<18:21:01,  1.85s/it]
Outer training loop:  29%|##8       | 14261/50000 [8:10:29<18:05:07,  1.82s/it]
Outer training loop:  29%|##8       | 14262/50000 [8:10:30<17:33:16,  1.77s/it]
Outer training loop:  29%|##8       | 14263/50000 [8:10:32<17:54:10,  1.80s/it]
Outer training loop:  29%|##8       | 14264/50000 [8:10:34<17:44:59,  1.79s/it]
Outer training loop:  29%|##8       | 14265/50000 [8:10:36<17:44:51,  1.79s/it]
Outer training loop:  29%|##8       | 14266/50000 [8:10:37<17:45:13,  1.79s/it]
Outer training loop:  29%|##8       | 14267/50000 [8:10:39<18:17:31,  1.84s/it]
Outer training loop:  29%|##8       | 14268/50000 [8:10:41<18:05:06,  1.82s/it]
Outer training loop:  29%|##8       | 14269/50000 [8:10:43<18:12:00,  1.83s/it]
Outer training loop:  29%|##8       | 14270/50000 [8:10:45<18:37:37,  1.88s/it]
Outer training loop:  29%|##8       | 14271/50000 [8:10:47<18:17:23,  1.84s/it]
Outer training loop:  29%|##8       | 14272/50000 [8:10:49<18:04:59,  1.82s/it]
Outer training loop:  29%|##8       | 14273/50000 [8:10:50<18:14:59,  1.84s/it]
Outer training loop:  29%|##8       | 14274/50000 [8:10:52<18:32:49,  1.87s/it]
Outer training loop:  29%|##8       | 14275/50000 [8:10:54<18:09:25,  1.83s/it]
Outer training loop:  29%|##8       | 14276/50000 [8:10:56<17:56:04,  1.81s/it]
Outer training loop:  29%|##8       | 14277/50000 [8:10:58<18:16:04,  1.84s/it]
Outer training loop:  29%|##8       | 14278/50000 [8:11:00<18:13:53,  1.84s/it]
Outer training loop:  29%|##8       | 14279/50000 [8:11:01<17:52:09,  1.80s/it]
Outer training loop:  29%|##8       | 14280/50000 [8:11:03<18:23:15,  1.85s/it]
Outer training loop:  29%|##8       | 14281/50000 [8:11:05<18:04:27,  1.82s/it]
Outer training loop:  29%|##8       | 14282/50000 [8:11:07<18:16:24,  1.84s/it]
Outer training loop:  29%|##8       | 14283/50000 [8:11:09<18:16:07,  1.84s/it]
Outer training loop:  29%|##8       | 14284/50000 [8:11:11<18:05:13,  1.82s/it]
Outer training loop:  29%|##8       | 14285/50000 [8:11:12<18:05:06,  1.82s/it]
Outer training loop:  29%|##8       | 14286/50000 [8:11:14<18:20:52,  1.85s/it]
Outer training loop:  29%|##8       | 14287/50000 [8:11:16<18:06:37,  1.83s/it]
Outer training loop:  29%|##8       | 14288/50000 [8:11:18<17:54:01,  1.80s/it]
Outer training loop:  29%|##8       | 14289/50000 [8:11:20<17:58:02,  1.81s/it]
Outer training loop:  29%|##8       | 14290/50000 [8:11:22<18:08:28,  1.83s/it]
Outer training loop:  29%|##8       | 14291/50000 [8:11:23<17:56:59,  1.81s/it]
Outer training loop:  29%|##8       | 14292/50000 [8:11:25<17:30:15,  1.76s/it]
Outer training loop:  29%|##8       | 14293/50000 [8:11:27<17:52:45,  1.80s/it]
Outer training loop:  29%|##8       | 14294/50000 [8:11:29<17:38:08,  1.78s/it]
Outer training loop:  29%|##8       | 14295/50000 [8:11:30<17:47:12,  1.79s/it]
Outer training loop:  29%|##8       | 14296/50000 [8:11:32<18:27:30,  1.86s/it]
Outer training loop:  29%|##8       | 14297/50000 [8:11:34<18:13:59,  1.84s/it]
Outer training loop:  29%|##8       | 14298/50000 [8:11:36<18:11:00,  1.83s/it]
Outer training loop:  29%|##8       | 14299/50000 [8:11:38<18:24:31,  1.86s/it]
Outer training loop:  29%|##8       | 14300/50000 [8:11:40<18:05:59,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  29%|##8       | 14301/50000 [8:11:47<33:44:57,  3.40s/it]
Outer training loop:  29%|##8       | 14302/50000 [8:11:49<29:27:13,  2.97s/it]
Outer training loop:  29%|##8       | 14303/50000 [8:11:51<26:09:14,  2.64s/it]
Outer training loop:  29%|##8       | 14304/50000 [8:11:52<23:45:36,  2.40s/it]
Outer training loop:  29%|##8       | 14305/50000 [8:11:54<21:40:51,  2.19s/it]
Outer training loop:  29%|##8       | 14306/50000 [8:11:56<20:25:46,  2.06s/it]
Outer training loop:  29%|##8       | 14307/50000 [8:11:58<20:21:33,  2.05s/it]
Outer training loop:  29%|##8       | 14308/50000 [8:12:00<19:18:10,  1.95s/it]
Outer training loop:  29%|##8       | 14309/50000 [8:12:01<18:19:38,  1.85s/it]
Outer training loop:  29%|##8       | 14310/50000 [8:12:03<17:59:37,  1.82s/it]
Outer training loop:  29%|##8       | 14311/50000 [8:12:05<18:21:01,  1.85s/it]
Outer training loop:  29%|##8       | 14312/50000 [8:12:07<17:56:43,  1.81s/it]
Outer training loop:  29%|##8       | 14313/50000 [8:12:08<17:46:08,  1.79s/it]
Outer training loop:  29%|##8       | 14314/50000 [8:12:10<17:56:14,  1.81s/it]
Outer training loop:  29%|##8       | 14315/50000 [8:12:12<18:07:53,  1.83s/it]
Outer training loop:  29%|##8       | 14316/50000 [8:12:14<17:41:44,  1.79s/it]
Outer training loop:  29%|##8       | 14317/50000 [8:12:16<18:15:53,  1.84s/it]
Outer training loop:  29%|##8       | 14318/50000 [8:12:18<18:02:09,  1.82s/it]
Outer training loop:  29%|##8       | 14319/50000 [8:12:19<17:29:16,  1.76s/it]
Outer training loop:  29%|##8       | 14320/50000 [8:12:21<17:33:38,  1.77s/it]
Outer training loop:  29%|##8       | 14321/50000 [8:12:23<18:17:28,  1.85s/it]
Outer training loop:  29%|##8       | 14322/50000 [8:12:25<18:00:27,  1.82s/it]
Outer training loop:  29%|##8       | 14323/50000 [8:12:26<17:43:50,  1.79s/it]
Outer training loop:  29%|##8       | 14324/50000 [8:12:28<17:55:33,  1.81s/it]
Outer training loop:  29%|##8       | 14325/50000 [8:12:30<17:16:13,  1.74s/it]
Outer training loop:  29%|##8       | 14326/50000 [8:12:32<17:31:52,  1.77s/it]
Outer training loop:  29%|##8       | 14327/50000 [8:12:33<17:25:40,  1.76s/it]
Outer training loop:  29%|##8       | 14328/50000 [8:12:35<17:54:13,  1.81s/it]
Outer training loop:  29%|##8       | 14329/50000 [8:12:37<17:40:21,  1.78s/it]
Outer training loop:  29%|##8       | 14330/50000 [8:12:39<17:29:59,  1.77s/it]
Outer training loop:  29%|##8       | 14331/50000 [8:12:41<18:11:23,  1.84s/it]
Outer training loop:  29%|##8       | 14332/50000 [8:12:43<18:04:45,  1.82s/it]
Outer training loop:  29%|##8       | 14333/50000 [8:12:44<17:43:41,  1.79s/it]
Outer training loop:  29%|##8       | 14334/50000 [8:12:46<18:18:42,  1.85s/it]
Outer training loop:  29%|##8       | 14335/50000 [8:12:48<17:51:16,  1.80s/it]
Outer training loop:  29%|##8       | 14336/50000 [8:12:50<17:40:33,  1.78s/it]
Outer training loop:  29%|##8       | 14337/50000 [8:12:51<17:24:07,  1.76s/it]
Outer training loop:  29%|##8       | 14338/50000 [8:12:53<17:47:24,  1.80s/it]
Outer training loop:  29%|##8       | 14339/50000 [8:12:55<17:34:59,  1.78s/it]
Outer training loop:  29%|##8       | 14340/50000 [8:12:57<17:39:54,  1.78s/it]
Outer training loop:  29%|##8       | 14341/50000 [8:12:59<17:53:50,  1.81s/it]
Outer training loop:  29%|##8       | 14342/50000 [8:13:00<17:38:26,  1.78s/it]
Outer training loop:  29%|##8       | 14343/50000 [8:13:02<17:32:41,  1.77s/it]
Outer training loop:  29%|##8       | 14344/50000 [8:13:04<18:04:06,  1.82s/it]
Outer training loop:  29%|##8       | 14345/50000 [8:13:06<17:59:35,  1.82s/it]
Outer training loop:  29%|##8       | 14346/50000 [8:13:08<17:52:06,  1.80s/it]
Outer training loop:  29%|##8       | 14347/50000 [8:13:10<18:22:03,  1.85s/it]
Outer training loop:  29%|##8       | 14348/50000 [8:13:11<18:03:41,  1.82s/it]
Outer training loop:  29%|##8       | 14349/50000 [8:13:13<17:57:25,  1.81s/it]
Outer training loop:  29%|##8       | 14350/50000 [8:13:15<17:43:51,  1.79s/it]
Outer training loop:  29%|##8       | 14351/50000 [8:13:17<18:08:48,  1.83s/it]
Outer training loop:  29%|##8       | 14352/50000 [8:13:19<17:39:59,  1.78s/it]
Outer training loop:  29%|##8       | 14353/50000 [8:13:20<17:38:04,  1.78s/it]
Outer training loop:  29%|##8       | 14354/50000 [8:13:22<17:42:53,  1.79s/it]
Outer training loop:  29%|##8       | 14355/50000 [8:13:24<17:23:50,  1.76s/it]
Outer training loop:  29%|##8       | 14356/50000 [8:13:26<17:41:06,  1.79s/it]
Outer training loop:  29%|##8       | 14357/50000 [8:13:28<17:59:13,  1.82s/it]
Outer training loop:  29%|##8       | 14358/50000 [8:13:29<17:17:46,  1.75s/it]
Outer training loop:  29%|##8       | 14359/50000 [8:13:31<17:18:39,  1.75s/it]
Outer training loop:  29%|##8       | 14360/50000 [8:13:33<17:34:37,  1.78s/it]
Outer training loop:  29%|##8       | 14361/50000 [8:13:34<17:19:23,  1.75s/it]
Outer training loop:  29%|##8       | 14362/50000 [8:13:36<17:02:42,  1.72s/it]
Outer training loop:  29%|##8       | 14363/50000 [8:13:38<17:06:00,  1.73s/it]
Outer training loop:  29%|##8       | 14364/50000 [8:13:40<17:25:29,  1.76s/it]
Outer training loop:  29%|##8       | 14365/50000 [8:13:41<17:12:20,  1.74s/it]
Outer training loop:  29%|##8       | 14366/50000 [8:13:43<17:10:34,  1.74s/it]
Outer training loop:  29%|##8       | 14367/50000 [8:13:45<17:57:25,  1.81s/it]
Outer training loop:  29%|##8       | 14368/50000 [8:13:47<17:37:45,  1.78s/it]
Outer training loop:  29%|##8       | 14369/50000 [8:13:49<17:45:16,  1.79s/it]
Outer training loop:  29%|##8       | 14370/50000 [8:13:51<18:11:00,  1.84s/it]
Outer training loop:  29%|##8       | 14371/50000 [8:13:52<18:07:01,  1.83s/it]
Outer training loop:  29%|##8       | 14372/50000 [8:13:54<17:46:34,  1.80s/it]
Outer training loop:  29%|##8       | 14373/50000 [8:13:56<17:43:50,  1.79s/it]
Outer training loop:  29%|##8       | 14374/50000 [8:13:58<18:08:05,  1.83s/it]
Outer training loop:  29%|##8       | 14375/50000 [8:14:00<17:59:43,  1.82s/it]
Outer training loop:  29%|##8       | 14376/50000 [8:14:01<17:44:30,  1.79s/it]
Outer training loop:  29%|##8       | 14377/50000 [8:14:03<17:58:23,  1.82s/it]
Outer training loop:  29%|##8       | 14378/50000 [8:14:05<17:46:30,  1.80s/it]
Outer training loop:  29%|##8       | 14379/50000 [8:14:07<17:26:26,  1.76s/it]
Outer training loop:  29%|##8       | 14380/50000 [8:14:08<17:18:06,  1.75s/it]
Outer training loop:  29%|##8       | 14381/50000 [8:14:10<17:44:32,  1.79s/it]
Outer training loop:  29%|##8       | 14382/50000 [8:14:12<17:50:46,  1.80s/it]
Outer training loop:  29%|##8       | 14383/50000 [8:14:14<17:41:16,  1.79s/it]
Outer training loop:  29%|##8       | 14384/50000 [8:14:16<18:05:28,  1.83s/it]
Outer training loop:  29%|##8       | 14385/50000 [8:14:17<17:41:11,  1.79s/it]
Outer training loop:  29%|##8       | 14386/50000 [8:14:19<17:36:51,  1.78s/it]
Outer training loop:  29%|##8       | 14387/50000 [8:14:21<17:37:59,  1.78s/it]
Outer training loop:  29%|##8       | 14388/50000 [8:14:23<18:29:23,  1.87s/it]
Outer training loop:  29%|##8       | 14389/50000 [8:14:25<18:21:48,  1.86s/it]
Outer training loop:  29%|##8       | 14390/50000 [8:14:27<18:05:17,  1.83s/it]
Outer training loop:  29%|##8       | 14391/50000 [8:14:29<18:04:35,  1.83s/it]
Outer training loop:  29%|##8       | 14392/50000 [8:14:30<17:30:11,  1.77s/it]
Outer training loop:  29%|##8       | 14393/50000 [8:14:32<17:23:18,  1.76s/it]
Outer training loop:  29%|##8       | 14394/50000 [8:14:34<17:40:11,  1.79s/it]
Outer training loop:  29%|##8       | 14395/50000 [8:14:35<17:26:06,  1.76s/it]
Outer training loop:  29%|##8       | 14396/50000 [8:14:37<17:16:04,  1.75s/it]
Outer training loop:  29%|##8       | 14397/50000 [8:14:39<17:04:24,  1.73s/it]
Outer training loop:  29%|##8       | 14398/50000 [8:14:41<17:38:02,  1.78s/it]
Outer training loop:  29%|##8       | 14399/50000 [8:14:43<17:48:59,  1.80s/it]
Outer training loop:  29%|##8       | 14400/50000 [8:14:44<17:46:15,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.66s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.68s/it]

Outer training loop:  29%|##8       | 14401/50000 [8:14:51<33:16:08,  3.36s/it]
Outer training loop:  29%|##8       | 14402/50000 [8:14:53<28:59:57,  2.93s/it]
Outer training loop:  29%|##8       | 14403/50000 [8:14:55<25:17:35,  2.56s/it]
Outer training loop:  29%|##8       | 14404/50000 [8:14:57<23:05:49,  2.34s/it]
Outer training loop:  29%|##8       | 14405/50000 [8:14:59<22:00:58,  2.23s/it]
Outer training loop:  29%|##8       | 14406/50000 [8:15:01<20:37:40,  2.09s/it]
Outer training loop:  29%|##8       | 14407/50000 [8:15:02<19:55:48,  2.02s/it]
Outer training loop:  29%|##8       | 14408/50000 [8:15:04<19:29:55,  1.97s/it]
Outer training loop:  29%|##8       | 14409/50000 [8:15:06<19:28:42,  1.97s/it]
Outer training loop:  29%|##8       | 14410/50000 [8:15:08<18:44:36,  1.90s/it]
Outer training loop:  29%|##8       | 14411/50000 [8:15:10<18:26:39,  1.87s/it]
Outer training loop:  29%|##8       | 14412/50000 [8:15:12<18:30:39,  1.87s/it]
Outer training loop:  29%|##8       | 14413/50000 [8:15:13<18:16:17,  1.85s/it]
Outer training loop:  29%|##8       | 14414/50000 [8:15:15<17:38:17,  1.78s/it]
Outer training loop:  29%|##8       | 14415/50000 [8:15:17<17:36:22,  1.78s/it]
Outer training loop:  29%|##8       | 14416/50000 [8:15:19<18:07:36,  1.83s/it]
Outer training loop:  29%|##8       | 14417/50000 [8:15:21<17:56:57,  1.82s/it]
Outer training loop:  29%|##8       | 14418/50000 [8:15:23<18:21:39,  1.86s/it]
Outer training loop:  29%|##8       | 14419/50000 [8:15:24<18:22:15,  1.86s/it]
Outer training loop:  29%|##8       | 14420/50000 [8:15:26<18:54:56,  1.91s/it]
Outer training loop:  29%|##8       | 14421/50000 [8:15:28<18:22:35,  1.86s/it]
Outer training loop:  29%|##8       | 14422/50000 [8:15:30<17:48:55,  1.80s/it]
Outer training loop:  29%|##8       | 14423/50000 [8:15:32<18:09:13,  1.84s/it]
Outer training loop:  29%|##8       | 14424/50000 [8:15:34<18:18:09,  1.85s/it]
Outer training loop:  29%|##8       | 14425/50000 [8:15:35<18:15:33,  1.85s/it]
Outer training loop:  29%|##8       | 14426/50000 [8:15:37<18:12:56,  1.84s/it]
Outer training loop:  29%|##8       | 14427/50000 [8:15:39<17:58:26,  1.82s/it]
Outer training loop:  29%|##8       | 14428/50000 [8:15:41<17:45:52,  1.80s/it]
Outer training loop:  29%|##8       | 14429/50000 [8:15:43<17:57:50,  1.82s/it]
Outer training loop:  29%|##8       | 14430/50000 [8:15:45<18:31:43,  1.88s/it]
Outer training loop:  29%|##8       | 14431/50000 [8:15:46<18:17:45,  1.85s/it]
Outer training loop:  29%|##8       | 14432/50000 [8:15:48<18:28:27,  1.87s/it]
Outer training loop:  29%|##8       | 14433/50000 [8:15:50<18:57:28,  1.92s/it]
Outer training loop:  29%|##8       | 14434/50000 [8:15:52<18:43:08,  1.89s/it]
Outer training loop:  29%|##8       | 14435/50000 [8:15:54<18:07:40,  1.83s/it]
Outer training loop:  29%|##8       | 14436/50000 [8:15:56<18:31:46,  1.88s/it]
Outer training loop:  29%|##8       | 14437/50000 [8:15:58<18:10:06,  1.84s/it]
Outer training loop:  29%|##8       | 14438/50000 [8:15:59<17:51:26,  1.81s/it]
Outer training loop:  29%|##8       | 14439/50000 [8:16:01<18:16:40,  1.85s/it]
Outer training loop:  29%|##8       | 14440/50000 [8:16:03<18:07:10,  1.83s/it]
Outer training loop:  29%|##8       | 14441/50000 [8:16:05<17:54:39,  1.81s/it]
Outer training loop:  29%|##8       | 14442/50000 [8:16:07<17:37:16,  1.78s/it]
Outer training loop:  29%|##8       | 14443/50000 [8:16:09<17:55:05,  1.81s/it]
Outer training loop:  29%|##8       | 14444/50000 [8:16:10<17:34:44,  1.78s/it]
Outer training loop:  29%|##8       | 14445/50000 [8:16:12<17:19:10,  1.75s/it]
Outer training loop:  29%|##8       | 14446/50000 [8:16:14<17:37:51,  1.79s/it]
Outer training loop:  29%|##8       | 14447/50000 [8:16:15<17:22:04,  1.76s/it]
Outer training loop:  29%|##8       | 14448/50000 [8:16:17<17:09:23,  1.74s/it]
Outer training loop:  29%|##8       | 14449/50000 [8:16:19<17:53:38,  1.81s/it]
Outer training loop:  29%|##8       | 14450/50000 [8:16:21<17:34:20,  1.78s/it]
Outer training loop:  29%|##8       | 14451/50000 [8:16:23<17:43:40,  1.80s/it]
Outer training loop:  29%|##8       | 14452/50000 [8:16:25<18:46:42,  1.90s/it]
Outer training loop:  29%|##8       | 14453/50000 [8:16:27<18:09:41,  1.84s/it]
Outer training loop:  29%|##8       | 14454/50000 [8:16:28<18:17:02,  1.85s/it]
Outer training loop:  29%|##8       | 14455/50000 [8:16:30<18:01:29,  1.83s/it]
Outer training loop:  29%|##8       | 14456/50000 [8:16:32<18:22:52,  1.86s/it]
Outer training loop:  29%|##8       | 14457/50000 [8:16:34<18:11:48,  1.84s/it]
Outer training loop:  29%|##8       | 14458/50000 [8:16:36<17:45:30,  1.80s/it]
Outer training loop:  29%|##8       | 14459/50000 [8:16:38<18:29:14,  1.87s/it]
Outer training loop:  29%|##8       | 14460/50000 [8:16:39<18:07:14,  1.84s/it]
Outer training loop:  29%|##8       | 14461/50000 [8:16:41<17:57:24,  1.82s/it]
Outer training loop:  29%|##8       | 14462/50000 [8:16:43<17:50:49,  1.81s/it]
Outer training loop:  29%|##8       | 14463/50000 [8:16:45<17:43:23,  1.80s/it]
Outer training loop:  29%|##8       | 14464/50000 [8:16:46<17:31:30,  1.78s/it]
Outer training loop:  29%|##8       | 14465/50000 [8:16:48<17:26:56,  1.77s/it]
Outer training loop:  29%|##8       | 14466/50000 [8:16:50<17:44:22,  1.80s/it]
Outer training loop:  29%|##8       | 14467/50000 [8:16:52<17:02:38,  1.73s/it]
Outer training loop:  29%|##8       | 14468/50000 [8:16:54<17:36:19,  1.78s/it]
Outer training loop:  29%|##8       | 14469/50000 [8:16:55<17:35:40,  1.78s/it]
Outer training loop:  29%|##8       | 14470/50000 [8:16:57<18:00:08,  1.82s/it]
Outer training loop:  29%|##8       | 14471/50000 [8:16:59<17:53:27,  1.81s/it]
Outer training loop:  29%|##8       | 14472/50000 [8:17:01<17:53:07,  1.81s/it]
Outer training loop:  29%|##8       | 14473/50000 [8:17:03<17:50:24,  1.81s/it]
Outer training loop:  29%|##8       | 14474/50000 [8:17:04<17:50:13,  1.81s/it]
Outer training loop:  29%|##8       | 14475/50000 [8:17:06<17:55:19,  1.82s/it]
Outer training loop:  29%|##8       | 14476/50000 [8:17:08<18:23:55,  1.86s/it]
Outer training loop:  29%|##8       | 14477/50000 [8:17:10<18:19:15,  1.86s/it]
Outer training loop:  29%|##8       | 14478/50000 [8:17:12<17:40:49,  1.79s/it]
Outer training loop:  29%|##8       | 14479/50000 [8:17:14<18:12:52,  1.85s/it]
Outer training loop:  29%|##8       | 14480/50000 [8:17:15<17:55:18,  1.82s/it]
Outer training loop:  29%|##8       | 14481/50000 [8:17:17<17:24:14,  1.76s/it]
Outer training loop:  29%|##8       | 14482/50000 [8:17:19<17:05:47,  1.73s/it]
Outer training loop:  29%|##8       | 14483/50000 [8:17:21<17:44:14,  1.80s/it]
Outer training loop:  29%|##8       | 14484/50000 [8:17:22<17:35:39,  1.78s/it]
Outer training loop:  29%|##8       | 14485/50000 [8:17:24<17:41:24,  1.79s/it]
Outer training loop:  29%|##8       | 14486/50000 [8:17:26<17:54:30,  1.82s/it]
Outer training loop:  29%|##8       | 14487/50000 [8:17:28<18:19:01,  1.86s/it]
Outer training loop:  29%|##8       | 14488/50000 [8:17:30<18:04:56,  1.83s/it]
Outer training loop:  29%|##8       | 14489/50000 [8:17:32<18:18:04,  1.86s/it]
Outer training loop:  29%|##8       | 14490/50000 [8:17:34<17:50:00,  1.81s/it]
Outer training loop:  29%|##8       | 14491/50000 [8:17:35<17:45:37,  1.80s/it]
Outer training loop:  29%|##8       | 14492/50000 [8:17:37<17:34:50,  1.78s/it]
Outer training loop:  29%|##8       | 14493/50000 [8:17:39<18:06:45,  1.84s/it]
Outer training loop:  29%|##8       | 14494/50000 [8:17:41<17:43:33,  1.80s/it]
Outer training loop:  29%|##8       | 14495/50000 [8:17:42<17:40:33,  1.79s/it]
Outer training loop:  29%|##8       | 14496/50000 [8:17:44<17:40:19,  1.79s/it]
Outer training loop:  29%|##8       | 14497/50000 [8:17:46<17:30:26,  1.78s/it]
Outer training loop:  29%|##8       | 14498/50000 [8:17:48<17:38:04,  1.79s/it]
Outer training loop:  29%|##8       | 14499/50000 [8:17:50<17:43:18,  1.80s/it]
Outer training loop:  29%|##9       | 14500/50000 [8:17:52<18:11:21,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.66s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:12,  1.62s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:13,  1.65s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:21,  1.76s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:17,  1.75s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:15,  1.74s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:18,  1.80s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:16,  1.80s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:14,  1.79s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:13,  1.81s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:14,  1.84s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:11,  1.82s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:08,  1.81s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:02,  1.75s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:04,  1.81s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:06,  1.86s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:00,  1.80s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:57,  1.77s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:54,  1.76s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:57,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:56,  1.85s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:51,  1.81s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:46,  1.75s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:44,  1.75s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:47,  1.83s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:46,  1.83s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:42,  1.79s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:41,  1.81s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:39,  1.80s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:38,  1.83s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:35,  1.80s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:33,  1.80s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:30,  1.77s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:28,  1.78s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:25,  1.74s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:24,  1.81s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:21,  1.77s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:19,  1.76s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:17,  1.77s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.73s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:12,  1.72s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:15,  1.84s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:12,  1.80s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:08,  1.76s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:05,  1.73s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:03,  1.70s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:00,  1.68s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<00:59,  1.69s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:01,  1.81s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:58,  1.79s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:56,  1.75s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:54,  1.75s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:51,  1.72s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:49,  1.71s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:47,  1.70s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:47,  1.83s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:45,  1.81s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.77s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.76s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:38,  1.74s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:36,  1.73s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:34,  1.73s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:30,  1.77s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.75s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:25,  1.73s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:23,  1.71s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.74s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:20,  1.72s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:18,  1.71s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:16,  1.81s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.81s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.79s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.79s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.78s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.82s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.81s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.76s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:14,  1.64s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:13,  1.64s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:11,  1.64s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:13,  1.68s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:12,  1.70s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:11,  1.71s/it]

Evaluating policy:   8%|8         | 7/83 [00:11<02:10,  1.72s/it]

Evaluating policy:  10%|9         | 8/83 [00:13<02:09,  1.72s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:08,  1.74s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:14,  1.86s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:08,  1.81s/it]

Evaluating policy:  16%|#5        | 13/83 [00:22<02:05,  1.79s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:07,  1.84s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:04,  1.83s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:01,  1.82s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.80s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:55,  1.78s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:53,  1.80s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:50,  1.78s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:48,  1.78s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:51,  1.86s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:47,  1.83s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.78s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:40,  1.77s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:36,  1.73s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:40,  1.83s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.79s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.77s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:33,  1.79s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:31,  1.79s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:28,  1.77s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:27,  1.83s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:25,  1.83s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:23,  1.81s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:21,  1.80s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.75s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:20,  1.87s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:15,  1.84s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:12,  1.81s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:10,  1.80s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:08,  1.80s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:10,  1.90s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:02,  1.80s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<01:01,  1.80s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:57,  1.75s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:56,  1.77s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:53,  1.73s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:51,  1.72s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:51,  1.85s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:48,  1.81s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:46,  1.79s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:44,  1.78s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.75s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:39,  1.74s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:40,  1.84s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:37,  1.78s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:34,  1.75s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:32,  1.72s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:30,  1.71s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.72s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.74s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:25,  1.73s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:23,  1.81s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:21,  1.79s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:19,  1.74s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.73s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:15,  1.73s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:13,  1.74s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.76s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.75s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:09,  1.87s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.87s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.85s/it]

Evaluating policy:  98%|#########7| 81/83 [02:25<00:03,  1.85s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.79s/it]

Outer training loop:  29%|##9       | 14501/50000 [8:22:55<911:25:48, 92.43s/it]
Outer training loop:  29%|##9       | 14502/50000 [8:22:57<643:24:59, 65.25s/it]
Outer training loop:  29%|##9       | 14503/50000 [8:22:59<455:47:03, 46.22s/it]
Outer training loop:  29%|##9       | 14504/50000 [8:23:01<324:48:50, 32.94s/it]
Outer training loop:  29%|##9       | 14505/50000 [8:23:03<232:30:26, 23.58s/it]
Outer training loop:  29%|##9       | 14506/50000 [8:23:05<169:19:01, 17.17s/it]
Outer training loop:  29%|##9       | 14507/50000 [8:23:07<123:25:12, 12.52s/it]
Outer training loop:  29%|##9       | 14508/50000 [8:23:08<91:51:11,  9.32s/it] 
Outer training loop:  29%|##9       | 14509/50000 [8:23:11<70:29:00,  7.15s/it]
Outer training loop:  29%|##9       | 14510/50000 [8:23:12<54:28:00,  5.52s/it]
Outer training loop:  29%|##9       | 14511/50000 [8:23:14<43:23:24,  4.40s/it]
Outer training loop:  29%|##9       | 14512/50000 [8:23:16<36:06:02,  3.66s/it]
Outer training loop:  29%|##9       | 14513/50000 [8:23:18<30:38:27,  3.11s/it]
Outer training loop:  29%|##9       | 14514/50000 [8:23:20<27:06:02,  2.75s/it]
Outer training loop:  29%|##9       | 14515/50000 [8:23:22<24:11:39,  2.45s/it]
Outer training loop:  29%|##9       | 14516/50000 [8:23:23<22:36:23,  2.29s/it]
Outer training loop:  29%|##9       | 14517/50000 [8:23:25<21:16:45,  2.16s/it]
Outer training loop:  29%|##9       | 14518/50000 [8:23:27<19:57:52,  2.03s/it]
Outer training loop:  29%|##9       | 14519/50000 [8:23:29<19:48:42,  2.01s/it]
Outer training loop:  29%|##9       | 14520/50000 [8:23:31<18:59:51,  1.93s/it]
Outer training loop:  29%|##9       | 14521/50000 [8:23:32<18:25:50,  1.87s/it]
Outer training loop:  29%|##9       | 14522/50000 [8:23:34<18:49:29,  1.91s/it]
Outer training loop:  29%|##9       | 14523/50000 [8:23:36<18:29:55,  1.88s/it]
Outer training loop:  29%|##9       | 14524/50000 [8:23:38<18:04:03,  1.83s/it]
Outer training loop:  29%|##9       | 14525/50000 [8:23:40<18:13:03,  1.85s/it]
Outer training loop:  29%|##9       | 14526/50000 [8:23:42<18:04:02,  1.83s/it]
Outer training loop:  29%|##9       | 14527/50000 [8:23:43<17:57:19,  1.82s/it]
Outer training loop:  29%|##9       | 14528/50000 [8:23:45<17:50:16,  1.81s/it]
Outer training loop:  29%|##9       | 14529/50000 [8:23:47<18:00:36,  1.83s/it]
Outer training loop:  29%|##9       | 14530/50000 [8:23:49<17:27:47,  1.77s/it]
Outer training loop:  29%|##9       | 14531/50000 [8:23:50<17:17:10,  1.75s/it]
Outer training loop:  29%|##9       | 14532/50000 [8:23:52<17:25:22,  1.77s/it]
Outer training loop:  29%|##9       | 14533/50000 [8:23:54<17:37:48,  1.79s/it]
Outer training loop:  29%|##9       | 14534/50000 [8:23:56<17:29:41,  1.78s/it]
Outer training loop:  29%|##9       | 14535/50000 [8:23:58<17:43:35,  1.80s/it]
Outer training loop:  29%|##9       | 14536/50000 [8:23:59<17:42:55,  1.80s/it]
Outer training loop:  29%|##9       | 14537/50000 [8:24:01<17:27:07,  1.77s/it]
Outer training loop:  29%|##9       | 14538/50000 [8:24:03<16:59:17,  1.72s/it]
Outer training loop:  29%|##9       | 14539/50000 [8:24:05<17:44:01,  1.80s/it]
Outer training loop:  29%|##9       | 14540/50000 [8:24:07<17:38:00,  1.79s/it]
Outer training loop:  29%|##9       | 14541/50000 [8:24:08<17:27:21,  1.77s/it]
Outer training loop:  29%|##9       | 14542/50000 [8:24:10<17:55:03,  1.82s/it]
Outer training loop:  29%|##9       | 14543/50000 [8:24:12<17:26:52,  1.77s/it]
Outer training loop:  29%|##9       | 14544/50000 [8:24:13<17:00:15,  1.73s/it]
Outer training loop:  29%|##9       | 14545/50000 [8:24:15<16:55:29,  1.72s/it]
Outer training loop:  29%|##9       | 14546/50000 [8:24:17<17:40:03,  1.79s/it]
Outer training loop:  29%|##9       | 14547/50000 [8:24:19<17:37:52,  1.79s/it]
Outer training loop:  29%|##9       | 14548/50000 [8:24:21<17:41:47,  1.80s/it]
Outer training loop:  29%|##9       | 14549/50000 [8:24:23<18:45:41,  1.91s/it]
Outer training loop:  29%|##9       | 14550/50000 [8:24:25<18:19:22,  1.86s/it]
Outer training loop:  29%|##9       | 14551/50000 [8:24:26<17:53:39,  1.82s/it]
Outer training loop:  29%|##9       | 14552/50000 [8:24:28<17:54:03,  1.82s/it]
Outer training loop:  29%|##9       | 14553/50000 [8:24:30<18:07:22,  1.84s/it]
Outer training loop:  29%|##9       | 14554/50000 [8:24:32<17:59:01,  1.83s/it]
Outer training loop:  29%|##9       | 14555/50000 [8:24:34<17:47:19,  1.81s/it]
Outer training loop:  29%|##9       | 14556/50000 [8:24:36<17:57:44,  1.82s/it]
Outer training loop:  29%|##9       | 14557/50000 [8:24:37<17:35:30,  1.79s/it]
Outer training loop:  29%|##9       | 14558/50000 [8:24:39<17:20:56,  1.76s/it]
Outer training loop:  29%|##9       | 14559/50000 [8:24:41<18:05:49,  1.84s/it]
Outer training loop:  29%|##9       | 14560/50000 [8:24:43<17:46:32,  1.81s/it]
Outer training loop:  29%|##9       | 14561/50000 [8:24:44<17:15:10,  1.75s/it]
Outer training loop:  29%|##9       | 14562/50000 [8:24:46<17:04:20,  1.73s/it]
Outer training loop:  29%|##9       | 14563/50000 [8:24:48<17:36:39,  1.79s/it]
Outer training loop:  29%|##9       | 14564/50000 [8:24:50<17:20:24,  1.76s/it]
Outer training loop:  29%|##9       | 14565/50000 [8:24:51<17:15:49,  1.75s/it]
Outer training loop:  29%|##9       | 14566/50000 [8:24:53<17:16:36,  1.76s/it]
Outer training loop:  29%|##9       | 14567/50000 [8:24:55<17:31:36,  1.78s/it]
Outer training loop:  29%|##9       | 14568/50000 [8:24:57<17:29:37,  1.78s/it]
Outer training loop:  29%|##9       | 14569/50000 [8:24:58<17:24:04,  1.77s/it]
Outer training loop:  29%|##9       | 14570/50000 [8:25:00<17:44:37,  1.80s/it]
Outer training loop:  29%|##9       | 14571/50000 [8:25:02<17:36:12,  1.79s/it]
Outer training loop:  29%|##9       | 14572/50000 [8:25:04<17:26:38,  1.77s/it]
Outer training loop:  29%|##9       | 14573/50000 [8:25:06<18:07:39,  1.84s/it]
Outer training loop:  29%|##9       | 14574/50000 [8:25:08<17:54:05,  1.82s/it]
Outer training loop:  29%|##9       | 14575/50000 [8:25:09<17:38:49,  1.79s/it]
Outer training loop:  29%|##9       | 14576/50000 [8:25:11<17:17:22,  1.76s/it]
Outer training loop:  29%|##9       | 14577/50000 [8:25:13<17:33:53,  1.79s/it]
Outer training loop:  29%|##9       | 14578/50000 [8:25:15<17:12:05,  1.75s/it]
Outer training loop:  29%|##9       | 14579/50000 [8:25:16<17:07:40,  1.74s/it]
Outer training loop:  29%|##9       | 14580/50000 [8:25:18<17:29:24,  1.78s/it]
Outer training loop:  29%|##9       | 14581/50000 [8:25:20<17:30:31,  1.78s/it]
Outer training loop:  29%|##9       | 14582/50000 [8:25:22<17:09:52,  1.74s/it]
Outer training loop:  29%|##9       | 14583/50000 [8:25:23<16:50:42,  1.71s/it]
Outer training loop:  29%|##9       | 14584/50000 [8:25:25<17:38:49,  1.79s/it]
Outer training loop:  29%|##9       | 14585/50000 [8:25:27<17:42:32,  1.80s/it]
Outer training loop:  29%|##9       | 14586/50000 [8:25:29<17:29:22,  1.78s/it]
Outer training loop:  29%|##9       | 14587/50000 [8:25:31<17:58:00,  1.83s/it]
Outer training loop:  29%|##9       | 14588/50000 [8:25:32<17:36:45,  1.79s/it]
Outer training loop:  29%|##9       | 14589/50000 [8:25:34<17:31:06,  1.78s/it]
Outer training loop:  29%|##9       | 14590/50000 [8:25:36<17:15:21,  1.75s/it]
Outer training loop:  29%|##9       | 14591/50000 [8:25:38<17:31:06,  1.78s/it]
Outer training loop:  29%|##9       | 14592/50000 [8:25:39<17:13:04,  1.75s/it]
Outer training loop:  29%|##9       | 14593/50000 [8:25:41<17:02:55,  1.73s/it]
Outer training loop:  29%|##9       | 14594/50000 [8:25:43<17:08:40,  1.74s/it]
Outer training loop:  29%|##9       | 14595/50000 [8:25:45<17:03:55,  1.74s/it]
Outer training loop:  29%|##9       | 14596/50000 [8:25:46<16:58:53,  1.73s/it]
Outer training loop:  29%|##9       | 14597/50000 [8:25:48<17:47:00,  1.81s/it]
Outer training loop:  29%|##9       | 14598/50000 [8:25:50<17:33:45,  1.79s/it]
Outer training loop:  29%|##9       | 14599/50000 [8:25:52<17:59:55,  1.83s/it]
Outer training loop:  29%|##9       | 14600/50000 [8:25:54<17:24:49,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.70s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  29%|##9       | 14601/50000 [8:26:01<33:41:31,  3.43s/it]
Outer training loop:  29%|##9       | 14602/50000 [8:26:02<28:34:00,  2.91s/it]
Outer training loop:  29%|##9       | 14603/50000 [8:26:04<24:55:26,  2.53s/it]
Outer training loop:  29%|##9       | 14604/50000 [8:26:06<22:23:56,  2.28s/it]
Outer training loop:  29%|##9       | 14605/50000 [8:26:08<21:19:59,  2.17s/it]
Outer training loop:  29%|##9       | 14606/50000 [8:26:10<20:19:51,  2.07s/it]
Outer training loop:  29%|##9       | 14607/50000 [8:26:11<19:26:32,  1.98s/it]
Outer training loop:  29%|##9       | 14608/50000 [8:26:13<19:29:21,  1.98s/it]
Outer training loop:  29%|##9       | 14609/50000 [8:26:15<18:53:39,  1.92s/it]
Outer training loop:  29%|##9       | 14610/50000 [8:26:17<18:18:42,  1.86s/it]
Outer training loop:  29%|##9       | 14611/50000 [8:26:19<17:45:26,  1.81s/it]
Outer training loop:  29%|##9       | 14612/50000 [8:26:20<18:06:54,  1.84s/it]
Outer training loop:  29%|##9       | 14613/50000 [8:26:22<17:53:32,  1.82s/it]
Outer training loop:  29%|##9       | 14614/50000 [8:26:24<17:20:51,  1.76s/it]
Outer training loop:  29%|##9       | 14615/50000 [8:26:26<17:37:51,  1.79s/it]
Outer training loop:  29%|##9       | 14616/50000 [8:26:28<17:36:19,  1.79s/it]
Outer training loop:  29%|##9       | 14617/50000 [8:26:29<17:21:35,  1.77s/it]
Outer training loop:  29%|##9       | 14618/50000 [8:26:31<16:56:29,  1.72s/it]
Outer training loop:  29%|##9       | 14619/50000 [8:26:33<17:28:39,  1.78s/it]
Outer training loop:  29%|##9       | 14620/50000 [8:26:35<17:29:16,  1.78s/it]
Outer training loop:  29%|##9       | 14621/50000 [8:26:36<17:34:39,  1.79s/it]
Outer training loop:  29%|##9       | 14622/50000 [8:26:38<17:49:17,  1.81s/it]
Outer training loop:  29%|##9       | 14623/50000 [8:26:40<17:27:52,  1.78s/it]
Outer training loop:  29%|##9       | 14624/50000 [8:26:42<17:26:16,  1.77s/it]
Outer training loop:  29%|##9       | 14625/50000 [8:26:44<17:43:23,  1.80s/it]
Outer training loop:  29%|##9       | 14626/50000 [8:26:45<17:24:18,  1.77s/it]
Outer training loop:  29%|##9       | 14627/50000 [8:26:47<17:06:50,  1.74s/it]
Outer training loop:  29%|##9       | 14628/50000 [8:26:49<17:18:19,  1.76s/it]
Outer training loop:  29%|##9       | 14629/50000 [8:26:51<17:51:11,  1.82s/it]
Outer training loop:  29%|##9       | 14630/50000 [8:26:52<17:25:44,  1.77s/it]
Outer training loop:  29%|##9       | 14631/50000 [8:26:54<16:53:38,  1.72s/it]
Outer training loop:  29%|##9       | 14632/50000 [8:26:56<17:13:45,  1.75s/it]
Outer training loop:  29%|##9       | 14633/50000 [8:26:57<16:57:01,  1.73s/it]
Outer training loop:  29%|##9       | 14634/50000 [8:26:59<17:00:28,  1.73s/it]
Outer training loop:  29%|##9       | 14635/50000 [8:27:01<17:31:46,  1.78s/it]
Outer training loop:  29%|##9       | 14636/50000 [8:27:03<17:07:03,  1.74s/it]
Outer training loop:  29%|##9       | 14637/50000 [8:27:04<16:58:24,  1.73s/it]
Outer training loop:  29%|##9       | 14638/50000 [8:27:06<16:59:56,  1.73s/it]
Outer training loop:  29%|##9       | 14639/50000 [8:27:08<17:51:29,  1.82s/it]
Outer training loop:  29%|##9       | 14640/50000 [8:27:10<17:37:54,  1.80s/it]
Outer training loop:  29%|##9       | 14641/50000 [8:27:12<17:36:56,  1.79s/it]
Outer training loop:  29%|##9       | 14642/50000 [8:27:14<18:20:14,  1.87s/it]
Outer training loop:  29%|##9       | 14643/50000 [8:27:15<17:52:49,  1.82s/it]
Outer training loop:  29%|##9       | 14644/50000 [8:27:17<17:31:52,  1.79s/it]
Outer training loop:  29%|##9       | 14645/50000 [8:27:19<17:49:56,  1.82s/it]
Outer training loop:  29%|##9       | 14646/50000 [8:27:21<17:54:40,  1.82s/it]
Outer training loop:  29%|##9       | 14647/50000 [8:27:23<17:42:54,  1.80s/it]
Outer training loop:  29%|##9       | 14648/50000 [8:27:25<18:13:43,  1.86s/it]
Outer training loop:  29%|##9       | 14649/50000 [8:27:26<17:43:58,  1.81s/it]
Outer training loop:  29%|##9       | 14650/50000 [8:27:28<17:33:30,  1.79s/it]
Outer training loop:  29%|##9       | 14651/50000 [8:27:30<17:40:30,  1.80s/it]
Outer training loop:  29%|##9       | 14652/50000 [8:27:32<18:03:30,  1.84s/it]
Outer training loop:  29%|##9       | 14653/50000 [8:27:34<17:52:44,  1.82s/it]
Outer training loop:  29%|##9       | 14654/50000 [8:27:35<17:23:49,  1.77s/it]
Outer training loop:  29%|##9       | 14655/50000 [8:27:37<17:50:29,  1.82s/it]
Outer training loop:  29%|##9       | 14656/50000 [8:27:39<17:48:43,  1.81s/it]
Outer training loop:  29%|##9       | 14657/50000 [8:27:41<17:24:02,  1.77s/it]
Outer training loop:  29%|##9       | 14658/50000 [8:27:43<18:01:13,  1.84s/it]
Outer training loop:  29%|##9       | 14659/50000 [8:27:44<17:50:22,  1.82s/it]
Outer training loop:  29%|##9       | 14660/50000 [8:27:46<18:14:07,  1.86s/it]
Outer training loop:  29%|##9       | 14661/50000 [8:27:48<18:23:30,  1.87s/it]
Outer training loop:  29%|##9       | 14662/50000 [8:27:50<18:52:07,  1.92s/it]
Outer training loop:  29%|##9       | 14663/50000 [8:27:52<18:08:07,  1.85s/it]
Outer training loop:  29%|##9       | 14664/50000 [8:27:54<17:50:56,  1.82s/it]
Outer training loop:  29%|##9       | 14665/50000 [8:27:55<17:35:37,  1.79s/it]
Outer training loop:  29%|##9       | 14666/50000 [8:27:57<18:08:45,  1.85s/it]
Outer training loop:  29%|##9       | 14667/50000 [8:27:59<17:54:04,  1.82s/it]
Outer training loop:  29%|##9       | 14668/50000 [8:28:01<17:53:01,  1.82s/it]
Outer training loop:  29%|##9       | 14669/50000 [8:28:03<18:28:39,  1.88s/it]
Outer training loop:  29%|##9       | 14670/50000 [8:28:05<18:35:39,  1.89s/it]
Outer training loop:  29%|##9       | 14671/50000 [8:28:07<17:49:25,  1.82s/it]
Outer training loop:  29%|##9       | 14672/50000 [8:28:09<18:10:40,  1.85s/it]
Outer training loop:  29%|##9       | 14673/50000 [8:28:10<18:02:56,  1.84s/it]
Outer training loop:  29%|##9       | 14674/50000 [8:28:12<17:48:30,  1.81s/it]
Outer training loop:  29%|##9       | 14675/50000 [8:28:14<17:33:40,  1.79s/it]
Outer training loop:  29%|##9       | 14676/50000 [8:28:16<17:55:34,  1.83s/it]
Outer training loop:  29%|##9       | 14677/50000 [8:28:18<17:50:06,  1.82s/it]
Outer training loop:  29%|##9       | 14678/50000 [8:28:19<17:35:04,  1.79s/it]
Outer training loop:  29%|##9       | 14679/50000 [8:28:21<17:36:06,  1.79s/it]
Outer training loop:  29%|##9       | 14680/50000 [8:28:23<17:20:17,  1.77s/it]
Outer training loop:  29%|##9       | 14681/50000 [8:28:25<17:17:20,  1.76s/it]
Outer training loop:  29%|##9       | 14682/50000 [8:28:26<17:49:08,  1.82s/it]
Outer training loop:  29%|##9       | 14683/50000 [8:28:28<17:44:18,  1.81s/it]
Outer training loop:  29%|##9       | 14684/50000 [8:28:30<17:09:04,  1.75s/it]
Outer training loop:  29%|##9       | 14685/50000 [8:28:32<17:24:26,  1.77s/it]
Outer training loop:  29%|##9       | 14686/50000 [8:28:34<17:30:05,  1.78s/it]
Outer training loop:  29%|##9       | 14687/50000 [8:28:35<17:11:00,  1.75s/it]
Outer training loop:  29%|##9       | 14688/50000 [8:28:37<17:15:26,  1.76s/it]
Outer training loop:  29%|##9       | 14689/50000 [8:28:39<17:27:38,  1.78s/it]
Outer training loop:  29%|##9       | 14690/50000 [8:28:41<17:24:10,  1.77s/it]
Outer training loop:  29%|##9       | 14691/50000 [8:28:42<16:58:56,  1.73s/it]
Outer training loop:  29%|##9       | 14692/50000 [8:28:44<17:37:47,  1.80s/it]
Outer training loop:  29%|##9       | 14693/50000 [8:28:46<17:03:14,  1.74s/it]
Outer training loop:  29%|##9       | 14694/50000 [8:28:48<17:09:10,  1.75s/it]
Outer training loop:  29%|##9       | 14695/50000 [8:28:49<17:24:12,  1.77s/it]
Outer training loop:  29%|##9       | 14696/50000 [8:28:51<17:37:28,  1.80s/it]
Outer training loop:  29%|##9       | 14697/50000 [8:28:53<17:44:19,  1.81s/it]
Outer training loop:  29%|##9       | 14698/50000 [8:28:55<17:57:41,  1.83s/it]
Outer training loop:  29%|##9       | 14699/50000 [8:28:57<17:34:40,  1.79s/it]
Outer training loop:  29%|##9       | 14700/50000 [8:28:58<17:28:09,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.73s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.68s/it]

Outer training loop:  29%|##9       | 14701/50000 [8:29:05<32:54:17,  3.36s/it]
Outer training loop:  29%|##9       | 14702/50000 [8:29:07<28:30:52,  2.91s/it]
Outer training loop:  29%|##9       | 14703/50000 [8:29:09<24:51:48,  2.54s/it]
Outer training loop:  29%|##9       | 14704/50000 [8:29:11<23:00:10,  2.35s/it]
Outer training loop:  29%|##9       | 14705/50000 [8:29:13<21:45:33,  2.22s/it]
Outer training loop:  29%|##9       | 14706/50000 [8:29:15<20:23:01,  2.08s/it]
Outer training loop:  29%|##9       | 14707/50000 [8:29:16<19:24:37,  1.98s/it]
Outer training loop:  29%|##9       | 14708/50000 [8:29:18<18:59:29,  1.94s/it]
Outer training loop:  29%|##9       | 14709/50000 [8:29:20<18:24:32,  1.88s/it]
Outer training loop:  29%|##9       | 14710/50000 [8:29:22<17:51:45,  1.82s/it]
Outer training loop:  29%|##9       | 14711/50000 [8:29:23<17:33:30,  1.79s/it]
Outer training loop:  29%|##9       | 14712/50000 [8:29:25<18:00:32,  1.84s/it]
Outer training loop:  29%|##9       | 14713/50000 [8:29:27<17:25:44,  1.78s/it]
Outer training loop:  29%|##9       | 14714/50000 [8:29:29<17:16:33,  1.76s/it]
Outer training loop:  29%|##9       | 14715/50000 [8:29:30<17:28:52,  1.78s/it]
Outer training loop:  29%|##9       | 14716/50000 [8:29:32<17:24:32,  1.78s/it]
Outer training loop:  29%|##9       | 14717/50000 [8:29:34<17:02:47,  1.74s/it]
Outer training loop:  29%|##9       | 14718/50000 [8:29:36<17:45:30,  1.81s/it]
Outer training loop:  29%|##9       | 14719/50000 [8:29:37<17:21:22,  1.77s/it]
Outer training loop:  29%|##9       | 14720/50000 [8:29:39<17:26:40,  1.78s/it]
Outer training loop:  29%|##9       | 14721/50000 [8:29:41<17:04:54,  1.74s/it]
Outer training loop:  29%|##9       | 14722/50000 [8:29:43<17:28:34,  1.78s/it]
Outer training loop:  29%|##9       | 14723/50000 [8:29:45<17:14:45,  1.76s/it]
Outer training loop:  29%|##9       | 14724/50000 [8:29:46<16:58:48,  1.73s/it]
Outer training loop:  29%|##9       | 14725/50000 [8:29:48<17:31:10,  1.79s/it]
Outer training loop:  29%|##9       | 14726/50000 [8:29:50<17:14:58,  1.76s/it]
Outer training loop:  29%|##9       | 14727/50000 [8:29:51<17:00:49,  1.74s/it]
Outer training loop:  29%|##9       | 14728/50000 [8:29:53<17:02:18,  1.74s/it]
Outer training loop:  29%|##9       | 14729/50000 [8:29:55<18:05:58,  1.85s/it]
Outer training loop:  29%|##9       | 14730/50000 [8:29:57<18:13:45,  1.86s/it]
Outer training loop:  29%|##9       | 14731/50000 [8:29:59<17:40:38,  1.80s/it]
Outer training loop:  29%|##9       | 14732/50000 [8:30:01<17:43:40,  1.81s/it]
Outer training loop:  29%|##9       | 14733/50000 [8:30:02<17:25:52,  1.78s/it]
Outer training loop:  29%|##9       | 14734/50000 [8:30:04<17:04:51,  1.74s/it]
Outer training loop:  29%|##9       | 14735/50000 [8:30:06<17:47:57,  1.82s/it]
Outer training loop:  29%|##9       | 14736/50000 [8:30:08<17:29:13,  1.79s/it]
Outer training loop:  29%|##9       | 14737/50000 [8:30:10<17:17:39,  1.77s/it]
Outer training loop:  29%|##9       | 14738/50000 [8:30:11<17:35:08,  1.80s/it]
Outer training loop:  29%|##9       | 14739/50000 [8:30:13<17:16:24,  1.76s/it]
Outer training loop:  29%|##9       | 14740/50000 [8:30:15<16:54:30,  1.73s/it]
Outer training loop:  29%|##9       | 14741/50000 [8:30:16<16:46:37,  1.71s/it]
Outer training loop:  29%|##9       | 14742/50000 [8:30:18<17:07:14,  1.75s/it]
Outer training loop:  29%|##9       | 14743/50000 [8:30:20<17:37:08,  1.80s/it]
Outer training loop:  29%|##9       | 14744/50000 [8:30:22<17:25:23,  1.78s/it]
Outer training loop:  29%|##9       | 14745/50000 [8:30:24<17:41:59,  1.81s/it]
Outer training loop:  29%|##9       | 14746/50000 [8:30:25<17:29:10,  1.79s/it]
Outer training loop:  29%|##9       | 14747/50000 [8:30:27<17:37:35,  1.80s/it]
Outer training loop:  29%|##9       | 14748/50000 [8:30:29<17:23:27,  1.78s/it]
Outer training loop:  29%|##9       | 14749/50000 [8:30:31<17:41:35,  1.81s/it]
Outer training loop:  30%|##9       | 14750/50000 [8:30:33<17:26:29,  1.78s/it]
Outer training loop:  30%|##9       | 14751/50000 [8:30:34<17:29:33,  1.79s/it]
Outer training loop:  30%|##9       | 14752/50000 [8:30:36<17:41:45,  1.81s/it]
Outer training loop:  30%|##9       | 14753/50000 [8:30:38<17:47:11,  1.82s/it]
Outer training loop:  30%|##9       | 14754/50000 [8:30:40<17:38:48,  1.80s/it]
Outer training loop:  30%|##9       | 14755/50000 [8:30:42<17:33:43,  1.79s/it]
Outer training loop:  30%|##9       | 14756/50000 [8:30:44<17:53:42,  1.83s/it]
Outer training loop:  30%|##9       | 14757/50000 [8:30:45<17:48:30,  1.82s/it]
Outer training loop:  30%|##9       | 14758/50000 [8:30:47<17:42:26,  1.81s/it]
Outer training loop:  30%|##9       | 14759/50000 [8:30:49<18:21:52,  1.88s/it]
Outer training loop:  30%|##9       | 14760/50000 [8:30:51<18:10:39,  1.86s/it]
Outer training loop:  30%|##9       | 14761/50000 [8:30:53<17:45:40,  1.81s/it]
Outer training loop:  30%|##9       | 14762/50000 [8:30:55<18:06:15,  1.85s/it]
Outer training loop:  30%|##9       | 14763/50000 [8:30:56<17:36:06,  1.80s/it]
Outer training loop:  30%|##9       | 14764/50000 [8:30:58<17:08:09,  1.75s/it]
Outer training loop:  30%|##9       | 14765/50000 [8:31:00<17:12:12,  1.76s/it]
Outer training loop:  30%|##9       | 14766/50000 [8:31:02<17:52:03,  1.83s/it]
Outer training loop:  30%|##9       | 14767/50000 [8:31:04<17:50:39,  1.82s/it]
Outer training loop:  30%|##9       | 14768/50000 [8:31:05<17:40:05,  1.81s/it]
Outer training loop:  30%|##9       | 14769/50000 [8:31:07<18:09:47,  1.86s/it]
Outer training loop:  30%|##9       | 14770/50000 [8:31:09<17:41:17,  1.81s/it]
Outer training loop:  30%|##9       | 14771/50000 [8:31:11<17:32:09,  1.79s/it]
Outer training loop:  30%|##9       | 14772/50000 [8:31:13<17:45:35,  1.81s/it]
Outer training loop:  30%|##9       | 14773/50000 [8:31:14<17:34:06,  1.80s/it]
Outer training loop:  30%|##9       | 14774/50000 [8:31:16<17:04:11,  1.74s/it]
Outer training loop:  30%|##9       | 14775/50000 [8:31:18<17:02:34,  1.74s/it]
Outer training loop:  30%|##9       | 14776/50000 [8:31:20<17:20:20,  1.77s/it]
Outer training loop:  30%|##9       | 14777/50000 [8:31:21<17:09:52,  1.75s/it]
Outer training loop:  30%|##9       | 14778/50000 [8:31:23<16:42:46,  1.71s/it]
Outer training loop:  30%|##9       | 14779/50000 [8:31:25<17:25:37,  1.78s/it]
Outer training loop:  30%|##9       | 14780/50000 [8:31:27<17:40:34,  1.81s/it]
Outer training loop:  30%|##9       | 14781/50000 [8:31:28<17:31:44,  1.79s/it]
Outer training loop:  30%|##9       | 14782/50000 [8:31:30<17:47:28,  1.82s/it]
Outer training loop:  30%|##9       | 14783/50000 [8:31:32<17:16:30,  1.77s/it]
Outer training loop:  30%|##9       | 14784/50000 [8:31:34<17:29:43,  1.79s/it]
Outer training loop:  30%|##9       | 14785/50000 [8:31:36<17:52:26,  1.83s/it]
Outer training loop:  30%|##9       | 14786/50000 [8:31:38<17:46:08,  1.82s/it]
Outer training loop:  30%|##9       | 14787/50000 [8:31:39<17:27:28,  1.78s/it]
Outer training loop:  30%|##9       | 14788/50000 [8:31:41<17:24:50,  1.78s/it]
Outer training loop:  30%|##9       | 14789/50000 [8:31:43<18:03:39,  1.85s/it]
Outer training loop:  30%|##9       | 14790/50000 [8:31:45<17:27:21,  1.78s/it]
Outer training loop:  30%|##9       | 14791/50000 [8:31:46<17:23:42,  1.78s/it]
Outer training loop:  30%|##9       | 14792/50000 [8:31:48<17:37:45,  1.80s/it]
Outer training loop:  30%|##9       | 14793/50000 [8:31:50<17:59:29,  1.84s/it]
Outer training loop:  30%|##9       | 14794/50000 [8:31:52<17:43:18,  1.81s/it]
Outer training loop:  30%|##9       | 14795/50000 [8:31:54<17:28:08,  1.79s/it]
Outer training loop:  30%|##9       | 14796/50000 [8:31:56<18:01:19,  1.84s/it]
Outer training loop:  30%|##9       | 14797/50000 [8:31:57<17:38:56,  1.80s/it]
Outer training loop:  30%|##9       | 14798/50000 [8:31:59<17:16:17,  1.77s/it]
Outer training loop:  30%|##9       | 14799/50000 [8:32:01<17:29:07,  1.79s/it]
Outer training loop:  30%|##9       | 14800/50000 [8:32:03<17:06:15,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.70s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  30%|##9       | 14801/50000 [8:32:10<33:00:30,  3.38s/it]
Outer training loop:  30%|##9       | 14802/50000 [8:32:11<28:06:44,  2.88s/it]
Outer training loop:  30%|##9       | 14803/50000 [8:32:13<25:08:45,  2.57s/it]
Outer training loop:  30%|##9       | 14804/50000 [8:32:15<23:17:04,  2.38s/it]
Outer training loop:  30%|##9       | 14805/50000 [8:32:17<21:26:21,  2.19s/it]
Outer training loop:  30%|##9       | 14806/50000 [8:32:19<20:08:57,  2.06s/it]
Outer training loop:  30%|##9       | 14807/50000 [8:32:20<19:05:06,  1.95s/it]
Outer training loop:  30%|##9       | 14808/50000 [8:32:22<19:12:03,  1.96s/it]
Outer training loop:  30%|##9       | 14809/50000 [8:32:24<18:32:12,  1.90s/it]
Outer training loop:  30%|##9       | 14810/50000 [8:32:26<18:07:18,  1.85s/it]
Outer training loop:  30%|##9       | 14811/50000 [8:32:28<18:35:29,  1.90s/it]
Outer training loop:  30%|##9       | 14812/50000 [8:32:29<17:40:50,  1.81s/it]
Outer training loop:  30%|##9       | 14813/50000 [8:32:31<17:27:29,  1.79s/it]
Outer training loop:  30%|##9       | 14814/50000 [8:32:33<17:52:12,  1.83s/it]
Outer training loop:  30%|##9       | 14815/50000 [8:32:35<17:24:57,  1.78s/it]
Outer training loop:  30%|##9       | 14816/50000 [8:32:37<17:29:06,  1.79s/it]
Outer training loop:  30%|##9       | 14817/50000 [8:32:39<18:02:31,  1.85s/it]
Outer training loop:  30%|##9       | 14818/50000 [8:32:40<17:26:23,  1.78s/it]
Outer training loop:  30%|##9       | 14819/50000 [8:32:42<17:18:57,  1.77s/it]
Outer training loop:  30%|##9       | 14820/50000 [8:32:44<17:05:49,  1.75s/it]
Outer training loop:  30%|##9       | 14821/50000 [8:32:46<17:18:24,  1.77s/it]
Outer training loop:  30%|##9       | 14822/50000 [8:32:47<17:15:21,  1.77s/it]
Outer training loop:  30%|##9       | 14823/50000 [8:32:49<17:09:25,  1.76s/it]
Outer training loop:  30%|##9       | 14824/50000 [8:32:51<17:37:10,  1.80s/it]
Outer training loop:  30%|##9       | 14825/50000 [8:32:53<17:10:48,  1.76s/it]
Outer training loop:  30%|##9       | 14826/50000 [8:32:54<17:05:51,  1.75s/it]
Outer training loop:  30%|##9       | 14827/50000 [8:32:56<17:37:35,  1.80s/it]
Outer training loop:  30%|##9       | 14828/50000 [8:32:58<17:21:13,  1.78s/it]
Outer training loop:  30%|##9       | 14829/50000 [8:33:00<17:04:06,  1.75s/it]
Outer training loop:  30%|##9       | 14830/50000 [8:33:01<17:04:47,  1.75s/it]
Outer training loop:  30%|##9       | 14831/50000 [8:33:03<17:35:46,  1.80s/it]
Outer training loop:  30%|##9       | 14832/50000 [8:33:05<17:34:41,  1.80s/it]
Outer training loop:  30%|##9       | 14833/50000 [8:33:07<17:16:35,  1.77s/it]
Outer training loop:  30%|##9       | 14834/50000 [8:33:09<17:29:31,  1.79s/it]
Outer training loop:  30%|##9       | 14835/50000 [8:33:10<17:19:10,  1.77s/it]
Outer training loop:  30%|##9       | 14836/50000 [8:33:12<17:06:27,  1.75s/it]
Outer training loop:  30%|##9       | 14837/50000 [8:33:14<17:07:09,  1.75s/it]
Outer training loop:  30%|##9       | 14838/50000 [8:33:16<17:25:37,  1.78s/it]
Outer training loop:  30%|##9       | 14839/50000 [8:33:17<17:06:40,  1.75s/it]
Outer training loop:  30%|##9       | 14840/50000 [8:33:19<17:18:13,  1.77s/it]
Outer training loop:  30%|##9       | 14841/50000 [8:33:21<17:40:18,  1.81s/it]
Outer training loop:  30%|##9       | 14842/50000 [8:33:23<17:30:55,  1.79s/it]
Outer training loop:  30%|##9       | 14843/50000 [8:33:25<17:11:39,  1.76s/it]
Outer training loop:  30%|##9       | 14844/50000 [8:33:26<17:11:27,  1.76s/it]
Outer training loop:  30%|##9       | 14845/50000 [8:33:28<17:33:29,  1.80s/it]
Outer training loop:  30%|##9       | 14846/50000 [8:33:30<17:19:51,  1.77s/it]
Outer training loop:  30%|##9       | 14847/50000 [8:33:32<17:05:22,  1.75s/it]
Outer training loop:  30%|##9       | 14848/50000 [8:33:33<16:58:10,  1.74s/it]
Outer training loop:  30%|##9       | 14849/50000 [8:33:35<17:30:20,  1.79s/it]
Outer training loop:  30%|##9       | 14850/50000 [8:33:37<17:21:45,  1.78s/it]
Outer training loop:  30%|##9       | 14851/50000 [8:33:39<17:19:43,  1.77s/it]
Outer training loop:  30%|##9       | 14852/50000 [8:33:41<17:51:16,  1.83s/it]
Outer training loop:  30%|##9       | 14853/50000 [8:33:42<17:44:37,  1.82s/it]
Outer training loop:  30%|##9       | 14854/50000 [8:33:44<17:20:01,  1.78s/it]
Outer training loop:  30%|##9       | 14855/50000 [8:33:46<17:56:34,  1.84s/it]
Outer training loop:  30%|##9       | 14856/50000 [8:33:48<17:42:16,  1.81s/it]
Outer training loop:  30%|##9       | 14857/50000 [8:33:50<17:19:49,  1.78s/it]
Outer training loop:  30%|##9       | 14858/50000 [8:33:51<16:54:26,  1.73s/it]
Outer training loop:  30%|##9       | 14859/50000 [8:33:53<17:23:27,  1.78s/it]
Outer training loop:  30%|##9       | 14860/50000 [8:33:55<17:03:28,  1.75s/it]
Outer training loop:  30%|##9       | 14861/50000 [8:33:56<16:53:24,  1.73s/it]
Outer training loop:  30%|##9       | 14862/50000 [8:33:58<17:22:57,  1.78s/it]
Outer training loop:  30%|##9       | 14863/50000 [8:34:00<16:56:42,  1.74s/it]
Outer training loop:  30%|##9       | 14864/50000 [8:34:02<16:38:12,  1.70s/it]
Outer training loop:  30%|##9       | 14865/50000 [8:34:03<17:06:37,  1.75s/it]
Outer training loop:  30%|##9       | 14866/50000 [8:34:05<16:52:33,  1.73s/it]
Outer training loop:  30%|##9       | 14867/50000 [8:34:07<16:53:32,  1.73s/it]
Outer training loop:  30%|##9       | 14868/50000 [8:34:09<17:25:56,  1.79s/it]
Outer training loop:  30%|##9       | 14869/50000 [8:34:11<17:32:24,  1.80s/it]
Outer training loop:  30%|##9       | 14870/50000 [8:34:12<16:59:30,  1.74s/it]
Outer training loop:  30%|##9       | 14871/50000 [8:34:14<17:25:54,  1.79s/it]
Outer training loop:  30%|##9       | 14872/50000 [8:34:16<17:02:25,  1.75s/it]
Outer training loop:  30%|##9       | 14873/50000 [8:34:17<16:54:07,  1.73s/it]
Outer training loop:  30%|##9       | 14874/50000 [8:34:19<16:49:02,  1.72s/it]
Outer training loop:  30%|##9       | 14875/50000 [8:34:21<17:18:05,  1.77s/it]
Outer training loop:  30%|##9       | 14876/50000 [8:34:23<17:06:38,  1.75s/it]
Outer training loop:  30%|##9       | 14877/50000 [8:34:24<16:58:22,  1.74s/it]
Outer training loop:  30%|##9       | 14878/50000 [8:34:26<17:26:06,  1.79s/it]
Outer training loop:  30%|##9       | 14879/50000 [8:34:28<17:16:40,  1.77s/it]
Outer training loop:  30%|##9       | 14880/50000 [8:34:30<17:24:16,  1.78s/it]
Outer training loop:  30%|##9       | 14881/50000 [8:34:32<17:52:49,  1.83s/it]
Outer training loop:  30%|##9       | 14882/50000 [8:34:34<17:25:43,  1.79s/it]
Outer training loop:  30%|##9       | 14883/50000 [8:34:35<17:22:16,  1.78s/it]
Outer training loop:  30%|##9       | 14884/50000 [8:34:37<17:04:34,  1.75s/it]
Outer training loop:  30%|##9       | 14885/50000 [8:34:39<17:43:57,  1.82s/it]
Outer training loop:  30%|##9       | 14886/50000 [8:34:41<17:49:13,  1.83s/it]
Outer training loop:  30%|##9       | 14887/50000 [8:34:42<17:11:32,  1.76s/it]
Outer training loop:  30%|##9       | 14888/50000 [8:34:44<17:29:25,  1.79s/it]
Outer training loop:  30%|##9       | 14889/50000 [8:34:46<17:19:06,  1.78s/it]
Outer training loop:  30%|##9       | 14890/50000 [8:34:48<17:43:25,  1.82s/it]
Outer training loop:  30%|##9       | 14891/50000 [8:34:50<18:04:15,  1.85s/it]
Outer training loop:  30%|##9       | 14892/50000 [8:34:52<17:43:53,  1.82s/it]
Outer training loop:  30%|##9       | 14893/50000 [8:34:53<17:31:27,  1.80s/it]
Outer training loop:  30%|##9       | 14894/50000 [8:34:55<17:17:12,  1.77s/it]
Outer training loop:  30%|##9       | 14895/50000 [8:34:57<17:34:34,  1.80s/it]
Outer training loop:  30%|##9       | 14896/50000 [8:34:59<17:20:02,  1.78s/it]
Outer training loop:  30%|##9       | 14897/50000 [8:35:00<16:54:38,  1.73s/it]
Outer training loop:  30%|##9       | 14898/50000 [8:35:02<17:14:30,  1.77s/it]
Outer training loop:  30%|##9       | 14899/50000 [8:35:04<17:09:39,  1.76s/it]
Outer training loop:  30%|##9       | 14900/50000 [8:35:06<17:11:21,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.87s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  30%|##9       | 14901/50000 [8:35:13<32:17:45,  3.31s/it]
Outer training loop:  30%|##9       | 14902/50000 [8:35:14<27:53:21,  2.86s/it]
Outer training loop:  30%|##9       | 14903/50000 [8:35:16<25:05:49,  2.57s/it]
Outer training loop:  30%|##9       | 14904/50000 [8:35:18<22:33:28,  2.31s/it]
Outer training loop:  30%|##9       | 14905/50000 [8:35:20<20:59:28,  2.15s/it]
Outer training loop:  30%|##9       | 14906/50000 [8:35:22<20:33:49,  2.11s/it]
Outer training loop:  30%|##9       | 14907/50000 [8:35:23<19:17:23,  1.98s/it]
Outer training loop:  30%|##9       | 14908/50000 [8:35:25<18:29:31,  1.90s/it]
Outer training loop:  30%|##9       | 14909/50000 [8:35:27<18:20:16,  1.88s/it]
Outer training loop:  30%|##9       | 14910/50000 [8:35:29<18:04:51,  1.85s/it]
Outer training loop:  30%|##9       | 14911/50000 [8:35:31<17:47:25,  1.83s/it]
Outer training loop:  30%|##9       | 14912/50000 [8:35:32<17:29:13,  1.79s/it]
Outer training loop:  30%|##9       | 14913/50000 [8:35:34<17:37:50,  1.81s/it]
Outer training loop:  30%|##9       | 14914/50000 [8:35:36<17:22:50,  1.78s/it]
Outer training loop:  30%|##9       | 14915/50000 [8:35:38<17:14:49,  1.77s/it]
Outer training loop:  30%|##9       | 14916/50000 [8:35:40<17:42:53,  1.82s/it]
Outer training loop:  30%|##9       | 14917/50000 [8:35:41<17:20:39,  1.78s/it]
Outer training loop:  30%|##9       | 14918/50000 [8:35:43<17:16:25,  1.77s/it]
Outer training loop:  30%|##9       | 14919/50000 [8:35:45<17:54:23,  1.84s/it]
Outer training loop:  30%|##9       | 14920/50000 [8:35:47<17:17:09,  1.77s/it]
Outer training loop:  30%|##9       | 14921/50000 [8:35:48<17:03:09,  1.75s/it]
Outer training loop:  30%|##9       | 14922/50000 [8:35:50<17:44:04,  1.82s/it]
Outer training loop:  30%|##9       | 14923/50000 [8:35:52<17:18:07,  1.78s/it]
Outer training loop:  30%|##9       | 14924/50000 [8:35:54<16:57:28,  1.74s/it]
Outer training loop:  30%|##9       | 14925/50000 [8:35:56<17:40:22,  1.81s/it]
Outer training loop:  30%|##9       | 14926/50000 [8:35:57<17:18:26,  1.78s/it]
Outer training loop:  30%|##9       | 14927/50000 [8:35:59<17:02:07,  1.75s/it]
Outer training loop:  30%|##9       | 14928/50000 [8:36:01<16:48:39,  1.73s/it]
Outer training loop:  30%|##9       | 14929/50000 [8:36:03<17:23:34,  1.79s/it]
Outer training loop:  30%|##9       | 14930/50000 [8:36:04<17:19:27,  1.78s/it]
Outer training loop:  30%|##9       | 14931/50000 [8:36:06<17:04:14,  1.75s/it]
Outer training loop:  30%|##9       | 14932/50000 [8:36:08<17:25:08,  1.79s/it]
Outer training loop:  30%|##9       | 14933/50000 [8:36:10<17:28:12,  1.79s/it]
Outer training loop:  30%|##9       | 14934/50000 [8:36:11<16:59:14,  1.74s/it]
Outer training loop:  30%|##9       | 14935/50000 [8:36:13<17:20:10,  1.78s/it]
Outer training loop:  30%|##9       | 14936/50000 [8:36:15<17:07:54,  1.76s/it]
Outer training loop:  30%|##9       | 14937/50000 [8:36:17<17:35:59,  1.81s/it]
Outer training loop:  30%|##9       | 14938/50000 [8:36:19<17:37:23,  1.81s/it]
Outer training loop:  30%|##9       | 14939/50000 [8:36:21<17:54:58,  1.84s/it]
Outer training loop:  30%|##9       | 14940/50000 [8:36:22<17:25:11,  1.79s/it]
Outer training loop:  30%|##9       | 14941/50000 [8:36:24<17:33:33,  1.80s/it]
Outer training loop:  30%|##9       | 14942/50000 [8:36:26<18:10:55,  1.87s/it]
Outer training loop:  30%|##9       | 14943/50000 [8:36:28<17:52:47,  1.84s/it]
Outer training loop:  30%|##9       | 14944/50000 [8:36:30<17:52:31,  1.84s/it]
Outer training loop:  30%|##9       | 14945/50000 [8:36:32<17:57:43,  1.84s/it]
Outer training loop:  30%|##9       | 14946/50000 [8:36:33<17:16:45,  1.77s/it]
Outer training loop:  30%|##9       | 14947/50000 [8:36:35<17:46:37,  1.83s/it]
Outer training loop:  30%|##9       | 14948/50000 [8:36:37<17:33:27,  1.80s/it]
Outer training loop:  30%|##9       | 14949/50000 [8:36:39<17:34:45,  1.81s/it]
Outer training loop:  30%|##9       | 14950/50000 [8:36:40<17:36:51,  1.81s/it]
Outer training loop:  30%|##9       | 14951/50000 [8:36:42<17:42:58,  1.82s/it]
Outer training loop:  30%|##9       | 14952/50000 [8:36:44<18:40:49,  1.92s/it]
Outer training loop:  30%|##9       | 14953/50000 [8:36:46<18:34:54,  1.91s/it]
Outer training loop:  30%|##9       | 14954/50000 [8:36:48<18:28:51,  1.90s/it]
Outer training loop:  30%|##9       | 14955/50000 [8:36:50<18:58:35,  1.95s/it]
Outer training loop:  30%|##9       | 14956/50000 [8:36:52<18:45:03,  1.93s/it]
Outer training loop:  30%|##9       | 14957/50000 [8:36:54<18:33:54,  1.91s/it]
Outer training loop:  30%|##9       | 14958/50000 [8:36:56<18:44:29,  1.93s/it]
Outer training loop:  30%|##9       | 14959/50000 [8:36:58<18:31:45,  1.90s/it]
Outer training loop:  30%|##9       | 14960/50000 [8:37:00<18:17:59,  1.88s/it]
Outer training loop:  30%|##9       | 14961/50000 [8:37:02<18:17:40,  1.88s/it]
Outer training loop:  30%|##9       | 14962/50000 [8:37:04<18:36:44,  1.91s/it]
Outer training loop:  30%|##9       | 14963/50000 [8:37:05<18:01:25,  1.85s/it]
Outer training loop:  30%|##9       | 14964/50000 [8:37:07<18:03:15,  1.86s/it]
Outer training loop:  30%|##9       | 14965/50000 [8:37:09<17:44:26,  1.82s/it]
Outer training loop:  30%|##9       | 14966/50000 [8:37:11<18:38:50,  1.92s/it]
Outer training loop:  30%|##9       | 14967/50000 [8:37:13<18:51:58,  1.94s/it]
Outer training loop:  30%|##9       | 14968/50000 [8:37:15<18:32:20,  1.91s/it]
Outer training loop:  30%|##9       | 14969/50000 [8:37:17<18:35:35,  1.91s/it]
Outer training loop:  30%|##9       | 14970/50000 [8:37:19<18:23:30,  1.89s/it]
Outer training loop:  30%|##9       | 14971/50000 [8:37:20<18:08:55,  1.87s/it]
Outer training loop:  30%|##9       | 14972/50000 [8:37:22<17:38:17,  1.81s/it]
Outer training loop:  30%|##9       | 14973/50000 [8:37:24<18:04:10,  1.86s/it]
Outer training loop:  30%|##9       | 14974/50000 [8:37:26<17:35:44,  1.81s/it]
Outer training loop:  30%|##9       | 14975/50000 [8:37:27<17:02:27,  1.75s/it]
Outer training loop:  30%|##9       | 14976/50000 [8:37:29<17:37:03,  1.81s/it]
Outer training loop:  30%|##9       | 14977/50000 [8:37:31<17:22:13,  1.79s/it]
Outer training loop:  30%|##9       | 14978/50000 [8:37:33<17:02:41,  1.75s/it]
Outer training loop:  30%|##9       | 14979/50000 [8:37:35<17:24:43,  1.79s/it]
Outer training loop:  30%|##9       | 14980/50000 [8:37:37<18:23:42,  1.89s/it]
Outer training loop:  30%|##9       | 14981/50000 [8:37:39<18:08:36,  1.87s/it]
Outer training loop:  30%|##9       | 14982/50000 [8:37:40<17:36:54,  1.81s/it]
Outer training loop:  30%|##9       | 14983/50000 [8:37:42<17:51:10,  1.84s/it]
Outer training loop:  30%|##9       | 14984/50000 [8:37:44<17:37:36,  1.81s/it]
Outer training loop:  30%|##9       | 14985/50000 [8:37:46<17:46:34,  1.83s/it]
Outer training loop:  30%|##9       | 14986/50000 [8:37:48<18:18:42,  1.88s/it]
Outer training loop:  30%|##9       | 14987/50000 [8:37:50<18:16:05,  1.88s/it]
Outer training loop:  30%|##9       | 14988/50000 [8:37:51<17:43:58,  1.82s/it]
Outer training loop:  30%|##9       | 14989/50000 [8:37:53<17:49:02,  1.83s/it]
Outer training loop:  30%|##9       | 14990/50000 [8:37:55<17:50:41,  1.83s/it]
Outer training loop:  30%|##9       | 14991/50000 [8:37:57<17:45:57,  1.83s/it]
Outer training loop:  30%|##9       | 14992/50000 [8:37:58<17:23:17,  1.79s/it]
Outer training loop:  30%|##9       | 14993/50000 [8:38:00<17:45:37,  1.83s/it]
Outer training loop:  30%|##9       | 14994/50000 [8:38:02<17:31:14,  1.80s/it]
Outer training loop:  30%|##9       | 14995/50000 [8:38:04<17:23:32,  1.79s/it]
Outer training loop:  30%|##9       | 14996/50000 [8:38:06<17:48:31,  1.83s/it]
Outer training loop:  30%|##9       | 14997/50000 [8:38:08<17:54:05,  1.84s/it]
Outer training loop:  30%|##9       | 14998/50000 [8:38:09<17:40:02,  1.82s/it]
Outer training loop:  30%|##9       | 14999/50000 [8:38:11<17:23:10,  1.79s/it]
Outer training loop:  30%|###       | 15000/50000 [8:38:13<17:31:32,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.66s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.69s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.75s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.77s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.86s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.82s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:21,  1.81s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:18,  1.80s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:21,  1.86s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:17,  1.84s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:13,  1.83s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.89s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:12,  1.87s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:08,  1.83s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:05,  1.82s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:06,  1.85s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:01,  1.82s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<01:59,  1.81s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:55,  1.78s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:58,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:56,  1.85s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:51,  1.80s/it]

Evaluating policy:  27%|##6       | 22/83 [00:40<01:47,  1.77s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:46,  1.77s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:46,  1.84s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:44,  1.83s/it]

Evaluating policy:  33%|###2      | 27/83 [00:49<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:51<01:38,  1.79s/it]

Evaluating policy:  35%|###4      | 29/83 [00:53<01:38,  1.83s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:37,  1.83s/it]

Evaluating policy:  37%|###7      | 31/83 [00:56<01:34,  1.81s/it]

Evaluating policy:  39%|###8      | 32/83 [00:58<01:29,  1.76s/it]

Evaluating policy:  40%|###9      | 33/83 [01:00<01:31,  1.83s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:27,  1.79s/it]

Evaluating policy:  42%|####2     | 35/83 [01:04<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:27,  1.86s/it]

Evaluating policy:  45%|####4     | 37/83 [01:07<01:23,  1.82s/it]

Evaluating policy:  46%|####5     | 38/83 [01:09<01:21,  1.80s/it]

Evaluating policy:  47%|####6     | 39/83 [01:11<01:19,  1.82s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:16,  1.78s/it]

Evaluating policy:  49%|####9     | 41/83 [01:14<01:14,  1.76s/it]

Evaluating policy:  51%|#####     | 42/83 [01:16<01:17,  1.90s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:18<01:13,  1.84s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:20<01:10,  1.80s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:06,  1.76s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:23<01:04,  1.74s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:25<01:02,  1.74s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:27<01:00,  1.72s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:29<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<00:59,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:32<00:57,  1.79s/it]

Evaluating policy:  63%|######2   | 52/83 [01:34<00:55,  1.80s/it]

Evaluating policy:  64%|######3   | 53/83 [01:36<00:53,  1.80s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:51,  1.76s/it]

Evaluating policy:  66%|######6   | 55/83 [01:39<00:49,  1.79s/it]

Evaluating policy:  67%|######7   | 56/83 [01:41<00:50,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:43<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:45<00:45,  1.83s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:47<00:43,  1.80s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:49<00:41,  1.82s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:50<00:40,  1.82s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:52<00:38,  1.81s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:54<00:35,  1.80s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:56<00:36,  1.91s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:58<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:00<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:01<00:28,  1.78s/it]

Evaluating policy:  82%|########1 | 68/83 [02:03<00:26,  1.78s/it]

Evaluating policy:  83%|########3 | 69/83 [02:05<00:24,  1.75s/it]

Evaluating policy:  84%|########4 | 70/83 [02:07<00:23,  1.78s/it]

Evaluating policy:  86%|########5 | 71/83 [02:08<00:21,  1.76s/it]

Evaluating policy:  87%|########6 | 72/83 [02:10<00:19,  1.76s/it]

Evaluating policy:  88%|########7 | 73/83 [02:12<00:19,  1.91s/it]

Evaluating policy:  89%|########9 | 74/83 [02:14<00:16,  1.87s/it]

Evaluating policy:  90%|######### | 75/83 [02:16<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:18<00:12,  1.84s/it]

Evaluating policy:  93%|#########2| 77/83 [02:20<00:10,  1.82s/it]

Evaluating policy:  94%|#########3| 78/83 [02:21<00:09,  1.81s/it]

Evaluating policy:  95%|#########5| 79/83 [02:23<00:07,  1.83s/it]

Evaluating policy:  96%|#########6| 80/83 [02:25<00:05,  1.83s/it]

Evaluating policy:  98%|#########7| 81/83 [02:27<00:03,  1.81s/it]

Evaluating policy:  99%|#########8| 82/83 [02:29<00:01,  1.99s/it]

Evaluating policy: 100%|##########| 83/83 [02:31<00:00,  1.93s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:17,  1.68s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:17,  1.70s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:16,  1.71s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:18,  1.75s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.76s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:14,  1.75s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:14,  1.78s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:11,  1.75s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:08,  1.74s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:21,  1.94s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:16,  1.89s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:11,  1.85s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:08,  1.83s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:05,  1.84s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:02,  1.82s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.79s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:55,  1.77s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:56,  1.83s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:54,  1.83s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:51,  1.80s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:49,  1.79s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:52,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:45,  1.82s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:42,  1.80s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:38,  1.77s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:41,  1.84s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:37,  1.80s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:35,  1.80s/it]

Evaluating policy:  37%|###7      | 31/83 [00:56<01:32,  1.77s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:29,  1.75s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:27,  1.75s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:29,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:03<01:26,  1.80s/it]

Evaluating policy:  43%|####3     | 36/83 [01:05<01:25,  1.82s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:21,  1.78s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:18,  1.74s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:15,  1.71s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:15,  1.79s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:13,  1.79s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:10,  1.77s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:08,  1.77s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:07,  1.76s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:04,  1.75s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:07,  1.87s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<01:01,  1.82s/it]

Evaluating policy:  60%|######    | 50/83 [01:30<01:00,  1.85s/it]

Evaluating policy:  61%|######1   | 51/83 [01:32<00:58,  1.84s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:53,  1.79s/it]

Evaluating policy:  65%|######5   | 54/83 [01:37<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:39<00:51,  1.85s/it]

Evaluating policy:  67%|######7   | 56/83 [01:41<00:49,  1.84s/it]

Evaluating policy:  69%|######8   | 57/83 [01:43<00:47,  1.82s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:45,  1.82s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:46<00:43,  1.81s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:48<00:41,  1.79s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:50<00:39,  1.77s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:52<00:40,  1.92s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:54<00:37,  1.90s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:56<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:57<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:59<00:30,  1.82s/it]

Evaluating policy:  81%|########  | 67/83 [02:01<00:29,  1.84s/it]

Evaluating policy:  82%|########1 | 68/83 [02:03<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:05<00:25,  1.83s/it]

Evaluating policy:  84%|########4 | 70/83 [02:07<00:26,  2.01s/it]

Evaluating policy:  86%|########5 | 71/83 [02:09<00:23,  1.98s/it]

Evaluating policy:  87%|########6 | 72/83 [02:11<00:21,  1.93s/it]

Evaluating policy:  88%|########7 | 73/83 [02:13<00:19,  1.94s/it]

Evaluating policy:  89%|########9 | 74/83 [02:15<00:17,  1.92s/it]

Evaluating policy:  90%|######### | 75/83 [02:17<00:15,  1.93s/it]

Evaluating policy:  92%|#########1| 76/83 [02:18<00:13,  1.91s/it]

Evaluating policy:  93%|#########2| 77/83 [02:20<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:22<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:24<00:07,  1.81s/it]

Evaluating policy:  96%|#########6| 80/83 [02:26<00:05,  1.94s/it]

Evaluating policy:  98%|#########7| 81/83 [02:28<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:29<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:31<00:00,  1.85s/it]

Outer training loop:  30%|###       | 15001/50000 [8:43:23<917:37:07, 94.39s/it]
Outer training loop:  30%|###       | 15002/50000 [8:43:25<647:14:26, 66.58s/it]
Outer training loop:  30%|###       | 15003/50000 [8:43:27<458:26:08, 47.16s/it]
Outer training loop:  30%|###       | 15004/50000 [8:43:29<325:55:43, 33.53s/it]
Outer training loop:  30%|###       | 15005/50000 [8:43:31<233:51:44, 24.06s/it]
Outer training loop:  30%|###       | 15006/50000 [8:43:33<169:35:40, 17.45s/it]
Outer training loop:  30%|###       | 15007/50000 [8:43:35<126:20:12, 13.00s/it]
Outer training loop:  30%|###       | 15008/50000 [8:43:37<93:48:23,  9.65s/it] 
Outer training loop:  30%|###       | 15009/50000 [8:43:39<70:58:34,  7.30s/it]
Outer training loop:  30%|###       | 15010/50000 [8:43:41<54:34:28,  5.61s/it]
Outer training loop:  30%|###       | 15011/50000 [8:43:43<44:07:45,  4.54s/it]
Outer training loop:  30%|###       | 15012/50000 [8:43:45<36:23:08,  3.74s/it]
Outer training loop:  30%|###       | 15013/50000 [8:43:46<30:25:38,  3.13s/it]
Outer training loop:  30%|###       | 15014/50000 [8:43:48<27:30:57,  2.83s/it]
Outer training loop:  30%|###       | 15015/50000 [8:43:50<24:41:27,  2.54s/it]
Outer training loop:  30%|###       | 15016/50000 [8:43:52<22:06:56,  2.28s/it]
Outer training loop:  30%|###       | 15017/50000 [8:43:54<20:45:33,  2.14s/it]
Outer training loop:  30%|###       | 15018/50000 [8:43:56<19:50:02,  2.04s/it]
Outer training loop:  30%|###       | 15019/50000 [8:43:57<19:05:37,  1.97s/it]
Outer training loop:  30%|###       | 15020/50000 [8:43:59<18:08:39,  1.87s/it]
Outer training loop:  30%|###       | 15021/50000 [8:44:01<18:31:44,  1.91s/it]
Outer training loop:  30%|###       | 15022/50000 [8:44:03<18:12:59,  1.87s/it]
Outer training loop:  30%|###       | 15023/50000 [8:44:05<18:14:22,  1.88s/it]
Outer training loop:  30%|###       | 15024/50000 [8:44:07<18:36:33,  1.92s/it]
Outer training loop:  30%|###       | 15025/50000 [8:44:08<18:18:36,  1.88s/it]
Outer training loop:  30%|###       | 15026/50000 [8:44:10<17:54:28,  1.84s/it]
Outer training loop:  30%|###       | 15027/50000 [8:44:12<17:36:11,  1.81s/it]
Outer training loop:  30%|###       | 15028/50000 [8:44:14<17:57:47,  1.85s/it]
Outer training loop:  30%|###       | 15029/50000 [8:44:16<18:02:56,  1.86s/it]
Outer training loop:  30%|###       | 15030/50000 [8:44:18<17:44:27,  1.83s/it]
Outer training loop:  30%|###       | 15031/50000 [8:44:20<18:48:53,  1.94s/it]
Outer training loop:  30%|###       | 15032/50000 [8:44:21<18:15:41,  1.88s/it]
Outer training loop:  30%|###       | 15033/50000 [8:44:23<18:15:52,  1.88s/it]
Outer training loop:  30%|###       | 15034/50000 [8:44:25<18:08:31,  1.87s/it]
Outer training loop:  30%|###       | 15035/50000 [8:44:27<18:22:57,  1.89s/it]
Outer training loop:  30%|###       | 15036/50000 [8:44:29<18:25:09,  1.90s/it]
Outer training loop:  30%|###       | 15037/50000 [8:44:31<18:09:37,  1.87s/it]
Outer training loop:  30%|###       | 15038/50000 [8:44:33<18:20:40,  1.89s/it]
Outer training loop:  30%|###       | 15039/50000 [8:44:35<18:23:41,  1.89s/it]
Outer training loop:  30%|###       | 15040/50000 [8:44:37<18:24:11,  1.90s/it]
Outer training loop:  30%|###       | 15041/50000 [8:44:39<18:57:30,  1.95s/it]
Outer training loop:  30%|###       | 15042/50000 [8:44:40<18:31:24,  1.91s/it]
Outer training loop:  30%|###       | 15043/50000 [8:44:42<18:07:18,  1.87s/it]
Outer training loop:  30%|###       | 15044/50000 [8:44:44<18:32:07,  1.91s/it]
Outer training loop:  30%|###       | 15045/50000 [8:44:46<18:38:18,  1.92s/it]
Outer training loop:  30%|###       | 15046/50000 [8:44:48<18:12:57,  1.88s/it]
Outer training loop:  30%|###       | 15047/50000 [8:44:50<17:57:50,  1.85s/it]
Outer training loop:  30%|###       | 15048/50000 [8:44:52<18:09:02,  1.87s/it]
Outer training loop:  30%|###       | 15049/50000 [8:44:53<17:38:03,  1.82s/it]
Outer training loop:  30%|###       | 15050/50000 [8:44:55<17:39:31,  1.82s/it]
Outer training loop:  30%|###       | 15051/50000 [8:44:57<17:48:52,  1.84s/it]
Outer training loop:  30%|###       | 15052/50000 [8:44:59<17:33:38,  1.81s/it]
Outer training loop:  30%|###       | 15053/50000 [8:45:01<17:27:58,  1.80s/it]
Outer training loop:  30%|###       | 15054/50000 [8:45:02<17:17:33,  1.78s/it]
Outer training loop:  30%|###       | 15055/50000 [8:45:04<18:09:31,  1.87s/it]
Outer training loop:  30%|###       | 15056/50000 [8:45:06<18:03:42,  1.86s/it]
Outer training loop:  30%|###       | 15057/50000 [8:45:08<17:29:43,  1.80s/it]
Outer training loop:  30%|###       | 15058/50000 [8:45:10<18:27:43,  1.90s/it]
Outer training loop:  30%|###       | 15059/50000 [8:45:12<18:10:57,  1.87s/it]
Outer training loop:  30%|###       | 15060/50000 [8:45:14<17:41:26,  1.82s/it]
Outer training loop:  30%|###       | 15061/50000 [8:45:16<18:18:38,  1.89s/it]
Outer training loop:  30%|###       | 15062/50000 [8:45:17<18:00:30,  1.86s/it]
Outer training loop:  30%|###       | 15063/50000 [8:45:19<17:41:04,  1.82s/it]
Outer training loop:  30%|###       | 15064/50000 [8:45:21<17:41:43,  1.82s/it]
Outer training loop:  30%|###       | 15065/50000 [8:45:23<18:11:00,  1.87s/it]
Outer training loop:  30%|###       | 15066/50000 [8:45:25<18:29:43,  1.91s/it]
Outer training loop:  30%|###       | 15067/50000 [8:45:27<17:47:21,  1.83s/it]
Outer training loop:  30%|###       | 15068/50000 [8:45:29<18:38:03,  1.92s/it]
Outer training loop:  30%|###       | 15069/50000 [8:45:30<18:13:09,  1.88s/it]
Outer training loop:  30%|###       | 15070/50000 [8:45:32<18:01:16,  1.86s/it]
Outer training loop:  30%|###       | 15071/50000 [8:45:34<18:42:41,  1.93s/it]
Outer training loop:  30%|###       | 15072/50000 [8:45:36<18:34:00,  1.91s/it]
Outer training loop:  30%|###       | 15073/50000 [8:45:38<18:10:25,  1.87s/it]
Outer training loop:  30%|###       | 15074/50000 [8:45:40<17:41:16,  1.82s/it]
Outer training loop:  30%|###       | 15075/50000 [8:45:42<18:00:21,  1.86s/it]
Outer training loop:  30%|###       | 15076/50000 [8:45:43<17:39:15,  1.82s/it]
Outer training loop:  30%|###       | 15077/50000 [8:45:45<17:43:13,  1.83s/it]
Outer training loop:  30%|###       | 15078/50000 [8:45:47<18:05:36,  1.87s/it]
Outer training loop:  30%|###       | 15079/50000 [8:45:49<17:45:22,  1.83s/it]
Outer training loop:  30%|###       | 15080/50000 [8:45:51<18:02:35,  1.86s/it]
Outer training loop:  30%|###       | 15081/50000 [8:45:53<18:09:49,  1.87s/it]
Outer training loop:  30%|###       | 15082/50000 [8:45:55<17:45:16,  1.83s/it]
Outer training loop:  30%|###       | 15083/50000 [8:45:56<17:16:20,  1.78s/it]
Outer training loop:  30%|###       | 15084/50000 [8:45:58<17:29:50,  1.80s/it]
Outer training loop:  30%|###       | 15085/50000 [8:46:00<18:06:40,  1.87s/it]
Outer training loop:  30%|###       | 15086/50000 [8:46:02<18:03:00,  1.86s/it]
Outer training loop:  30%|###       | 15087/50000 [8:46:04<17:46:30,  1.83s/it]
Outer training loop:  30%|###       | 15088/50000 [8:46:06<18:08:24,  1.87s/it]
Outer training loop:  30%|###       | 15089/50000 [8:46:07<17:39:09,  1.82s/it]
Outer training loop:  30%|###       | 15090/50000 [8:46:09<17:51:44,  1.84s/it]
Outer training loop:  30%|###       | 15091/50000 [8:46:11<18:35:49,  1.92s/it]
Outer training loop:  30%|###       | 15092/50000 [8:46:13<18:05:41,  1.87s/it]
Outer training loop:  30%|###       | 15093/50000 [8:46:15<17:47:50,  1.84s/it]
Outer training loop:  30%|###       | 15094/50000 [8:46:17<17:23:30,  1.79s/it]
Outer training loop:  30%|###       | 15095/50000 [8:46:18<17:52:55,  1.84s/it]
Outer training loop:  30%|###       | 15096/50000 [8:46:20<17:28:46,  1.80s/it]
Outer training loop:  30%|###       | 15097/50000 [8:46:22<17:18:04,  1.78s/it]
Outer training loop:  30%|###       | 15098/50000 [8:46:24<16:55:11,  1.75s/it]
Outer training loop:  30%|###       | 15099/50000 [8:46:25<17:08:36,  1.77s/it]
Outer training loop:  30%|###       | 15100/50000 [8:46:27<17:06:26,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  30%|###       | 15101/50000 [8:46:34<32:04:32,  3.31s/it]
Outer training loop:  30%|###       | 15102/50000 [8:46:36<28:11:43,  2.91s/it]
Outer training loop:  30%|###       | 15103/50000 [8:46:38<24:24:45,  2.52s/it]
Outer training loop:  30%|###       | 15104/50000 [8:46:39<22:02:56,  2.27s/it]
Outer training loop:  30%|###       | 15105/50000 [8:46:41<20:40:58,  2.13s/it]
Outer training loop:  30%|###       | 15106/50000 [8:46:43<19:48:33,  2.04s/it]
Outer training loop:  30%|###       | 15107/50000 [8:46:45<18:58:26,  1.96s/it]
Outer training loop:  30%|###       | 15108/50000 [8:46:46<18:20:36,  1.89s/it]
Outer training loop:  30%|###       | 15109/50000 [8:46:48<18:20:01,  1.89s/it]
Outer training loop:  30%|###       | 15110/50000 [8:46:50<17:35:47,  1.82s/it]
Outer training loop:  30%|###       | 15111/50000 [8:46:52<17:15:22,  1.78s/it]
Outer training loop:  30%|###       | 15112/50000 [8:46:53<17:09:54,  1.77s/it]
Outer training loop:  30%|###       | 15113/50000 [8:46:55<16:54:57,  1.75s/it]
Outer training loop:  30%|###       | 15114/50000 [8:46:57<16:38:13,  1.72s/it]
Outer training loop:  30%|###       | 15115/50000 [8:46:59<17:57:54,  1.85s/it]
Outer training loop:  30%|###       | 15116/50000 [8:47:01<17:23:06,  1.79s/it]
Outer training loop:  30%|###       | 15117/50000 [8:47:02<17:10:43,  1.77s/it]
Outer training loop:  30%|###       | 15118/50000 [8:47:04<17:01:58,  1.76s/it]
Outer training loop:  30%|###       | 15119/50000 [8:47:06<17:34:11,  1.81s/it]
Outer training loop:  30%|###       | 15120/50000 [8:47:08<17:16:41,  1.78s/it]
Outer training loop:  30%|###       | 15121/50000 [8:47:09<17:08:58,  1.77s/it]
Outer training loop:  30%|###       | 15122/50000 [8:47:11<17:34:53,  1.81s/it]
Outer training loop:  30%|###       | 15123/50000 [8:47:13<17:50:58,  1.84s/it]
Outer training loop:  30%|###       | 15124/50000 [8:47:15<17:37:54,  1.82s/it]
Outer training loop:  30%|###       | 15125/50000 [8:47:17<18:11:54,  1.88s/it]
Outer training loop:  30%|###       | 15126/50000 [8:47:19<18:18:05,  1.89s/it]
Outer training loop:  30%|###       | 15127/50000 [8:47:21<18:07:27,  1.87s/it]
Outer training loop:  30%|###       | 15128/50000 [8:47:23<18:26:42,  1.90s/it]
Outer training loop:  30%|###       | 15129/50000 [8:47:25<17:57:19,  1.85s/it]
Outer training loop:  30%|###       | 15130/50000 [8:47:26<17:29:15,  1.81s/it]
Outer training loop:  30%|###       | 15131/50000 [8:47:28<17:56:33,  1.85s/it]
Outer training loop:  30%|###       | 15132/50000 [8:47:30<17:52:28,  1.85s/it]
Outer training loop:  30%|###       | 15133/50000 [8:47:32<17:44:36,  1.83s/it]
Outer training loop:  30%|###       | 15134/50000 [8:47:34<17:55:25,  1.85s/it]
Outer training loop:  30%|###       | 15135/50000 [8:47:36<18:44:49,  1.94s/it]
Outer training loop:  30%|###       | 15136/50000 [8:47:38<18:58:45,  1.96s/it]
Outer training loop:  30%|###       | 15137/50000 [8:47:40<18:22:00,  1.90s/it]
Outer training loop:  30%|###       | 15138/50000 [8:47:42<18:35:56,  1.92s/it]
Outer training loop:  30%|###       | 15139/50000 [8:47:43<17:58:23,  1.86s/it]
Outer training loop:  30%|###       | 15140/50000 [8:47:45<17:57:38,  1.85s/it]
Outer training loop:  30%|###       | 15141/50000 [8:47:47<17:37:31,  1.82s/it]
Outer training loop:  30%|###       | 15142/50000 [8:47:49<17:55:14,  1.85s/it]
Outer training loop:  30%|###       | 15143/50000 [8:47:51<17:41:03,  1.83s/it]
Outer training loop:  30%|###       | 15144/50000 [8:47:52<17:29:51,  1.81s/it]
Outer training loop:  30%|###       | 15145/50000 [8:47:54<17:27:08,  1.80s/it]
Outer training loop:  30%|###       | 15146/50000 [8:47:56<18:16:41,  1.89s/it]
Outer training loop:  30%|###       | 15147/50000 [8:47:58<17:47:25,  1.84s/it]
Outer training loop:  30%|###       | 15148/50000 [8:48:00<18:00:46,  1.86s/it]
Outer training loop:  30%|###       | 15149/50000 [8:48:02<18:27:06,  1.91s/it]
Outer training loop:  30%|###       | 15150/50000 [8:48:04<18:24:18,  1.90s/it]
Outer training loop:  30%|###       | 15151/50000 [8:48:06<18:32:45,  1.92s/it]
Outer training loop:  30%|###       | 15152/50000 [8:48:08<18:19:16,  1.89s/it]
Outer training loop:  30%|###       | 15153/50000 [8:48:09<18:25:57,  1.90s/it]
Outer training loop:  30%|###       | 15154/50000 [8:48:11<17:43:56,  1.83s/it]
Outer training loop:  30%|###       | 15155/50000 [8:48:13<17:40:22,  1.83s/it]
Outer training loop:  30%|###       | 15156/50000 [8:48:15<17:46:58,  1.84s/it]
Outer training loop:  30%|###       | 15157/50000 [8:48:17<17:45:27,  1.83s/it]
Outer training loop:  30%|###       | 15158/50000 [8:48:18<17:31:35,  1.81s/it]
Outer training loop:  30%|###       | 15159/50000 [8:48:20<17:51:36,  1.85s/it]
Outer training loop:  30%|###       | 15160/50000 [8:48:22<17:50:45,  1.84s/it]
Outer training loop:  30%|###       | 15161/50000 [8:48:24<17:51:49,  1.85s/it]
Outer training loop:  30%|###       | 15162/50000 [8:48:26<18:15:37,  1.89s/it]
Outer training loop:  30%|###       | 15163/50000 [8:48:28<18:11:22,  1.88s/it]
Outer training loop:  30%|###       | 15164/50000 [8:48:30<18:05:21,  1.87s/it]
Outer training loop:  30%|###       | 15165/50000 [8:48:32<18:47:20,  1.94s/it]
Outer training loop:  30%|###       | 15166/50000 [8:48:34<18:18:30,  1.89s/it]
Outer training loop:  30%|###       | 15167/50000 [8:48:35<18:10:02,  1.88s/it]
Outer training loop:  30%|###       | 15168/50000 [8:48:37<17:48:45,  1.84s/it]
Outer training loop:  30%|###       | 15169/50000 [8:48:39<18:14:41,  1.89s/it]
Outer training loop:  30%|###       | 15170/50000 [8:48:41<17:58:53,  1.86s/it]
Outer training loop:  30%|###       | 15171/50000 [8:48:43<17:47:17,  1.84s/it]
Outer training loop:  30%|###       | 15172/50000 [8:48:45<18:03:24,  1.87s/it]
Outer training loop:  30%|###       | 15173/50000 [8:48:46<17:32:13,  1.81s/it]
Outer training loop:  30%|###       | 15174/50000 [8:48:48<17:26:06,  1.80s/it]
Outer training loop:  30%|###       | 15175/50000 [8:48:50<17:56:10,  1.85s/it]
Outer training loop:  30%|###       | 15176/50000 [8:48:52<17:43:21,  1.83s/it]
Outer training loop:  30%|###       | 15177/50000 [8:48:54<17:07:20,  1.77s/it]
Outer training loop:  30%|###       | 15178/50000 [8:48:55<17:07:27,  1.77s/it]
Outer training loop:  30%|###       | 15179/50000 [8:48:57<17:55:55,  1.85s/it]
Outer training loop:  30%|###       | 15180/50000 [8:48:59<17:25:17,  1.80s/it]
Outer training loop:  30%|###       | 15181/50000 [8:49:01<17:12:07,  1.78s/it]
Outer training loop:  30%|###       | 15182/50000 [8:49:03<17:01:08,  1.76s/it]
Outer training loop:  30%|###       | 15183/50000 [8:49:05<17:55:47,  1.85s/it]
Outer training loop:  30%|###       | 15184/50000 [8:49:06<17:43:06,  1.83s/it]
Outer training loop:  30%|###       | 15185/50000 [8:49:08<17:19:59,  1.79s/it]
Outer training loop:  30%|###       | 15186/50000 [8:49:10<18:08:14,  1.88s/it]
Outer training loop:  30%|###       | 15187/50000 [8:49:12<17:58:13,  1.86s/it]
Outer training loop:  30%|###       | 15188/50000 [8:49:14<17:23:05,  1.80s/it]
Outer training loop:  30%|###       | 15189/50000 [8:49:16<17:43:13,  1.83s/it]
Outer training loop:  30%|###       | 15190/50000 [8:49:17<17:11:34,  1.78s/it]
Outer training loop:  30%|###       | 15191/50000 [8:49:19<17:22:16,  1.80s/it]
Outer training loop:  30%|###       | 15192/50000 [8:49:21<17:17:23,  1.79s/it]
Outer training loop:  30%|###       | 15193/50000 [8:49:23<17:29:31,  1.81s/it]
Outer training loop:  30%|###       | 15194/50000 [8:49:24<17:26:03,  1.80s/it]
Outer training loop:  30%|###       | 15195/50000 [8:49:26<17:07:42,  1.77s/it]
Outer training loop:  30%|###       | 15196/50000 [8:49:28<17:28:22,  1.81s/it]
Outer training loop:  30%|###       | 15197/50000 [8:49:30<17:07:18,  1.77s/it]
Outer training loop:  30%|###       | 15198/50000 [8:49:31<16:46:15,  1.73s/it]
Outer training loop:  30%|###       | 15199/50000 [8:49:33<17:07:04,  1.77s/it]
Outer training loop:  30%|###       | 15200/50000 [8:49:35<16:57:30,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.62s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.71s/it]

Outer training loop:  30%|###       | 15201/50000 [8:49:42<32:04:50,  3.32s/it]
Outer training loop:  30%|###       | 15202/50000 [8:49:44<27:18:45,  2.83s/it]
Outer training loop:  30%|###       | 15203/50000 [8:49:46<24:56:27,  2.58s/it]
Outer training loop:  30%|###       | 15204/50000 [8:49:47<22:24:33,  2.32s/it]
Outer training loop:  30%|###       | 15205/50000 [8:49:49<20:44:43,  2.15s/it]
Outer training loop:  30%|###       | 15206/50000 [8:49:51<19:24:02,  2.01s/it]
Outer training loop:  30%|###       | 15207/50000 [8:49:53<19:11:51,  1.99s/it]
Outer training loop:  30%|###       | 15208/50000 [8:49:54<18:36:52,  1.93s/it]
Outer training loop:  30%|###       | 15209/50000 [8:49:56<18:05:59,  1.87s/it]
Outer training loop:  30%|###       | 15210/50000 [8:49:58<17:50:08,  1.85s/it]
Outer training loop:  30%|###       | 15211/50000 [8:50:00<17:35:32,  1.82s/it]
Outer training loop:  30%|###       | 15212/50000 [8:50:02<17:49:23,  1.84s/it]
Outer training loop:  30%|###       | 15213/50000 [8:50:03<17:23:14,  1.80s/it]
Outer training loop:  30%|###       | 15214/50000 [8:50:05<17:49:50,  1.85s/it]
Outer training loop:  30%|###       | 15215/50000 [8:50:07<17:26:57,  1.81s/it]
Outer training loop:  30%|###       | 15216/50000 [8:50:09<17:15:15,  1.79s/it]
Outer training loop:  30%|###       | 15217/50000 [8:50:11<17:36:09,  1.82s/it]
Outer training loop:  30%|###       | 15218/50000 [8:50:12<17:04:11,  1.77s/it]
Outer training loop:  30%|###       | 15219/50000 [8:50:14<16:41:03,  1.73s/it]
Outer training loop:  30%|###       | 15220/50000 [8:50:16<16:59:15,  1.76s/it]
Outer training loop:  30%|###       | 15221/50000 [8:50:17<16:46:00,  1.74s/it]
Outer training loop:  30%|###       | 15222/50000 [8:50:19<16:46:25,  1.74s/it]
Outer training loop:  30%|###       | 15223/50000 [8:50:21<16:47:02,  1.74s/it]
Outer training loop:  30%|###       | 15224/50000 [8:50:23<17:28:13,  1.81s/it]
Outer training loop:  30%|###       | 15225/50000 [8:50:25<17:21:44,  1.80s/it]
Outer training loop:  30%|###       | 15226/50000 [8:50:26<17:09:32,  1.78s/it]
Outer training loop:  30%|###       | 15227/50000 [8:50:28<17:32:53,  1.82s/it]
Outer training loop:  30%|###       | 15228/50000 [8:50:30<17:14:50,  1.79s/it]
Outer training loop:  30%|###       | 15229/50000 [8:50:32<16:47:46,  1.74s/it]
Outer training loop:  30%|###       | 15230/50000 [8:50:33<16:39:50,  1.73s/it]
Outer training loop:  30%|###       | 15231/50000 [8:50:35<17:15:28,  1.79s/it]
Outer training loop:  30%|###       | 15232/50000 [8:50:37<16:53:58,  1.75s/it]
Outer training loop:  30%|###       | 15233/50000 [8:50:39<16:59:04,  1.76s/it]
Outer training loop:  30%|###       | 15234/50000 [8:50:41<17:22:26,  1.80s/it]
Outer training loop:  30%|###       | 15235/50000 [8:50:42<17:16:49,  1.79s/it]
Outer training loop:  30%|###       | 15236/50000 [8:50:44<17:16:19,  1.79s/it]
Outer training loop:  30%|###       | 15237/50000 [8:50:46<17:50:08,  1.85s/it]
Outer training loop:  30%|###       | 15238/50000 [8:50:48<17:31:59,  1.82s/it]
Outer training loop:  30%|###       | 15239/50000 [8:50:50<17:10:02,  1.78s/it]
Outer training loop:  30%|###       | 15240/50000 [8:50:51<16:57:44,  1.76s/it]
Outer training loop:  30%|###       | 15241/50000 [8:50:53<17:40:37,  1.83s/it]
Outer training loop:  30%|###       | 15242/50000 [8:50:55<17:19:29,  1.79s/it]
Outer training loop:  30%|###       | 15243/50000 [8:50:57<17:15:13,  1.79s/it]
Outer training loop:  30%|###       | 15244/50000 [8:50:59<17:34:32,  1.82s/it]
Outer training loop:  30%|###       | 15245/50000 [8:51:00<17:24:30,  1.80s/it]
Outer training loop:  30%|###       | 15246/50000 [8:51:02<17:01:35,  1.76s/it]
Outer training loop:  30%|###       | 15247/50000 [8:51:04<16:53:07,  1.75s/it]
Outer training loop:  30%|###       | 15248/50000 [8:51:06<17:14:35,  1.79s/it]
Outer training loop:  30%|###       | 15249/50000 [8:51:07<17:08:58,  1.78s/it]
Outer training loop:  30%|###       | 15250/50000 [8:51:09<16:52:57,  1.75s/it]
Outer training loop:  31%|###       | 15251/50000 [8:51:11<17:44:26,  1.84s/it]
Outer training loop:  31%|###       | 15252/50000 [8:51:13<17:27:56,  1.81s/it]
Outer training loop:  31%|###       | 15253/50000 [8:51:15<17:28:52,  1.81s/it]
Outer training loop:  31%|###       | 15254/50000 [8:51:16<17:00:45,  1.76s/it]
Outer training loop:  31%|###       | 15255/50000 [8:51:18<17:15:58,  1.79s/it]
Outer training loop:  31%|###       | 15256/50000 [8:51:20<17:29:03,  1.81s/it]
Outer training loop:  31%|###       | 15257/50000 [8:51:22<17:10:37,  1.78s/it]
Outer training loop:  31%|###       | 15258/50000 [8:51:24<17:42:14,  1.83s/it]
Outer training loop:  31%|###       | 15259/50000 [8:51:26<17:44:22,  1.84s/it]
Outer training loop:  31%|###       | 15260/50000 [8:51:27<17:20:17,  1.80s/it]
Outer training loop:  31%|###       | 15261/50000 [8:51:29<17:22:24,  1.80s/it]
Outer training loop:  31%|###       | 15262/50000 [8:51:31<18:13:58,  1.89s/it]
Outer training loop:  31%|###       | 15263/50000 [8:51:33<18:03:28,  1.87s/it]
Outer training loop:  31%|###       | 15264/50000 [8:51:35<17:36:32,  1.82s/it]
Outer training loop:  31%|###       | 15265/50000 [8:51:37<17:50:33,  1.85s/it]
Outer training loop:  31%|###       | 15266/50000 [8:51:38<17:14:43,  1.79s/it]
Outer training loop:  31%|###       | 15267/50000 [8:51:40<17:01:20,  1.76s/it]
Outer training loop:  31%|###       | 15268/50000 [8:51:42<17:31:55,  1.82s/it]
Outer training loop:  31%|###       | 15269/50000 [8:51:44<17:30:21,  1.81s/it]
Outer training loop:  31%|###       | 15270/50000 [8:51:46<17:28:26,  1.81s/it]
Outer training loop:  31%|###       | 15271/50000 [8:51:47<17:12:07,  1.78s/it]
Outer training loop:  31%|###       | 15272/50000 [8:51:49<17:38:32,  1.83s/it]
Outer training loop:  31%|###       | 15273/50000 [8:51:51<17:26:16,  1.81s/it]
Outer training loop:  31%|###       | 15274/50000 [8:51:53<17:23:00,  1.80s/it]
Outer training loop:  31%|###       | 15275/50000 [8:51:55<17:42:33,  1.84s/it]
Outer training loop:  31%|###       | 15276/50000 [8:51:56<17:08:25,  1.78s/it]
Outer training loop:  31%|###       | 15277/50000 [8:51:58<16:46:22,  1.74s/it]
Outer training loop:  31%|###       | 15278/50000 [8:52:00<17:20:50,  1.80s/it]
Outer training loop:  31%|###       | 15279/50000 [8:52:02<16:58:53,  1.76s/it]
Outer training loop:  31%|###       | 15280/50000 [8:52:03<17:01:02,  1.76s/it]
Outer training loop:  31%|###       | 15281/50000 [8:52:05<17:03:53,  1.77s/it]
Outer training loop:  31%|###       | 15282/50000 [8:52:07<17:14:29,  1.79s/it]
Outer training loop:  31%|###       | 15283/50000 [8:52:09<17:15:35,  1.79s/it]
Outer training loop:  31%|###       | 15284/50000 [8:52:10<17:06:43,  1.77s/it]
Outer training loop:  31%|###       | 15285/50000 [8:52:12<17:33:54,  1.82s/it]
Outer training loop:  31%|###       | 15286/50000 [8:52:14<17:02:12,  1.77s/it]
Outer training loop:  31%|###       | 15287/50000 [8:52:16<16:31:36,  1.71s/it]
Outer training loop:  31%|###       | 15288/50000 [8:52:18<17:09:04,  1.78s/it]
Outer training loop:  31%|###       | 15289/50000 [8:52:19<17:14:19,  1.79s/it]
Outer training loop:  31%|###       | 15290/50000 [8:52:21<16:50:45,  1.75s/it]
Outer training loop:  31%|###       | 15291/50000 [8:52:23<17:09:59,  1.78s/it]
Outer training loop:  31%|###       | 15292/50000 [8:52:25<17:43:17,  1.84s/it]
Outer training loop:  31%|###       | 15293/50000 [8:52:27<17:17:28,  1.79s/it]
Outer training loop:  31%|###       | 15294/50000 [8:52:28<17:13:24,  1.79s/it]
Outer training loop:  31%|###       | 15295/50000 [8:52:30<17:32:15,  1.82s/it]
Outer training loop:  31%|###       | 15296/50000 [8:52:32<17:19:24,  1.80s/it]
Outer training loop:  31%|###       | 15297/50000 [8:52:34<17:13:53,  1.79s/it]
Outer training loop:  31%|###       | 15298/50000 [8:52:36<17:42:59,  1.84s/it]
Outer training loop:  31%|###       | 15299/50000 [8:52:37<17:23:51,  1.80s/it]
Outer training loop:  31%|###       | 15300/50000 [8:52:39<17:18:17,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.64s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.68s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.65s/it]

Outer training loop:  31%|###       | 15301/50000 [8:52:46<32:13:51,  3.34s/it]
Outer training loop:  31%|###       | 15302/50000 [8:52:48<28:22:33,  2.94s/it]
Outer training loop:  31%|###       | 15303/50000 [8:52:50<24:57:34,  2.59s/it]
Outer training loop:  31%|###       | 15304/50000 [8:52:52<22:25:48,  2.33s/it]
Outer training loop:  31%|###       | 15305/50000 [8:52:53<21:02:06,  2.18s/it]
Outer training loop:  31%|###       | 15306/50000 [8:52:55<19:44:01,  2.05s/it]
Outer training loop:  31%|###       | 15307/50000 [8:52:57<18:53:52,  1.96s/it]
Outer training loop:  31%|###       | 15308/50000 [8:52:59<18:29:46,  1.92s/it]
Outer training loop:  31%|###       | 15309/50000 [8:53:01<18:46:17,  1.95s/it]
Outer training loop:  31%|###       | 15310/50000 [8:53:02<17:58:15,  1.86s/it]
Outer training loop:  31%|###       | 15311/50000 [8:53:04<17:41:32,  1.84s/it]
Outer training loop:  31%|###       | 15312/50000 [8:53:06<18:04:47,  1.88s/it]
Outer training loop:  31%|###       | 15313/50000 [8:53:08<18:04:49,  1.88s/it]
Outer training loop:  31%|###       | 15314/50000 [8:53:10<18:08:07,  1.88s/it]
Outer training loop:  31%|###       | 15315/50000 [8:53:12<18:00:32,  1.87s/it]
Outer training loop:  31%|###       | 15316/50000 [8:53:14<17:34:11,  1.82s/it]
Outer training loop:  31%|###       | 15317/50000 [8:53:15<17:12:26,  1.79s/it]
Outer training loop:  31%|###       | 15318/50000 [8:53:17<17:00:08,  1.76s/it]
Outer training loop:  31%|###       | 15319/50000 [8:53:19<17:35:17,  1.83s/it]
Outer training loop:  31%|###       | 15320/50000 [8:53:21<17:16:50,  1.79s/it]
Outer training loop:  31%|###       | 15321/50000 [8:53:22<17:00:15,  1.77s/it]
Outer training loop:  31%|###       | 15322/50000 [8:53:24<17:14:57,  1.79s/it]
Outer training loop:  31%|###       | 15323/50000 [8:53:26<16:48:48,  1.75s/it]
Outer training loop:  31%|###       | 15324/50000 [8:53:28<17:09:24,  1.78s/it]
Outer training loop:  31%|###       | 15325/50000 [8:53:30<17:19:18,  1.80s/it]
Outer training loop:  31%|###       | 15326/50000 [8:53:31<17:07:30,  1.78s/it]
Outer training loop:  31%|###       | 15327/50000 [8:53:33<17:03:17,  1.77s/it]
Outer training loop:  31%|###       | 15328/50000 [8:53:35<16:47:11,  1.74s/it]
Outer training loop:  31%|###       | 15329/50000 [8:53:37<17:29:44,  1.82s/it]
Outer training loop:  31%|###       | 15330/50000 [8:53:38<17:05:02,  1.77s/it]
Outer training loop:  31%|###       | 15331/50000 [8:53:40<17:10:43,  1.78s/it]
Outer training loop:  31%|###       | 15332/50000 [8:53:42<17:46:26,  1.85s/it]
Outer training loop:  31%|###       | 15333/50000 [8:53:44<17:20:36,  1.80s/it]
Outer training loop:  31%|###       | 15334/50000 [8:53:46<17:08:32,  1.78s/it]
Outer training loop:  31%|###       | 15335/50000 [8:53:48<17:35:00,  1.83s/it]
Outer training loop:  31%|###       | 15336/50000 [8:53:49<17:29:13,  1.82s/it]
Outer training loop:  31%|###       | 15337/50000 [8:53:51<17:32:39,  1.82s/it]
Outer training loop:  31%|###       | 15338/50000 [8:53:53<17:49:02,  1.85s/it]
Outer training loop:  31%|###       | 15339/50000 [8:53:55<17:11:06,  1.78s/it]
Outer training loop:  31%|###       | 15340/50000 [8:53:56<16:57:32,  1.76s/it]
Outer training loop:  31%|###       | 15341/50000 [8:53:58<17:12:53,  1.79s/it]
Outer training loop:  31%|###       | 15342/50000 [8:54:00<16:59:40,  1.77s/it]
Outer training loop:  31%|###       | 15343/50000 [8:54:02<17:23:46,  1.81s/it]
Outer training loop:  31%|###       | 15344/50000 [8:54:04<17:05:48,  1.78s/it]
Outer training loop:  31%|###       | 15345/50000 [8:54:05<17:30:17,  1.82s/it]
Outer training loop:  31%|###       | 15346/50000 [8:54:07<17:14:35,  1.79s/it]
Outer training loop:  31%|###       | 15347/50000 [8:54:09<16:55:25,  1.76s/it]
Outer training loop:  31%|###       | 15348/50000 [8:54:11<17:30:07,  1.82s/it]
Outer training loop:  31%|###       | 15349/50000 [8:54:13<17:20:23,  1.80s/it]
Outer training loop:  31%|###       | 15350/50000 [8:54:14<16:53:42,  1.76s/it]
Outer training loop:  31%|###       | 15351/50000 [8:54:16<17:21:47,  1.80s/it]
Outer training loop:  31%|###       | 15352/50000 [8:54:18<16:53:52,  1.76s/it]
Outer training loop:  31%|###       | 15353/50000 [8:54:19<16:36:28,  1.73s/it]
Outer training loop:  31%|###       | 15354/50000 [8:54:21<16:19:16,  1.70s/it]
Outer training loop:  31%|###       | 15355/50000 [8:54:23<16:38:35,  1.73s/it]
Outer training loop:  31%|###       | 15356/50000 [8:54:25<16:26:28,  1.71s/it]
Outer training loop:  31%|###       | 15357/50000 [8:54:26<16:28:06,  1.71s/it]
Outer training loop:  31%|###       | 15358/50000 [8:54:28<16:59:01,  1.76s/it]
Outer training loop:  31%|###       | 15359/50000 [8:54:30<17:19:52,  1.80s/it]
Outer training loop:  31%|###       | 15360/50000 [8:54:32<17:08:35,  1.78s/it]
Outer training loop:  31%|###       | 15361/50000 [8:54:34<17:45:30,  1.85s/it]
Outer training loop:  31%|###       | 15362/50000 [8:54:36<17:32:59,  1.82s/it]
Outer training loop:  31%|###       | 15363/50000 [8:54:37<17:20:10,  1.80s/it]
Outer training loop:  31%|###       | 15364/50000 [8:54:39<16:59:27,  1.77s/it]
Outer training loop:  31%|###       | 15365/50000 [8:54:41<17:26:04,  1.81s/it]
Outer training loop:  31%|###       | 15366/50000 [8:54:43<16:47:04,  1.74s/it]
Outer training loop:  31%|###       | 15367/50000 [8:54:44<16:40:59,  1.73s/it]
Outer training loop:  31%|###       | 15368/50000 [8:54:46<17:13:59,  1.79s/it]
Outer training loop:  31%|###       | 15369/50000 [8:54:48<17:14:49,  1.79s/it]
Outer training loop:  31%|###       | 15370/50000 [8:54:50<17:25:11,  1.81s/it]
Outer training loop:  31%|###       | 15371/50000 [8:54:52<17:33:30,  1.83s/it]
Outer training loop:  31%|###       | 15372/50000 [8:54:53<17:17:14,  1.80s/it]
Outer training loop:  31%|###       | 15373/50000 [8:54:55<17:02:33,  1.77s/it]
Outer training loop:  31%|###       | 15374/50000 [8:54:57<17:09:23,  1.78s/it]
Outer training loop:  31%|###       | 15375/50000 [8:54:59<17:33:59,  1.83s/it]
Outer training loop:  31%|###       | 15376/50000 [8:55:01<17:14:13,  1.79s/it]
Outer training loop:  31%|###       | 15377/50000 [8:55:02<17:17:42,  1.80s/it]
Outer training loop:  31%|###       | 15378/50000 [8:55:04<17:06:12,  1.78s/it]
Outer training loop:  31%|###       | 15379/50000 [8:55:06<17:35:45,  1.83s/it]
Outer training loop:  31%|###       | 15380/50000 [8:55:08<17:16:44,  1.80s/it]
Outer training loop:  31%|###       | 15381/50000 [8:55:10<17:09:19,  1.78s/it]
Outer training loop:  31%|###       | 15382/50000 [8:55:11<17:38:07,  1.83s/it]
Outer training loop:  31%|###       | 15383/50000 [8:55:13<17:16:33,  1.80s/it]
Outer training loop:  31%|###       | 15384/50000 [8:55:15<17:04:47,  1.78s/it]
Outer training loop:  31%|###       | 15385/50000 [8:55:17<17:36:49,  1.83s/it]
Outer training loop:  31%|###       | 15386/50000 [8:55:19<17:27:01,  1.81s/it]
Outer training loop:  31%|###       | 15387/50000 [8:55:20<17:23:48,  1.81s/it]
Outer training loop:  31%|###       | 15388/50000 [8:55:22<16:57:19,  1.76s/it]
Outer training loop:  31%|###       | 15389/50000 [8:55:24<17:22:48,  1.81s/it]
Outer training loop:  31%|###       | 15390/50000 [8:55:26<16:58:57,  1.77s/it]
Outer training loop:  31%|###       | 15391/50000 [8:55:27<17:00:06,  1.77s/it]
Outer training loop:  31%|###       | 15392/50000 [8:55:29<17:00:14,  1.77s/it]
Outer training loop:  31%|###       | 15393/50000 [8:55:31<17:19:33,  1.80s/it]
Outer training loop:  31%|###       | 15394/50000 [8:55:33<17:13:20,  1.79s/it]
Outer training loop:  31%|###       | 15395/50000 [8:55:35<17:02:26,  1.77s/it]
Outer training loop:  31%|###       | 15396/50000 [8:55:37<17:25:05,  1.81s/it]
Outer training loop:  31%|###       | 15397/50000 [8:55:38<17:02:54,  1.77s/it]
Outer training loop:  31%|###       | 15398/50000 [8:55:40<16:51:11,  1.75s/it]
Outer training loop:  31%|###       | 15399/50000 [8:55:42<17:37:41,  1.83s/it]
Outer training loop:  31%|###       | 15400/50000 [8:55:44<17:20:15,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.65s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  31%|###       | 15401/50000 [8:55:51<32:04:08,  3.34s/it]
Outer training loop:  31%|###       | 15402/50000 [8:55:52<27:31:14,  2.86s/it]
Outer training loop:  31%|###       | 15403/50000 [8:55:54<24:48:55,  2.58s/it]
Outer training loop:  31%|###       | 15404/50000 [8:55:56<22:28:48,  2.34s/it]
Outer training loop:  31%|###       | 15405/50000 [8:55:58<20:37:26,  2.15s/it]
Outer training loop:  31%|###       | 15406/50000 [8:56:00<20:20:15,  2.12s/it]
Outer training loop:  31%|###       | 15407/50000 [8:56:02<19:32:14,  2.03s/it]
Outer training loop:  31%|###       | 15408/50000 [8:56:03<18:14:17,  1.90s/it]
Outer training loop:  31%|###       | 15409/50000 [8:56:05<18:28:22,  1.92s/it]
Outer training loop:  31%|###       | 15410/50000 [8:56:07<17:29:38,  1.82s/it]
Outer training loop:  31%|###       | 15411/50000 [8:56:09<17:36:47,  1.83s/it]
Outer training loop:  31%|###       | 15412/50000 [8:56:10<17:21:43,  1.81s/it]
Outer training loop:  31%|###       | 15413/50000 [8:56:12<17:48:46,  1.85s/it]
Outer training loop:  31%|###       | 15414/50000 [8:56:14<17:46:30,  1.85s/it]
Outer training loop:  31%|###       | 15415/50000 [8:56:16<17:05:21,  1.78s/it]
Outer training loop:  31%|###       | 15416/50000 [8:56:18<17:17:43,  1.80s/it]
Outer training loop:  31%|###       | 15417/50000 [8:56:19<17:10:12,  1.79s/it]
Outer training loop:  31%|###       | 15418/50000 [8:56:21<17:08:58,  1.79s/it]
Outer training loop:  31%|###       | 15419/50000 [8:56:23<17:12:53,  1.79s/it]
Outer training loop:  31%|###       | 15420/50000 [8:56:25<17:19:36,  1.80s/it]
Outer training loop:  31%|###       | 15421/50000 [8:56:27<17:07:50,  1.78s/it]
Outer training loop:  31%|###       | 15422/50000 [8:56:28<16:51:08,  1.75s/it]
Outer training loop:  31%|###       | 15423/50000 [8:56:30<17:26:39,  1.82s/it]
Outer training loop:  31%|###       | 15424/50000 [8:56:32<17:20:00,  1.80s/it]
Outer training loop:  31%|###       | 15425/50000 [8:56:34<16:57:34,  1.77s/it]
Outer training loop:  31%|###       | 15426/50000 [8:56:36<17:26:56,  1.82s/it]
Outer training loop:  31%|###       | 15427/50000 [8:56:37<17:20:02,  1.80s/it]
Outer training loop:  31%|###       | 15428/50000 [8:56:39<16:45:39,  1.75s/it]
Outer training loop:  31%|###       | 15429/50000 [8:56:41<16:48:37,  1.75s/it]
Outer training loop:  31%|###       | 15430/50000 [8:56:43<17:29:39,  1.82s/it]
Outer training loop:  31%|###       | 15431/50000 [8:56:45<17:26:41,  1.82s/it]
Outer training loop:  31%|###       | 15432/50000 [8:56:46<17:15:12,  1.80s/it]
Outer training loop:  31%|###       | 15433/50000 [8:56:48<17:27:54,  1.82s/it]
Outer training loop:  31%|###       | 15434/50000 [8:56:50<17:23:44,  1.81s/it]
Outer training loop:  31%|###       | 15435/50000 [8:56:52<16:56:29,  1.76s/it]
Outer training loop:  31%|###       | 15436/50000 [8:56:53<17:11:42,  1.79s/it]
Outer training loop:  31%|###       | 15437/50000 [8:56:55<16:57:32,  1.77s/it]
Outer training loop:  31%|###       | 15438/50000 [8:56:57<16:41:45,  1.74s/it]
Outer training loop:  31%|###       | 15439/50000 [8:56:59<17:15:02,  1.80s/it]
Outer training loop:  31%|###       | 15440/50000 [8:57:00<16:57:47,  1.77s/it]
Outer training loop:  31%|###       | 15441/50000 [8:57:02<17:05:04,  1.78s/it]
Outer training loop:  31%|###       | 15442/50000 [8:57:04<16:33:56,  1.73s/it]
Outer training loop:  31%|###       | 15443/50000 [8:57:06<17:19:43,  1.81s/it]
Outer training loop:  31%|###       | 15444/50000 [8:57:07<16:50:30,  1.75s/it]
Outer training loop:  31%|###       | 15445/50000 [8:57:09<16:28:20,  1.72s/it]
Outer training loop:  31%|###       | 15446/50000 [8:57:11<17:05:11,  1.78s/it]
Outer training loop:  31%|###       | 15447/50000 [8:57:13<16:40:41,  1.74s/it]
Outer training loop:  31%|###       | 15448/50000 [8:57:14<16:37:45,  1.73s/it]
Outer training loop:  31%|###       | 15449/50000 [8:57:16<16:56:13,  1.76s/it]
Outer training loop:  31%|###       | 15450/50000 [8:57:18<16:37:17,  1.73s/it]
Outer training loop:  31%|###       | 15451/50000 [8:57:20<16:37:13,  1.73s/it]
Outer training loop:  31%|###       | 15452/50000 [8:57:21<16:38:33,  1.73s/it]
Outer training loop:  31%|###       | 15453/50000 [8:57:23<16:58:48,  1.77s/it]
Outer training loop:  31%|###       | 15454/50000 [8:57:25<16:48:00,  1.75s/it]
Outer training loop:  31%|###       | 15455/50000 [8:57:27<16:38:18,  1.73s/it]
Outer training loop:  31%|###       | 15456/50000 [8:57:29<17:27:06,  1.82s/it]
Outer training loop:  31%|###       | 15457/50000 [8:57:30<16:52:06,  1.76s/it]
Outer training loop:  31%|###       | 15458/50000 [8:57:32<16:52:38,  1.76s/it]
Outer training loop:  31%|###       | 15459/50000 [8:57:34<16:51:25,  1.76s/it]
Outer training loop:  31%|###       | 15460/50000 [8:57:36<17:17:02,  1.80s/it]
Outer training loop:  31%|###       | 15461/50000 [8:57:38<17:34:59,  1.83s/it]
Outer training loop:  31%|###       | 15462/50000 [8:57:39<17:23:50,  1.81s/it]
Outer training loop:  31%|###       | 15463/50000 [8:57:41<17:41:47,  1.84s/it]
Outer training loop:  31%|###       | 15464/50000 [8:57:43<17:40:17,  1.84s/it]
Outer training loop:  31%|###       | 15465/50000 [8:57:45<17:26:28,  1.82s/it]
Outer training loop:  31%|###       | 15466/50000 [8:57:47<17:54:18,  1.87s/it]
Outer training loop:  31%|###       | 15467/50000 [8:57:49<17:21:28,  1.81s/it]
Outer training loop:  31%|###       | 15468/50000 [8:57:50<17:00:23,  1.77s/it]
Outer training loop:  31%|###       | 15469/50000 [8:57:52<17:16:55,  1.80s/it]
Outer training loop:  31%|###       | 15470/50000 [8:57:54<16:49:49,  1.75s/it]
Outer training loop:  31%|###       | 15471/50000 [8:57:56<16:57:38,  1.77s/it]
Outer training loop:  31%|###       | 15472/50000 [8:57:57<17:29:10,  1.82s/it]
Outer training loop:  31%|###       | 15473/50000 [8:57:59<17:07:54,  1.79s/it]
Outer training loop:  31%|###       | 15474/50000 [8:58:01<16:38:23,  1.74s/it]
Outer training loop:  31%|###       | 15475/50000 [8:58:03<16:45:32,  1.75s/it]
Outer training loop:  31%|###       | 15476/50000 [8:58:04<17:07:53,  1.79s/it]
Outer training loop:  31%|###       | 15477/50000 [8:58:06<16:50:48,  1.76s/it]
Outer training loop:  31%|###       | 15478/50000 [8:58:08<16:34:41,  1.73s/it]
Outer training loop:  31%|###       | 15479/50000 [8:58:10<17:03:36,  1.78s/it]
Outer training loop:  31%|###       | 15480/50000 [8:58:11<16:52:04,  1.76s/it]
Outer training loop:  31%|###       | 15481/50000 [8:58:13<16:47:10,  1.75s/it]
Outer training loop:  31%|###       | 15482/50000 [8:58:15<17:14:06,  1.80s/it]
Outer training loop:  31%|###       | 15483/50000 [8:58:17<17:19:16,  1.81s/it]
Outer training loop:  31%|###       | 15484/50000 [8:58:19<17:02:59,  1.78s/it]
Outer training loop:  31%|###       | 15485/50000 [8:58:20<16:46:15,  1.75s/it]
Outer training loop:  31%|###       | 15486/50000 [8:58:22<17:24:30,  1.82s/it]
Outer training loop:  31%|###       | 15487/50000 [8:58:24<17:11:56,  1.79s/it]
Outer training loop:  31%|###       | 15488/50000 [8:58:26<16:59:51,  1.77s/it]
Outer training loop:  31%|###       | 15489/50000 [8:58:28<17:27:55,  1.82s/it]
Outer training loop:  31%|###       | 15490/50000 [8:58:29<17:02:22,  1.78s/it]
Outer training loop:  31%|###       | 15491/50000 [8:58:31<16:57:04,  1.77s/it]
Outer training loop:  31%|###       | 15492/50000 [8:58:33<17:32:35,  1.83s/it]
Outer training loop:  31%|###       | 15493/50000 [8:58:35<17:11:00,  1.79s/it]
Outer training loop:  31%|###       | 15494/50000 [8:58:36<16:54:01,  1.76s/it]
Outer training loop:  31%|###       | 15495/50000 [8:58:38<17:02:26,  1.78s/it]
Outer training loop:  31%|###       | 15496/50000 [8:58:40<16:47:54,  1.75s/it]
Outer training loop:  31%|###       | 15497/50000 [8:58:42<16:54:33,  1.76s/it]
Outer training loop:  31%|###       | 15498/50000 [8:58:43<16:43:59,  1.75s/it]
Outer training loop:  31%|###       | 15499/50000 [8:58:45<17:26:53,  1.82s/it]
Outer training loop:  31%|###1      | 15500/50000 [8:58:47<17:00:24,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.87s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:18,  1.69s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.79s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:20,  1.77s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:23,  1.84s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:19,  1.81s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:15,  1.79s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:13,  1.78s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:17,  1.86s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:12,  1.81s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:10,  1.82s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.79s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:07,  1.82s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:04,  1.80s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<01:59,  1.76s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:54,  1.71s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:57,  1.78s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:53,  1.75s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:50,  1.73s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:49,  1.74s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:47,  1.73s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:49,  1.79s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:47,  1.79s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:43,  1.75s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.78s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:44,  1.84s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:39,  1.81s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:34,  1.75s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:32,  1.75s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.77s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:29,  1.80s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:26,  1.76s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:23,  1.73s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:22,  1.75s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:23,  1.82s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:19,  1.78s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:17,  1.75s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:13,  1.71s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:11,  1.69s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:10,  1.72s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:08,  1.71s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:10,  1.82s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:06,  1.75s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:03,  1.70s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:01,  1.70s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:24<00:59,  1.69s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<00:56,  1.67s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:55,  1.67s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:57,  1.81s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:57,  1.85s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:53,  1.79s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.73s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:47,  1.74s/it]

Evaluating policy:  69%|######8   | 57/83 [01:40<00:46,  1.77s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:43,  1.74s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:43,  1.83s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:41,  1.78s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:38,  1.77s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:36,  1.74s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:34,  1.74s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:32,  1.72s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:54<00:30,  1.72s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:29,  1.71s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.77s/it]

Evaluating policy:  84%|########4 | 70/83 [02:03<00:23,  1.77s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:20,  1.72s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:18,  1.71s/it]

Evaluating policy:  88%|########7 | 73/83 [02:08<00:16,  1.70s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:15,  1.71s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:13,  1.75s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:11,  1.86s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:09,  1.81s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.84s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.82s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.77s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 83/83 [02:26<00:00,  1.73s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:19,  1.70s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:30,  1.86s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:25,  1.82s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.81s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:17,  1.77s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:19,  1.81s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:16,  1.80s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:11,  1.76s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:08,  1.74s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:10,  1.79s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:09,  1.79s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:04,  1.76s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:01,  1.73s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:04,  1.80s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:01,  1.79s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:58,  1.78s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:55,  1.76s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:58,  1.82s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:56,  1.82s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:52,  1.78s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:48,  1.75s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:46,  1.74s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:48,  1.82s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:45,  1.78s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:41,  1.76s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:39,  1.74s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:37,  1.74s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:41,  1.84s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.79s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:34,  1.79s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.75s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:28,  1.74s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:28,  1.77s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:25,  1.81s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:21,  1.78s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:18,  1.74s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:14,  1.70s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.72s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:15,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:11,  1.79s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.78s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:07,  1.78s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:04,  1.75s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:02,  1.79s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:58,  1.77s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:56,  1.77s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:54,  1.75s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:52,  1.75s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:50,  1.80s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:48,  1.78s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:45,  1.74s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:43,  1.76s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.76s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:39,  1.73s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:37,  1.72s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.79s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:33,  1.77s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:31,  1.75s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.73s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:28,  1.75s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:26,  1.76s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:25,  1.82s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:25,  1.96s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:20,  1.83s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:16,  1.79s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.78s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.77s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.75s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.73s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.92s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.82s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.86s/it]

Outer training loop:  31%|###1      | 15501/50000 [9:03:50<882:04:21, 92.05s/it]
Outer training loop:  31%|###1      | 15502/50000 [9:03:51<622:19:19, 64.94s/it]
Outer training loop:  31%|###1      | 15503/50000 [9:03:53<440:22:52, 45.96s/it]
Outer training loop:  31%|###1      | 15504/50000 [9:03:55<313:13:33, 32.69s/it]
Outer training loop:  31%|###1      | 15505/50000 [9:03:57<224:26:19, 23.42s/it]
Outer training loop:  31%|###1      | 15506/50000 [9:03:58<162:08:08, 16.92s/it]
Outer training loop:  31%|###1      | 15507/50000 [9:04:01<119:54:00, 12.51s/it]
Outer training loop:  31%|###1      | 15508/50000 [9:04:02<89:01:30,  9.29s/it] 
Outer training loop:  31%|###1      | 15509/50000 [9:04:04<67:21:16,  7.03s/it]
Outer training loop:  31%|###1      | 15510/50000 [9:04:06<52:51:52,  5.52s/it]
Outer training loop:  31%|###1      | 15511/50000 [9:04:08<41:53:02,  4.37s/it]
Outer training loop:  31%|###1      | 15512/50000 [9:04:10<34:09:35,  3.57s/it]
Outer training loop:  31%|###1      | 15513/50000 [9:04:11<29:09:19,  3.04s/it]
Outer training loop:  31%|###1      | 15514/50000 [9:04:13<25:15:04,  2.64s/it]
Outer training loop:  31%|###1      | 15515/50000 [9:04:15<22:28:56,  2.35s/it]
Outer training loop:  31%|###1      | 15516/50000 [9:04:16<20:26:15,  2.13s/it]
Outer training loop:  31%|###1      | 15517/50000 [9:04:18<19:52:29,  2.07s/it]
Outer training loop:  31%|###1      | 15518/50000 [9:04:20<19:18:12,  2.02s/it]
Outer training loop:  31%|###1      | 15519/50000 [9:04:22<18:24:45,  1.92s/it]
Outer training loop:  31%|###1      | 15520/50000 [9:04:24<17:48:07,  1.86s/it]
Outer training loop:  31%|###1      | 15521/50000 [9:04:26<18:22:28,  1.92s/it]
Outer training loop:  31%|###1      | 15522/50000 [9:04:27<17:52:49,  1.87s/it]
Outer training loop:  31%|###1      | 15523/50000 [9:04:29<17:20:49,  1.81s/it]
Outer training loop:  31%|###1      | 15524/50000 [9:04:31<17:25:26,  1.82s/it]
Outer training loop:  31%|###1      | 15525/50000 [9:04:33<17:10:00,  1.79s/it]
Outer training loop:  31%|###1      | 15526/50000 [9:04:34<17:04:43,  1.78s/it]
Outer training loop:  31%|###1      | 15527/50000 [9:04:36<17:37:13,  1.84s/it]
Outer training loop:  31%|###1      | 15528/50000 [9:04:38<17:18:40,  1.81s/it]
Outer training loop:  31%|###1      | 15529/50000 [9:04:40<17:22:56,  1.82s/it]
Outer training loop:  31%|###1      | 15530/50000 [9:04:42<17:02:21,  1.78s/it]
Outer training loop:  31%|###1      | 15531/50000 [9:04:43<17:13:00,  1.80s/it]
Outer training loop:  31%|###1      | 15532/50000 [9:04:45<17:24:38,  1.82s/it]
Outer training loop:  31%|###1      | 15533/50000 [9:04:47<17:14:03,  1.80s/it]
Outer training loop:  31%|###1      | 15534/50000 [9:04:49<17:51:19,  1.87s/it]
Outer training loop:  31%|###1      | 15535/50000 [9:04:51<17:46:23,  1.86s/it]
Outer training loop:  31%|###1      | 15536/50000 [9:04:53<17:20:33,  1.81s/it]
Outer training loop:  31%|###1      | 15537/50000 [9:04:55<17:46:49,  1.86s/it]
Outer training loop:  31%|###1      | 15538/50000 [9:04:56<17:41:16,  1.85s/it]
Outer training loop:  31%|###1      | 15539/50000 [9:04:58<17:29:29,  1.83s/it]
Outer training loop:  31%|###1      | 15540/50000 [9:05:00<17:14:43,  1.80s/it]
Outer training loop:  31%|###1      | 15541/50000 [9:05:02<17:34:06,  1.84s/it]
Outer training loop:  31%|###1      | 15542/50000 [9:05:04<17:24:24,  1.82s/it]
Outer training loop:  31%|###1      | 15543/50000 [9:05:05<17:15:59,  1.80s/it]
Outer training loop:  31%|###1      | 15544/50000 [9:05:07<16:57:45,  1.77s/it]
Outer training loop:  31%|###1      | 15545/50000 [9:05:09<17:32:01,  1.83s/it]
Outer training loop:  31%|###1      | 15546/50000 [9:05:11<17:15:13,  1.80s/it]
Outer training loop:  31%|###1      | 15547/50000 [9:05:13<17:23:13,  1.82s/it]
Outer training loop:  31%|###1      | 15548/50000 [9:05:15<17:48:09,  1.86s/it]
Outer training loop:  31%|###1      | 15549/50000 [9:05:16<17:24:53,  1.82s/it]
Outer training loop:  31%|###1      | 15550/50000 [9:05:18<17:17:33,  1.81s/it]
Outer training loop:  31%|###1      | 15551/50000 [9:05:20<17:25:51,  1.82s/it]
Outer training loop:  31%|###1      | 15552/50000 [9:05:22<17:10:31,  1.79s/it]
Outer training loop:  31%|###1      | 15553/50000 [9:05:23<16:36:44,  1.74s/it]
Outer training loop:  31%|###1      | 15554/50000 [9:05:25<16:34:54,  1.73s/it]
Outer training loop:  31%|###1      | 15555/50000 [9:05:27<17:01:13,  1.78s/it]
Outer training loop:  31%|###1      | 15556/50000 [9:05:29<16:59:36,  1.78s/it]
Outer training loop:  31%|###1      | 15557/50000 [9:05:31<17:04:33,  1.78s/it]
Outer training loop:  31%|###1      | 15558/50000 [9:05:32<17:27:39,  1.83s/it]
Outer training loop:  31%|###1      | 15559/50000 [9:05:34<17:01:34,  1.78s/it]
Outer training loop:  31%|###1      | 15560/50000 [9:05:36<17:01:48,  1.78s/it]
Outer training loop:  31%|###1      | 15561/50000 [9:05:38<16:56:13,  1.77s/it]
Outer training loop:  31%|###1      | 15562/50000 [9:05:40<17:25:40,  1.82s/it]
Outer training loop:  31%|###1      | 15563/50000 [9:05:41<17:06:41,  1.79s/it]
Outer training loop:  31%|###1      | 15564/50000 [9:05:43<17:01:26,  1.78s/it]
Outer training loop:  31%|###1      | 15565/50000 [9:05:45<17:17:38,  1.81s/it]
Outer training loop:  31%|###1      | 15566/50000 [9:05:47<16:38:51,  1.74s/it]
Outer training loop:  31%|###1      | 15567/50000 [9:05:48<16:25:59,  1.72s/it]
Outer training loop:  31%|###1      | 15568/50000 [9:05:50<16:44:08,  1.75s/it]
Outer training loop:  31%|###1      | 15569/50000 [9:05:52<16:29:31,  1.72s/it]
Outer training loop:  31%|###1      | 15570/50000 [9:05:54<16:49:45,  1.76s/it]
Outer training loop:  31%|###1      | 15571/50000 [9:05:55<16:36:38,  1.74s/it]
Outer training loop:  31%|###1      | 15572/50000 [9:05:57<17:12:32,  1.80s/it]
Outer training loop:  31%|###1      | 15573/50000 [9:05:59<17:16:40,  1.81s/it]
Outer training loop:  31%|###1      | 15574/50000 [9:06:01<16:56:20,  1.77s/it]
Outer training loop:  31%|###1      | 15575/50000 [9:06:03<17:10:07,  1.80s/it]
Outer training loop:  31%|###1      | 15576/50000 [9:06:04<16:30:15,  1.73s/it]
Outer training loop:  31%|###1      | 15577/50000 [9:06:06<16:35:42,  1.74s/it]
Outer training loop:  31%|###1      | 15578/50000 [9:06:07<16:09:35,  1.69s/it]
Outer training loop:  31%|###1      | 15579/50000 [9:06:09<16:45:54,  1.75s/it]
Outer training loop:  31%|###1      | 15580/50000 [9:06:11<16:41:09,  1.75s/it]
Outer training loop:  31%|###1      | 15581/50000 [9:06:13<16:50:37,  1.76s/it]
Outer training loop:  31%|###1      | 15582/50000 [9:06:15<17:07:32,  1.79s/it]
Outer training loop:  31%|###1      | 15583/50000 [9:06:16<16:49:53,  1.76s/it]
Outer training loop:  31%|###1      | 15584/50000 [9:06:18<16:48:49,  1.76s/it]
Outer training loop:  31%|###1      | 15585/50000 [9:06:20<16:56:48,  1.77s/it]
Outer training loop:  31%|###1      | 15586/50000 [9:06:22<17:22:50,  1.82s/it]
Outer training loop:  31%|###1      | 15587/50000 [9:06:24<17:36:48,  1.84s/it]
Outer training loop:  31%|###1      | 15588/50000 [9:06:26<17:29:34,  1.83s/it]
Outer training loop:  31%|###1      | 15589/50000 [9:06:27<17:43:27,  1.85s/it]
Outer training loop:  31%|###1      | 15590/50000 [9:06:29<17:13:39,  1.80s/it]
Outer training loop:  31%|###1      | 15591/50000 [9:06:31<16:49:58,  1.76s/it]
Outer training loop:  31%|###1      | 15592/50000 [9:06:33<16:42:57,  1.75s/it]
Outer training loop:  31%|###1      | 15593/50000 [9:06:35<17:33:08,  1.84s/it]
Outer training loop:  31%|###1      | 15594/50000 [9:06:36<17:14:38,  1.80s/it]
Outer training loop:  31%|###1      | 15595/50000 [9:06:38<16:56:26,  1.77s/it]
Outer training loop:  31%|###1      | 15596/50000 [9:06:40<17:18:02,  1.81s/it]
Outer training loop:  31%|###1      | 15597/50000 [9:06:42<17:00:24,  1.78s/it]
Outer training loop:  31%|###1      | 15598/50000 [9:06:43<16:59:01,  1.78s/it]
Outer training loop:  31%|###1      | 15599/50000 [9:06:45<17:08:55,  1.79s/it]
Outer training loop:  31%|###1      | 15600/50000 [9:06:47<16:43:01,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  31%|###1      | 15601/50000 [9:06:54<31:55:51,  3.34s/it]
Outer training loop:  31%|###1      | 15602/50000 [9:06:56<27:15:47,  2.85s/it]
Outer training loop:  31%|###1      | 15603/50000 [9:06:58<24:33:00,  2.57s/it]
Outer training loop:  31%|###1      | 15604/50000 [9:06:59<22:05:04,  2.31s/it]
Outer training loop:  31%|###1      | 15605/50000 [9:07:01<20:28:01,  2.14s/it]
Outer training loop:  31%|###1      | 15606/50000 [9:07:03<19:20:40,  2.02s/it]
Outer training loop:  31%|###1      | 15607/50000 [9:07:05<19:20:16,  2.02s/it]
Outer training loop:  31%|###1      | 15608/50000 [9:07:07<18:30:32,  1.94s/it]
Outer training loop:  31%|###1      | 15609/50000 [9:07:08<17:46:42,  1.86s/it]
Outer training loop:  31%|###1      | 15610/50000 [9:07:10<18:00:53,  1.89s/it]
Outer training loop:  31%|###1      | 15611/50000 [9:07:12<17:35:09,  1.84s/it]
Outer training loop:  31%|###1      | 15612/50000 [9:07:14<17:06:12,  1.79s/it]
Outer training loop:  31%|###1      | 15613/50000 [9:07:15<17:19:26,  1.81s/it]
Outer training loop:  31%|###1      | 15614/50000 [9:07:17<17:01:35,  1.78s/it]
Outer training loop:  31%|###1      | 15615/50000 [9:07:19<16:43:31,  1.75s/it]
Outer training loop:  31%|###1      | 15616/50000 [9:07:21<16:50:36,  1.76s/it]
Outer training loop:  31%|###1      | 15617/50000 [9:07:22<17:13:04,  1.80s/it]
Outer training loop:  31%|###1      | 15618/50000 [9:07:24<16:44:45,  1.75s/it]
Outer training loop:  31%|###1      | 15619/50000 [9:07:26<16:38:39,  1.74s/it]
Outer training loop:  31%|###1      | 15620/50000 [9:07:28<16:55:25,  1.77s/it]
Outer training loop:  31%|###1      | 15621/50000 [9:07:29<16:44:07,  1.75s/it]
Outer training loop:  31%|###1      | 15622/50000 [9:07:31<16:34:50,  1.74s/it]
Outer training loop:  31%|###1      | 15623/50000 [9:07:33<16:41:16,  1.75s/it]
Outer training loop:  31%|###1      | 15624/50000 [9:07:35<17:02:20,  1.78s/it]
Outer training loop:  31%|###1      | 15625/50000 [9:07:37<17:07:56,  1.79s/it]
Outer training loop:  31%|###1      | 15626/50000 [9:07:38<16:48:54,  1.76s/it]
Outer training loop:  31%|###1      | 15627/50000 [9:07:40<17:31:12,  1.83s/it]
Outer training loop:  31%|###1      | 15628/50000 [9:07:42<17:13:11,  1.80s/it]
Outer training loop:  31%|###1      | 15629/50000 [9:07:44<16:55:34,  1.77s/it]
Outer training loop:  31%|###1      | 15630/50000 [9:07:45<16:40:46,  1.75s/it]
Outer training loop:  31%|###1      | 15631/50000 [9:07:47<16:56:32,  1.77s/it]
Outer training loop:  31%|###1      | 15632/50000 [9:07:49<16:52:36,  1.77s/it]
Outer training loop:  31%|###1      | 15633/50000 [9:07:51<16:57:39,  1.78s/it]
Outer training loop:  31%|###1      | 15634/50000 [9:07:53<18:02:27,  1.89s/it]
Outer training loop:  31%|###1      | 15635/50000 [9:07:55<17:47:11,  1.86s/it]
Outer training loop:  31%|###1      | 15636/50000 [9:07:56<17:13:53,  1.81s/it]
Outer training loop:  31%|###1      | 15637/50000 [9:07:58<17:41:39,  1.85s/it]
Outer training loop:  31%|###1      | 15638/50000 [9:08:00<17:10:52,  1.80s/it]
Outer training loop:  31%|###1      | 15639/50000 [9:08:02<16:58:19,  1.78s/it]
Outer training loop:  31%|###1      | 15640/50000 [9:08:03<16:41:15,  1.75s/it]
Outer training loop:  31%|###1      | 15641/50000 [9:08:05<17:12:08,  1.80s/it]
Outer training loop:  31%|###1      | 15642/50000 [9:08:07<16:47:16,  1.76s/it]
Outer training loop:  31%|###1      | 15643/50000 [9:08:09<16:40:58,  1.75s/it]
Outer training loop:  31%|###1      | 15644/50000 [9:08:11<16:53:08,  1.77s/it]
Outer training loop:  31%|###1      | 15645/50000 [9:08:12<16:38:54,  1.74s/it]
Outer training loop:  31%|###1      | 15646/50000 [9:08:14<16:37:39,  1.74s/it]
Outer training loop:  31%|###1      | 15647/50000 [9:08:16<16:25:41,  1.72s/it]
Outer training loop:  31%|###1      | 15648/50000 [9:08:18<17:09:21,  1.80s/it]
Outer training loop:  31%|###1      | 15649/50000 [9:08:19<16:54:08,  1.77s/it]
Outer training loop:  31%|###1      | 15650/50000 [9:08:21<16:47:33,  1.76s/it]
Outer training loop:  31%|###1      | 15651/50000 [9:08:23<17:15:10,  1.81s/it]
Outer training loop:  31%|###1      | 15652/50000 [9:08:25<17:31:30,  1.84s/it]
Outer training loop:  31%|###1      | 15653/50000 [9:08:27<17:26:36,  1.83s/it]
Outer training loop:  31%|###1      | 15654/50000 [9:08:28<17:09:35,  1.80s/it]
Outer training loop:  31%|###1      | 15655/50000 [9:08:30<17:22:17,  1.82s/it]
Outer training loop:  31%|###1      | 15656/50000 [9:08:32<16:44:59,  1.76s/it]
Outer training loop:  31%|###1      | 15657/50000 [9:08:34<16:33:51,  1.74s/it]
Outer training loop:  31%|###1      | 15658/50000 [9:08:36<17:10:17,  1.80s/it]
Outer training loop:  31%|###1      | 15659/50000 [9:08:37<16:52:54,  1.77s/it]
Outer training loop:  31%|###1      | 15660/50000 [9:08:39<16:43:37,  1.75s/it]
Outer training loop:  31%|###1      | 15661/50000 [9:08:41<17:16:50,  1.81s/it]
Outer training loop:  31%|###1      | 15662/50000 [9:08:43<17:13:57,  1.81s/it]
Outer training loop:  31%|###1      | 15663/50000 [9:08:44<16:56:21,  1.78s/it]
Outer training loop:  31%|###1      | 15664/50000 [9:08:46<16:53:18,  1.77s/it]
Outer training loop:  31%|###1      | 15665/50000 [9:08:48<17:22:51,  1.82s/it]
Outer training loop:  31%|###1      | 15666/50000 [9:08:50<17:10:53,  1.80s/it]
Outer training loop:  31%|###1      | 15667/50000 [9:08:52<17:11:01,  1.80s/it]
Outer training loop:  31%|###1      | 15668/50000 [9:08:54<17:26:23,  1.83s/it]
Outer training loop:  31%|###1      | 15669/50000 [9:08:55<17:00:32,  1.78s/it]
Outer training loop:  31%|###1      | 15670/50000 [9:08:57<16:49:58,  1.77s/it]
Outer training loop:  31%|###1      | 15671/50000 [9:08:59<16:45:00,  1.76s/it]
Outer training loop:  31%|###1      | 15672/50000 [9:09:01<17:08:00,  1.80s/it]
Outer training loop:  31%|###1      | 15673/50000 [9:09:02<17:02:37,  1.79s/it]
Outer training loop:  31%|###1      | 15674/50000 [9:09:04<17:13:32,  1.81s/it]
Outer training loop:  31%|###1      | 15675/50000 [9:09:06<17:42:17,  1.86s/it]
Outer training loop:  31%|###1      | 15676/50000 [9:09:08<16:56:41,  1.78s/it]
Outer training loop:  31%|###1      | 15677/50000 [9:09:10<16:48:42,  1.76s/it]
Outer training loop:  31%|###1      | 15678/50000 [9:09:11<16:57:11,  1.78s/it]
Outer training loop:  31%|###1      | 15679/50000 [9:09:13<16:49:12,  1.76s/it]
Outer training loop:  31%|###1      | 15680/50000 [9:09:15<16:54:23,  1.77s/it]
Outer training loop:  31%|###1      | 15681/50000 [9:09:17<16:50:55,  1.77s/it]
Outer training loop:  31%|###1      | 15682/50000 [9:09:18<16:58:38,  1.78s/it]
Outer training loop:  31%|###1      | 15683/50000 [9:09:20<17:00:34,  1.78s/it]
Outer training loop:  31%|###1      | 15684/50000 [9:09:22<17:28:04,  1.83s/it]
Outer training loop:  31%|###1      | 15685/50000 [9:09:24<17:51:46,  1.87s/it]
Outer training loop:  31%|###1      | 15686/50000 [9:09:26<17:25:36,  1.83s/it]
Outer training loop:  31%|###1      | 15687/50000 [9:09:28<17:02:02,  1.79s/it]
Outer training loop:  31%|###1      | 15688/50000 [9:09:29<17:28:41,  1.83s/it]
Outer training loop:  31%|###1      | 15689/50000 [9:09:31<17:23:53,  1.83s/it]
Outer training loop:  31%|###1      | 15690/50000 [9:09:33<17:03:42,  1.79s/it]
Outer training loop:  31%|###1      | 15691/50000 [9:09:35<17:23:09,  1.82s/it]
Outer training loop:  31%|###1      | 15692/50000 [9:09:37<17:02:54,  1.79s/it]
Outer training loop:  31%|###1      | 15693/50000 [9:09:38<17:07:32,  1.80s/it]
Outer training loop:  31%|###1      | 15694/50000 [9:09:40<17:27:03,  1.83s/it]
Outer training loop:  31%|###1      | 15695/50000 [9:09:42<17:12:21,  1.81s/it]
Outer training loop:  31%|###1      | 15696/50000 [9:09:44<16:41:29,  1.75s/it]
Outer training loop:  31%|###1      | 15697/50000 [9:09:45<16:18:14,  1.71s/it]
Outer training loop:  31%|###1      | 15698/50000 [9:09:47<16:55:22,  1.78s/it]
Outer training loop:  31%|###1      | 15699/50000 [9:09:49<16:44:40,  1.76s/it]
Outer training loop:  31%|###1      | 15700/50000 [9:09:51<16:30:11,  1.73s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.70s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.67s/it]

Outer training loop:  31%|###1      | 15701/50000 [9:09:58<31:14:49,  3.28s/it]
Outer training loop:  31%|###1      | 15702/50000 [9:09:59<27:10:39,  2.85s/it]
Outer training loop:  31%|###1      | 15703/50000 [9:10:01<23:51:18,  2.50s/it]
Outer training loop:  31%|###1      | 15704/50000 [9:10:03<21:30:04,  2.26s/it]
Outer training loop:  31%|###1      | 15705/50000 [9:10:05<20:12:27,  2.12s/it]
Outer training loop:  31%|###1      | 15706/50000 [9:10:06<19:00:35,  2.00s/it]
Outer training loop:  31%|###1      | 15707/50000 [9:10:08<18:15:14,  1.92s/it]
Outer training loop:  31%|###1      | 15708/50000 [9:10:10<17:42:47,  1.86s/it]
Outer training loop:  31%|###1      | 15709/50000 [9:10:12<17:43:26,  1.86s/it]
Outer training loop:  31%|###1      | 15710/50000 [9:10:13<17:12:43,  1.81s/it]
Outer training loop:  31%|###1      | 15711/50000 [9:10:15<16:59:05,  1.78s/it]
Outer training loop:  31%|###1      | 15712/50000 [9:10:17<17:26:27,  1.83s/it]
Outer training loop:  31%|###1      | 15713/50000 [9:10:19<17:09:26,  1.80s/it]
Outer training loop:  31%|###1      | 15714/50000 [9:10:20<16:49:41,  1.77s/it]
Outer training loop:  31%|###1      | 15715/50000 [9:10:22<17:21:24,  1.82s/it]
Outer training loop:  31%|###1      | 15716/50000 [9:10:24<16:53:13,  1.77s/it]
Outer training loop:  31%|###1      | 15717/50000 [9:10:26<17:04:20,  1.79s/it]
Outer training loop:  31%|###1      | 15718/50000 [9:10:28<17:16:41,  1.81s/it]
Outer training loop:  31%|###1      | 15719/50000 [9:10:29<16:56:34,  1.78s/it]
Outer training loop:  31%|###1      | 15720/50000 [9:10:31<16:44:35,  1.76s/it]
Outer training loop:  31%|###1      | 15721/50000 [9:10:33<16:33:06,  1.74s/it]
Outer training loop:  31%|###1      | 15722/50000 [9:10:35<16:56:02,  1.78s/it]
Outer training loop:  31%|###1      | 15723/50000 [9:10:36<16:51:19,  1.77s/it]
Outer training loop:  31%|###1      | 15724/50000 [9:10:38<16:42:21,  1.75s/it]
Outer training loop:  31%|###1      | 15725/50000 [9:10:40<17:10:36,  1.80s/it]
Outer training loop:  31%|###1      | 15726/50000 [9:10:42<16:50:03,  1.77s/it]
Outer training loop:  31%|###1      | 15727/50000 [9:10:43<16:45:30,  1.76s/it]
Outer training loop:  31%|###1      | 15728/50000 [9:10:45<16:46:08,  1.76s/it]
Outer training loop:  31%|###1      | 15729/50000 [9:10:47<16:30:53,  1.73s/it]
Outer training loop:  31%|###1      | 15730/50000 [9:10:49<16:26:14,  1.73s/it]
Outer training loop:  31%|###1      | 15731/50000 [9:10:50<16:38:56,  1.75s/it]
Outer training loop:  31%|###1      | 15732/50000 [9:10:52<16:55:33,  1.78s/it]
Outer training loop:  31%|###1      | 15733/50000 [9:10:54<16:57:33,  1.78s/it]
Outer training loop:  31%|###1      | 15734/50000 [9:10:56<16:40:21,  1.75s/it]
Outer training loop:  31%|###1      | 15735/50000 [9:10:58<17:11:22,  1.81s/it]
Outer training loop:  31%|###1      | 15736/50000 [9:10:59<16:55:16,  1.78s/it]
Outer training loop:  31%|###1      | 15737/50000 [9:11:01<16:55:29,  1.78s/it]
Outer training loop:  31%|###1      | 15738/50000 [9:11:03<17:15:21,  1.81s/it]
Outer training loop:  31%|###1      | 15739/50000 [9:11:05<17:05:51,  1.80s/it]
Outer training loop:  31%|###1      | 15740/50000 [9:11:07<17:10:29,  1.80s/it]
Outer training loop:  31%|###1      | 15741/50000 [9:11:08<17:17:33,  1.82s/it]
Outer training loop:  31%|###1      | 15742/50000 [9:11:10<17:22:02,  1.83s/it]
Outer training loop:  31%|###1      | 15743/50000 [9:11:12<17:11:55,  1.81s/it]
Outer training loop:  31%|###1      | 15744/50000 [9:11:14<16:39:30,  1.75s/it]
Outer training loop:  31%|###1      | 15745/50000 [9:11:16<16:59:38,  1.79s/it]
Outer training loop:  31%|###1      | 15746/50000 [9:11:17<16:42:19,  1.76s/it]
Outer training loop:  31%|###1      | 15747/50000 [9:11:19<16:34:24,  1.74s/it]
Outer training loop:  31%|###1      | 15748/50000 [9:11:21<17:02:02,  1.79s/it]
Outer training loop:  31%|###1      | 15749/50000 [9:11:23<16:42:04,  1.76s/it]
Outer training loop:  32%|###1      | 15750/50000 [9:11:24<16:55:20,  1.78s/it]
Outer training loop:  32%|###1      | 15751/50000 [9:11:26<16:43:31,  1.76s/it]
Outer training loop:  32%|###1      | 15752/50000 [9:11:28<17:15:24,  1.81s/it]
Outer training loop:  32%|###1      | 15753/50000 [9:11:30<17:11:25,  1.81s/it]
Outer training loop:  32%|###1      | 15754/50000 [9:11:31<16:43:58,  1.76s/it]
Outer training loop:  32%|###1      | 15755/50000 [9:11:33<17:09:00,  1.80s/it]
Outer training loop:  32%|###1      | 15756/50000 [9:11:35<17:00:01,  1.79s/it]
Outer training loop:  32%|###1      | 15757/50000 [9:11:37<16:40:38,  1.75s/it]
Outer training loop:  32%|###1      | 15758/50000 [9:11:39<17:05:28,  1.80s/it]
Outer training loop:  32%|###1      | 15759/50000 [9:11:41<17:12:19,  1.81s/it]
Outer training loop:  32%|###1      | 15760/50000 [9:11:42<17:15:31,  1.81s/it]
Outer training loop:  32%|###1      | 15761/50000 [9:11:44<16:59:16,  1.79s/it]
Outer training loop:  32%|###1      | 15762/50000 [9:11:46<17:36:21,  1.85s/it]
Outer training loop:  32%|###1      | 15763/50000 [9:11:48<17:07:46,  1.80s/it]
Outer training loop:  32%|###1      | 15764/50000 [9:11:49<16:54:19,  1.78s/it]
Outer training loop:  32%|###1      | 15765/50000 [9:11:51<17:36:04,  1.85s/it]
Outer training loop:  32%|###1      | 15766/50000 [9:11:53<17:07:21,  1.80s/it]
Outer training loop:  32%|###1      | 15767/50000 [9:11:55<16:53:52,  1.78s/it]
Outer training loop:  32%|###1      | 15768/50000 [9:11:57<17:05:09,  1.80s/it]
Outer training loop:  32%|###1      | 15769/50000 [9:11:58<16:40:03,  1.75s/it]
Outer training loop:  32%|###1      | 15770/50000 [9:12:00<16:11:40,  1.70s/it]
Outer training loop:  32%|###1      | 15771/50000 [9:12:02<16:24:12,  1.73s/it]
Outer training loop:  32%|###1      | 15772/50000 [9:12:03<16:19:17,  1.72s/it]
Outer training loop:  32%|###1      | 15773/50000 [9:12:05<16:26:40,  1.73s/it]
Outer training loop:  32%|###1      | 15774/50000 [9:12:07<16:42:20,  1.76s/it]
Outer training loop:  32%|###1      | 15775/50000 [9:12:09<17:00:41,  1.79s/it]
Outer training loop:  32%|###1      | 15776/50000 [9:12:11<16:36:11,  1.75s/it]
Outer training loop:  32%|###1      | 15777/50000 [9:12:12<16:16:13,  1.71s/it]
Outer training loop:  32%|###1      | 15778/50000 [9:12:14<16:11:44,  1.70s/it]
Outer training loop:  32%|###1      | 15779/50000 [9:12:16<16:39:34,  1.75s/it]
Outer training loop:  32%|###1      | 15780/50000 [9:12:17<16:30:12,  1.74s/it]
Outer training loop:  32%|###1      | 15781/50000 [9:12:19<17:01:42,  1.79s/it]
Outer training loop:  32%|###1      | 15782/50000 [9:12:21<17:09:46,  1.81s/it]
Outer training loop:  32%|###1      | 15783/50000 [9:12:23<16:58:40,  1.79s/it]
Outer training loop:  32%|###1      | 15784/50000 [9:12:25<16:42:28,  1.76s/it]
Outer training loop:  32%|###1      | 15785/50000 [9:12:26<16:33:03,  1.74s/it]
Outer training loop:  32%|###1      | 15786/50000 [9:12:28<17:04:20,  1.80s/it]
Outer training loop:  32%|###1      | 15787/50000 [9:12:30<17:00:01,  1.79s/it]
Outer training loop:  32%|###1      | 15788/50000 [9:12:32<16:46:37,  1.77s/it]
Outer training loop:  32%|###1      | 15789/50000 [9:12:34<17:13:57,  1.81s/it]
Outer training loop:  32%|###1      | 15790/50000 [9:12:36<17:21:04,  1.83s/it]
Outer training loop:  32%|###1      | 15791/50000 [9:12:37<16:50:07,  1.77s/it]
Outer training loop:  32%|###1      | 15792/50000 [9:12:39<17:30:12,  1.84s/it]
Outer training loop:  32%|###1      | 15793/50000 [9:12:41<17:19:44,  1.82s/it]
Outer training loop:  32%|###1      | 15794/50000 [9:12:43<17:04:21,  1.80s/it]
Outer training loop:  32%|###1      | 15795/50000 [9:12:45<17:23:49,  1.83s/it]
Outer training loop:  32%|###1      | 15796/50000 [9:12:46<16:56:23,  1.78s/it]
Outer training loop:  32%|###1      | 15797/50000 [9:12:48<16:38:58,  1.75s/it]
Outer training loop:  32%|###1      | 15798/50000 [9:12:50<16:32:28,  1.74s/it]
Outer training loop:  32%|###1      | 15799/50000 [9:12:52<16:51:51,  1.78s/it]
Outer training loop:  32%|###1      | 15800/50000 [9:12:53<16:43:29,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.66s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]

Outer training loop:  32%|###1      | 15801/50000 [9:13:00<31:46:20,  3.34s/it]
Outer training loop:  32%|###1      | 15802/50000 [9:13:02<27:15:06,  2.87s/it]
Outer training loop:  32%|###1      | 15803/50000 [9:13:04<24:40:50,  2.60s/it]
Outer training loop:  32%|###1      | 15804/50000 [9:13:06<22:07:08,  2.33s/it]
Outer training loop:  32%|###1      | 15805/50000 [9:13:07<20:36:26,  2.17s/it]
Outer training loop:  32%|###1      | 15806/50000 [9:13:09<19:04:08,  2.01s/it]
Outer training loop:  32%|###1      | 15807/50000 [9:13:11<18:49:08,  1.98s/it]
Outer training loop:  32%|###1      | 15808/50000 [9:13:13<18:01:02,  1.90s/it]
Outer training loop:  32%|###1      | 15809/50000 [9:13:15<17:45:37,  1.87s/it]
Outer training loop:  32%|###1      | 15810/50000 [9:13:16<17:21:57,  1.83s/it]
Outer training loop:  32%|###1      | 15811/50000 [9:13:18<17:29:53,  1.84s/it]
Outer training loop:  32%|###1      | 15812/50000 [9:13:20<17:51:00,  1.88s/it]
Outer training loop:  32%|###1      | 15813/50000 [9:13:22<17:34:17,  1.85s/it]
Outer training loop:  32%|###1      | 15814/50000 [9:13:24<17:34:32,  1.85s/it]
Outer training loop:  32%|###1      | 15815/50000 [9:13:26<17:24:41,  1.83s/it]
Outer training loop:  32%|###1      | 15816/50000 [9:13:27<16:49:03,  1.77s/it]
Outer training loop:  32%|###1      | 15817/50000 [9:13:29<17:23:31,  1.83s/it]
Outer training loop:  32%|###1      | 15818/50000 [9:13:31<17:04:12,  1.80s/it]
Outer training loop:  32%|###1      | 15819/50000 [9:13:33<16:39:55,  1.76s/it]
Outer training loop:  32%|###1      | 15820/50000 [9:13:34<16:30:19,  1.74s/it]
Outer training loop:  32%|###1      | 15821/50000 [9:13:36<17:12:11,  1.81s/it]
Outer training loop:  32%|###1      | 15822/50000 [9:13:38<16:52:48,  1.78s/it]
Outer training loop:  32%|###1      | 15823/50000 [9:13:40<16:52:19,  1.78s/it]
Outer training loop:  32%|###1      | 15824/50000 [9:13:42<17:32:41,  1.85s/it]
Outer training loop:  32%|###1      | 15825/50000 [9:13:43<17:10:35,  1.81s/it]
Outer training loop:  32%|###1      | 15826/50000 [9:13:45<17:02:24,  1.80s/it]
Outer training loop:  32%|###1      | 15827/50000 [9:13:47<16:47:11,  1.77s/it]
Outer training loop:  32%|###1      | 15828/50000 [9:13:49<17:56:33,  1.89s/it]
Outer training loop:  32%|###1      | 15829/50000 [9:13:51<17:29:15,  1.84s/it]
Outer training loop:  32%|###1      | 15830/50000 [9:13:53<17:49:30,  1.88s/it]
Outer training loop:  32%|###1      | 15831/50000 [9:13:55<17:37:20,  1.86s/it]
Outer training loop:  32%|###1      | 15832/50000 [9:13:56<17:11:09,  1.81s/it]
Outer training loop:  32%|###1      | 15833/50000 [9:13:58<16:46:28,  1.77s/it]
Outer training loop:  32%|###1      | 15834/50000 [9:14:00<17:18:37,  1.82s/it]
Outer training loop:  32%|###1      | 15835/50000 [9:14:02<17:08:31,  1.81s/it]
Outer training loop:  32%|###1      | 15836/50000 [9:14:03<16:52:10,  1.78s/it]
Outer training loop:  32%|###1      | 15837/50000 [9:14:05<17:17:05,  1.82s/it]
Outer training loop:  32%|###1      | 15838/50000 [9:14:07<17:01:54,  1.79s/it]
Outer training loop:  32%|###1      | 15839/50000 [9:14:09<16:47:57,  1.77s/it]
Outer training loop:  32%|###1      | 15840/50000 [9:14:10<16:36:23,  1.75s/it]
Outer training loop:  32%|###1      | 15841/50000 [9:14:12<17:04:30,  1.80s/it]
Outer training loop:  32%|###1      | 15842/50000 [9:14:14<16:33:00,  1.74s/it]
Outer training loop:  32%|###1      | 15843/50000 [9:14:16<16:35:35,  1.75s/it]
Outer training loop:  32%|###1      | 15844/50000 [9:14:18<17:28:54,  1.84s/it]
Outer training loop:  32%|###1      | 15845/50000 [9:14:20<17:11:06,  1.81s/it]
Outer training loop:  32%|###1      | 15846/50000 [9:14:21<17:05:33,  1.80s/it]
Outer training loop:  32%|###1      | 15847/50000 [9:14:23<17:36:31,  1.86s/it]
Outer training loop:  32%|###1      | 15848/50000 [9:14:25<17:14:39,  1.82s/it]
Outer training loop:  32%|###1      | 15849/50000 [9:14:27<16:44:04,  1.76s/it]
Outer training loop:  32%|###1      | 15850/50000 [9:14:29<17:18:44,  1.83s/it]
Outer training loop:  32%|###1      | 15851/50000 [9:14:30<17:13:16,  1.82s/it]
Outer training loop:  32%|###1      | 15852/50000 [9:14:32<17:01:43,  1.80s/it]
Outer training loop:  32%|###1      | 15853/50000 [9:14:34<16:48:56,  1.77s/it]
Outer training loop:  32%|###1      | 15854/50000 [9:14:36<17:06:18,  1.80s/it]
Outer training loop:  32%|###1      | 15855/50000 [9:14:37<16:47:28,  1.77s/it]
Outer training loop:  32%|###1      | 15856/50000 [9:14:39<16:33:35,  1.75s/it]
Outer training loop:  32%|###1      | 15857/50000 [9:14:41<16:59:42,  1.79s/it]
Outer training loop:  32%|###1      | 15858/50000 [9:14:43<16:30:16,  1.74s/it]
Outer training loop:  32%|###1      | 15859/50000 [9:14:44<16:12:42,  1.71s/it]
Outer training loop:  32%|###1      | 15860/50000 [9:14:46<16:56:44,  1.79s/it]
Outer training loop:  32%|###1      | 15861/50000 [9:14:48<16:54:39,  1.78s/it]
Outer training loop:  32%|###1      | 15862/50000 [9:14:50<16:52:19,  1.78s/it]
Outer training loop:  32%|###1      | 15863/50000 [9:14:52<16:41:45,  1.76s/it]
Outer training loop:  32%|###1      | 15864/50000 [9:14:53<17:03:34,  1.80s/it]
Outer training loop:  32%|###1      | 15865/50000 [9:14:55<16:48:39,  1.77s/it]
Outer training loop:  32%|###1      | 15866/50000 [9:14:57<16:26:25,  1.73s/it]
Outer training loop:  32%|###1      | 15867/50000 [9:14:59<16:49:46,  1.78s/it]
Outer training loop:  32%|###1      | 15868/50000 [9:15:00<16:24:21,  1.73s/it]
Outer training loop:  32%|###1      | 15869/50000 [9:15:02<16:17:51,  1.72s/it]
Outer training loop:  32%|###1      | 15870/50000 [9:15:04<16:57:48,  1.79s/it]
Outer training loop:  32%|###1      | 15871/50000 [9:15:06<16:37:03,  1.75s/it]
Outer training loop:  32%|###1      | 15872/50000 [9:15:07<16:22:13,  1.73s/it]
Outer training loop:  32%|###1      | 15873/50000 [9:15:09<16:19:41,  1.72s/it]
Outer training loop:  32%|###1      | 15874/50000 [9:15:11<16:33:43,  1.75s/it]
Outer training loop:  32%|###1      | 15875/50000 [9:15:12<16:22:27,  1.73s/it]
Outer training loop:  32%|###1      | 15876/50000 [9:15:14<16:32:42,  1.75s/it]
Outer training loop:  32%|###1      | 15877/50000 [9:15:16<17:05:39,  1.80s/it]
Outer training loop:  32%|###1      | 15878/50000 [9:15:18<17:12:03,  1.81s/it]
Outer training loop:  32%|###1      | 15879/50000 [9:15:20<17:02:58,  1.80s/it]
Outer training loop:  32%|###1      | 15880/50000 [9:15:21<16:48:13,  1.77s/it]
Outer training loop:  32%|###1      | 15881/50000 [9:15:23<16:45:40,  1.77s/it]
Outer training loop:  32%|###1      | 15882/50000 [9:15:25<16:38:08,  1.76s/it]
Outer training loop:  32%|###1      | 15883/50000 [9:15:27<16:33:09,  1.75s/it]
Outer training loop:  32%|###1      | 15884/50000 [9:15:29<17:06:35,  1.81s/it]
Outer training loop:  32%|###1      | 15885/50000 [9:15:30<16:50:27,  1.78s/it]
Outer training loop:  32%|###1      | 15886/50000 [9:15:32<16:34:29,  1.75s/it]
Outer training loop:  32%|###1      | 15887/50000 [9:15:34<16:26:52,  1.74s/it]
Outer training loop:  32%|###1      | 15888/50000 [9:15:36<17:04:43,  1.80s/it]
Outer training loop:  32%|###1      | 15889/50000 [9:15:37<16:50:28,  1.78s/it]
Outer training loop:  32%|###1      | 15890/50000 [9:15:39<16:53:29,  1.78s/it]
Outer training loop:  32%|###1      | 15891/50000 [9:15:41<17:06:33,  1.81s/it]
Outer training loop:  32%|###1      | 15892/50000 [9:15:43<16:47:23,  1.77s/it]
Outer training loop:  32%|###1      | 15893/50000 [9:15:45<16:59:13,  1.79s/it]
Outer training loop:  32%|###1      | 15894/50000 [9:15:47<17:26:53,  1.84s/it]
Outer training loop:  32%|###1      | 15895/50000 [9:15:48<17:15:17,  1.82s/it]
Outer training loop:  32%|###1      | 15896/50000 [9:15:50<16:54:52,  1.79s/it]
Outer training loop:  32%|###1      | 15897/50000 [9:15:52<17:22:08,  1.83s/it]
Outer training loop:  32%|###1      | 15898/50000 [9:15:54<16:53:18,  1.78s/it]
Outer training loop:  32%|###1      | 15899/50000 [9:15:56<17:11:45,  1.82s/it]
Outer training loop:  32%|###1      | 15900/50000 [9:15:57<17:23:09,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.59s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.66s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.71s/it]

Outer training loop:  32%|###1      | 15901/50000 [9:16:04<32:10:00,  3.40s/it]
Outer training loop:  32%|###1      | 15902/50000 [9:16:06<27:13:11,  2.87s/it]
Outer training loop:  32%|###1      | 15903/50000 [9:16:08<23:55:34,  2.53s/it]
Outer training loop:  32%|###1      | 15904/50000 [9:16:10<22:00:33,  2.32s/it]
Outer training loop:  32%|###1      | 15905/50000 [9:16:11<20:22:31,  2.15s/it]
Outer training loop:  32%|###1      | 15906/50000 [9:16:13<19:34:36,  2.07s/it]
Outer training loop:  32%|###1      | 15907/50000 [9:16:15<18:45:28,  1.98s/it]
Outer training loop:  32%|###1      | 15908/50000 [9:16:17<18:24:39,  1.94s/it]
Outer training loop:  32%|###1      | 15909/50000 [9:16:19<17:22:02,  1.83s/it]
Outer training loop:  32%|###1      | 15910/50000 [9:16:20<17:09:06,  1.81s/it]
Outer training loop:  32%|###1      | 15911/50000 [9:16:22<16:57:54,  1.79s/it]
Outer training loop:  32%|###1      | 15912/50000 [9:16:24<16:47:00,  1.77s/it]
Outer training loop:  32%|###1      | 15913/50000 [9:16:25<16:32:40,  1.75s/it]
Outer training loop:  32%|###1      | 15914/50000 [9:16:27<16:51:39,  1.78s/it]
Outer training loop:  32%|###1      | 15915/50000 [9:16:29<16:31:15,  1.74s/it]
Outer training loop:  32%|###1      | 15916/50000 [9:16:31<16:18:42,  1.72s/it]
Outer training loop:  32%|###1      | 15917/50000 [9:16:32<16:29:08,  1.74s/it]
Outer training loop:  32%|###1      | 15918/50000 [9:16:34<17:09:46,  1.81s/it]
Outer training loop:  32%|###1      | 15919/50000 [9:16:36<17:02:33,  1.80s/it]
Outer training loop:  32%|###1      | 15920/50000 [9:16:38<17:04:31,  1.80s/it]
Outer training loop:  32%|###1      | 15921/50000 [9:16:40<17:11:57,  1.82s/it]
Outer training loop:  32%|###1      | 15922/50000 [9:16:42<16:58:41,  1.79s/it]
Outer training loop:  32%|###1      | 15923/50000 [9:16:43<16:31:43,  1.75s/it]
Outer training loop:  32%|###1      | 15924/50000 [9:16:45<16:44:56,  1.77s/it]
Outer training loop:  32%|###1      | 15925/50000 [9:16:47<17:03:21,  1.80s/it]
Outer training loop:  32%|###1      | 15926/50000 [9:16:49<17:02:04,  1.80s/it]
Outer training loop:  32%|###1      | 15927/50000 [9:16:51<17:26:17,  1.84s/it]
Outer training loop:  32%|###1      | 15928/50000 [9:16:52<17:02:55,  1.80s/it]
Outer training loop:  32%|###1      | 15929/50000 [9:16:54<16:43:34,  1.77s/it]
Outer training loop:  32%|###1      | 15930/50000 [9:16:56<16:21:33,  1.73s/it]
Outer training loop:  32%|###1      | 15931/50000 [9:16:58<16:50:18,  1.78s/it]
Outer training loop:  32%|###1      | 15932/50000 [9:16:59<16:16:00,  1.72s/it]
Outer training loop:  32%|###1      | 15933/50000 [9:17:01<16:26:30,  1.74s/it]
Outer training loop:  32%|###1      | 15934/50000 [9:17:03<17:02:38,  1.80s/it]
Outer training loop:  32%|###1      | 15935/50000 [9:17:05<16:44:40,  1.77s/it]
Outer training loop:  32%|###1      | 15936/50000 [9:17:06<16:56:02,  1.79s/it]
Outer training loop:  32%|###1      | 15937/50000 [9:17:08<16:59:54,  1.80s/it]
Outer training loop:  32%|###1      | 15938/50000 [9:17:10<16:57:53,  1.79s/it]
Outer training loop:  32%|###1      | 15939/50000 [9:17:12<17:01:05,  1.80s/it]
Outer training loop:  32%|###1      | 15940/50000 [9:17:14<17:13:06,  1.82s/it]
Outer training loop:  32%|###1      | 15941/50000 [9:17:15<16:47:26,  1.77s/it]
Outer training loop:  32%|###1      | 15942/50000 [9:17:17<16:45:55,  1.77s/it]
Outer training loop:  32%|###1      | 15943/50000 [9:17:19<16:58:38,  1.79s/it]
Outer training loop:  32%|###1      | 15944/50000 [9:17:21<17:11:22,  1.82s/it]
Outer training loop:  32%|###1      | 15945/50000 [9:17:23<16:51:41,  1.78s/it]
Outer training loop:  32%|###1      | 15946/50000 [9:17:24<16:40:05,  1.76s/it]
Outer training loop:  32%|###1      | 15947/50000 [9:17:26<17:17:19,  1.83s/it]
Outer training loop:  32%|###1      | 15948/50000 [9:17:28<16:50:59,  1.78s/it]
Outer training loop:  32%|###1      | 15949/50000 [9:17:30<16:31:46,  1.75s/it]
Outer training loop:  32%|###1      | 15950/50000 [9:17:31<16:19:05,  1.73s/it]
Outer training loop:  32%|###1      | 15951/50000 [9:17:33<16:52:03,  1.78s/it]
Outer training loop:  32%|###1      | 15952/50000 [9:17:35<16:50:40,  1.78s/it]
Outer training loop:  32%|###1      | 15953/50000 [9:17:37<17:00:33,  1.80s/it]
Outer training loop:  32%|###1      | 15954/50000 [9:17:39<17:20:22,  1.83s/it]
Outer training loop:  32%|###1      | 15955/50000 [9:17:40<17:00:30,  1.80s/it]
Outer training loop:  32%|###1      | 15956/50000 [9:17:42<16:50:12,  1.78s/it]
Outer training loop:  32%|###1      | 15957/50000 [9:17:44<17:17:14,  1.83s/it]
Outer training loop:  32%|###1      | 15958/50000 [9:17:46<17:22:56,  1.84s/it]
Outer training loop:  32%|###1      | 15959/50000 [9:17:48<16:52:21,  1.78s/it]
Outer training loop:  32%|###1      | 15960/50000 [9:17:49<17:03:06,  1.80s/it]
Outer training loop:  32%|###1      | 15961/50000 [9:17:51<16:55:20,  1.79s/it]
Outer training loop:  32%|###1      | 15962/50000 [9:17:53<16:35:55,  1.76s/it]
Outer training loop:  32%|###1      | 15963/50000 [9:17:55<16:33:44,  1.75s/it]
Outer training loop:  32%|###1      | 15964/50000 [9:17:56<16:50:11,  1.78s/it]
Outer training loop:  32%|###1      | 15965/50000 [9:17:58<16:36:38,  1.76s/it]
Outer training loop:  32%|###1      | 15966/50000 [9:18:00<16:35:14,  1.75s/it]
Outer training loop:  32%|###1      | 15967/50000 [9:18:02<16:48:15,  1.78s/it]
Outer training loop:  32%|###1      | 15968/50000 [9:18:03<16:28:29,  1.74s/it]
Outer training loop:  32%|###1      | 15969/50000 [9:18:05<16:24:52,  1.74s/it]
Outer training loop:  32%|###1      | 15970/50000 [9:18:07<17:04:08,  1.81s/it]
Outer training loop:  32%|###1      | 15971/50000 [9:18:09<16:44:36,  1.77s/it]
Outer training loop:  32%|###1      | 15972/50000 [9:18:10<16:31:20,  1.75s/it]
Outer training loop:  32%|###1      | 15973/50000 [9:18:12<16:19:40,  1.73s/it]
Outer training loop:  32%|###1      | 15974/50000 [9:18:14<17:15:25,  1.83s/it]
Outer training loop:  32%|###1      | 15975/50000 [9:18:16<17:29:25,  1.85s/it]
Outer training loop:  32%|###1      | 15976/50000 [9:18:18<17:26:46,  1.85s/it]
Outer training loop:  32%|###1      | 15977/50000 [9:18:20<17:40:56,  1.87s/it]
Outer training loop:  32%|###1      | 15978/50000 [9:18:22<17:01:36,  1.80s/it]
Outer training loop:  32%|###1      | 15979/50000 [9:18:23<16:48:05,  1.78s/it]
Outer training loop:  32%|###1      | 15980/50000 [9:18:25<17:17:25,  1.83s/it]
Outer training loop:  32%|###1      | 15981/50000 [9:18:27<17:04:24,  1.81s/it]
Outer training loop:  32%|###1      | 15982/50000 [9:18:29<16:51:41,  1.78s/it]
Outer training loop:  32%|###1      | 15983/50000 [9:18:30<16:30:12,  1.75s/it]
Outer training loop:  32%|###1      | 15984/50000 [9:18:32<16:48:31,  1.78s/it]
Outer training loop:  32%|###1      | 15985/50000 [9:18:34<16:36:31,  1.76s/it]
Outer training loop:  32%|###1      | 15986/50000 [9:18:36<16:42:47,  1.77s/it]
Outer training loop:  32%|###1      | 15987/50000 [9:18:38<17:03:40,  1.81s/it]
Outer training loop:  32%|###1      | 15988/50000 [9:18:39<16:47:01,  1.78s/it]
Outer training loop:  32%|###1      | 15989/50000 [9:18:41<16:56:03,  1.79s/it]
Outer training loop:  32%|###1      | 15990/50000 [9:18:43<17:24:55,  1.84s/it]
Outer training loop:  32%|###1      | 15991/50000 [9:18:45<17:01:08,  1.80s/it]
Outer training loop:  32%|###1      | 15992/50000 [9:18:47<16:45:18,  1.77s/it]
Outer training loop:  32%|###1      | 15993/50000 [9:18:48<17:18:34,  1.83s/it]
Outer training loop:  32%|###1      | 15994/50000 [9:18:50<17:01:53,  1.80s/it]
Outer training loop:  32%|###1      | 15995/50000 [9:18:52<16:46:36,  1.78s/it]
Outer training loop:  32%|###1      | 15996/50000 [9:18:54<16:58:20,  1.80s/it]
Outer training loop:  32%|###1      | 15997/50000 [9:18:56<17:02:15,  1.80s/it]
Outer training loop:  32%|###1      | 15998/50000 [9:18:57<16:49:51,  1.78s/it]
Outer training loop:  32%|###1      | 15999/50000 [9:18:59<16:28:36,  1.74s/it]
Outer training loop:  32%|###2      | 16000/50000 [9:19:01<16:57:02,  1.79s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.75s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.74s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:24,  1.81s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:19,  1.77s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:16,  1.75s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:18,  1.80s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:15,  1.79s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:13,  1.78s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:12,  1.79s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:13,  1.83s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:09,  1.79s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:05,  1.77s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:01,  1.74s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:04,  1.80s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:02,  1.79s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:56,  1.74s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:54,  1.73s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:52,  1.73s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:54,  1.80s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:53,  1.80s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:52,  1.82s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:47,  1.77s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:51,  1.86s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:46,  1.81s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:44,  1.80s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.79s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:40,  1.79s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:43,  1.88s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:37,  1.80s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:34,  1.79s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:33,  1.79s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:31,  1.79s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:29,  1.79s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:26,  1.80s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:24,  1.79s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:20,  1.74s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:17,  1.73s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.75s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:16,  1.81s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:14,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:12,  1.82s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.79s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:06,  1.76s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:03,  1.72s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:05,  1.83s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:02,  1.78s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<00:59,  1.75s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:56,  1.73s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:55,  1.73s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:53,  1.74s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:52,  1.74s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:50,  1.80s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:47,  1.77s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:46,  1.79s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:44,  1.78s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:41,  1.74s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.74s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:38,  1.74s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.78s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:32,  1.74s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:30,  1.70s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:28,  1.70s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.70s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:25,  1.73s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:23,  1.69s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:19,  1.80s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.77s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:15,  1.76s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.76s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.76s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.77s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.77s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.80s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.94s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.85s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.78s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:20,  1.71s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:17,  1.70s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:13,  1.67s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:14,  1.70s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:10,  1.67s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:09,  1.68s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:21,  1.86s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:11,  1.77s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:06,  1.74s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:12,  1.83s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.80s/it]

Evaluating policy:  16%|#5        | 13/83 [00:22<02:02,  1.76s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:02,  1.77s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:04,  1.83s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:00,  1.80s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.79s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:55,  1.78s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:56,  1.82s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:52,  1.79s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:49,  1.76s/it]

Evaluating policy:  27%|##6       | 22/83 [00:38<01:46,  1.74s/it]

Evaluating policy:  28%|##7       | 23/83 [00:40<01:44,  1.74s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:47,  1.81s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.79s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:40,  1.77s/it]

Evaluating policy:  33%|###2      | 27/83 [00:47<01:36,  1.73s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:35,  1.74s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:37,  1.81s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:34,  1.78s/it]

Evaluating policy:  37%|###7      | 31/83 [00:54<01:30,  1.74s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:27,  1.71s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:26,  1.72s/it]

Evaluating policy:  41%|####      | 34/83 [00:59<01:23,  1.71s/it]

Evaluating policy:  42%|####2     | 35/83 [01:01<01:27,  1.81s/it]

Evaluating policy:  43%|####3     | 36/83 [01:03<01:25,  1.82s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:22,  1.79s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:20,  1.79s/it]

Evaluating policy:  47%|####6     | 39/83 [01:08<01:16,  1.73s/it]

Evaluating policy:  48%|####8     | 40/83 [01:10<01:15,  1.75s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:17,  1.85s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:12,  1.82s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.79s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:08,  1.80s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:05,  1.76s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:03,  1.75s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:59,  1.79s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:57,  1.79s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:54,  1.77s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:52,  1.76s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:49,  1.72s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:51,  1.85s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:46,  1.78s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:44,  1.78s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.76s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.75s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:37,  1.73s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:36,  1.72s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:37,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:31,  1.76s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.75s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:28,  1.77s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:26,  1.75s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.78s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.76s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:22,  1.89s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.77s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:15,  1.74s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.75s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.75s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.73s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.70s/it]

Evaluating policy:  95%|#########5| 79/83 [02:19<00:06,  1.72s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.73s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.80s/it]

Outer training loop:  32%|###2      | 16001/50000 [9:24:03<869:02:50, 92.02s/it]
Outer training loop:  32%|###2      | 16002/50000 [9:24:05<613:14:30, 64.94s/it]
Outer training loop:  32%|###2      | 16003/50000 [9:24:07<433:48:27, 45.94s/it]
Outer training loop:  32%|###2      | 16004/50000 [9:24:09<308:31:30, 32.67s/it]
Outer training loop:  32%|###2      | 16005/50000 [9:24:10<221:07:17, 23.42s/it]
Outer training loop:  32%|###2      | 16006/50000 [9:24:12<160:04:38, 16.95s/it]
Outer training loop:  32%|###2      | 16007/50000 [9:24:14<116:55:07, 12.38s/it]
Outer training loop:  32%|###2      | 16008/50000 [9:24:16<86:49:19,  9.20s/it] 
Outer training loop:  32%|###2      | 16009/50000 [9:24:18<67:04:57,  7.10s/it]
Outer training loop:  32%|###2      | 16010/50000 [9:24:20<51:52:15,  5.49s/it]
Outer training loop:  32%|###2      | 16011/50000 [9:24:21<41:10:24,  4.36s/it]
Outer training loop:  32%|###2      | 16012/50000 [9:24:23<34:12:02,  3.62s/it]
Outer training loop:  32%|###2      | 16013/50000 [9:24:25<28:57:58,  3.07s/it]
Outer training loop:  32%|###2      | 16014/50000 [9:24:27<25:13:33,  2.67s/it]
Outer training loop:  32%|###2      | 16015/50000 [9:24:29<23:09:33,  2.45s/it]
Outer training loop:  32%|###2      | 16016/50000 [9:24:30<21:03:43,  2.23s/it]
Outer training loop:  32%|###2      | 16017/50000 [9:24:32<19:36:56,  2.08s/it]
Outer training loop:  32%|###2      | 16018/50000 [9:24:34<19:02:31,  2.02s/it]
Outer training loop:  32%|###2      | 16019/50000 [9:24:36<18:33:55,  1.97s/it]
Outer training loop:  32%|###2      | 16020/50000 [9:24:38<18:04:23,  1.91s/it]
Outer training loop:  32%|###2      | 16021/50000 [9:24:39<17:22:19,  1.84s/it]
Outer training loop:  32%|###2      | 16022/50000 [9:24:41<17:23:30,  1.84s/it]
Outer training loop:  32%|###2      | 16023/50000 [9:24:43<16:50:41,  1.78s/it]
Outer training loop:  32%|###2      | 16024/50000 [9:24:45<16:30:17,  1.75s/it]
Outer training loop:  32%|###2      | 16025/50000 [9:24:46<17:07:13,  1.81s/it]
Outer training loop:  32%|###2      | 16026/50000 [9:24:48<16:41:41,  1.77s/it]
Outer training loop:  32%|###2      | 16027/50000 [9:24:50<16:43:05,  1.77s/it]
Outer training loop:  32%|###2      | 16028/50000 [9:24:52<17:06:00,  1.81s/it]
Outer training loop:  32%|###2      | 16029/50000 [9:24:54<16:50:59,  1.79s/it]
Outer training loop:  32%|###2      | 16030/50000 [9:24:55<16:51:32,  1.79s/it]
Outer training loop:  32%|###2      | 16031/50000 [9:24:57<16:30:37,  1.75s/it]
Outer training loop:  32%|###2      | 16032/50000 [9:24:59<16:32:58,  1.75s/it]
Outer training loop:  32%|###2      | 16033/50000 [9:25:01<16:33:19,  1.75s/it]
Outer training loop:  32%|###2      | 16034/50000 [9:25:02<16:14:11,  1.72s/it]
Outer training loop:  32%|###2      | 16035/50000 [9:25:04<16:50:22,  1.78s/it]
Outer training loop:  32%|###2      | 16036/50000 [9:25:06<16:27:24,  1.74s/it]
Outer training loop:  32%|###2      | 16037/50000 [9:25:07<16:13:57,  1.72s/it]
Outer training loop:  32%|###2      | 16038/50000 [9:25:09<16:41:51,  1.77s/it]
Outer training loop:  32%|###2      | 16039/50000 [9:25:11<16:42:22,  1.77s/it]
Outer training loop:  32%|###2      | 16040/50000 [9:25:13<16:25:14,  1.74s/it]
Outer training loop:  32%|###2      | 16041/50000 [9:25:14<16:20:07,  1.73s/it]
Outer training loop:  32%|###2      | 16042/50000 [9:25:16<16:56:53,  1.80s/it]
Outer training loop:  32%|###2      | 16043/50000 [9:25:18<16:55:04,  1.79s/it]
Outer training loop:  32%|###2      | 16044/50000 [9:25:20<16:33:27,  1.76s/it]
Outer training loop:  32%|###2      | 16045/50000 [9:25:22<16:47:55,  1.78s/it]
Outer training loop:  32%|###2      | 16046/50000 [9:25:23<16:26:52,  1.74s/it]
Outer training loop:  32%|###2      | 16047/50000 [9:25:25<16:14:04,  1.72s/it]
Outer training loop:  32%|###2      | 16048/50000 [9:25:27<16:53:47,  1.79s/it]
Outer training loop:  32%|###2      | 16049/50000 [9:25:29<16:41:15,  1.77s/it]
Outer training loop:  32%|###2      | 16050/50000 [9:25:30<16:44:22,  1.78s/it]
Outer training loop:  32%|###2      | 16051/50000 [9:25:32<16:48:41,  1.78s/it]
Outer training loop:  32%|###2      | 16052/50000 [9:25:34<16:44:49,  1.78s/it]
Outer training loop:  32%|###2      | 16053/50000 [9:25:36<16:49:44,  1.78s/it]
Outer training loop:  32%|###2      | 16054/50000 [9:25:38<16:29:38,  1.75s/it]
Outer training loop:  32%|###2      | 16055/50000 [9:25:39<16:56:43,  1.80s/it]
Outer training loop:  32%|###2      | 16056/50000 [9:25:41<16:57:11,  1.80s/it]
Outer training loop:  32%|###2      | 16057/50000 [9:25:43<16:30:51,  1.75s/it]
Outer training loop:  32%|###2      | 16058/50000 [9:25:44<16:09:50,  1.71s/it]
Outer training loop:  32%|###2      | 16059/50000 [9:25:46<16:57:55,  1.80s/it]
Outer training loop:  32%|###2      | 16060/50000 [9:25:48<16:47:30,  1.78s/it]
Outer training loop:  32%|###2      | 16061/50000 [9:25:50<16:36:15,  1.76s/it]
Outer training loop:  32%|###2      | 16062/50000 [9:25:52<16:55:03,  1.79s/it]
Outer training loop:  32%|###2      | 16063/50000 [9:25:53<16:33:16,  1.76s/it]
Outer training loop:  32%|###2      | 16064/50000 [9:25:55<16:27:17,  1.75s/it]
Outer training loop:  32%|###2      | 16065/50000 [9:25:57<16:52:55,  1.79s/it]
Outer training loop:  32%|###2      | 16066/50000 [9:25:59<16:47:31,  1.78s/it]
Outer training loop:  32%|###2      | 16067/50000 [9:26:01<16:45:39,  1.78s/it]
Outer training loop:  32%|###2      | 16068/50000 [9:26:02<16:18:24,  1.73s/it]
Outer training loop:  32%|###2      | 16069/50000 [9:26:04<17:11:58,  1.82s/it]
Outer training loop:  32%|###2      | 16070/50000 [9:26:06<16:43:20,  1.77s/it]
Outer training loop:  32%|###2      | 16071/50000 [9:26:08<16:33:54,  1.76s/it]
Outer training loop:  32%|###2      | 16072/50000 [9:26:10<17:03:30,  1.81s/it]
Outer training loop:  32%|###2      | 16073/50000 [9:26:11<16:40:54,  1.77s/it]
Outer training loop:  32%|###2      | 16074/50000 [9:26:13<16:22:25,  1.74s/it]
Outer training loop:  32%|###2      | 16075/50000 [9:26:15<16:21:11,  1.74s/it]
Outer training loop:  32%|###2      | 16076/50000 [9:26:16<16:37:08,  1.76s/it]
Outer training loop:  32%|###2      | 16077/50000 [9:26:18<16:27:00,  1.75s/it]
Outer training loop:  32%|###2      | 16078/50000 [9:26:20<16:19:42,  1.73s/it]
Outer training loop:  32%|###2      | 16079/50000 [9:26:22<16:42:37,  1.77s/it]
Outer training loop:  32%|###2      | 16080/50000 [9:26:23<16:28:11,  1.75s/it]
Outer training loop:  32%|###2      | 16081/50000 [9:26:25<16:29:24,  1.75s/it]
Outer training loop:  32%|###2      | 16082/50000 [9:26:27<16:50:48,  1.79s/it]
Outer training loop:  32%|###2      | 16083/50000 [9:26:29<16:37:58,  1.77s/it]
Outer training loop:  32%|###2      | 16084/50000 [9:26:31<16:33:04,  1.76s/it]
Outer training loop:  32%|###2      | 16085/50000 [9:26:32<16:12:09,  1.72s/it]
Outer training loop:  32%|###2      | 16086/50000 [9:26:34<17:00:48,  1.81s/it]
Outer training loop:  32%|###2      | 16087/50000 [9:26:36<16:50:52,  1.79s/it]
Outer training loop:  32%|###2      | 16088/50000 [9:26:38<16:47:04,  1.78s/it]
Outer training loop:  32%|###2      | 16089/50000 [9:26:40<17:11:09,  1.82s/it]
Outer training loop:  32%|###2      | 16090/50000 [9:26:41<16:38:18,  1.77s/it]
Outer training loop:  32%|###2      | 16091/50000 [9:26:43<16:22:47,  1.74s/it]
Outer training loop:  32%|###2      | 16092/50000 [9:26:45<16:46:31,  1.78s/it]
Outer training loop:  32%|###2      | 16093/50000 [9:26:47<16:37:30,  1.77s/it]
Outer training loop:  32%|###2      | 16094/50000 [9:26:48<16:29:04,  1.75s/it]
Outer training loop:  32%|###2      | 16095/50000 [9:26:50<16:28:00,  1.75s/it]
Outer training loop:  32%|###2      | 16096/50000 [9:26:52<16:43:37,  1.78s/it]
Outer training loop:  32%|###2      | 16097/50000 [9:26:54<16:31:23,  1.75s/it]
Outer training loop:  32%|###2      | 16098/50000 [9:26:55<16:30:35,  1.75s/it]
Outer training loop:  32%|###2      | 16099/50000 [9:26:57<16:14:12,  1.72s/it]
Outer training loop:  32%|###2      | 16100/50000 [9:26:59<16:42:42,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.60s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.69s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.66s/it]

Outer training loop:  32%|###2      | 16101/50000 [9:27:06<31:06:46,  3.30s/it]
Outer training loop:  32%|###2      | 16102/50000 [9:27:07<26:29:26,  2.81s/it]
Outer training loop:  32%|###2      | 16103/50000 [9:27:09<24:06:15,  2.56s/it]
Outer training loop:  32%|###2      | 16104/50000 [9:27:11<22:03:08,  2.34s/it]
Outer training loop:  32%|###2      | 16105/50000 [9:27:13<20:02:44,  2.13s/it]
Outer training loop:  32%|###2      | 16106/50000 [9:27:15<18:54:01,  2.01s/it]
Outer training loop:  32%|###2      | 16107/50000 [9:27:17<18:47:51,  2.00s/it]
Outer training loop:  32%|###2      | 16108/50000 [9:27:18<17:58:55,  1.91s/it]
Outer training loop:  32%|###2      | 16109/50000 [9:27:20<17:36:55,  1.87s/it]
Outer training loop:  32%|###2      | 16110/50000 [9:27:22<17:07:50,  1.82s/it]
Outer training loop:  32%|###2      | 16111/50000 [9:27:24<17:25:39,  1.85s/it]
Outer training loop:  32%|###2      | 16112/50000 [9:27:25<17:15:48,  1.83s/it]
Outer training loop:  32%|###2      | 16113/50000 [9:27:27<17:19:05,  1.84s/it]
Outer training loop:  32%|###2      | 16114/50000 [9:27:29<17:42:31,  1.88s/it]
Outer training loop:  32%|###2      | 16115/50000 [9:27:31<17:21:55,  1.84s/it]
Outer training loop:  32%|###2      | 16116/50000 [9:27:33<17:09:53,  1.82s/it]
Outer training loop:  32%|###2      | 16117/50000 [9:27:35<17:28:21,  1.86s/it]
Outer training loop:  32%|###2      | 16118/50000 [9:27:36<17:04:47,  1.81s/it]
Outer training loop:  32%|###2      | 16119/50000 [9:27:38<16:45:42,  1.78s/it]
Outer training loop:  32%|###2      | 16120/50000 [9:27:40<16:43:17,  1.78s/it]
Outer training loop:  32%|###2      | 16121/50000 [9:27:42<16:51:49,  1.79s/it]
Outer training loop:  32%|###2      | 16122/50000 [9:27:43<16:29:27,  1.75s/it]
Outer training loop:  32%|###2      | 16123/50000 [9:27:45<16:23:36,  1.74s/it]
Outer training loop:  32%|###2      | 16124/50000 [9:27:47<16:42:38,  1.78s/it]
Outer training loop:  32%|###2      | 16125/50000 [9:27:49<16:27:53,  1.75s/it]
Outer training loop:  32%|###2      | 16126/50000 [9:27:50<16:27:14,  1.75s/it]
Outer training loop:  32%|###2      | 16127/50000 [9:27:52<16:25:59,  1.75s/it]
Outer training loop:  32%|###2      | 16128/50000 [9:27:54<16:59:14,  1.81s/it]
Outer training loop:  32%|###2      | 16129/50000 [9:27:56<16:56:04,  1.80s/it]
Outer training loop:  32%|###2      | 16130/50000 [9:27:58<16:39:49,  1.77s/it]
Outer training loop:  32%|###2      | 16131/50000 [9:28:00<17:17:04,  1.84s/it]
Outer training loop:  32%|###2      | 16132/50000 [9:28:01<16:49:28,  1.79s/it]
Outer training loop:  32%|###2      | 16133/50000 [9:28:03<16:24:24,  1.74s/it]
Outer training loop:  32%|###2      | 16134/50000 [9:28:05<16:14:04,  1.73s/it]
Outer training loop:  32%|###2      | 16135/50000 [9:28:06<16:45:24,  1.78s/it]
Outer training loop:  32%|###2      | 16136/50000 [9:28:08<16:29:37,  1.75s/it]
Outer training loop:  32%|###2      | 16137/50000 [9:28:10<16:26:04,  1.75s/it]
Outer training loop:  32%|###2      | 16138/50000 [9:28:12<16:45:26,  1.78s/it]
Outer training loop:  32%|###2      | 16139/50000 [9:28:13<16:37:33,  1.77s/it]
Outer training loop:  32%|###2      | 16140/50000 [9:28:15<16:29:42,  1.75s/it]
Outer training loop:  32%|###2      | 16141/50000 [9:28:17<16:41:46,  1.78s/it]
Outer training loop:  32%|###2      | 16142/50000 [9:28:19<16:23:44,  1.74s/it]
Outer training loop:  32%|###2      | 16143/50000 [9:28:20<16:24:38,  1.74s/it]
Outer training loop:  32%|###2      | 16144/50000 [9:28:22<16:34:07,  1.76s/it]
Outer training loop:  32%|###2      | 16145/50000 [9:28:24<16:52:57,  1.80s/it]
Outer training loop:  32%|###2      | 16146/50000 [9:28:26<16:54:54,  1.80s/it]
Outer training loop:  32%|###2      | 16147/50000 [9:28:28<16:28:47,  1.75s/it]
Outer training loop:  32%|###2      | 16148/50000 [9:28:29<16:49:36,  1.79s/it]
Outer training loop:  32%|###2      | 16149/50000 [9:28:31<16:28:30,  1.75s/it]
Outer training loop:  32%|###2      | 16150/50000 [9:28:33<16:26:02,  1.75s/it]
Outer training loop:  32%|###2      | 16151/50000 [9:28:35<16:33:42,  1.76s/it]
Outer training loop:  32%|###2      | 16152/50000 [9:28:37<17:15:44,  1.84s/it]
Outer training loop:  32%|###2      | 16153/50000 [9:28:39<17:22:09,  1.85s/it]
Outer training loop:  32%|###2      | 16154/50000 [9:28:40<16:56:37,  1.80s/it]
Outer training loop:  32%|###2      | 16155/50000 [9:28:42<17:19:41,  1.84s/it]
Outer training loop:  32%|###2      | 16156/50000 [9:28:44<17:04:38,  1.82s/it]
Outer training loop:  32%|###2      | 16157/50000 [9:28:46<16:33:27,  1.76s/it]
Outer training loop:  32%|###2      | 16158/50000 [9:28:47<16:51:54,  1.79s/it]
Outer training loop:  32%|###2      | 16159/50000 [9:28:49<16:50:14,  1.79s/it]
Outer training loop:  32%|###2      | 16160/50000 [9:28:51<16:34:01,  1.76s/it]
Outer training loop:  32%|###2      | 16161/50000 [9:28:53<16:30:18,  1.76s/it]
Outer training loop:  32%|###2      | 16162/50000 [9:28:55<17:23:07,  1.85s/it]
Outer training loop:  32%|###2      | 16163/50000 [9:28:56<16:57:39,  1.80s/it]
Outer training loop:  32%|###2      | 16164/50000 [9:28:58<16:47:02,  1.79s/it]
Outer training loop:  32%|###2      | 16165/50000 [9:29:00<16:49:59,  1.79s/it]
Outer training loop:  32%|###2      | 16166/50000 [9:29:02<16:39:18,  1.77s/it]
Outer training loop:  32%|###2      | 16167/50000 [9:29:03<16:33:57,  1.76s/it]
Outer training loop:  32%|###2      | 16168/50000 [9:29:05<16:53:02,  1.80s/it]
Outer training loop:  32%|###2      | 16169/50000 [9:29:07<16:53:08,  1.80s/it]
Outer training loop:  32%|###2      | 16170/50000 [9:29:09<16:57:04,  1.80s/it]
Outer training loop:  32%|###2      | 16171/50000 [9:29:11<16:36:57,  1.77s/it]
Outer training loop:  32%|###2      | 16172/50000 [9:29:12<16:50:17,  1.79s/it]
Outer training loop:  32%|###2      | 16173/50000 [9:29:14<16:38:29,  1.77s/it]
Outer training loop:  32%|###2      | 16174/50000 [9:29:16<16:18:17,  1.74s/it]
Outer training loop:  32%|###2      | 16175/50000 [9:29:18<16:32:44,  1.76s/it]
Outer training loop:  32%|###2      | 16176/50000 [9:29:19<16:26:27,  1.75s/it]
Outer training loop:  32%|###2      | 16177/50000 [9:29:21<16:12:11,  1.72s/it]
Outer training loop:  32%|###2      | 16178/50000 [9:29:23<16:12:37,  1.73s/it]
Outer training loop:  32%|###2      | 16179/50000 [9:29:25<16:58:16,  1.81s/it]
Outer training loop:  32%|###2      | 16180/50000 [9:29:26<16:46:33,  1.79s/it]
Outer training loop:  32%|###2      | 16181/50000 [9:29:28<16:40:23,  1.77s/it]
Outer training loop:  32%|###2      | 16182/50000 [9:29:30<17:04:52,  1.82s/it]
Outer training loop:  32%|###2      | 16183/50000 [9:29:32<16:44:28,  1.78s/it]
Outer training loop:  32%|###2      | 16184/50000 [9:29:34<16:27:28,  1.75s/it]
Outer training loop:  32%|###2      | 16185/50000 [9:29:35<16:39:42,  1.77s/it]
Outer training loop:  32%|###2      | 16186/50000 [9:29:37<17:03:56,  1.82s/it]
Outer training loop:  32%|###2      | 16187/50000 [9:29:39<16:57:27,  1.81s/it]
Outer training loop:  32%|###2      | 16188/50000 [9:29:41<16:51:47,  1.80s/it]
Outer training loop:  32%|###2      | 16189/50000 [9:29:43<17:00:44,  1.81s/it]
Outer training loop:  32%|###2      | 16190/50000 [9:29:44<16:45:38,  1.78s/it]
Outer training loop:  32%|###2      | 16191/50000 [9:29:46<16:26:58,  1.75s/it]
Outer training loop:  32%|###2      | 16192/50000 [9:29:48<17:02:43,  1.82s/it]
Outer training loop:  32%|###2      | 16193/50000 [9:29:50<16:34:16,  1.76s/it]
Outer training loop:  32%|###2      | 16194/50000 [9:29:51<16:36:27,  1.77s/it]
Outer training loop:  32%|###2      | 16195/50000 [9:29:53<16:53:31,  1.80s/it]
Outer training loop:  32%|###2      | 16196/50000 [9:29:55<16:32:51,  1.76s/it]
Outer training loop:  32%|###2      | 16197/50000 [9:29:57<16:16:43,  1.73s/it]
Outer training loop:  32%|###2      | 16198/50000 [9:29:59<16:35:50,  1.77s/it]
Outer training loop:  32%|###2      | 16199/50000 [9:30:00<16:11:47,  1.73s/it]
Outer training loop:  32%|###2      | 16200/50000 [9:30:02<16:13:26,  1.73s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.91s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  32%|###2      | 16201/50000 [9:30:09<31:43:17,  3.38s/it]
Outer training loop:  32%|###2      | 16202/50000 [9:30:11<27:46:33,  2.96s/it]
Outer training loop:  32%|###2      | 16203/50000 [9:30:13<24:01:17,  2.56s/it]
Outer training loop:  32%|###2      | 16204/50000 [9:30:14<21:38:12,  2.30s/it]
Outer training loop:  32%|###2      | 16205/50000 [9:30:16<19:40:23,  2.10s/it]
Outer training loop:  32%|###2      | 16206/50000 [9:30:18<19:20:46,  2.06s/it]
Outer training loop:  32%|###2      | 16207/50000 [9:30:20<18:24:48,  1.96s/it]
Outer training loop:  32%|###2      | 16208/50000 [9:30:22<17:51:03,  1.90s/it]
Outer training loop:  32%|###2      | 16209/50000 [9:30:23<17:39:24,  1.88s/it]
Outer training loop:  32%|###2      | 16210/50000 [9:30:25<17:33:12,  1.87s/it]
Outer training loop:  32%|###2      | 16211/50000 [9:30:27<17:12:08,  1.83s/it]
Outer training loop:  32%|###2      | 16212/50000 [9:30:29<17:16:30,  1.84s/it]
Outer training loop:  32%|###2      | 16213/50000 [9:30:30<16:44:57,  1.78s/it]
Outer training loop:  32%|###2      | 16214/50000 [9:30:32<16:41:55,  1.78s/it]
Outer training loop:  32%|###2      | 16215/50000 [9:30:34<16:46:37,  1.79s/it]
Outer training loop:  32%|###2      | 16216/50000 [9:30:36<17:26:04,  1.86s/it]
Outer training loop:  32%|###2      | 16217/50000 [9:30:38<16:58:21,  1.81s/it]
Outer training loop:  32%|###2      | 16218/50000 [9:30:40<16:56:04,  1.80s/it]
Outer training loop:  32%|###2      | 16219/50000 [9:30:41<16:45:23,  1.79s/it]
Outer training loop:  32%|###2      | 16220/50000 [9:30:43<17:02:12,  1.82s/it]
Outer training loop:  32%|###2      | 16221/50000 [9:30:45<16:37:27,  1.77s/it]
Outer training loop:  32%|###2      | 16222/50000 [9:30:47<16:42:20,  1.78s/it]
Outer training loop:  32%|###2      | 16223/50000 [9:30:49<17:04:05,  1.82s/it]
Outer training loop:  32%|###2      | 16224/50000 [9:30:50<16:49:12,  1.79s/it]
Outer training loop:  32%|###2      | 16225/50000 [9:30:52<16:25:43,  1.75s/it]
Outer training loop:  32%|###2      | 16226/50000 [9:30:54<16:59:04,  1.81s/it]
Outer training loop:  32%|###2      | 16227/50000 [9:30:55<16:24:45,  1.75s/it]
Outer training loop:  32%|###2      | 16228/50000 [9:30:57<16:14:01,  1.73s/it]
Outer training loop:  32%|###2      | 16229/50000 [9:30:59<16:29:04,  1.76s/it]
Outer training loop:  32%|###2      | 16230/50000 [9:31:01<16:19:04,  1.74s/it]
Outer training loop:  32%|###2      | 16231/50000 [9:31:02<16:21:50,  1.74s/it]
Outer training loop:  32%|###2      | 16232/50000 [9:31:04<16:22:09,  1.75s/it]
Outer training loop:  32%|###2      | 16233/50000 [9:31:06<16:46:52,  1.79s/it]
Outer training loop:  32%|###2      | 16234/50000 [9:31:08<16:38:59,  1.78s/it]
Outer training loop:  32%|###2      | 16235/50000 [9:31:10<16:40:57,  1.78s/it]
Outer training loop:  32%|###2      | 16236/50000 [9:31:12<17:06:09,  1.82s/it]
Outer training loop:  32%|###2      | 16237/50000 [9:31:13<16:46:29,  1.79s/it]
Outer training loop:  32%|###2      | 16238/50000 [9:31:15<16:43:01,  1.78s/it]
Outer training loop:  32%|###2      | 16239/50000 [9:31:17<16:28:33,  1.76s/it]
Outer training loop:  32%|###2      | 16240/50000 [9:31:19<16:45:25,  1.79s/it]
Outer training loop:  32%|###2      | 16241/50000 [9:31:20<16:33:55,  1.77s/it]
Outer training loop:  32%|###2      | 16242/50000 [9:31:22<16:15:16,  1.73s/it]
Outer training loop:  32%|###2      | 16243/50000 [9:31:24<16:37:33,  1.77s/it]
Outer training loop:  32%|###2      | 16244/50000 [9:31:25<16:17:34,  1.74s/it]
Outer training loop:  32%|###2      | 16245/50000 [9:31:27<16:21:08,  1.74s/it]
Outer training loop:  32%|###2      | 16246/50000 [9:31:29<16:37:06,  1.77s/it]
Outer training loop:  32%|###2      | 16247/50000 [9:31:31<16:20:13,  1.74s/it]
Outer training loop:  32%|###2      | 16248/50000 [9:31:32<16:10:21,  1.72s/it]
Outer training loop:  32%|###2      | 16249/50000 [9:31:34<16:38:36,  1.78s/it]
Outer training loop:  32%|###2      | 16250/50000 [9:31:36<17:01:22,  1.82s/it]
Outer training loop:  33%|###2      | 16251/50000 [9:31:38<16:45:10,  1.79s/it]
Outer training loop:  33%|###2      | 16252/50000 [9:31:40<16:15:52,  1.73s/it]
Outer training loop:  33%|###2      | 16253/50000 [9:31:41<16:36:59,  1.77s/it]
Outer training loop:  33%|###2      | 16254/50000 [9:31:43<16:29:42,  1.76s/it]
Outer training loop:  33%|###2      | 16255/50000 [9:31:45<16:24:56,  1.75s/it]
Outer training loop:  33%|###2      | 16256/50000 [9:31:47<16:33:14,  1.77s/it]
Outer training loop:  33%|###2      | 16257/50000 [9:31:49<16:47:45,  1.79s/it]
Outer training loop:  33%|###2      | 16258/50000 [9:31:50<16:44:42,  1.79s/it]
Outer training loop:  33%|###2      | 16259/50000 [9:31:52<16:29:24,  1.76s/it]
Outer training loop:  33%|###2      | 16260/50000 [9:31:54<16:48:59,  1.79s/it]
Outer training loop:  33%|###2      | 16261/50000 [9:31:56<16:57:46,  1.81s/it]
Outer training loop:  33%|###2      | 16262/50000 [9:31:57<16:48:38,  1.79s/it]
Outer training loop:  33%|###2      | 16263/50000 [9:31:59<17:08:43,  1.83s/it]
Outer training loop:  33%|###2      | 16264/50000 [9:32:01<16:51:50,  1.80s/it]
Outer training loop:  33%|###2      | 16265/50000 [9:32:03<16:28:22,  1.76s/it]
Outer training loop:  33%|###2      | 16266/50000 [9:32:04<16:09:19,  1.72s/it]
Outer training loop:  33%|###2      | 16267/50000 [9:32:06<16:41:59,  1.78s/it]
Outer training loop:  33%|###2      | 16268/50000 [9:32:08<16:27:52,  1.76s/it]
Outer training loop:  33%|###2      | 16269/50000 [9:32:10<16:33:32,  1.77s/it]
Outer training loop:  33%|###2      | 16270/50000 [9:32:12<16:51:46,  1.80s/it]
Outer training loop:  33%|###2      | 16271/50000 [9:32:13<16:31:16,  1.76s/it]
Outer training loop:  33%|###2      | 16272/50000 [9:32:15<16:35:36,  1.77s/it]
Outer training loop:  33%|###2      | 16273/50000 [9:32:17<17:12:27,  1.84s/it]
Outer training loop:  33%|###2      | 16274/50000 [9:32:19<17:05:48,  1.82s/it]
Outer training loop:  33%|###2      | 16275/50000 [9:32:21<16:47:25,  1.79s/it]
Outer training loop:  33%|###2      | 16276/50000 [9:32:22<16:44:09,  1.79s/it]
Outer training loop:  33%|###2      | 16277/50000 [9:32:24<17:04:23,  1.82s/it]
Outer training loop:  33%|###2      | 16278/50000 [9:32:26<16:52:59,  1.80s/it]
Outer training loop:  33%|###2      | 16279/50000 [9:32:28<16:54:26,  1.81s/it]
Outer training loop:  33%|###2      | 16280/50000 [9:32:30<17:00:16,  1.82s/it]
Outer training loop:  33%|###2      | 16281/50000 [9:32:31<16:33:06,  1.77s/it]
Outer training loop:  33%|###2      | 16282/50000 [9:32:33<16:19:13,  1.74s/it]
Outer training loop:  33%|###2      | 16283/50000 [9:32:35<16:12:31,  1.73s/it]
Outer training loop:  33%|###2      | 16284/50000 [9:32:37<16:41:21,  1.78s/it]
Outer training loop:  33%|###2      | 16285/50000 [9:32:38<16:34:53,  1.77s/it]
Outer training loop:  33%|###2      | 16286/50000 [9:32:40<16:21:24,  1.75s/it]
Outer training loop:  33%|###2      | 16287/50000 [9:32:42<16:40:25,  1.78s/it]
Outer training loop:  33%|###2      | 16288/50000 [9:32:44<16:30:13,  1.76s/it]
Outer training loop:  33%|###2      | 16289/50000 [9:32:46<17:04:17,  1.82s/it]
Outer training loop:  33%|###2      | 16290/50000 [9:32:47<16:53:39,  1.80s/it]
Outer training loop:  33%|###2      | 16291/50000 [9:32:49<16:38:51,  1.78s/it]
Outer training loop:  33%|###2      | 16292/50000 [9:32:51<16:36:01,  1.77s/it]
Outer training loop:  33%|###2      | 16293/50000 [9:32:53<16:39:12,  1.78s/it]
Outer training loop:  33%|###2      | 16294/50000 [9:32:54<16:07:51,  1.72s/it]
Outer training loop:  33%|###2      | 16295/50000 [9:32:56<16:08:20,  1.72s/it]
Outer training loop:  33%|###2      | 16296/50000 [9:32:58<16:22:09,  1.75s/it]
Outer training loop:  33%|###2      | 16297/50000 [9:33:00<16:14:07,  1.73s/it]
Outer training loop:  33%|###2      | 16298/50000 [9:33:01<16:13:27,  1.73s/it]
Outer training loop:  33%|###2      | 16299/50000 [9:33:03<16:16:30,  1.74s/it]
Outer training loop:  33%|###2      | 16300/50000 [9:33:05<17:00:52,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.64s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.69s/it]

Outer training loop:  33%|###2      | 16301/50000 [9:33:12<31:18:09,  3.34s/it]
Outer training loop:  33%|###2      | 16302/50000 [9:33:14<26:35:35,  2.84s/it]
Outer training loop:  33%|###2      | 16303/50000 [9:33:15<23:44:09,  2.54s/it]
Outer training loop:  33%|###2      | 16304/50000 [9:33:17<22:11:31,  2.37s/it]
Outer training loop:  33%|###2      | 16305/50000 [9:33:19<20:25:50,  2.18s/it]
Outer training loop:  33%|###2      | 16306/50000 [9:33:21<19:22:52,  2.07s/it]
Outer training loop:  33%|###2      | 16307/50000 [9:33:23<18:17:08,  1.95s/it]
Outer training loop:  33%|###2      | 16308/50000 [9:33:24<17:57:55,  1.92s/it]
Outer training loop:  33%|###2      | 16309/50000 [9:33:26<17:30:24,  1.87s/it]
Outer training loop:  33%|###2      | 16310/50000 [9:33:28<17:06:36,  1.83s/it]
Outer training loop:  33%|###2      | 16311/50000 [9:33:30<17:15:41,  1.84s/it]
Outer training loop:  33%|###2      | 16312/50000 [9:33:32<16:46:03,  1.79s/it]
Outer training loop:  33%|###2      | 16313/50000 [9:33:33<16:46:30,  1.79s/it]
Outer training loop:  33%|###2      | 16314/50000 [9:33:35<17:20:55,  1.85s/it]
Outer training loop:  33%|###2      | 16315/50000 [9:33:37<16:56:13,  1.81s/it]
Outer training loop:  33%|###2      | 16316/50000 [9:33:39<16:44:01,  1.79s/it]
Outer training loop:  33%|###2      | 16317/50000 [9:33:40<16:14:53,  1.74s/it]
Outer training loop:  33%|###2      | 16318/50000 [9:33:42<16:30:24,  1.76s/it]
Outer training loop:  33%|###2      | 16319/50000 [9:33:44<16:33:07,  1.77s/it]
Outer training loop:  33%|###2      | 16320/50000 [9:33:46<16:28:11,  1.76s/it]
Outer training loop:  33%|###2      | 16321/50000 [9:33:48<17:08:09,  1.83s/it]
Outer training loop:  33%|###2      | 16322/50000 [9:33:49<16:54:29,  1.81s/it]
Outer training loop:  33%|###2      | 16323/50000 [9:33:51<16:36:47,  1.78s/it]
Outer training loop:  33%|###2      | 16324/50000 [9:33:53<16:17:13,  1.74s/it]
Outer training loop:  33%|###2      | 16325/50000 [9:33:55<16:29:33,  1.76s/it]
Outer training loop:  33%|###2      | 16326/50000 [9:33:56<16:37:36,  1.78s/it]
Outer training loop:  33%|###2      | 16327/50000 [9:33:58<16:13:55,  1.74s/it]
Outer training loop:  33%|###2      | 16328/50000 [9:34:00<16:27:57,  1.76s/it]
Outer training loop:  33%|###2      | 16329/50000 [9:34:02<16:18:39,  1.74s/it]
Outer training loop:  33%|###2      | 16330/50000 [9:34:03<16:26:34,  1.76s/it]
Outer training loop:  33%|###2      | 16331/50000 [9:34:05<16:27:54,  1.76s/it]
Outer training loop:  33%|###2      | 16332/50000 [9:34:07<16:46:15,  1.79s/it]
Outer training loop:  33%|###2      | 16333/50000 [9:34:09<16:22:12,  1.75s/it]
Outer training loop:  33%|###2      | 16334/50000 [9:34:10<16:22:00,  1.75s/it]
Outer training loop:  33%|###2      | 16335/50000 [9:34:12<16:49:10,  1.80s/it]
Outer training loop:  33%|###2      | 16336/50000 [9:34:14<16:30:02,  1.76s/it]
Outer training loop:  33%|###2      | 16337/50000 [9:34:16<16:19:35,  1.75s/it]
Outer training loop:  33%|###2      | 16338/50000 [9:34:18<16:22:52,  1.75s/it]
Outer training loop:  33%|###2      | 16339/50000 [9:34:19<16:56:11,  1.81s/it]
Outer training loop:  33%|###2      | 16340/50000 [9:34:21<16:40:43,  1.78s/it]
Outer training loop:  33%|###2      | 16341/50000 [9:34:23<16:15:26,  1.74s/it]
Outer training loop:  33%|###2      | 16342/50000 [9:34:25<16:34:39,  1.77s/it]
Outer training loop:  33%|###2      | 16343/50000 [9:34:27<16:50:22,  1.80s/it]
Outer training loop:  33%|###2      | 16344/50000 [9:34:28<16:16:34,  1.74s/it]
Outer training loop:  33%|###2      | 16345/50000 [9:34:30<16:14:27,  1.74s/it]
Outer training loop:  33%|###2      | 16346/50000 [9:34:32<16:46:17,  1.79s/it]
Outer training loop:  33%|###2      | 16347/50000 [9:34:34<16:54:23,  1.81s/it]
Outer training loop:  33%|###2      | 16348/50000 [9:34:35<16:37:25,  1.78s/it]
Outer training loop:  33%|###2      | 16349/50000 [9:34:37<16:32:19,  1.77s/it]
Outer training loop:  33%|###2      | 16350/50000 [9:34:39<16:20:43,  1.75s/it]
Outer training loop:  33%|###2      | 16351/50000 [9:34:41<16:32:36,  1.77s/it]
Outer training loop:  33%|###2      | 16352/50000 [9:34:43<16:58:02,  1.82s/it]
Outer training loop:  33%|###2      | 16353/50000 [9:34:44<16:19:42,  1.75s/it]
Outer training loop:  33%|###2      | 16354/50000 [9:34:46<16:05:15,  1.72s/it]
Outer training loop:  33%|###2      | 16355/50000 [9:34:48<16:10:10,  1.73s/it]
Outer training loop:  33%|###2      | 16356/50000 [9:34:50<16:50:18,  1.80s/it]
Outer training loop:  33%|###2      | 16357/50000 [9:34:51<16:28:03,  1.76s/it]
Outer training loop:  33%|###2      | 16358/50000 [9:34:53<16:18:58,  1.75s/it]
Outer training loop:  33%|###2      | 16359/50000 [9:34:55<16:47:14,  1.80s/it]
Outer training loop:  33%|###2      | 16360/50000 [9:34:57<16:38:00,  1.78s/it]
Outer training loop:  33%|###2      | 16361/50000 [9:34:58<16:57:41,  1.82s/it]
Outer training loop:  33%|###2      | 16362/50000 [9:35:00<16:54:29,  1.81s/it]
Outer training loop:  33%|###2      | 16363/50000 [9:35:02<17:14:50,  1.85s/it]
Outer training loop:  33%|###2      | 16364/50000 [9:35:04<16:56:33,  1.81s/it]
Outer training loop:  33%|###2      | 16365/50000 [9:35:06<16:41:45,  1.79s/it]
Outer training loop:  33%|###2      | 16366/50000 [9:35:08<17:05:21,  1.83s/it]
Outer training loop:  33%|###2      | 16367/50000 [9:35:09<16:43:43,  1.79s/it]
Outer training loop:  33%|###2      | 16368/50000 [9:35:11<16:28:09,  1.76s/it]
Outer training loop:  33%|###2      | 16369/50000 [9:35:13<16:09:47,  1.73s/it]
Outer training loop:  33%|###2      | 16370/50000 [9:35:14<16:36:30,  1.78s/it]
Outer training loop:  33%|###2      | 16371/50000 [9:35:16<16:06:59,  1.73s/it]
Outer training loop:  33%|###2      | 16372/50000 [9:35:18<16:11:57,  1.73s/it]
Outer training loop:  33%|###2      | 16373/50000 [9:35:20<16:43:30,  1.79s/it]
Outer training loop:  33%|###2      | 16374/50000 [9:35:22<16:32:18,  1.77s/it]
Outer training loop:  33%|###2      | 16375/50000 [9:35:23<16:22:22,  1.75s/it]
Outer training loop:  33%|###2      | 16376/50000 [9:35:25<16:27:49,  1.76s/it]
Outer training loop:  33%|###2      | 16377/50000 [9:35:27<16:41:00,  1.79s/it]
Outer training loop:  33%|###2      | 16378/50000 [9:35:29<16:42:59,  1.79s/it]
Outer training loop:  33%|###2      | 16379/50000 [9:35:30<16:46:04,  1.80s/it]
Outer training loop:  33%|###2      | 16380/50000 [9:35:32<17:01:25,  1.82s/it]
Outer training loop:  33%|###2      | 16381/50000 [9:35:34<16:51:09,  1.80s/it]
Outer training loop:  33%|###2      | 16382/50000 [9:35:36<16:47:15,  1.80s/it]
Outer training loop:  33%|###2      | 16383/50000 [9:35:38<16:49:36,  1.80s/it]
Outer training loop:  33%|###2      | 16384/50000 [9:35:39<16:31:40,  1.77s/it]
Outer training loop:  33%|###2      | 16385/50000 [9:35:41<16:23:06,  1.75s/it]
Outer training loop:  33%|###2      | 16386/50000 [9:35:43<16:38:34,  1.78s/it]
Outer training loop:  33%|###2      | 16387/50000 [9:35:45<16:52:02,  1.81s/it]
Outer training loop:  33%|###2      | 16388/50000 [9:35:46<16:22:03,  1.75s/it]
Outer training loop:  33%|###2      | 16389/50000 [9:35:48<16:16:11,  1.74s/it]
Outer training loop:  33%|###2      | 16390/50000 [9:35:50<16:50:38,  1.80s/it]
Outer training loop:  33%|###2      | 16391/50000 [9:35:52<16:27:15,  1.76s/it]
Outer training loop:  33%|###2      | 16392/50000 [9:35:54<16:28:39,  1.77s/it]
Outer training loop:  33%|###2      | 16393/50000 [9:35:55<16:08:26,  1.73s/it]
Outer training loop:  33%|###2      | 16394/50000 [9:35:57<16:45:24,  1.80s/it]
Outer training loop:  33%|###2      | 16395/50000 [9:35:59<16:29:50,  1.77s/it]
Outer training loop:  33%|###2      | 16396/50000 [9:36:01<16:30:43,  1.77s/it]
Outer training loop:  33%|###2      | 16397/50000 [9:36:03<17:24:56,  1.87s/it]
Outer training loop:  33%|###2      | 16398/50000 [9:36:04<17:07:04,  1.83s/it]
Outer training loop:  33%|###2      | 16399/50000 [9:36:06<16:34:50,  1.78s/it]
Outer training loop:  33%|###2      | 16400/50000 [9:36:08<16:05:42,  1.72s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  33%|###2      | 16401/50000 [9:36:15<31:35:24,  3.38s/it]
Outer training loop:  33%|###2      | 16402/50000 [9:36:17<26:48:56,  2.87s/it]
Outer training loop:  33%|###2      | 16403/50000 [9:36:18<23:28:22,  2.52s/it]
Outer training loop:  33%|###2      | 16404/50000 [9:36:20<21:23:38,  2.29s/it]
Outer training loop:  33%|###2      | 16405/50000 [9:36:22<20:17:42,  2.17s/it]
Outer training loop:  33%|###2      | 16406/50000 [9:36:24<19:21:32,  2.07s/it]
Outer training loop:  33%|###2      | 16407/50000 [9:36:26<18:27:54,  1.98s/it]
Outer training loop:  33%|###2      | 16408/50000 [9:36:27<18:11:41,  1.95s/it]
Outer training loop:  33%|###2      | 16409/50000 [9:36:29<17:32:22,  1.88s/it]
Outer training loop:  33%|###2      | 16410/50000 [9:36:31<17:04:40,  1.83s/it]
Outer training loop:  33%|###2      | 16411/50000 [9:36:33<17:07:54,  1.84s/it]
Outer training loop:  33%|###2      | 16412/50000 [9:36:34<16:34:39,  1.78s/it]
Outer training loop:  33%|###2      | 16413/50000 [9:36:36<16:17:02,  1.75s/it]
Outer training loop:  33%|###2      | 16414/50000 [9:36:38<16:48:51,  1.80s/it]
Outer training loop:  33%|###2      | 16415/50000 [9:36:40<16:38:25,  1.78s/it]
Outer training loop:  33%|###2      | 16416/50000 [9:36:41<16:12:51,  1.74s/it]
Outer training loop:  33%|###2      | 16417/50000 [9:36:43<16:17:03,  1.75s/it]
Outer training loop:  33%|###2      | 16418/50000 [9:36:45<16:42:55,  1.79s/it]
Outer training loop:  33%|###2      | 16419/50000 [9:36:47<16:28:35,  1.77s/it]
Outer training loop:  33%|###2      | 16420/50000 [9:36:48<16:09:36,  1.73s/it]
Outer training loop:  33%|###2      | 16421/50000 [9:36:50<16:40:56,  1.79s/it]
Outer training loop:  33%|###2      | 16422/50000 [9:36:52<16:19:51,  1.75s/it]
Outer training loop:  33%|###2      | 16423/50000 [9:36:54<16:12:42,  1.74s/it]
Outer training loop:  33%|###2      | 16424/50000 [9:36:56<16:45:23,  1.80s/it]
Outer training loop:  33%|###2      | 16425/50000 [9:36:57<16:23:22,  1.76s/it]
Outer training loop:  33%|###2      | 16426/50000 [9:36:59<16:20:27,  1.75s/it]
Outer training loop:  33%|###2      | 16427/50000 [9:37:01<16:04:13,  1.72s/it]
Outer training loop:  33%|###2      | 16428/50000 [9:37:03<16:31:08,  1.77s/it]
Outer training loop:  33%|###2      | 16429/50000 [9:37:04<16:37:05,  1.78s/it]
Outer training loop:  33%|###2      | 16430/50000 [9:37:06<16:23:16,  1.76s/it]
Outer training loop:  33%|###2      | 16431/50000 [9:37:08<16:35:31,  1.78s/it]
Outer training loop:  33%|###2      | 16432/50000 [9:37:10<16:20:01,  1.75s/it]
Outer training loop:  33%|###2      | 16433/50000 [9:37:11<16:21:07,  1.75s/it]
Outer training loop:  33%|###2      | 16434/50000 [9:37:13<16:49:44,  1.80s/it]
Outer training loop:  33%|###2      | 16435/50000 [9:37:15<16:51:13,  1.81s/it]
Outer training loop:  33%|###2      | 16436/50000 [9:37:17<16:27:10,  1.76s/it]
Outer training loop:  33%|###2      | 16437/50000 [9:37:19<16:54:45,  1.81s/it]
Outer training loop:  33%|###2      | 16438/50000 [9:37:20<16:38:11,  1.78s/it]
Outer training loop:  33%|###2      | 16439/50000 [9:37:22<16:32:01,  1.77s/it]
Outer training loop:  33%|###2      | 16440/50000 [9:37:24<16:35:28,  1.78s/it]
Outer training loop:  33%|###2      | 16441/50000 [9:37:26<16:49:50,  1.81s/it]
Outer training loop:  33%|###2      | 16442/50000 [9:37:27<16:06:59,  1.73s/it]
Outer training loop:  33%|###2      | 16443/50000 [9:37:29<16:33:45,  1.78s/it]
Outer training loop:  33%|###2      | 16444/50000 [9:37:31<17:08:41,  1.84s/it]
Outer training loop:  33%|###2      | 16445/50000 [9:37:33<16:49:41,  1.81s/it]
Outer training loop:  33%|###2      | 16446/50000 [9:37:35<16:47:42,  1.80s/it]
Outer training loop:  33%|###2      | 16447/50000 [9:37:37<17:00:36,  1.83s/it]
Outer training loop:  33%|###2      | 16448/50000 [9:37:38<16:55:01,  1.82s/it]
Outer training loop:  33%|###2      | 16449/50000 [9:37:40<16:36:40,  1.78s/it]
Outer training loop:  33%|###2      | 16450/50000 [9:37:42<17:19:51,  1.86s/it]
Outer training loop:  33%|###2      | 16451/50000 [9:37:44<17:41:33,  1.90s/it]
Outer training loop:  33%|###2      | 16452/50000 [9:37:46<17:07:01,  1.84s/it]
Outer training loop:  33%|###2      | 16453/50000 [9:37:48<16:42:01,  1.79s/it]
Outer training loop:  33%|###2      | 16454/50000 [9:37:49<16:59:50,  1.82s/it]
Outer training loop:  33%|###2      | 16455/50000 [9:37:51<16:43:57,  1.80s/it]
Outer training loop:  33%|###2      | 16456/50000 [9:37:53<16:41:56,  1.79s/it]
Outer training loop:  33%|###2      | 16457/50000 [9:37:55<16:35:51,  1.78s/it]
Outer training loop:  33%|###2      | 16458/50000 [9:37:57<17:08:53,  1.84s/it]
Outer training loop:  33%|###2      | 16459/50000 [9:37:58<16:57:46,  1.82s/it]
Outer training loop:  33%|###2      | 16460/50000 [9:38:00<16:43:24,  1.80s/it]
Outer training loop:  33%|###2      | 16461/50000 [9:38:02<16:52:24,  1.81s/it]
Outer training loop:  33%|###2      | 16462/50000 [9:38:04<16:46:24,  1.80s/it]
Outer training loop:  33%|###2      | 16463/50000 [9:38:06<16:35:42,  1.78s/it]
Outer training loop:  33%|###2      | 16464/50000 [9:38:07<16:46:47,  1.80s/it]
Outer training loop:  33%|###2      | 16465/50000 [9:38:09<16:35:14,  1.78s/it]
Outer training loop:  33%|###2      | 16466/50000 [9:38:11<16:38:11,  1.79s/it]
Outer training loop:  33%|###2      | 16467/50000 [9:38:13<16:46:56,  1.80s/it]
Outer training loop:  33%|###2      | 16468/50000 [9:38:14<16:28:27,  1.77s/it]
Outer training loop:  33%|###2      | 16469/50000 [9:38:16<16:23:18,  1.76s/it]
Outer training loop:  33%|###2      | 16470/50000 [9:38:18<16:05:24,  1.73s/it]
Outer training loop:  33%|###2      | 16471/50000 [9:38:20<16:46:23,  1.80s/it]
Outer training loop:  33%|###2      | 16472/50000 [9:38:21<16:19:01,  1.75s/it]
Outer training loop:  33%|###2      | 16473/50000 [9:38:23<16:13:52,  1.74s/it]
Outer training loop:  33%|###2      | 16474/50000 [9:38:25<16:39:43,  1.79s/it]
Outer training loop:  33%|###2      | 16475/50000 [9:38:27<16:28:19,  1.77s/it]
Outer training loop:  33%|###2      | 16476/50000 [9:38:28<16:11:12,  1.74s/it]
Outer training loop:  33%|###2      | 16477/50000 [9:38:30<16:10:04,  1.74s/it]
Outer training loop:  33%|###2      | 16478/50000 [9:38:32<16:16:20,  1.75s/it]
Outer training loop:  33%|###2      | 16479/50000 [9:38:34<16:12:18,  1.74s/it]
Outer training loop:  33%|###2      | 16480/50000 [9:38:35<15:58:10,  1.72s/it]
Outer training loop:  33%|###2      | 16481/50000 [9:38:37<16:30:03,  1.77s/it]
Outer training loop:  33%|###2      | 16482/50000 [9:38:39<16:38:53,  1.79s/it]
Outer training loop:  33%|###2      | 16483/50000 [9:38:41<16:32:53,  1.78s/it]
Outer training loop:  33%|###2      | 16484/50000 [9:38:43<16:57:09,  1.82s/it]
Outer training loop:  33%|###2      | 16485/50000 [9:38:45<16:44:08,  1.80s/it]
Outer training loop:  33%|###2      | 16486/50000 [9:38:46<16:32:13,  1.78s/it]
Outer training loop:  33%|###2      | 16487/50000 [9:38:48<16:27:07,  1.77s/it]
Outer training loop:  33%|###2      | 16488/50000 [9:38:50<17:06:30,  1.84s/it]
Outer training loop:  33%|###2      | 16489/50000 [9:38:52<16:44:05,  1.80s/it]
Outer training loop:  33%|###2      | 16490/50000 [9:38:54<16:50:13,  1.81s/it]
Outer training loop:  33%|###2      | 16491/50000 [9:38:55<16:30:25,  1.77s/it]
Outer training loop:  33%|###2      | 16492/50000 [9:38:57<16:16:51,  1.75s/it]
Outer training loop:  33%|###2      | 16493/50000 [9:38:59<16:04:39,  1.73s/it]
Outer training loop:  33%|###2      | 16494/50000 [9:39:01<16:56:11,  1.82s/it]
Outer training loop:  33%|###2      | 16495/50000 [9:39:02<16:36:07,  1.78s/it]
Outer training loop:  33%|###2      | 16496/50000 [9:39:04<16:25:40,  1.77s/it]
Outer training loop:  33%|###2      | 16497/50000 [9:39:06<17:03:29,  1.83s/it]
Outer training loop:  33%|###2      | 16498/50000 [9:39:08<16:59:38,  1.83s/it]
Outer training loop:  33%|###2      | 16499/50000 [9:39:10<16:39:39,  1.79s/it]
Outer training loop:  33%|###3      | 16500/50000 [9:39:11<17:00:23,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.72s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.77s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:22,  1.75s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:20,  1.76s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.82s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:21,  1.81s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.78s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:14,  1.80s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:14,  1.82s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:09,  1.78s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:10,  1.82s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:08,  1.80s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:04,  1.78s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:02,  1.77s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:02,  1.81s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:57,  1.76s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:56,  1.77s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:53,  1.74s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:51,  1.74s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:55,  1.84s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:51,  1.79s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:47,  1.76s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:45,  1.76s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:45,  1.81s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.78s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:41,  1.80s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:37,  1.77s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:34,  1.75s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:34,  1.83s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:30,  1.78s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:28,  1.76s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:23,  1.71s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:26,  1.80s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:23,  1.78s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:20,  1.76s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:18,  1.74s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.74s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:15,  1.75s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:12,  1.73s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:16,  1.86s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:12,  1.82s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.78s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:05,  1.73s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:03,  1.73s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:01,  1.70s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<00:58,  1.69s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:58,  1.76s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:56,  1.75s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:54,  1.77s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:52,  1.74s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:49,  1.72s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.73s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:49,  1.84s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:45,  1.81s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.78s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.77s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:38,  1.75s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:36,  1.74s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:34,  1.75s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:30,  1.77s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:27,  1.74s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:26,  1.75s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.72s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.74s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:20,  1.69s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:18,  1.67s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.81s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.80s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.78s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.75s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.78s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.78s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.75s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.88s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:09,  1.58s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:10,  1.61s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:10,  1.63s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:11,  1.67s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:09,  1.66s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:09,  1.68s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:14,  1.77s/it]

Evaluating policy:  10%|9         | 8/83 [00:13<02:11,  1.75s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:09,  1.75s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:06,  1.73s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:20,  1.95s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:13,  1.88s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:09,  1.84s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:05,  1.85s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:03,  1.84s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:59,  1.81s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:59,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:55,  1.83s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:51,  1.79s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:47,  1.76s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:46,  1.77s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:49,  1.85s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:46,  1.83s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:42,  1.80s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:39,  1.78s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:37,  1.77s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:36,  1.82s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:32,  1.77s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:28,  1.73s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:26,  1.73s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:23,  1.71s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:26,  1.84s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:23,  1.82s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:18,  1.75s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:15,  1.72s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.73s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:17,  1.85s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:15,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:11,  1.80s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.79s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:08,  1.80s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:05,  1.77s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:03,  1.76s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:01,  1.82s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:58,  1.77s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:56,  1.77s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:54,  1.77s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:52,  1.74s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:49,  1.70s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:44,  1.80s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.79s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:40,  1.77s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:38,  1.75s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:36,  1.76s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:53<00:37,  1.88s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:32,  1.80s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:29,  1.76s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:28,  1.75s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:25,  1.73s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:24,  1.77s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:22,  1.77s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:22,  1.89s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.79s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:15,  1.75s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:13,  1.75s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.74s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.74s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.69s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:06,  1.72s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.83s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.77s/it]

Outer training loop:  33%|###3      | 16501/50000 [9:44:15<858:04:37, 92.21s/it]
Outer training loop:  33%|###3      | 16502/50000 [9:44:16<605:23:50, 65.06s/it]
Outer training loop:  33%|###3      | 16503/50000 [9:44:18<428:33:02, 46.06s/it]
Outer training loop:  33%|###3      | 16504/50000 [9:44:20<304:57:38, 32.78s/it]
Outer training loop:  33%|###3      | 16505/50000 [9:44:21<218:05:10, 23.44s/it]
Outer training loop:  33%|###3      | 16506/50000 [9:44:23<157:39:02, 16.94s/it]
Outer training loop:  33%|###3      | 16507/50000 [9:44:25<114:53:50, 12.35s/it]
Outer training loop:  33%|###3      | 16508/50000 [9:44:27<86:57:38,  9.35s/it] 
Outer training loop:  33%|###3      | 16509/50000 [9:44:29<65:37:20,  7.05s/it]
Outer training loop:  33%|###3      | 16510/50000 [9:44:31<50:39:42,  5.45s/it]
Outer training loop:  33%|###3      | 16511/50000 [9:44:33<40:47:09,  4.38s/it]
Outer training loop:  33%|###3      | 16512/50000 [9:44:34<33:30:16,  3.60s/it]
Outer training loop:  33%|###3      | 16513/50000 [9:44:36<28:19:20,  3.04s/it]
Outer training loop:  33%|###3      | 16514/50000 [9:44:38<25:04:25,  2.70s/it]
Outer training loop:  33%|###3      | 16515/50000 [9:44:40<22:14:59,  2.39s/it]
Outer training loop:  33%|###3      | 16516/50000 [9:44:41<20:28:32,  2.20s/it]
Outer training loop:  33%|###3      | 16517/50000 [9:44:43<19:15:11,  2.07s/it]
Outer training loop:  33%|###3      | 16518/50000 [9:44:45<18:38:23,  2.00s/it]
Outer training loop:  33%|###3      | 16519/50000 [9:44:47<17:49:07,  1.92s/it]
Outer training loop:  33%|###3      | 16520/50000 [9:44:48<16:58:40,  1.83s/it]
Outer training loop:  33%|###3      | 16521/50000 [9:44:50<17:23:51,  1.87s/it]
Outer training loop:  33%|###3      | 16522/50000 [9:44:52<17:07:44,  1.84s/it]
Outer training loop:  33%|###3      | 16523/50000 [9:44:54<16:36:18,  1.79s/it]
Outer training loop:  33%|###3      | 16524/50000 [9:44:56<16:59:31,  1.83s/it]
Outer training loop:  33%|###3      | 16525/50000 [9:44:58<17:10:05,  1.85s/it]
Outer training loop:  33%|###3      | 16526/50000 [9:44:59<16:31:28,  1.78s/it]
Outer training loop:  33%|###3      | 16527/50000 [9:45:01<16:44:14,  1.80s/it]
Outer training loop:  33%|###3      | 16528/50000 [9:45:03<16:20:46,  1.76s/it]
Outer training loop:  33%|###3      | 16529/50000 [9:45:04<16:15:31,  1.75s/it]
Outer training loop:  33%|###3      | 16530/50000 [9:45:06<16:17:07,  1.75s/it]
Outer training loop:  33%|###3      | 16531/50000 [9:45:08<16:55:06,  1.82s/it]
Outer training loop:  33%|###3      | 16532/50000 [9:45:10<16:33:11,  1.78s/it]
Outer training loop:  33%|###3      | 16533/50000 [9:45:12<16:36:54,  1.79s/it]
Outer training loop:  33%|###3      | 16534/50000 [9:45:14<17:04:25,  1.84s/it]
Outer training loop:  33%|###3      | 16535/50000 [9:45:15<16:56:24,  1.82s/it]
Outer training loop:  33%|###3      | 16536/50000 [9:45:17<16:46:53,  1.81s/it]
Outer training loop:  33%|###3      | 16537/50000 [9:45:19<17:02:56,  1.83s/it]
Outer training loop:  33%|###3      | 16538/50000 [9:45:21<16:34:37,  1.78s/it]
Outer training loop:  33%|###3      | 16539/50000 [9:45:22<16:34:37,  1.78s/it]
Outer training loop:  33%|###3      | 16540/50000 [9:45:24<16:09:30,  1.74s/it]
Outer training loop:  33%|###3      | 16541/50000 [9:45:26<16:31:32,  1.78s/it]
Outer training loop:  33%|###3      | 16542/50000 [9:45:28<16:09:42,  1.74s/it]
Outer training loop:  33%|###3      | 16543/50000 [9:45:29<16:02:51,  1.73s/it]
Outer training loop:  33%|###3      | 16544/50000 [9:45:31<16:52:48,  1.82s/it]
Outer training loop:  33%|###3      | 16545/50000 [9:45:33<16:29:48,  1.78s/it]
Outer training loop:  33%|###3      | 16546/50000 [9:45:35<16:34:09,  1.78s/it]
Outer training loop:  33%|###3      | 16547/50000 [9:45:37<16:44:00,  1.80s/it]
Outer training loop:  33%|###3      | 16548/50000 [9:45:38<16:15:28,  1.75s/it]
Outer training loop:  33%|###3      | 16549/50000 [9:45:40<16:09:46,  1.74s/it]
Outer training loop:  33%|###3      | 16550/50000 [9:45:42<15:46:54,  1.70s/it]
Outer training loop:  33%|###3      | 16551/50000 [9:45:43<16:05:29,  1.73s/it]
Outer training loop:  33%|###3      | 16552/50000 [9:45:45<15:59:22,  1.72s/it]
Outer training loop:  33%|###3      | 16553/50000 [9:45:47<16:10:23,  1.74s/it]
Outer training loop:  33%|###3      | 16554/50000 [9:45:49<15:55:14,  1.71s/it]
Outer training loop:  33%|###3      | 16555/50000 [9:45:50<15:42:40,  1.69s/it]
Outer training loop:  33%|###3      | 16556/50000 [9:45:52<15:42:48,  1.69s/it]
Outer training loop:  33%|###3      | 16557/50000 [9:45:54<16:31:55,  1.78s/it]
Outer training loop:  33%|###3      | 16558/50000 [9:45:56<16:19:12,  1.76s/it]
Outer training loop:  33%|###3      | 16559/50000 [9:45:57<16:23:51,  1.77s/it]
Outer training loop:  33%|###3      | 16560/50000 [9:45:59<16:02:29,  1.73s/it]
Outer training loop:  33%|###3      | 16561/50000 [9:46:01<16:28:25,  1.77s/it]
Outer training loop:  33%|###3      | 16562/50000 [9:46:03<16:09:27,  1.74s/it]
Outer training loop:  33%|###3      | 16563/50000 [9:46:04<16:05:41,  1.73s/it]
Outer training loop:  33%|###3      | 16564/50000 [9:46:06<16:23:45,  1.77s/it]
Outer training loop:  33%|###3      | 16565/50000 [9:46:08<16:09:35,  1.74s/it]
Outer training loop:  33%|###3      | 16566/50000 [9:46:09<16:06:30,  1.73s/it]
Outer training loop:  33%|###3      | 16567/50000 [9:46:11<16:16:39,  1.75s/it]
Outer training loop:  33%|###3      | 16568/50000 [9:46:13<15:59:58,  1.72s/it]
Outer training loop:  33%|###3      | 16569/50000 [9:46:15<15:34:33,  1.68s/it]
Outer training loop:  33%|###3      | 16570/50000 [9:46:16<15:27:39,  1.66s/it]
Outer training loop:  33%|###3      | 16571/50000 [9:46:18<16:16:06,  1.75s/it]
Outer training loop:  33%|###3      | 16572/50000 [9:46:20<16:18:16,  1.76s/it]
Outer training loop:  33%|###3      | 16573/50000 [9:46:22<16:13:13,  1.75s/it]
Outer training loop:  33%|###3      | 16574/50000 [9:46:23<16:00:38,  1.72s/it]
Outer training loop:  33%|###3      | 16575/50000 [9:46:25<16:58:52,  1.83s/it]
Outer training loop:  33%|###3      | 16576/50000 [9:46:27<16:53:16,  1.82s/it]
Outer training loop:  33%|###3      | 16577/50000 [9:46:29<16:36:42,  1.79s/it]
Outer training loop:  33%|###3      | 16578/50000 [9:46:31<16:39:58,  1.80s/it]
Outer training loop:  33%|###3      | 16579/50000 [9:46:32<16:17:47,  1.76s/it]
Outer training loop:  33%|###3      | 16580/50000 [9:46:34<16:14:32,  1.75s/it]
Outer training loop:  33%|###3      | 16581/50000 [9:46:36<16:36:42,  1.79s/it]
Outer training loop:  33%|###3      | 16582/50000 [9:46:38<16:27:30,  1.77s/it]
Outer training loop:  33%|###3      | 16583/50000 [9:46:39<16:34:56,  1.79s/it]
Outer training loop:  33%|###3      | 16584/50000 [9:46:41<16:41:15,  1.80s/it]
Outer training loop:  33%|###3      | 16585/50000 [9:46:43<16:35:58,  1.79s/it]
Outer training loop:  33%|###3      | 16586/50000 [9:46:45<16:22:31,  1.76s/it]
Outer training loop:  33%|###3      | 16587/50000 [9:46:46<16:01:30,  1.73s/it]
Outer training loop:  33%|###3      | 16588/50000 [9:46:48<16:32:35,  1.78s/it]
Outer training loop:  33%|###3      | 16589/50000 [9:46:50<16:36:32,  1.79s/it]
Outer training loop:  33%|###3      | 16590/50000 [9:46:52<16:22:42,  1.76s/it]
Outer training loop:  33%|###3      | 16591/50000 [9:46:54<16:49:08,  1.81s/it]
Outer training loop:  33%|###3      | 16592/50000 [9:46:56<16:40:45,  1.80s/it]
Outer training loop:  33%|###3      | 16593/50000 [9:46:57<16:06:07,  1.74s/it]
Outer training loop:  33%|###3      | 16594/50000 [9:46:59<15:59:55,  1.72s/it]
Outer training loop:  33%|###3      | 16595/50000 [9:47:01<16:24:43,  1.77s/it]
Outer training loop:  33%|###3      | 16596/50000 [9:47:02<16:18:23,  1.76s/it]
Outer training loop:  33%|###3      | 16597/50000 [9:47:04<16:11:11,  1.74s/it]
Outer training loop:  33%|###3      | 16598/50000 [9:47:06<16:24:58,  1.77s/it]
Outer training loop:  33%|###3      | 16599/50000 [9:47:08<15:55:17,  1.72s/it]
Outer training loop:  33%|###3      | 16600/50000 [9:47:09<16:01:10,  1.73s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  33%|###3      | 16601/50000 [9:47:16<30:31:55,  3.29s/it]
Outer training loop:  33%|###3      | 16602/50000 [9:47:18<26:21:44,  2.84s/it]
Outer training loop:  33%|###3      | 16603/50000 [9:47:20<23:50:03,  2.57s/it]
Outer training loop:  33%|###3      | 16604/50000 [9:47:22<21:29:20,  2.32s/it]
Outer training loop:  33%|###3      | 16605/50000 [9:47:23<19:46:05,  2.13s/it]
Outer training loop:  33%|###3      | 16606/50000 [9:47:25<19:07:37,  2.06s/it]
Outer training loop:  33%|###3      | 16607/50000 [9:47:27<18:12:32,  1.96s/it]
Outer training loop:  33%|###3      | 16608/50000 [9:47:29<17:26:34,  1.88s/it]
Outer training loop:  33%|###3      | 16609/50000 [9:47:30<16:40:13,  1.80s/it]
Outer training loop:  33%|###3      | 16610/50000 [9:47:32<16:47:22,  1.81s/it]
Outer training loop:  33%|###3      | 16611/50000 [9:47:34<16:27:03,  1.77s/it]
Outer training loop:  33%|###3      | 16612/50000 [9:47:36<16:15:35,  1.75s/it]
Outer training loop:  33%|###3      | 16613/50000 [9:47:37<16:30:23,  1.78s/it]
Outer training loop:  33%|###3      | 16614/50000 [9:47:39<16:12:11,  1.75s/it]
Outer training loop:  33%|###3      | 16615/50000 [9:47:41<16:11:20,  1.75s/it]
Outer training loop:  33%|###3      | 16616/50000 [9:47:42<15:59:37,  1.72s/it]
Outer training loop:  33%|###3      | 16617/50000 [9:47:44<16:30:02,  1.78s/it]
Outer training loop:  33%|###3      | 16618/50000 [9:47:46<16:22:36,  1.77s/it]
Outer training loop:  33%|###3      | 16619/50000 [9:47:48<16:09:32,  1.74s/it]
Outer training loop:  33%|###3      | 16620/50000 [9:47:50<16:33:40,  1.79s/it]
Outer training loop:  33%|###3      | 16621/50000 [9:47:51<16:26:15,  1.77s/it]
Outer training loop:  33%|###3      | 16622/50000 [9:47:53<17:00:22,  1.83s/it]
Outer training loop:  33%|###3      | 16623/50000 [9:47:55<16:45:46,  1.81s/it]
Outer training loop:  33%|###3      | 16624/50000 [9:47:57<17:06:36,  1.85s/it]
Outer training loop:  33%|###3      | 16625/50000 [9:47:59<16:36:59,  1.79s/it]
Outer training loop:  33%|###3      | 16626/50000 [9:48:00<16:18:37,  1.76s/it]
Outer training loop:  33%|###3      | 16627/50000 [9:48:02<16:44:58,  1.81s/it]
Outer training loop:  33%|###3      | 16628/50000 [9:48:04<16:32:37,  1.78s/it]
Outer training loop:  33%|###3      | 16629/50000 [9:48:06<16:35:04,  1.79s/it]
Outer training loop:  33%|###3      | 16630/50000 [9:48:08<16:43:48,  1.80s/it]
Outer training loop:  33%|###3      | 16631/50000 [9:48:09<16:19:04,  1.76s/it]
Outer training loop:  33%|###3      | 16632/50000 [9:48:11<16:05:15,  1.74s/it]
Outer training loop:  33%|###3      | 16633/50000 [9:48:13<16:38:38,  1.80s/it]
Outer training loop:  33%|###3      | 16634/50000 [9:48:15<16:22:03,  1.77s/it]
Outer training loop:  33%|###3      | 16635/50000 [9:48:16<16:16:07,  1.76s/it]
Outer training loop:  33%|###3      | 16636/50000 [9:48:18<16:15:51,  1.75s/it]
Outer training loop:  33%|###3      | 16637/50000 [9:48:20<17:14:44,  1.86s/it]
Outer training loop:  33%|###3      | 16638/50000 [9:48:22<16:53:20,  1.82s/it]
Outer training loop:  33%|###3      | 16639/50000 [9:48:24<16:25:52,  1.77s/it]
Outer training loop:  33%|###3      | 16640/50000 [9:48:26<16:56:39,  1.83s/it]
Outer training loop:  33%|###3      | 16641/50000 [9:48:27<16:35:47,  1.79s/it]
Outer training loop:  33%|###3      | 16642/50000 [9:48:29<16:25:13,  1.77s/it]
Outer training loop:  33%|###3      | 16643/50000 [9:48:31<16:40:34,  1.80s/it]
Outer training loop:  33%|###3      | 16644/50000 [9:48:33<16:24:47,  1.77s/it]
Outer training loop:  33%|###3      | 16645/50000 [9:48:34<16:17:20,  1.76s/it]
Outer training loop:  33%|###3      | 16646/50000 [9:48:36<16:35:48,  1.79s/it]
Outer training loop:  33%|###3      | 16647/50000 [9:48:38<16:19:11,  1.76s/it]
Outer training loop:  33%|###3      | 16648/50000 [9:48:40<16:09:58,  1.74s/it]
Outer training loop:  33%|###3      | 16649/50000 [9:48:41<15:58:26,  1.72s/it]
Outer training loop:  33%|###3      | 16650/50000 [9:48:43<16:34:36,  1.79s/it]
Outer training loop:  33%|###3      | 16651/50000 [9:48:45<16:19:15,  1.76s/it]
Outer training loop:  33%|###3      | 16652/50000 [9:48:47<16:10:27,  1.75s/it]
Outer training loop:  33%|###3      | 16653/50000 [9:48:48<16:03:33,  1.73s/it]
Outer training loop:  33%|###3      | 16654/50000 [9:48:50<16:39:23,  1.80s/it]
Outer training loop:  33%|###3      | 16655/50000 [9:48:52<16:13:14,  1.75s/it]
Outer training loop:  33%|###3      | 16656/50000 [9:48:54<16:15:27,  1.76s/it]
Outer training loop:  33%|###3      | 16657/50000 [9:48:56<16:55:39,  1.83s/it]
Outer training loop:  33%|###3      | 16658/50000 [9:48:57<16:31:55,  1.79s/it]
Outer training loop:  33%|###3      | 16659/50000 [9:48:59<16:17:38,  1.76s/it]
Outer training loop:  33%|###3      | 16660/50000 [9:49:01<17:00:50,  1.84s/it]
Outer training loop:  33%|###3      | 16661/50000 [9:49:03<16:43:21,  1.81s/it]
Outer training loop:  33%|###3      | 16662/50000 [9:49:05<16:51:20,  1.82s/it]
Outer training loop:  33%|###3      | 16663/50000 [9:49:07<16:59:55,  1.84s/it]
Outer training loop:  33%|###3      | 16664/50000 [9:49:08<16:45:18,  1.81s/it]
Outer training loop:  33%|###3      | 16665/50000 [9:49:10<16:38:47,  1.80s/it]
Outer training loop:  33%|###3      | 16666/50000 [9:49:12<16:21:56,  1.77s/it]
Outer training loop:  33%|###3      | 16667/50000 [9:49:14<16:39:00,  1.80s/it]
Outer training loop:  33%|###3      | 16668/50000 [9:49:15<16:13:10,  1.75s/it]
Outer training loop:  33%|###3      | 16669/50000 [9:49:17<16:12:17,  1.75s/it]
Outer training loop:  33%|###3      | 16670/50000 [9:49:19<15:55:41,  1.72s/it]
Outer training loop:  33%|###3      | 16671/50000 [9:49:21<16:31:06,  1.78s/it]
Outer training loop:  33%|###3      | 16672/50000 [9:49:23<16:53:22,  1.82s/it]
Outer training loop:  33%|###3      | 16673/50000 [9:49:24<16:40:18,  1.80s/it]
Outer training loop:  33%|###3      | 16674/50000 [9:49:26<16:55:36,  1.83s/it]
Outer training loop:  33%|###3      | 16675/50000 [9:49:28<16:19:02,  1.76s/it]
Outer training loop:  33%|###3      | 16676/50000 [9:49:29<16:03:55,  1.74s/it]
Outer training loop:  33%|###3      | 16677/50000 [9:49:31<16:21:03,  1.77s/it]
Outer training loop:  33%|###3      | 16678/50000 [9:49:33<16:17:30,  1.76s/it]
Outer training loop:  33%|###3      | 16679/50000 [9:49:35<16:17:44,  1.76s/it]
Outer training loop:  33%|###3      | 16680/50000 [9:49:37<16:41:43,  1.80s/it]
Outer training loop:  33%|###3      | 16681/50000 [9:49:39<16:39:08,  1.80s/it]
Outer training loop:  33%|###3      | 16682/50000 [9:49:40<16:34:23,  1.79s/it]
Outer training loop:  33%|###3      | 16683/50000 [9:49:42<16:31:13,  1.79s/it]
Outer training loop:  33%|###3      | 16684/50000 [9:49:44<16:39:14,  1.80s/it]
Outer training loop:  33%|###3      | 16685/50000 [9:49:46<16:19:11,  1.76s/it]
Outer training loop:  33%|###3      | 16686/50000 [9:49:47<16:28:59,  1.78s/it]
Outer training loop:  33%|###3      | 16687/50000 [9:49:49<16:41:26,  1.80s/it]
Outer training loop:  33%|###3      | 16688/50000 [9:49:51<16:39:38,  1.80s/it]
Outer training loop:  33%|###3      | 16689/50000 [9:49:53<16:31:23,  1.79s/it]
Outer training loop:  33%|###3      | 16690/50000 [9:49:55<16:50:53,  1.82s/it]
Outer training loop:  33%|###3      | 16691/50000 [9:49:57<16:49:22,  1.82s/it]
Outer training loop:  33%|###3      | 16692/50000 [9:49:58<16:44:33,  1.81s/it]
Outer training loop:  33%|###3      | 16693/50000 [9:50:00<16:20:55,  1.77s/it]
Outer training loop:  33%|###3      | 16694/50000 [9:50:02<16:48:15,  1.82s/it]
Outer training loop:  33%|###3      | 16695/50000 [9:50:04<16:37:40,  1.80s/it]
Outer training loop:  33%|###3      | 16696/50000 [9:50:05<16:20:56,  1.77s/it]
Outer training loop:  33%|###3      | 16697/50000 [9:50:07<16:36:15,  1.79s/it]
Outer training loop:  33%|###3      | 16698/50000 [9:50:09<16:29:56,  1.78s/it]
Outer training loop:  33%|###3      | 16699/50000 [9:50:11<16:16:26,  1.76s/it]
Outer training loop:  33%|###3      | 16700/50000 [9:50:13<16:38:49,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.61s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.65s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.68s/it]

Outer training loop:  33%|###3      | 16701/50000 [9:50:19<30:39:07,  3.31s/it]
Outer training loop:  33%|###3      | 16702/50000 [9:50:21<26:16:35,  2.84s/it]
Outer training loop:  33%|###3      | 16703/50000 [9:50:23<23:16:30,  2.52s/it]
Outer training loop:  33%|###3      | 16704/50000 [9:50:25<21:26:27,  2.32s/it]
Outer training loop:  33%|###3      | 16705/50000 [9:50:27<20:10:14,  2.18s/it]
Outer training loop:  33%|###3      | 16706/50000 [9:50:28<18:51:56,  2.04s/it]
Outer training loop:  33%|###3      | 16707/50000 [9:50:30<18:17:01,  1.98s/it]
Outer training loop:  33%|###3      | 16708/50000 [9:50:32<18:12:00,  1.97s/it]
Outer training loop:  33%|###3      | 16709/50000 [9:50:34<17:50:01,  1.93s/it]
Outer training loop:  33%|###3      | 16710/50000 [9:50:36<17:06:15,  1.85s/it]
Outer training loop:  33%|###3      | 16711/50000 [9:50:37<16:26:48,  1.78s/it]
Outer training loop:  33%|###3      | 16712/50000 [9:50:39<16:51:09,  1.82s/it]
Outer training loop:  33%|###3      | 16713/50000 [9:50:41<16:32:08,  1.79s/it]
Outer training loop:  33%|###3      | 16714/50000 [9:50:43<16:24:46,  1.78s/it]
Outer training loop:  33%|###3      | 16715/50000 [9:50:44<16:41:19,  1.81s/it]
Outer training loop:  33%|###3      | 16716/50000 [9:50:46<16:34:42,  1.79s/it]
Outer training loop:  33%|###3      | 16717/50000 [9:50:48<16:22:35,  1.77s/it]
Outer training loop:  33%|###3      | 16718/50000 [9:50:50<17:01:21,  1.84s/it]
Outer training loop:  33%|###3      | 16719/50000 [9:50:52<17:30:50,  1.89s/it]
Outer training loop:  33%|###3      | 16720/50000 [9:50:54<16:54:16,  1.83s/it]
Outer training loop:  33%|###3      | 16721/50000 [9:50:56<17:34:17,  1.90s/it]
Outer training loop:  33%|###3      | 16722/50000 [9:50:57<17:04:30,  1.85s/it]
Outer training loop:  33%|###3      | 16723/50000 [9:50:59<16:43:55,  1.81s/it]
Outer training loop:  33%|###3      | 16724/50000 [9:51:01<16:44:04,  1.81s/it]
Outer training loop:  33%|###3      | 16725/50000 [9:51:03<17:06:43,  1.85s/it]
Outer training loop:  33%|###3      | 16726/50000 [9:51:05<17:03:18,  1.85s/it]
Outer training loop:  33%|###3      | 16727/50000 [9:51:06<16:25:01,  1.78s/it]
Outer training loop:  33%|###3      | 16728/50000 [9:51:08<16:41:08,  1.81s/it]
Outer training loop:  33%|###3      | 16729/50000 [9:51:10<16:28:44,  1.78s/it]
Outer training loop:  33%|###3      | 16730/50000 [9:51:12<16:22:49,  1.77s/it]
Outer training loop:  33%|###3      | 16731/50000 [9:51:14<17:07:33,  1.85s/it]
Outer training loop:  33%|###3      | 16732/50000 [9:51:16<16:43:29,  1.81s/it]
Outer training loop:  33%|###3      | 16733/50000 [9:51:17<16:25:44,  1.78s/it]
Outer training loop:  33%|###3      | 16734/50000 [9:51:19<16:12:55,  1.75s/it]
Outer training loop:  33%|###3      | 16735/50000 [9:51:21<16:36:25,  1.80s/it]
Outer training loop:  33%|###3      | 16736/50000 [9:51:23<16:28:18,  1.78s/it]
Outer training loop:  33%|###3      | 16737/50000 [9:51:24<16:06:08,  1.74s/it]
Outer training loop:  33%|###3      | 16738/50000 [9:51:26<16:38:04,  1.80s/it]
Outer training loop:  33%|###3      | 16739/50000 [9:51:28<16:13:40,  1.76s/it]
Outer training loop:  33%|###3      | 16740/50000 [9:51:30<16:10:45,  1.75s/it]
Outer training loop:  33%|###3      | 16741/50000 [9:51:31<16:06:35,  1.74s/it]
Outer training loop:  33%|###3      | 16742/50000 [9:51:33<16:42:20,  1.81s/it]
Outer training loop:  33%|###3      | 16743/50000 [9:51:35<16:24:04,  1.78s/it]
Outer training loop:  33%|###3      | 16744/50000 [9:51:37<15:56:05,  1.72s/it]
Outer training loop:  33%|###3      | 16745/50000 [9:51:38<16:22:20,  1.77s/it]
Outer training loop:  33%|###3      | 16746/50000 [9:51:40<16:13:17,  1.76s/it]
Outer training loop:  33%|###3      | 16747/50000 [9:51:42<16:06:11,  1.74s/it]
Outer training loop:  33%|###3      | 16748/50000 [9:51:44<16:28:11,  1.78s/it]
Outer training loop:  33%|###3      | 16749/50000 [9:51:45<16:07:54,  1.75s/it]
Outer training loop:  34%|###3      | 16750/50000 [9:51:47<15:53:50,  1.72s/it]
Outer training loop:  34%|###3      | 16751/50000 [9:51:49<15:55:20,  1.72s/it]
Outer training loop:  34%|###3      | 16752/50000 [9:51:51<16:27:23,  1.78s/it]
Outer training loop:  34%|###3      | 16753/50000 [9:51:52<16:13:01,  1.76s/it]
Outer training loop:  34%|###3      | 16754/50000 [9:51:54<16:23:57,  1.78s/it]
Outer training loop:  34%|###3      | 16755/50000 [9:51:56<16:47:40,  1.82s/it]
Outer training loop:  34%|###3      | 16756/50000 [9:51:58<16:19:36,  1.77s/it]
Outer training loop:  34%|###3      | 16757/50000 [9:51:59<15:58:25,  1.73s/it]
Outer training loop:  34%|###3      | 16758/50000 [9:52:01<16:09:36,  1.75s/it]
Outer training loop:  34%|###3      | 16759/50000 [9:52:03<16:26:25,  1.78s/it]
Outer training loop:  34%|###3      | 16760/50000 [9:52:05<16:02:34,  1.74s/it]
Outer training loop:  34%|###3      | 16761/50000 [9:52:06<15:37:22,  1.69s/it]
Outer training loop:  34%|###3      | 16762/50000 [9:52:08<15:48:35,  1.71s/it]
Outer training loop:  34%|###3      | 16763/50000 [9:52:10<15:43:11,  1.70s/it]
Outer training loop:  34%|###3      | 16764/50000 [9:52:11<15:32:19,  1.68s/it]
Outer training loop:  34%|###3      | 16765/50000 [9:52:13<16:07:28,  1.75s/it]
Outer training loop:  34%|###3      | 16766/50000 [9:52:15<15:51:04,  1.72s/it]
Outer training loop:  34%|###3      | 16767/50000 [9:52:17<15:45:12,  1.71s/it]
Outer training loop:  34%|###3      | 16768/50000 [9:52:18<15:42:13,  1.70s/it]
Outer training loop:  34%|###3      | 16769/50000 [9:52:20<16:33:25,  1.79s/it]
Outer training loop:  34%|###3      | 16770/50000 [9:52:22<16:17:21,  1.76s/it]
Outer training loop:  34%|###3      | 16771/50000 [9:52:24<16:09:46,  1.75s/it]
Outer training loop:  34%|###3      | 16772/50000 [9:52:26<16:25:17,  1.78s/it]
Outer training loop:  34%|###3      | 16773/50000 [9:52:27<16:12:01,  1.76s/it]
Outer training loop:  34%|###3      | 16774/50000 [9:52:29<16:14:04,  1.76s/it]
Outer training loop:  34%|###3      | 16775/50000 [9:52:31<16:18:54,  1.77s/it]
Outer training loop:  34%|###3      | 16776/50000 [9:52:33<16:44:55,  1.81s/it]
Outer training loop:  34%|###3      | 16777/50000 [9:52:34<16:28:10,  1.78s/it]
Outer training loop:  34%|###3      | 16778/50000 [9:52:36<16:11:48,  1.76s/it]
Outer training loop:  34%|###3      | 16779/50000 [9:52:38<16:29:24,  1.79s/it]
Outer training loop:  34%|###3      | 16780/50000 [9:52:40<16:31:07,  1.79s/it]
Outer training loop:  34%|###3      | 16781/50000 [9:52:41<16:17:50,  1.77s/it]
Outer training loop:  34%|###3      | 16782/50000 [9:52:43<16:00:44,  1.74s/it]
Outer training loop:  34%|###3      | 16783/50000 [9:52:45<16:24:08,  1.78s/it]
Outer training loop:  34%|###3      | 16784/50000 [9:52:47<16:03:10,  1.74s/it]
Outer training loop:  34%|###3      | 16785/50000 [9:52:48<16:13:09,  1.76s/it]
Outer training loop:  34%|###3      | 16786/50000 [9:52:50<16:44:40,  1.81s/it]
Outer training loop:  34%|###3      | 16787/50000 [9:52:52<16:20:04,  1.77s/it]
Outer training loop:  34%|###3      | 16788/50000 [9:52:54<16:01:53,  1.74s/it]
Outer training loop:  34%|###3      | 16789/50000 [9:52:56<16:25:30,  1.78s/it]
Outer training loop:  34%|###3      | 16790/50000 [9:52:57<16:17:44,  1.77s/it]
Outer training loop:  34%|###3      | 16791/50000 [9:52:59<16:15:55,  1.76s/it]
Outer training loop:  34%|###3      | 16792/50000 [9:53:01<16:12:37,  1.76s/it]
Outer training loop:  34%|###3      | 16793/50000 [9:53:03<16:16:00,  1.76s/it]
Outer training loop:  34%|###3      | 16794/50000 [9:53:04<16:19:27,  1.77s/it]
Outer training loop:  34%|###3      | 16795/50000 [9:53:06<16:15:57,  1.76s/it]
Outer training loop:  34%|###3      | 16796/50000 [9:53:08<16:36:42,  1.80s/it]
Outer training loop:  34%|###3      | 16797/50000 [9:53:10<16:13:52,  1.76s/it]
Outer training loop:  34%|###3      | 16798/50000 [9:53:11<15:58:25,  1.73s/it]
Outer training loop:  34%|###3      | 16799/50000 [9:53:13<15:55:05,  1.73s/it]
Outer training loop:  34%|###3      | 16800/50000 [9:53:15<16:09:58,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.66s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.69s/it]

Outer training loop:  34%|###3      | 16801/50000 [9:53:22<30:28:33,  3.30s/it]
Outer training loop:  34%|###3      | 16802/50000 [9:53:24<25:57:14,  2.81s/it]
Outer training loop:  34%|###3      | 16803/50000 [9:53:25<23:02:05,  2.50s/it]
Outer training loop:  34%|###3      | 16804/50000 [9:53:27<21:19:34,  2.31s/it]
Outer training loop:  34%|###3      | 16805/50000 [9:53:29<19:19:50,  2.10s/it]
Outer training loop:  34%|###3      | 16806/50000 [9:53:31<18:22:08,  1.99s/it]
Outer training loop:  34%|###3      | 16807/50000 [9:53:32<18:04:43,  1.96s/it]
Outer training loop:  34%|###3      | 16808/50000 [9:53:34<17:20:47,  1.88s/it]
Outer training loop:  34%|###3      | 16809/50000 [9:53:36<16:51:50,  1.83s/it]
Outer training loop:  34%|###3      | 16810/50000 [9:53:38<16:48:23,  1.82s/it]
Outer training loop:  34%|###3      | 16811/50000 [9:53:40<17:05:44,  1.85s/it]
Outer training loop:  34%|###3      | 16812/50000 [9:53:41<16:41:12,  1.81s/it]
Outer training loop:  34%|###3      | 16813/50000 [9:53:43<16:30:14,  1.79s/it]
Outer training loop:  34%|###3      | 16814/50000 [9:53:45<16:22:25,  1.78s/it]
Outer training loop:  34%|###3      | 16815/50000 [9:53:47<16:21:50,  1.78s/it]
Outer training loop:  34%|###3      | 16816/50000 [9:53:48<16:51:36,  1.83s/it]
Outer training loop:  34%|###3      | 16817/50000 [9:53:50<16:37:20,  1.80s/it]
Outer training loop:  34%|###3      | 16818/50000 [9:53:52<16:53:16,  1.83s/it]
Outer training loop:  34%|###3      | 16819/50000 [9:53:54<16:19:52,  1.77s/it]
Outer training loop:  34%|###3      | 16820/50000 [9:53:55<15:57:57,  1.73s/it]
Outer training loop:  34%|###3      | 16821/50000 [9:53:57<16:47:47,  1.82s/it]
Outer training loop:  34%|###3      | 16822/50000 [9:53:59<16:47:30,  1.82s/it]
Outer training loop:  34%|###3      | 16823/50000 [9:54:01<16:39:41,  1.81s/it]
Outer training loop:  34%|###3      | 16824/50000 [9:54:03<17:04:27,  1.85s/it]
Outer training loop:  34%|###3      | 16825/50000 [9:54:05<16:53:41,  1.83s/it]
Outer training loop:  34%|###3      | 16826/50000 [9:54:06<16:40:05,  1.81s/it]
Outer training loop:  34%|###3      | 16827/50000 [9:54:08<16:54:44,  1.84s/it]
Outer training loop:  34%|###3      | 16828/50000 [9:54:10<16:25:22,  1.78s/it]
Outer training loop:  34%|###3      | 16829/50000 [9:54:12<16:08:20,  1.75s/it]
Outer training loop:  34%|###3      | 16830/50000 [9:54:13<15:56:35,  1.73s/it]
Outer training loop:  34%|###3      | 16831/50000 [9:54:15<16:08:12,  1.75s/it]
Outer training loop:  34%|###3      | 16832/50000 [9:54:17<16:10:59,  1.76s/it]
Outer training loop:  34%|###3      | 16833/50000 [9:54:19<16:04:17,  1.74s/it]
Outer training loop:  34%|###3      | 16834/50000 [9:54:20<15:54:36,  1.73s/it]
Outer training loop:  34%|###3      | 16835/50000 [9:54:22<16:07:34,  1.75s/it]
Outer training loop:  34%|###3      | 16836/50000 [9:54:24<16:03:34,  1.74s/it]
Outer training loop:  34%|###3      | 16837/50000 [9:54:26<15:54:29,  1.73s/it]
Outer training loop:  34%|###3      | 16838/50000 [9:54:27<16:06:23,  1.75s/it]
Outer training loop:  34%|###3      | 16839/50000 [9:54:29<15:59:40,  1.74s/it]
Outer training loop:  34%|###3      | 16840/50000 [9:54:31<16:05:35,  1.75s/it]
Outer training loop:  34%|###3      | 16841/50000 [9:54:33<16:23:11,  1.78s/it]
Outer training loop:  34%|###3      | 16842/50000 [9:54:35<16:22:03,  1.78s/it]
Outer training loop:  34%|###3      | 16843/50000 [9:54:36<16:13:54,  1.76s/it]
Outer training loop:  34%|###3      | 16844/50000 [9:54:38<16:25:01,  1.78s/it]
Outer training loop:  34%|###3      | 16845/50000 [9:54:40<16:15:50,  1.77s/it]
Outer training loop:  34%|###3      | 16846/50000 [9:54:42<16:12:59,  1.76s/it]
Outer training loop:  34%|###3      | 16847/50000 [9:54:44<16:47:51,  1.82s/it]
Outer training loop:  34%|###3      | 16848/50000 [9:54:45<16:26:39,  1.79s/it]
Outer training loop:  34%|###3      | 16849/50000 [9:54:47<16:18:05,  1.77s/it]
Outer training loop:  34%|###3      | 16850/50000 [9:54:49<16:20:37,  1.77s/it]
Outer training loop:  34%|###3      | 16851/50000 [9:54:51<17:03:59,  1.85s/it]
Outer training loop:  34%|###3      | 16852/50000 [9:54:52<16:28:41,  1.79s/it]
Outer training loop:  34%|###3      | 16853/50000 [9:54:54<15:57:59,  1.73s/it]
Outer training loop:  34%|###3      | 16854/50000 [9:54:56<16:38:13,  1.81s/it]
Outer training loop:  34%|###3      | 16855/50000 [9:54:58<16:15:00,  1.76s/it]
Outer training loop:  34%|###3      | 16856/50000 [9:54:59<16:14:02,  1.76s/it]
Outer training loop:  34%|###3      | 16857/50000 [9:55:01<16:43:19,  1.82s/it]
Outer training loop:  34%|###3      | 16858/50000 [9:55:03<16:21:56,  1.78s/it]
Outer training loop:  34%|###3      | 16859/50000 [9:55:05<16:25:05,  1.78s/it]
Outer training loop:  34%|###3      | 16860/50000 [9:55:07<16:19:42,  1.77s/it]
Outer training loop:  34%|###3      | 16861/50000 [9:55:08<16:34:39,  1.80s/it]
Outer training loop:  34%|###3      | 16862/50000 [9:55:10<16:28:28,  1.79s/it]
Outer training loop:  34%|###3      | 16863/50000 [9:55:12<16:24:39,  1.78s/it]
Outer training loop:  34%|###3      | 16864/50000 [9:55:14<16:30:38,  1.79s/it]
Outer training loop:  34%|###3      | 16865/50000 [9:55:15<16:13:26,  1.76s/it]
Outer training loop:  34%|###3      | 16866/50000 [9:55:18<16:56:30,  1.84s/it]
Outer training loop:  34%|###3      | 16867/50000 [9:55:19<16:24:47,  1.78s/it]
Outer training loop:  34%|###3      | 16868/50000 [9:55:21<16:56:13,  1.84s/it]
Outer training loop:  34%|###3      | 16869/50000 [9:55:23<16:13:52,  1.76s/it]
Outer training loop:  34%|###3      | 16870/50000 [9:55:25<16:16:25,  1.77s/it]
Outer training loop:  34%|###3      | 16871/50000 [9:55:26<16:36:44,  1.81s/it]
Outer training loop:  34%|###3      | 16872/50000 [9:55:28<16:34:52,  1.80s/it]
Outer training loop:  34%|###3      | 16873/50000 [9:55:30<16:08:24,  1.75s/it]
Outer training loop:  34%|###3      | 16874/50000 [9:55:32<16:35:03,  1.80s/it]
Outer training loop:  34%|###3      | 16875/50000 [9:55:34<16:28:07,  1.79s/it]
Outer training loop:  34%|###3      | 16876/50000 [9:55:35<16:17:09,  1.77s/it]
Outer training loop:  34%|###3      | 16877/50000 [9:55:37<16:13:59,  1.76s/it]
Outer training loop:  34%|###3      | 16878/50000 [9:55:39<16:44:02,  1.82s/it]
Outer training loop:  34%|###3      | 16879/50000 [9:55:41<16:34:07,  1.80s/it]
Outer training loop:  34%|###3      | 16880/50000 [9:55:42<16:18:53,  1.77s/it]
Outer training loop:  34%|###3      | 16881/50000 [9:55:44<16:37:56,  1.81s/it]
Outer training loop:  34%|###3      | 16882/50000 [9:55:46<16:14:19,  1.77s/it]
Outer training loop:  34%|###3      | 16883/50000 [9:55:48<15:57:03,  1.73s/it]
Outer training loop:  34%|###3      | 16884/50000 [9:55:50<16:28:37,  1.79s/it]
Outer training loop:  34%|###3      | 16885/50000 [9:55:51<16:37:48,  1.81s/it]
Outer training loop:  34%|###3      | 16886/50000 [9:55:53<16:26:27,  1.79s/it]
Outer training loop:  34%|###3      | 16887/50000 [9:55:55<16:23:32,  1.78s/it]
Outer training loop:  34%|###3      | 16888/50000 [9:55:57<16:54:50,  1.84s/it]
Outer training loop:  34%|###3      | 16889/50000 [9:55:59<16:27:55,  1.79s/it]
Outer training loop:  34%|###3      | 16890/50000 [9:56:00<16:15:40,  1.77s/it]
Outer training loop:  34%|###3      | 16891/50000 [9:56:02<16:39:05,  1.81s/it]
Outer training loop:  34%|###3      | 16892/50000 [9:56:04<16:28:48,  1.79s/it]
Outer training loop:  34%|###3      | 16893/50000 [9:56:06<16:17:57,  1.77s/it]
Outer training loop:  34%|###3      | 16894/50000 [9:56:07<16:08:38,  1.76s/it]
Outer training loop:  34%|###3      | 16895/50000 [9:56:09<16:33:16,  1.80s/it]
Outer training loop:  34%|###3      | 16896/50000 [9:56:11<16:24:25,  1.78s/it]
Outer training loop:  34%|###3      | 16897/50000 [9:56:13<15:56:42,  1.73s/it]
Outer training loop:  34%|###3      | 16898/50000 [9:56:14<16:04:37,  1.75s/it]
Outer training loop:  34%|###3      | 16899/50000 [9:56:16<15:53:15,  1.73s/it]
Outer training loop:  34%|###3      | 16900/50000 [9:56:18<15:59:52,  1.74s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.68s/it]

Outer training loop:  34%|###3      | 16901/50000 [9:56:25<30:40:32,  3.34s/it]
Outer training loop:  34%|###3      | 16902/50000 [9:56:27<26:40:54,  2.90s/it]
Outer training loop:  34%|###3      | 16903/50000 [9:56:29<23:26:38,  2.55s/it]
Outer training loop:  34%|###3      | 16904/50000 [9:56:30<20:59:34,  2.28s/it]
Outer training loop:  34%|###3      | 16905/50000 [9:56:32<19:40:51,  2.14s/it]
Outer training loop:  34%|###3      | 16906/50000 [9:56:34<18:38:56,  2.03s/it]
Outer training loop:  34%|###3      | 16907/50000 [9:56:36<17:57:43,  1.95s/it]
Outer training loop:  34%|###3      | 16908/50000 [9:56:37<17:37:29,  1.92s/it]
Outer training loop:  34%|###3      | 16909/50000 [9:56:39<17:14:32,  1.88s/it]
Outer training loop:  34%|###3      | 16910/50000 [9:56:41<16:45:13,  1.82s/it]
Outer training loop:  34%|###3      | 16911/50000 [9:56:43<16:59:09,  1.85s/it]
Outer training loop:  34%|###3      | 16912/50000 [9:56:45<16:40:14,  1.81s/it]
Outer training loop:  34%|###3      | 16913/50000 [9:56:46<16:55:34,  1.84s/it]
Outer training loop:  34%|###3      | 16914/50000 [9:56:48<16:37:34,  1.81s/it]
Outer training loop:  34%|###3      | 16915/50000 [9:56:50<16:37:13,  1.81s/it]
Outer training loop:  34%|###3      | 16916/50000 [9:56:52<16:18:39,  1.77s/it]
Outer training loop:  34%|###3      | 16917/50000 [9:56:53<15:57:36,  1.74s/it]
Outer training loop:  34%|###3      | 16918/50000 [9:56:55<16:35:06,  1.80s/it]
Outer training loop:  34%|###3      | 16919/50000 [9:56:57<16:25:06,  1.79s/it]
Outer training loop:  34%|###3      | 16920/50000 [9:56:59<16:17:19,  1.77s/it]
Outer training loop:  34%|###3      | 16921/50000 [9:57:00<16:06:36,  1.75s/it]
Outer training loop:  34%|###3      | 16922/50000 [9:57:02<16:26:20,  1.79s/it]
Outer training loop:  34%|###3      | 16923/50000 [9:57:04<16:10:36,  1.76s/it]
Outer training loop:  34%|###3      | 16924/50000 [9:57:06<16:02:07,  1.75s/it]
Outer training loop:  34%|###3      | 16925/50000 [9:57:08<16:42:48,  1.82s/it]
Outer training loop:  34%|###3      | 16926/50000 [9:57:09<16:31:38,  1.80s/it]
Outer training loop:  34%|###3      | 16927/50000 [9:57:11<16:21:01,  1.78s/it]
Outer training loop:  34%|###3      | 16928/50000 [9:57:13<16:55:19,  1.84s/it]
Outer training loop:  34%|###3      | 16929/50000 [9:57:15<16:40:55,  1.82s/it]
Outer training loop:  34%|###3      | 16930/50000 [9:57:17<16:11:46,  1.76s/it]
Outer training loop:  34%|###3      | 16931/50000 [9:57:18<16:03:56,  1.75s/it]
Outer training loop:  34%|###3      | 16932/50000 [9:57:20<16:23:49,  1.79s/it]
Outer training loop:  34%|###3      | 16933/50000 [9:57:22<16:15:20,  1.77s/it]
Outer training loop:  34%|###3      | 16934/50000 [9:57:24<16:07:51,  1.76s/it]
Outer training loop:  34%|###3      | 16935/50000 [9:57:26<16:36:16,  1.81s/it]
Outer training loop:  34%|###3      | 16936/50000 [9:57:27<16:22:16,  1.78s/it]
Outer training loop:  34%|###3      | 16937/50000 [9:57:29<16:10:16,  1.76s/it]
Outer training loop:  34%|###3      | 16938/50000 [9:57:31<16:12:17,  1.76s/it]
Outer training loop:  34%|###3      | 16939/50000 [9:57:32<16:01:18,  1.74s/it]
Outer training loop:  34%|###3      | 16940/50000 [9:57:34<15:54:48,  1.73s/it]
Outer training loop:  34%|###3      | 16941/50000 [9:57:36<15:58:10,  1.74s/it]
Outer training loop:  34%|###3      | 16942/50000 [9:57:38<16:18:21,  1.78s/it]
Outer training loop:  34%|###3      | 16943/50000 [9:57:40<16:16:29,  1.77s/it]
Outer training loop:  34%|###3      | 16944/50000 [9:57:41<16:16:59,  1.77s/it]
Outer training loop:  34%|###3      | 16945/50000 [9:57:43<16:32:07,  1.80s/it]
Outer training loop:  34%|###3      | 16946/50000 [9:57:45<16:26:40,  1.79s/it]
Outer training loop:  34%|###3      | 16947/50000 [9:57:47<16:23:27,  1.79s/it]
Outer training loop:  34%|###3      | 16948/50000 [9:57:49<16:51:50,  1.84s/it]
Outer training loop:  34%|###3      | 16949/50000 [9:57:51<16:54:26,  1.84s/it]
Outer training loop:  34%|###3      | 16950/50000 [9:57:52<16:25:00,  1.79s/it]
Outer training loop:  34%|###3      | 16951/50000 [9:57:54<16:09:28,  1.76s/it]
Outer training loop:  34%|###3      | 16952/50000 [9:57:56<16:33:30,  1.80s/it]
Outer training loop:  34%|###3      | 16953/50000 [9:57:58<16:19:01,  1.78s/it]
Outer training loop:  34%|###3      | 16954/50000 [9:57:59<16:07:41,  1.76s/it]
Outer training loop:  34%|###3      | 16955/50000 [9:58:01<16:25:47,  1.79s/it]
Outer training loop:  34%|###3      | 16956/50000 [9:58:03<16:04:54,  1.75s/it]
Outer training loop:  34%|###3      | 16957/50000 [9:58:05<16:06:16,  1.75s/it]
Outer training loop:  34%|###3      | 16958/50000 [9:58:06<16:27:23,  1.79s/it]
Outer training loop:  34%|###3      | 16959/50000 [9:58:08<16:16:28,  1.77s/it]
Outer training loop:  34%|###3      | 16960/50000 [9:58:10<16:05:57,  1.75s/it]
Outer training loop:  34%|###3      | 16961/50000 [9:58:12<15:56:01,  1.74s/it]
Outer training loop:  34%|###3      | 16962/50000 [9:58:13<16:12:14,  1.77s/it]
Outer training loop:  34%|###3      | 16963/50000 [9:58:15<16:17:12,  1.77s/it]
Outer training loop:  34%|###3      | 16964/50000 [9:58:17<16:23:03,  1.79s/it]
Outer training loop:  34%|###3      | 16965/50000 [9:58:19<16:43:28,  1.82s/it]
Outer training loop:  34%|###3      | 16966/50000 [9:58:21<16:20:35,  1.78s/it]
Outer training loop:  34%|###3      | 16967/50000 [9:58:22<16:26:45,  1.79s/it]
Outer training loop:  34%|###3      | 16968/50000 [9:58:24<16:44:37,  1.82s/it]
Outer training loop:  34%|###3      | 16969/50000 [9:58:26<16:35:00,  1.81s/it]
Outer training loop:  34%|###3      | 16970/50000 [9:58:28<16:22:06,  1.78s/it]
Outer training loop:  34%|###3      | 16971/50000 [9:58:30<16:45:55,  1.83s/it]
Outer training loop:  34%|###3      | 16972/50000 [9:58:31<16:30:33,  1.80s/it]
Outer training loop:  34%|###3      | 16973/50000 [9:58:33<16:10:29,  1.76s/it]
Outer training loop:  34%|###3      | 16974/50000 [9:58:35<16:34:04,  1.81s/it]
Outer training loop:  34%|###3      | 16975/50000 [9:58:37<16:12:31,  1.77s/it]
Outer training loop:  34%|###3      | 16976/50000 [9:58:38<16:06:12,  1.76s/it]
Outer training loop:  34%|###3      | 16977/50000 [9:58:40<16:30:04,  1.80s/it]
Outer training loop:  34%|###3      | 16978/50000 [9:58:42<16:03:20,  1.75s/it]
Outer training loop:  34%|###3      | 16979/50000 [9:58:44<16:00:53,  1.75s/it]
Outer training loop:  34%|###3      | 16980/50000 [9:58:46<16:30:41,  1.80s/it]
Outer training loop:  34%|###3      | 16981/50000 [9:58:47<16:33:19,  1.81s/it]
Outer training loop:  34%|###3      | 16982/50000 [9:58:49<16:33:40,  1.81s/it]
Outer training loop:  34%|###3      | 16983/50000 [9:58:51<16:15:13,  1.77s/it]
Outer training loop:  34%|###3      | 16984/50000 [9:58:53<16:39:12,  1.82s/it]
Outer training loop:  34%|###3      | 16985/50000 [9:58:55<16:28:10,  1.80s/it]
Outer training loop:  34%|###3      | 16986/50000 [9:58:56<16:26:29,  1.79s/it]
Outer training loop:  34%|###3      | 16987/50000 [9:58:58<16:45:51,  1.83s/it]
Outer training loop:  34%|###3      | 16988/50000 [9:59:00<16:44:08,  1.83s/it]
Outer training loop:  34%|###3      | 16989/50000 [9:59:02<16:26:37,  1.79s/it]
Outer training loop:  34%|###3      | 16990/50000 [9:59:04<16:29:57,  1.80s/it]
Outer training loop:  34%|###3      | 16991/50000 [9:59:05<16:07:21,  1.76s/it]
Outer training loop:  34%|###3      | 16992/50000 [9:59:07<15:43:43,  1.72s/it]
Outer training loop:  34%|###3      | 16993/50000 [9:59:09<16:04:13,  1.75s/it]
Outer training loop:  34%|###3      | 16994/50000 [9:59:11<16:00:46,  1.75s/it]
Outer training loop:  34%|###3      | 16995/50000 [9:59:12<15:54:21,  1.73s/it]
Outer training loop:  34%|###3      | 16996/50000 [9:59:14<15:54:16,  1.73s/it]
Outer training loop:  34%|###3      | 16997/50000 [9:59:16<16:12:11,  1.77s/it]
Outer training loop:  34%|###3      | 16998/50000 [9:59:18<16:11:37,  1.77s/it]
Outer training loop:  34%|###3      | 16999/50000 [9:59:19<16:11:00,  1.77s/it]
Outer training loop:  34%|###4      | 17000/50000 [9:59:21<16:33:15,  1.81s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.62s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.68s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:17,  1.67s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:15,  1.67s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:18,  1.74s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:16,  1.73s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:15,  1.73s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.77s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:13,  1.75s/it]

Evaluating policy:  10%|9         | 8/83 [00:13<02:09,  1.73s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:13,  1.80s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:14,  1.84s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:10,  1.82s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.79s/it]

Evaluating policy:  16%|#5        | 13/83 [00:22<02:02,  1.76s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:06,  1.84s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:03,  1.81s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:57,  1.75s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:55,  1.75s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:52,  1.73s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:57,  1.83s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:54,  1.82s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:50,  1.78s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:47,  1.76s/it]

Evaluating policy:  28%|##7       | 23/83 [00:40<01:45,  1.76s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.79s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:39,  1.75s/it]

Evaluating policy:  33%|###2      | 27/83 [00:47<01:39,  1.77s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:35,  1.73s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.79s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.77s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.77s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:29,  1.76s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:27,  1.75s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:24,  1.72s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:25,  1.79s/it]

Evaluating policy:  43%|####3     | 36/83 [01:03<01:23,  1.78s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:20,  1.74s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:17,  1.72s/it]

Evaluating policy:  47%|####6     | 39/83 [01:08<01:16,  1.74s/it]

Evaluating policy:  48%|####8     | 40/83 [01:10<01:12,  1.69s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:15,  1.80s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:13,  1.78s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:15<01:09,  1.73s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:17<01:06,  1.72s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:04,  1.69s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:03,  1.71s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:22<01:01,  1.71s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:24<01:03,  1.81s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<01:00,  1.77s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:57,  1.73s/it]

Evaluating policy:  61%|######1   | 51/83 [01:29<00:55,  1.73s/it]

Evaluating policy:  63%|######2   | 52/83 [01:31<00:54,  1.74s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:51,  1.73s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:50,  1.79s/it]

Evaluating policy:  67%|######7   | 56/83 [01:38<00:47,  1.77s/it]

Evaluating policy:  69%|######8   | 57/83 [01:40<00:46,  1.78s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:43,  1.75s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.78s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:45<00:40,  1.76s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:47<00:38,  1.75s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:36,  1.81s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:33,  1.77s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:54<00:31,  1.73s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:29,  1.71s/it]

Evaluating policy:  81%|########  | 67/83 [01:57<00:27,  1.69s/it]

Evaluating policy:  82%|########1 | 68/83 [01:59<00:25,  1.72s/it]

Evaluating policy:  83%|########3 | 69/83 [02:01<00:23,  1.69s/it]

Evaluating policy:  84%|########4 | 70/83 [02:03<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:22,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:19,  1.80s/it]

Evaluating policy:  88%|########7 | 73/83 [02:08<00:17,  1.76s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:15,  1.77s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:14,  1.78s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.77s/it]

Evaluating policy:  93%|#########2| 77/83 [02:15<00:10,  1.76s/it]

Evaluating policy:  94%|#########3| 78/83 [02:17<00:08,  1.76s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.91s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.84s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 83/83 [02:26<00:00,  1.77s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:18,  1.69s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:18,  1.71s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:15,  1.69s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:15,  1.72s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:10,  1.68s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:22,  1.85s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.83s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:12,  1.79s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:13,  1.83s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:10,  1.81s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:06,  1.78s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:02,  1.75s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:06,  1.83s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:02,  1.80s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:00,  1.80s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.79s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:55,  1.78s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:56,  1.81s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:52,  1.79s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:47,  1.74s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:45,  1.72s/it]

Evaluating policy:  28%|##7       | 23/83 [00:40<01:44,  1.74s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:46,  1.80s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.78s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:43,  1.81s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:38,  1.76s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:35,  1.74s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:38,  1.83s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:35,  1.81s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:33,  1.79s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:31,  1.79s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:27,  1.76s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:24,  1.80s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:22,  1.79s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:18,  1.75s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.73s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.74s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:14,  1.82s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:11,  1.78s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:08,  1.75s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:07,  1.76s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:04,  1.73s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:03,  1.81s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:01,  1.80s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:58,  1.78s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:57,  1.78s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:54,  1.77s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:52,  1.74s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:52,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:50,  1.79s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:48,  1.79s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:45,  1.74s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:43,  1.75s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:41,  1.75s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:39,  1.73s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:38,  1.81s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.76s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:33,  1.77s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:31,  1.74s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.73s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.74s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:25,  1.72s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.73s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:21,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:19,  1.76s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.75s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:15,  1.74s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:13,  1.75s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.78s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.76s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.72s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.84s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.80s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.78s/it]

Outer training loop:  34%|###4      | 17001/50000 [10:04:23<841:29:18, 91.80s/it]
Outer training loop:  34%|###4      | 17002/50000 [10:04:25<593:43:44, 64.77s/it]
Outer training loop:  34%|###4      | 17003/50000 [10:04:26<420:15:38, 45.85s/it]
Outer training loop:  34%|###4      | 17004/50000 [10:04:28<298:51:21, 32.61s/it]
Outer training loop:  34%|###4      | 17005/50000 [10:04:30<213:56:37, 23.34s/it]
Outer training loop:  34%|###4      | 17006/50000 [10:04:32<154:35:55, 16.87s/it]
Outer training loop:  34%|###4      | 17007/50000 [10:04:34<114:23:57, 12.48s/it]
Outer training loop:  34%|###4      | 17008/50000 [10:04:36<84:56:36,  9.27s/it] 
Outer training loop:  34%|###4      | 17009/50000 [10:04:37<63:53:09,  6.97s/it]
Outer training loop:  34%|###4      | 17010/50000 [10:04:39<49:24:57,  5.39s/it]
Outer training loop:  34%|###4      | 17011/50000 [10:04:41<39:45:40,  4.34s/it]
Outer training loop:  34%|###4      | 17012/50000 [10:04:42<32:16:47,  3.52s/it]
Outer training loop:  34%|###4      | 17013/50000 [10:04:44<27:22:43,  2.99s/it]
Outer training loop:  34%|###4      | 17014/50000 [10:04:46<24:13:14,  2.64s/it]
Outer training loop:  34%|###4      | 17015/50000 [10:04:48<21:37:17,  2.36s/it]
Outer training loop:  34%|###4      | 17016/50000 [10:04:49<19:32:27,  2.13s/it]
Outer training loop:  34%|###4      | 17017/50000 [10:04:51<18:55:07,  2.06s/it]
Outer training loop:  34%|###4      | 17018/50000 [10:04:53<18:03:51,  1.97s/it]
Outer training loop:  34%|###4      | 17019/50000 [10:04:55<17:25:17,  1.90s/it]
Outer training loop:  34%|###4      | 17020/50000 [10:04:57<17:03:58,  1.86s/it]
Outer training loop:  34%|###4      | 17021/50000 [10:04:58<16:48:50,  1.84s/it]
Outer training loop:  34%|###4      | 17022/50000 [10:05:00<16:13:45,  1.77s/it]
Outer training loop:  34%|###4      | 17023/50000 [10:05:02<16:35:57,  1.81s/it]
Outer training loop:  34%|###4      | 17024/50000 [10:05:03<16:00:04,  1.75s/it]
Outer training loop:  34%|###4      | 17025/50000 [10:05:05<16:09:56,  1.76s/it]
Outer training loop:  34%|###4      | 17026/50000 [10:05:07<16:20:04,  1.78s/it]
Outer training loop:  34%|###4      | 17027/50000 [10:05:09<16:31:23,  1.80s/it]
Outer training loop:  34%|###4      | 17028/50000 [10:05:11<16:36:42,  1.81s/it]
Outer training loop:  34%|###4      | 17029/50000 [10:05:12<16:28:44,  1.80s/it]
Outer training loop:  34%|###4      | 17030/50000 [10:05:14<16:42:56,  1.83s/it]
Outer training loop:  34%|###4      | 17031/50000 [10:05:16<16:21:25,  1.79s/it]
Outer training loop:  34%|###4      | 17032/50000 [10:05:18<16:07:19,  1.76s/it]
Outer training loop:  34%|###4      | 17033/50000 [10:05:20<16:27:09,  1.80s/it]
Outer training loop:  34%|###4      | 17034/50000 [10:05:21<16:15:12,  1.77s/it]
Outer training loop:  34%|###4      | 17035/50000 [10:05:23<15:55:21,  1.74s/it]
Outer training loop:  34%|###4      | 17036/50000 [10:05:25<16:03:58,  1.75s/it]
Outer training loop:  34%|###4      | 17037/50000 [10:05:27<16:16:22,  1.78s/it]
Outer training loop:  34%|###4      | 17038/50000 [10:05:28<16:12:39,  1.77s/it]
Outer training loop:  34%|###4      | 17039/50000 [10:05:30<15:52:58,  1.73s/it]
Outer training loop:  34%|###4      | 17040/50000 [10:05:32<16:11:05,  1.77s/it]
Outer training loop:  34%|###4      | 17041/50000 [10:05:34<16:07:25,  1.76s/it]
Outer training loop:  34%|###4      | 17042/50000 [10:05:35<16:06:16,  1.76s/it]
Outer training loop:  34%|###4      | 17043/50000 [10:05:37<15:58:12,  1.74s/it]
Outer training loop:  34%|###4      | 17044/50000 [10:05:39<16:36:35,  1.81s/it]
Outer training loop:  34%|###4      | 17045/50000 [10:05:41<16:17:42,  1.78s/it]
Outer training loop:  34%|###4      | 17046/50000 [10:05:42<15:51:37,  1.73s/it]
Outer training loop:  34%|###4      | 17047/50000 [10:05:44<16:24:52,  1.79s/it]
Outer training loop:  34%|###4      | 17048/50000 [10:05:46<15:58:35,  1.75s/it]
Outer training loop:  34%|###4      | 17049/50000 [10:05:48<16:10:23,  1.77s/it]
Outer training loop:  34%|###4      | 17050/50000 [10:05:50<16:26:03,  1.80s/it]
Outer training loop:  34%|###4      | 17051/50000 [10:05:52<16:43:58,  1.83s/it]
Outer training loop:  34%|###4      | 17052/50000 [10:05:53<16:23:43,  1.79s/it]
Outer training loop:  34%|###4      | 17053/50000 [10:05:55<16:01:54,  1.75s/it]
Outer training loop:  34%|###4      | 17054/50000 [10:05:57<16:23:58,  1.79s/it]
Outer training loop:  34%|###4      | 17055/50000 [10:05:59<16:12:22,  1.77s/it]
Outer training loop:  34%|###4      | 17056/50000 [10:06:00<16:14:39,  1.78s/it]
Outer training loop:  34%|###4      | 17057/50000 [10:06:02<16:03:34,  1.75s/it]
Outer training loop:  34%|###4      | 17058/50000 [10:06:04<16:27:51,  1.80s/it]
Outer training loop:  34%|###4      | 17059/50000 [10:06:06<16:28:01,  1.80s/it]
Outer training loop:  34%|###4      | 17060/50000 [10:06:07<16:18:49,  1.78s/it]
Outer training loop:  34%|###4      | 17061/50000 [10:06:09<16:22:57,  1.79s/it]
Outer training loop:  34%|###4      | 17062/50000 [10:06:11<16:14:41,  1.78s/it]
Outer training loop:  34%|###4      | 17063/50000 [10:06:13<15:40:48,  1.71s/it]
Outer training loop:  34%|###4      | 17064/50000 [10:06:15<16:29:13,  1.80s/it]
Outer training loop:  34%|###4      | 17065/50000 [10:06:16<15:47:53,  1.73s/it]
Outer training loop:  34%|###4      | 17066/50000 [10:06:18<15:34:49,  1.70s/it]
Outer training loop:  34%|###4      | 17067/50000 [10:06:19<15:32:41,  1.70s/it]
Outer training loop:  34%|###4      | 17068/50000 [10:06:21<16:00:26,  1.75s/it]
Outer training loop:  34%|###4      | 17069/50000 [10:06:23<15:41:11,  1.71s/it]
Outer training loop:  34%|###4      | 17070/50000 [10:06:25<15:50:02,  1.73s/it]
Outer training loop:  34%|###4      | 17071/50000 [10:06:26<15:47:36,  1.73s/it]
Outer training loop:  34%|###4      | 17072/50000 [10:06:28<16:21:08,  1.79s/it]
Outer training loop:  34%|###4      | 17073/50000 [10:06:30<16:10:06,  1.77s/it]
Outer training loop:  34%|###4      | 17074/50000 [10:06:32<16:00:26,  1.75s/it]
Outer training loop:  34%|###4      | 17075/50000 [10:06:34<16:37:01,  1.82s/it]
Outer training loop:  34%|###4      | 17076/50000 [10:06:36<16:18:32,  1.78s/it]
Outer training loop:  34%|###4      | 17077/50000 [10:06:37<16:07:56,  1.76s/it]
Outer training loop:  34%|###4      | 17078/50000 [10:06:39<15:56:54,  1.74s/it]
Outer training loop:  34%|###4      | 17079/50000 [10:06:41<16:26:00,  1.80s/it]
Outer training loop:  34%|###4      | 17080/50000 [10:06:43<16:15:57,  1.78s/it]
Outer training loop:  34%|###4      | 17081/50000 [10:06:44<15:47:13,  1.73s/it]
Outer training loop:  34%|###4      | 17082/50000 [10:06:46<16:11:36,  1.77s/it]
Outer training loop:  34%|###4      | 17083/50000 [10:06:48<16:03:56,  1.76s/it]
Outer training loop:  34%|###4      | 17084/50000 [10:06:49<15:44:56,  1.72s/it]
Outer training loop:  34%|###4      | 17085/50000 [10:06:51<16:15:53,  1.78s/it]
Outer training loop:  34%|###4      | 17086/50000 [10:06:53<16:09:06,  1.77s/it]
Outer training loop:  34%|###4      | 17087/50000 [10:06:55<15:56:55,  1.74s/it]
Outer training loop:  34%|###4      | 17088/50000 [10:06:57<15:55:47,  1.74s/it]
Outer training loop:  34%|###4      | 17089/50000 [10:06:58<16:11:15,  1.77s/it]
Outer training loop:  34%|###4      | 17090/50000 [10:07:00<16:05:44,  1.76s/it]
Outer training loop:  34%|###4      | 17091/50000 [10:07:02<16:01:37,  1.75s/it]
Outer training loop:  34%|###4      | 17092/50000 [10:07:04<16:37:27,  1.82s/it]
Outer training loop:  34%|###4      | 17093/50000 [10:07:06<16:24:53,  1.80s/it]
Outer training loop:  34%|###4      | 17094/50000 [10:07:07<16:44:13,  1.83s/it]
Outer training loop:  34%|###4      | 17095/50000 [10:07:09<17:02:19,  1.86s/it]
Outer training loop:  34%|###4      | 17096/50000 [10:07:11<16:39:50,  1.82s/it]
Outer training loop:  34%|###4      | 17097/50000 [10:07:13<16:00:20,  1.75s/it]
Outer training loop:  34%|###4      | 17098/50000 [10:07:14<15:57:47,  1.75s/it]
Outer training loop:  34%|###4      | 17099/50000 [10:07:16<16:42:10,  1.83s/it]
Outer training loop:  34%|###4      | 17100/50000 [10:07:18<16:33:08,  1.81s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  34%|###4      | 17101/50000 [10:07:25<30:52:04,  3.38s/it]
Outer training loop:  34%|###4      | 17102/50000 [10:07:27<26:22:29,  2.89s/it]
Outer training loop:  34%|###4      | 17103/50000 [10:07:29<23:44:22,  2.60s/it]
Outer training loop:  34%|###4      | 17104/50000 [10:07:31<21:17:47,  2.33s/it]
Outer training loop:  34%|###4      | 17105/50000 [10:07:32<19:53:13,  2.18s/it]
Outer training loop:  34%|###4      | 17106/50000 [10:07:34<19:17:35,  2.11s/it]
Outer training loop:  34%|###4      | 17107/50000 [10:07:36<18:26:44,  2.02s/it]
Outer training loop:  34%|###4      | 17108/50000 [10:07:38<17:23:12,  1.90s/it]
Outer training loop:  34%|###4      | 17109/50000 [10:07:40<17:23:43,  1.90s/it]
Outer training loop:  34%|###4      | 17110/50000 [10:07:41<16:49:04,  1.84s/it]
Outer training loop:  34%|###4      | 17111/50000 [10:07:43<16:13:28,  1.78s/it]
Outer training loop:  34%|###4      | 17112/50000 [10:07:45<15:56:49,  1.75s/it]
Outer training loop:  34%|###4      | 17113/50000 [10:07:47<16:19:06,  1.79s/it]
Outer training loop:  34%|###4      | 17114/50000 [10:07:48<16:20:19,  1.79s/it]
Outer training loop:  34%|###4      | 17115/50000 [10:07:50<16:14:50,  1.78s/it]
Outer training loop:  34%|###4      | 17116/50000 [10:07:52<16:38:26,  1.82s/it]
Outer training loop:  34%|###4      | 17117/50000 [10:07:54<16:27:42,  1.80s/it]
Outer training loop:  34%|###4      | 17118/50000 [10:07:56<16:12:33,  1.77s/it]
Outer training loop:  34%|###4      | 17119/50000 [10:07:58<16:38:48,  1.82s/it]
Outer training loop:  34%|###4      | 17120/50000 [10:07:59<16:23:36,  1.79s/it]
Outer training loop:  34%|###4      | 17121/50000 [10:08:01<16:10:59,  1.77s/it]
Outer training loop:  34%|###4      | 17122/50000 [10:08:03<16:34:32,  1.81s/it]
Outer training loop:  34%|###4      | 17123/50000 [10:08:05<16:15:50,  1.78s/it]
Outer training loop:  34%|###4      | 17124/50000 [10:08:06<16:15:51,  1.78s/it]
Outer training loop:  34%|###4      | 17125/50000 [10:08:08<16:30:18,  1.81s/it]
Outer training loop:  34%|###4      | 17126/50000 [10:08:10<16:10:27,  1.77s/it]
Outer training loop:  34%|###4      | 17127/50000 [10:08:12<15:45:30,  1.73s/it]
Outer training loop:  34%|###4      | 17128/50000 [10:08:13<15:55:41,  1.74s/it]
Outer training loop:  34%|###4      | 17129/50000 [10:08:15<16:00:06,  1.75s/it]
Outer training loop:  34%|###4      | 17130/50000 [10:08:17<16:03:02,  1.76s/it]
Outer training loop:  34%|###4      | 17131/50000 [10:08:19<15:59:45,  1.75s/it]
Outer training loop:  34%|###4      | 17132/50000 [10:08:21<16:37:16,  1.82s/it]
Outer training loop:  34%|###4      | 17133/50000 [10:08:22<16:35:31,  1.82s/it]
Outer training loop:  34%|###4      | 17134/50000 [10:08:24<16:06:53,  1.77s/it]
Outer training loop:  34%|###4      | 17135/50000 [10:08:26<16:01:20,  1.76s/it]
Outer training loop:  34%|###4      | 17136/50000 [10:08:28<16:34:48,  1.82s/it]
Outer training loop:  34%|###4      | 17137/50000 [10:08:29<16:25:27,  1.80s/it]
Outer training loop:  34%|###4      | 17138/50000 [10:08:31<16:06:19,  1.76s/it]
Outer training loop:  34%|###4      | 17139/50000 [10:08:33<16:19:37,  1.79s/it]
Outer training loop:  34%|###4      | 17140/50000 [10:08:35<16:09:03,  1.77s/it]
Outer training loop:  34%|###4      | 17141/50000 [10:08:37<16:25:36,  1.80s/it]
Outer training loop:  34%|###4      | 17142/50000 [10:08:38<16:21:31,  1.79s/it]
Outer training loop:  34%|###4      | 17143/50000 [10:08:40<16:41:32,  1.83s/it]
Outer training loop:  34%|###4      | 17144/50000 [10:08:42<16:24:46,  1.80s/it]
Outer training loop:  34%|###4      | 17145/50000 [10:08:44<16:21:43,  1.79s/it]
Outer training loop:  34%|###4      | 17146/50000 [10:08:46<16:34:18,  1.82s/it]
Outer training loop:  34%|###4      | 17147/50000 [10:08:47<16:04:42,  1.76s/it]
Outer training loop:  34%|###4      | 17148/50000 [10:08:49<16:07:09,  1.77s/it]
Outer training loop:  34%|###4      | 17149/50000 [10:08:51<16:42:44,  1.83s/it]
Outer training loop:  34%|###4      | 17150/50000 [10:08:53<16:15:39,  1.78s/it]
Outer training loop:  34%|###4      | 17151/50000 [10:08:54<16:09:32,  1.77s/it]
Outer training loop:  34%|###4      | 17152/50000 [10:08:56<16:43:13,  1.83s/it]
Outer training loop:  34%|###4      | 17153/50000 [10:08:58<16:23:51,  1.80s/it]
Outer training loop:  34%|###4      | 17154/50000 [10:09:00<16:10:44,  1.77s/it]
Outer training loop:  34%|###4      | 17155/50000 [10:09:02<15:57:24,  1.75s/it]
Outer training loop:  34%|###4      | 17156/50000 [10:09:03<16:21:20,  1.79s/it]
Outer training loop:  34%|###4      | 17157/50000 [10:09:05<16:21:16,  1.79s/it]
Outer training loop:  34%|###4      | 17158/50000 [10:09:07<16:05:52,  1.76s/it]
Outer training loop:  34%|###4      | 17159/50000 [10:09:09<16:23:19,  1.80s/it]
Outer training loop:  34%|###4      | 17160/50000 [10:09:11<16:07:31,  1.77s/it]
Outer training loop:  34%|###4      | 17161/50000 [10:09:12<16:11:50,  1.78s/it]
Outer training loop:  34%|###4      | 17162/50000 [10:09:14<16:26:17,  1.80s/it]
Outer training loop:  34%|###4      | 17163/50000 [10:09:16<16:14:55,  1.78s/it]
Outer training loop:  34%|###4      | 17164/50000 [10:09:18<16:13:29,  1.78s/it]
Outer training loop:  34%|###4      | 17165/50000 [10:09:19<16:01:13,  1.76s/it]
Outer training loop:  34%|###4      | 17166/50000 [10:09:21<16:36:26,  1.82s/it]
Outer training loop:  34%|###4      | 17167/50000 [10:09:23<16:21:38,  1.79s/it]
Outer training loop:  34%|###4      | 17168/50000 [10:09:25<16:10:19,  1.77s/it]
Outer training loop:  34%|###4      | 17169/50000 [10:09:27<16:35:10,  1.82s/it]
Outer training loop:  34%|###4      | 17170/50000 [10:09:28<16:17:31,  1.79s/it]
Outer training loop:  34%|###4      | 17171/50000 [10:09:30<15:46:15,  1.73s/it]
Outer training loop:  34%|###4      | 17172/50000 [10:09:32<16:02:20,  1.76s/it]
Outer training loop:  34%|###4      | 17173/50000 [10:09:34<15:51:54,  1.74s/it]
Outer training loop:  34%|###4      | 17174/50000 [10:09:35<15:49:21,  1.74s/it]
Outer training loop:  34%|###4      | 17175/50000 [10:09:37<15:34:45,  1.71s/it]
Outer training loop:  34%|###4      | 17176/50000 [10:09:39<16:03:20,  1.76s/it]
Outer training loop:  34%|###4      | 17177/50000 [10:09:41<15:57:11,  1.75s/it]
Outer training loop:  34%|###4      | 17178/50000 [10:09:42<15:53:39,  1.74s/it]
Outer training loop:  34%|###4      | 17179/50000 [10:09:44<15:44:11,  1.73s/it]
Outer training loop:  34%|###4      | 17180/50000 [10:09:46<16:24:24,  1.80s/it]
Outer training loop:  34%|###4      | 17181/50000 [10:09:48<16:02:31,  1.76s/it]
Outer training loop:  34%|###4      | 17182/50000 [10:09:49<16:04:56,  1.76s/it]
Outer training loop:  34%|###4      | 17183/50000 [10:09:51<16:44:40,  1.84s/it]
Outer training loop:  34%|###4      | 17184/50000 [10:09:53<16:27:28,  1.81s/it]
Outer training loop:  34%|###4      | 17185/50000 [10:09:55<16:18:36,  1.79s/it]
Outer training loop:  34%|###4      | 17186/50000 [10:09:57<16:06:02,  1.77s/it]
Outer training loop:  34%|###4      | 17187/50000 [10:09:59<16:39:17,  1.83s/it]
Outer training loop:  34%|###4      | 17188/50000 [10:10:00<16:27:44,  1.81s/it]
Outer training loop:  34%|###4      | 17189/50000 [10:10:02<16:19:27,  1.79s/it]
Outer training loop:  34%|###4      | 17190/50000 [10:10:04<16:56:41,  1.86s/it]
Outer training loop:  34%|###4      | 17191/50000 [10:10:06<17:14:09,  1.89s/it]
Outer training loop:  34%|###4      | 17192/50000 [10:10:08<16:59:09,  1.86s/it]
Outer training loop:  34%|###4      | 17193/50000 [10:10:10<17:07:50,  1.88s/it]
Outer training loop:  34%|###4      | 17194/50000 [10:10:12<16:50:00,  1.85s/it]
Outer training loop:  34%|###4      | 17195/50000 [10:10:13<16:22:24,  1.80s/it]
Outer training loop:  34%|###4      | 17196/50000 [10:10:15<16:20:24,  1.79s/it]
Outer training loop:  34%|###4      | 17197/50000 [10:10:17<16:54:51,  1.86s/it]
Outer training loop:  34%|###4      | 17198/50000 [10:10:19<16:51:46,  1.85s/it]
Outer training loop:  34%|###4      | 17199/50000 [10:10:21<16:42:09,  1.83s/it]
Outer training loop:  34%|###4      | 17200/50000 [10:10:23<17:02:15,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.64s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.67s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.69s/it]

Outer training loop:  34%|###4      | 17201/50000 [10:10:30<31:19:00,  3.44s/it]
Outer training loop:  34%|###4      | 17202/50000 [10:10:31<26:47:06,  2.94s/it]
Outer training loop:  34%|###4      | 17203/50000 [10:10:33<23:19:40,  2.56s/it]
Outer training loop:  34%|###4      | 17204/50000 [10:10:35<21:50:03,  2.40s/it]
Outer training loop:  34%|###4      | 17205/50000 [10:10:37<19:58:34,  2.19s/it]
Outer training loop:  34%|###4      | 17206/50000 [10:10:39<18:41:49,  2.05s/it]
Outer training loop:  34%|###4      | 17207/50000 [10:10:40<17:49:32,  1.96s/it]
Outer training loop:  34%|###4      | 17208/50000 [10:10:42<17:38:52,  1.94s/it]
Outer training loop:  34%|###4      | 17209/50000 [10:10:44<17:05:44,  1.88s/it]
Outer training loop:  34%|###4      | 17210/50000 [10:10:46<16:29:24,  1.81s/it]
Outer training loop:  34%|###4      | 17211/50000 [10:10:48<16:41:52,  1.83s/it]
Outer training loop:  34%|###4      | 17212/50000 [10:10:49<16:30:16,  1.81s/it]
Outer training loop:  34%|###4      | 17213/50000 [10:10:51<16:24:36,  1.80s/it]
Outer training loop:  34%|###4      | 17214/50000 [10:10:53<16:36:00,  1.82s/it]
Outer training loop:  34%|###4      | 17215/50000 [10:10:55<16:55:39,  1.86s/it]
Outer training loop:  34%|###4      | 17216/50000 [10:10:57<16:39:28,  1.83s/it]
Outer training loop:  34%|###4      | 17217/50000 [10:10:58<16:19:09,  1.79s/it]
Outer training loop:  34%|###4      | 17218/50000 [10:11:00<16:44:47,  1.84s/it]
Outer training loop:  34%|###4      | 17219/50000 [10:11:02<16:14:28,  1.78s/it]
Outer training loop:  34%|###4      | 17220/50000 [10:11:04<16:07:57,  1.77s/it]
Outer training loop:  34%|###4      | 17221/50000 [10:11:06<16:30:22,  1.81s/it]
Outer training loop:  34%|###4      | 17222/50000 [10:11:07<15:55:00,  1.75s/it]
Outer training loop:  34%|###4      | 17223/50000 [10:11:09<15:57:04,  1.75s/it]
Outer training loop:  34%|###4      | 17224/50000 [10:11:11<16:29:47,  1.81s/it]
Outer training loop:  34%|###4      | 17225/50000 [10:11:13<16:07:26,  1.77s/it]
Outer training loop:  34%|###4      | 17226/50000 [10:11:14<16:11:40,  1.78s/it]
Outer training loop:  34%|###4      | 17227/50000 [10:11:16<16:40:49,  1.83s/it]
Outer training loop:  34%|###4      | 17228/50000 [10:11:18<16:18:58,  1.79s/it]
Outer training loop:  34%|###4      | 17229/50000 [10:11:20<16:00:01,  1.76s/it]
Outer training loop:  34%|###4      | 17230/50000 [10:11:21<15:51:11,  1.74s/it]
Outer training loop:  34%|###4      | 17231/50000 [10:11:23<15:58:16,  1.75s/it]
Outer training loop:  34%|###4      | 17232/50000 [10:11:25<15:59:32,  1.76s/it]
Outer training loop:  34%|###4      | 17233/50000 [10:11:27<16:00:08,  1.76s/it]
Outer training loop:  34%|###4      | 17234/50000 [10:11:29<16:13:27,  1.78s/it]
Outer training loop:  34%|###4      | 17235/50000 [10:11:30<15:54:08,  1.75s/it]
Outer training loop:  34%|###4      | 17236/50000 [10:11:32<15:36:11,  1.71s/it]
Outer training loop:  34%|###4      | 17237/50000 [10:11:34<15:30:52,  1.70s/it]
Outer training loop:  34%|###4      | 17238/50000 [10:11:36<16:55:45,  1.86s/it]
Outer training loop:  34%|###4      | 17239/50000 [10:11:37<16:33:15,  1.82s/it]
Outer training loop:  34%|###4      | 17240/50000 [10:11:39<16:00:15,  1.76s/it]
Outer training loop:  34%|###4      | 17241/50000 [10:11:41<16:15:43,  1.79s/it]
Outer training loop:  34%|###4      | 17242/50000 [10:11:43<16:03:43,  1.77s/it]
Outer training loop:  34%|###4      | 17243/50000 [10:11:44<16:04:10,  1.77s/it]
Outer training loop:  34%|###4      | 17244/50000 [10:11:46<16:08:27,  1.77s/it]
Outer training loop:  34%|###4      | 17245/50000 [10:11:48<16:25:12,  1.80s/it]
Outer training loop:  34%|###4      | 17246/50000 [10:11:50<16:04:04,  1.77s/it]
Outer training loop:  34%|###4      | 17247/50000 [10:11:51<15:50:09,  1.74s/it]
Outer training loop:  34%|###4      | 17248/50000 [10:11:53<16:03:15,  1.76s/it]
Outer training loop:  34%|###4      | 17249/50000 [10:11:55<15:53:49,  1.75s/it]
Outer training loop:  34%|###4      | 17250/50000 [10:11:57<16:00:17,  1.76s/it]
Outer training loop:  35%|###4      | 17251/50000 [10:11:58<15:36:29,  1.72s/it]
Outer training loop:  35%|###4      | 17252/50000 [10:12:00<16:13:19,  1.78s/it]
Outer training loop:  35%|###4      | 17253/50000 [10:12:02<16:13:53,  1.78s/it]
Outer training loop:  35%|###4      | 17254/50000 [10:12:04<16:13:16,  1.78s/it]
Outer training loop:  35%|###4      | 17255/50000 [10:12:06<16:34:29,  1.82s/it]
Outer training loop:  35%|###4      | 17256/50000 [10:12:08<16:16:54,  1.79s/it]
Outer training loop:  35%|###4      | 17257/50000 [10:12:09<16:03:37,  1.77s/it]
Outer training loop:  35%|###4      | 17258/50000 [10:12:11<15:51:24,  1.74s/it]
Outer training loop:  35%|###4      | 17259/50000 [10:12:13<16:17:42,  1.79s/it]
Outer training loop:  35%|###4      | 17260/50000 [10:12:15<15:57:32,  1.75s/it]
Outer training loop:  35%|###4      | 17261/50000 [10:12:16<15:49:11,  1.74s/it]
Outer training loop:  35%|###4      | 17262/50000 [10:12:18<16:18:58,  1.79s/it]
Outer training loop:  35%|###4      | 17263/50000 [10:12:20<16:07:14,  1.77s/it]
Outer training loop:  35%|###4      | 17264/50000 [10:12:21<15:44:29,  1.73s/it]
Outer training loop:  35%|###4      | 17265/50000 [10:12:23<15:39:15,  1.72s/it]
Outer training loop:  35%|###4      | 17266/50000 [10:12:25<16:10:03,  1.78s/it]
Outer training loop:  35%|###4      | 17267/50000 [10:12:27<15:57:28,  1.76s/it]
Outer training loop:  35%|###4      | 17268/50000 [10:12:28<15:48:15,  1.74s/it]
Outer training loop:  35%|###4      | 17269/50000 [10:12:30<16:14:12,  1.79s/it]
Outer training loop:  35%|###4      | 17270/50000 [10:12:32<16:01:17,  1.76s/it]
Outer training loop:  35%|###4      | 17271/50000 [10:12:34<16:01:10,  1.76s/it]
Outer training loop:  35%|###4      | 17272/50000 [10:12:36<16:25:27,  1.81s/it]
Outer training loop:  35%|###4      | 17273/50000 [10:12:38<16:24:55,  1.81s/it]
Outer training loop:  35%|###4      | 17274/50000 [10:12:39<16:28:04,  1.81s/it]
Outer training loop:  35%|###4      | 17275/50000 [10:12:41<16:09:12,  1.78s/it]
Outer training loop:  35%|###4      | 17276/50000 [10:12:43<16:39:08,  1.83s/it]
Outer training loop:  35%|###4      | 17277/50000 [10:12:45<16:19:17,  1.80s/it]
Outer training loop:  35%|###4      | 17278/50000 [10:12:46<15:48:30,  1.74s/it]
Outer training loop:  35%|###4      | 17279/50000 [10:12:48<16:17:54,  1.79s/it]
Outer training loop:  35%|###4      | 17280/50000 [10:12:50<15:57:17,  1.76s/it]
Outer training loop:  35%|###4      | 17281/50000 [10:12:52<15:44:04,  1.73s/it]
Outer training loop:  35%|###4      | 17282/50000 [10:12:53<15:22:22,  1.69s/it]
Outer training loop:  35%|###4      | 17283/50000 [10:12:55<16:10:01,  1.78s/it]
Outer training loop:  35%|###4      | 17284/50000 [10:12:57<16:17:40,  1.79s/it]
Outer training loop:  35%|###4      | 17285/50000 [10:12:59<16:04:06,  1.77s/it]
Outer training loop:  35%|###4      | 17286/50000 [10:13:01<16:32:34,  1.82s/it]
Outer training loop:  35%|###4      | 17287/50000 [10:13:02<16:17:59,  1.79s/it]
Outer training loop:  35%|###4      | 17288/50000 [10:13:04<16:49:11,  1.85s/it]
Outer training loop:  35%|###4      | 17289/50000 [10:13:06<17:18:13,  1.90s/it]
Outer training loop:  35%|###4      | 17290/50000 [10:13:08<16:52:05,  1.86s/it]
Outer training loop:  35%|###4      | 17291/50000 [10:13:10<16:34:14,  1.82s/it]
Outer training loop:  35%|###4      | 17292/50000 [10:13:12<16:12:48,  1.78s/it]
Outer training loop:  35%|###4      | 17293/50000 [10:13:14<16:38:39,  1.83s/it]
Outer training loop:  35%|###4      | 17294/50000 [10:13:15<16:29:26,  1.82s/it]
Outer training loop:  35%|###4      | 17295/50000 [10:13:17<16:17:15,  1.79s/it]
Outer training loop:  35%|###4      | 17296/50000 [10:13:19<16:42:21,  1.84s/it]
Outer training loop:  35%|###4      | 17297/50000 [10:13:21<16:22:46,  1.80s/it]
Outer training loop:  35%|###4      | 17298/50000 [10:13:22<16:10:11,  1.78s/it]
Outer training loop:  35%|###4      | 17299/50000 [10:13:24<16:35:56,  1.83s/it]
Outer training loop:  35%|###4      | 17300/50000 [10:13:26<16:18:05,  1.79s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.67s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.69s/it]

Outer training loop:  35%|###4      | 17301/50000 [10:13:33<30:29:55,  3.36s/it]
Outer training loop:  35%|###4      | 17302/50000 [10:13:35<26:25:36,  2.91s/it]
Outer training loop:  35%|###4      | 17303/50000 [10:13:37<23:48:41,  2.62s/it]
Outer training loop:  35%|###4      | 17304/50000 [10:13:39<21:16:44,  2.34s/it]
Outer training loop:  35%|###4      | 17305/50000 [10:13:40<19:49:53,  2.18s/it]
Outer training loop:  35%|###4      | 17306/50000 [10:13:42<19:10:55,  2.11s/it]
Outer training loop:  35%|###4      | 17307/50000 [10:13:44<18:12:19,  2.00s/it]
Outer training loop:  35%|###4      | 17308/50000 [10:13:46<17:13:08,  1.90s/it]
Outer training loop:  35%|###4      | 17309/50000 [10:13:48<16:44:42,  1.84s/it]
Outer training loop:  35%|###4      | 17310/50000 [10:13:49<16:47:09,  1.85s/it]
Outer training loop:  35%|###4      | 17311/50000 [10:13:51<16:37:00,  1.83s/it]
Outer training loop:  35%|###4      | 17312/50000 [10:13:53<16:12:32,  1.79s/it]
Outer training loop:  35%|###4      | 17313/50000 [10:13:55<16:32:47,  1.82s/it]
Outer training loop:  35%|###4      | 17314/50000 [10:13:57<16:22:22,  1.80s/it]
Outer training loop:  35%|###4      | 17315/50000 [10:13:58<16:06:50,  1.77s/it]
Outer training loop:  35%|###4      | 17316/50000 [10:14:00<16:08:03,  1.78s/it]
Outer training loop:  35%|###4      | 17317/50000 [10:14:02<16:35:06,  1.83s/it]
Outer training loop:  35%|###4      | 17318/50000 [10:14:04<16:16:31,  1.79s/it]
Outer training loop:  35%|###4      | 17319/50000 [10:14:05<16:04:16,  1.77s/it]
Outer training loop:  35%|###4      | 17320/50000 [10:14:07<16:33:04,  1.82s/it]
Outer training loop:  35%|###4      | 17321/50000 [10:14:09<16:16:38,  1.79s/it]
Outer training loop:  35%|###4      | 17322/50000 [10:14:11<16:01:49,  1.77s/it]
Outer training loop:  35%|###4      | 17323/50000 [10:14:12<15:57:34,  1.76s/it]
Outer training loop:  35%|###4      | 17324/50000 [10:14:14<16:19:06,  1.80s/it]
Outer training loop:  35%|###4      | 17325/50000 [10:14:16<16:08:15,  1.78s/it]
Outer training loop:  35%|###4      | 17326/50000 [10:14:18<15:59:59,  1.76s/it]
Outer training loop:  35%|###4      | 17327/50000 [10:14:20<16:22:15,  1.80s/it]
Outer training loop:  35%|###4      | 17328/50000 [10:14:21<15:49:55,  1.74s/it]
Outer training loop:  35%|###4      | 17329/50000 [10:14:23<15:47:45,  1.74s/it]
Outer training loop:  35%|###4      | 17330/50000 [10:14:25<15:38:17,  1.72s/it]
Outer training loop:  35%|###4      | 17331/50000 [10:14:27<16:32:49,  1.82s/it]
Outer training loop:  35%|###4      | 17332/50000 [10:14:29<16:15:45,  1.79s/it]
Outer training loop:  35%|###4      | 17333/50000 [10:14:30<16:34:26,  1.83s/it]
Outer training loop:  35%|###4      | 17334/50000 [10:14:32<16:09:54,  1.78s/it]
Outer training loop:  35%|###4      | 17335/50000 [10:14:34<16:07:26,  1.78s/it]
Outer training loop:  35%|###4      | 17336/50000 [10:14:36<16:08:03,  1.78s/it]
Outer training loop:  35%|###4      | 17337/50000 [10:14:38<16:35:58,  1.83s/it]
Outer training loop:  35%|###4      | 17338/50000 [10:14:39<16:32:42,  1.82s/it]
Outer training loop:  35%|###4      | 17339/50000 [10:14:41<16:18:21,  1.80s/it]
Outer training loop:  35%|###4      | 17340/50000 [10:14:43<16:32:17,  1.82s/it]
Outer training loop:  35%|###4      | 17341/50000 [10:14:45<16:18:24,  1.80s/it]
Outer training loop:  35%|###4      | 17342/50000 [10:14:47<16:07:35,  1.78s/it]
Outer training loop:  35%|###4      | 17343/50000 [10:14:48<16:15:35,  1.79s/it]
Outer training loop:  35%|###4      | 17344/50000 [10:14:50<16:34:28,  1.83s/it]
Outer training loop:  35%|###4      | 17345/50000 [10:14:52<16:10:29,  1.78s/it]
Outer training loop:  35%|###4      | 17346/50000 [10:14:54<16:11:14,  1.78s/it]
Outer training loop:  35%|###4      | 17347/50000 [10:14:56<16:20:55,  1.80s/it]
Outer training loop:  35%|###4      | 17348/50000 [10:14:57<16:19:15,  1.80s/it]
Outer training loop:  35%|###4      | 17349/50000 [10:14:59<16:13:28,  1.79s/it]
Outer training loop:  35%|###4      | 17350/50000 [10:15:01<16:18:44,  1.80s/it]
Outer training loop:  35%|###4      | 17351/50000 [10:15:03<15:56:32,  1.76s/it]
Outer training loop:  35%|###4      | 17352/50000 [10:15:04<15:39:43,  1.73s/it]
Outer training loop:  35%|###4      | 17353/50000 [10:15:06<16:02:53,  1.77s/it]
Outer training loop:  35%|###4      | 17354/50000 [10:15:08<15:51:43,  1.75s/it]
Outer training loop:  35%|###4      | 17355/50000 [10:15:10<15:49:51,  1.75s/it]
Outer training loop:  35%|###4      | 17356/50000 [10:15:11<16:08:16,  1.78s/it]
Outer training loop:  35%|###4      | 17357/50000 [10:15:13<16:07:47,  1.78s/it]
Outer training loop:  35%|###4      | 17358/50000 [10:15:15<15:45:48,  1.74s/it]
Outer training loop:  35%|###4      | 17359/50000 [10:15:17<15:46:16,  1.74s/it]
Outer training loop:  35%|###4      | 17360/50000 [10:15:18<15:58:40,  1.76s/it]
Outer training loop:  35%|###4      | 17361/50000 [10:15:20<15:55:31,  1.76s/it]
Outer training loop:  35%|###4      | 17362/50000 [10:15:22<15:32:23,  1.71s/it]
Outer training loop:  35%|###4      | 17363/50000 [10:15:24<16:21:37,  1.80s/it]
Outer training loop:  35%|###4      | 17364/50000 [10:15:26<16:34:54,  1.83s/it]
Outer training loop:  35%|###4      | 17365/50000 [10:15:27<16:18:46,  1.80s/it]
Outer training loop:  35%|###4      | 17366/50000 [10:15:29<16:05:19,  1.77s/it]
Outer training loop:  35%|###4      | 17367/50000 [10:15:31<16:13:16,  1.79s/it]
Outer training loop:  35%|###4      | 17368/50000 [10:15:33<15:57:02,  1.76s/it]
Outer training loop:  35%|###4      | 17369/50000 [10:15:34<15:42:48,  1.73s/it]
Outer training loop:  35%|###4      | 17370/50000 [10:15:36<16:07:32,  1.78s/it]
Outer training loop:  35%|###4      | 17371/50000 [10:15:38<15:49:39,  1.75s/it]
Outer training loop:  35%|###4      | 17372/50000 [10:15:39<15:28:42,  1.71s/it]
Outer training loop:  35%|###4      | 17373/50000 [10:15:41<15:22:44,  1.70s/it]
Outer training loop:  35%|###4      | 17374/50000 [10:15:43<15:56:58,  1.76s/it]
Outer training loop:  35%|###4      | 17375/50000 [10:15:45<15:43:47,  1.74s/it]
Outer training loop:  35%|###4      | 17376/50000 [10:15:46<15:46:29,  1.74s/it]
Outer training loop:  35%|###4      | 17377/50000 [10:15:48<16:11:27,  1.79s/it]
Outer training loop:  35%|###4      | 17378/50000 [10:15:50<16:18:21,  1.80s/it]
Outer training loop:  35%|###4      | 17379/50000 [10:15:52<16:14:54,  1.79s/it]
Outer training loop:  35%|###4      | 17380/50000 [10:15:54<16:39:35,  1.84s/it]
Outer training loop:  35%|###4      | 17381/50000 [10:15:56<16:11:12,  1.79s/it]
Outer training loop:  35%|###4      | 17382/50000 [10:15:57<16:07:56,  1.78s/it]
Outer training loop:  35%|###4      | 17383/50000 [10:15:59<15:40:09,  1.73s/it]
Outer training loop:  35%|###4      | 17384/50000 [10:16:01<15:54:56,  1.76s/it]
Outer training loop:  35%|###4      | 17385/50000 [10:16:03<16:16:59,  1.80s/it]
Outer training loop:  35%|###4      | 17386/50000 [10:16:04<15:55:51,  1.76s/it]
Outer training loop:  35%|###4      | 17387/50000 [10:16:06<16:18:52,  1.80s/it]
Outer training loop:  35%|###4      | 17388/50000 [10:16:08<16:05:51,  1.78s/it]
Outer training loop:  35%|###4      | 17389/50000 [10:16:10<15:52:46,  1.75s/it]
Outer training loop:  35%|###4      | 17390/50000 [10:16:11<15:58:24,  1.76s/it]
Outer training loop:  35%|###4      | 17391/50000 [10:16:13<15:57:09,  1.76s/it]
Outer training loop:  35%|###4      | 17392/50000 [10:16:15<15:52:52,  1.75s/it]
Outer training loop:  35%|###4      | 17393/50000 [10:16:17<16:21:00,  1.81s/it]
Outer training loop:  35%|###4      | 17394/50000 [10:16:19<15:59:24,  1.77s/it]
Outer training loop:  35%|###4      | 17395/50000 [10:16:20<16:02:22,  1.77s/it]
Outer training loop:  35%|###4      | 17396/50000 [10:16:22<16:03:43,  1.77s/it]
Outer training loop:  35%|###4      | 17397/50000 [10:16:24<16:18:49,  1.80s/it]
Outer training loop:  35%|###4      | 17398/50000 [10:16:26<16:02:22,  1.77s/it]
Outer training loop:  35%|###4      | 17399/50000 [10:16:27<16:07:00,  1.78s/it]
Outer training loop:  35%|###4      | 17400/50000 [10:16:29<16:40:39,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.65s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  35%|###4      | 17401/50000 [10:16:36<30:32:36,  3.37s/it]
Outer training loop:  35%|###4      | 17402/50000 [10:16:38<25:59:53,  2.87s/it]
Outer training loop:  35%|###4      | 17403/50000 [10:16:40<23:02:43,  2.55s/it]
Outer training loop:  35%|###4      | 17404/50000 [10:16:42<21:30:24,  2.38s/it]
Outer training loop:  35%|###4      | 17405/50000 [10:16:44<19:58:59,  2.21s/it]
Outer training loop:  35%|###4      | 17406/50000 [10:16:45<18:32:02,  2.05s/it]
Outer training loop:  35%|###4      | 17407/50000 [10:16:47<17:41:16,  1.95s/it]
Outer training loop:  35%|###4      | 17408/50000 [10:16:49<17:22:14,  1.92s/it]
Outer training loop:  35%|###4      | 17409/50000 [10:16:51<16:39:38,  1.84s/it]
Outer training loop:  35%|###4      | 17410/50000 [10:16:52<16:26:56,  1.82s/it]
Outer training loop:  35%|###4      | 17411/50000 [10:16:54<16:49:01,  1.86s/it]
Outer training loop:  35%|###4      | 17412/50000 [10:16:56<16:45:37,  1.85s/it]
Outer training loop:  35%|###4      | 17413/50000 [10:16:58<16:28:00,  1.82s/it]
Outer training loop:  35%|###4      | 17414/50000 [10:17:00<16:08:27,  1.78s/it]
Outer training loop:  35%|###4      | 17415/50000 [10:17:01<16:22:25,  1.81s/it]
Outer training loop:  35%|###4      | 17416/50000 [10:17:03<16:15:00,  1.80s/it]
Outer training loop:  35%|###4      | 17417/50000 [10:17:05<16:05:58,  1.78s/it]
Outer training loop:  35%|###4      | 17418/50000 [10:17:07<16:20:37,  1.81s/it]
Outer training loop:  35%|###4      | 17419/50000 [10:17:08<15:57:09,  1.76s/it]
Outer training loop:  35%|###4      | 17420/50000 [10:17:10<15:55:44,  1.76s/it]
Outer training loop:  35%|###4      | 17421/50000 [10:17:12<15:46:14,  1.74s/it]
Outer training loop:  35%|###4      | 17422/50000 [10:17:14<16:19:57,  1.80s/it]
Outer training loop:  35%|###4      | 17423/50000 [10:17:16<16:08:43,  1.78s/it]
Outer training loop:  35%|###4      | 17424/50000 [10:17:17<16:10:09,  1.79s/it]
Outer training loop:  35%|###4      | 17425/50000 [10:17:19<16:22:58,  1.81s/it]
Outer training loop:  35%|###4      | 17426/50000 [10:17:21<16:09:43,  1.79s/it]
Outer training loop:  35%|###4      | 17427/50000 [10:17:23<16:12:12,  1.79s/it]
Outer training loop:  35%|###4      | 17428/50000 [10:17:25<16:06:17,  1.78s/it]
Outer training loop:  35%|###4      | 17429/50000 [10:17:27<16:38:11,  1.84s/it]
Outer training loop:  35%|###4      | 17430/50000 [10:17:28<16:11:05,  1.79s/it]
Outer training loop:  35%|###4      | 17431/50000 [10:17:30<16:06:38,  1.78s/it]
Outer training loop:  35%|###4      | 17432/50000 [10:17:32<16:38:59,  1.84s/it]
Outer training loop:  35%|###4      | 17433/50000 [10:17:34<16:16:02,  1.80s/it]
Outer training loop:  35%|###4      | 17434/50000 [10:17:35<16:13:11,  1.79s/it]
Outer training loop:  35%|###4      | 17435/50000 [10:17:37<15:59:27,  1.77s/it]
Outer training loop:  35%|###4      | 17436/50000 [10:17:39<16:33:17,  1.83s/it]
Outer training loop:  35%|###4      | 17437/50000 [10:17:41<16:17:23,  1.80s/it]
Outer training loop:  35%|###4      | 17438/50000 [10:17:43<16:07:48,  1.78s/it]
Outer training loop:  35%|###4      | 17439/50000 [10:17:45<16:30:51,  1.83s/it]
Outer training loop:  35%|###4      | 17440/50000 [10:17:46<16:15:41,  1.80s/it]
Outer training loop:  35%|###4      | 17441/50000 [10:17:48<16:08:14,  1.78s/it]
Outer training loop:  35%|###4      | 17442/50000 [10:17:50<16:15:08,  1.80s/it]
Outer training loop:  35%|###4      | 17443/50000 [10:17:52<16:28:41,  1.82s/it]
Outer training loop:  35%|###4      | 17444/50000 [10:17:54<16:29:36,  1.82s/it]
Outer training loop:  35%|###4      | 17445/50000 [10:17:55<16:18:36,  1.80s/it]
Outer training loop:  35%|###4      | 17446/50000 [10:17:57<16:48:20,  1.86s/it]
Outer training loop:  35%|###4      | 17447/50000 [10:17:59<16:26:21,  1.82s/it]
Outer training loop:  35%|###4      | 17448/50000 [10:18:01<16:10:34,  1.79s/it]
Outer training loop:  35%|###4      | 17449/50000 [10:18:02<16:02:05,  1.77s/it]
Outer training loop:  35%|###4      | 17450/50000 [10:18:04<15:40:57,  1.73s/it]
Outer training loop:  35%|###4      | 17451/50000 [10:18:06<15:25:33,  1.71s/it]
Outer training loop:  35%|###4      | 17452/50000 [10:18:07<15:22:33,  1.70s/it]
Outer training loop:  35%|###4      | 17453/50000 [10:18:09<15:46:21,  1.74s/it]
Outer training loop:  35%|###4      | 17454/50000 [10:18:11<15:54:20,  1.76s/it]
Outer training loop:  35%|###4      | 17455/50000 [10:18:13<15:55:39,  1.76s/it]
Outer training loop:  35%|###4      | 17456/50000 [10:18:15<16:43:32,  1.85s/it]
Outer training loop:  35%|###4      | 17457/50000 [10:18:17<16:20:30,  1.81s/it]
Outer training loop:  35%|###4      | 17458/50000 [10:18:18<16:17:28,  1.80s/it]
Outer training loop:  35%|###4      | 17459/50000 [10:18:20<16:04:13,  1.78s/it]
Outer training loop:  35%|###4      | 17460/50000 [10:18:22<16:12:41,  1.79s/it]
Outer training loop:  35%|###4      | 17461/50000 [10:18:24<16:01:03,  1.77s/it]
Outer training loop:  35%|###4      | 17462/50000 [10:18:25<15:57:59,  1.77s/it]
Outer training loop:  35%|###4      | 17463/50000 [10:18:27<16:39:22,  1.84s/it]
Outer training loop:  35%|###4      | 17464/50000 [10:18:29<16:24:24,  1.82s/it]
Outer training loop:  35%|###4      | 17465/50000 [10:18:31<16:11:04,  1.79s/it]
Outer training loop:  35%|###4      | 17466/50000 [10:18:33<15:59:22,  1.77s/it]
Outer training loop:  35%|###4      | 17467/50000 [10:18:35<16:18:34,  1.80s/it]
Outer training loop:  35%|###4      | 17468/50000 [10:18:36<15:55:48,  1.76s/it]
Outer training loop:  35%|###4      | 17469/50000 [10:18:38<16:04:03,  1.78s/it]
Outer training loop:  35%|###4      | 17470/50000 [10:18:40<16:28:53,  1.82s/it]
Outer training loop:  35%|###4      | 17471/50000 [10:18:42<16:23:54,  1.81s/it]
Outer training loop:  35%|###4      | 17472/50000 [10:18:43<15:53:43,  1.76s/it]
Outer training loop:  35%|###4      | 17473/50000 [10:18:45<16:22:26,  1.81s/it]
Outer training loop:  35%|###4      | 17474/50000 [10:18:47<15:59:13,  1.77s/it]
Outer training loop:  35%|###4      | 17475/50000 [10:18:49<16:09:18,  1.79s/it]
Outer training loop:  35%|###4      | 17476/50000 [10:18:51<16:11:17,  1.79s/it]
Outer training loop:  35%|###4      | 17477/50000 [10:18:52<16:18:08,  1.80s/it]
Outer training loop:  35%|###4      | 17478/50000 [10:18:54<16:03:39,  1.78s/it]
Outer training loop:  35%|###4      | 17479/50000 [10:18:56<16:03:44,  1.78s/it]
Outer training loop:  35%|###4      | 17480/50000 [10:18:58<16:16:26,  1.80s/it]
Outer training loop:  35%|###4      | 17481/50000 [10:18:59<15:56:27,  1.76s/it]
Outer training loop:  35%|###4      | 17482/50000 [10:19:01<16:31:13,  1.83s/it]
Outer training loop:  35%|###4      | 17483/50000 [10:19:03<16:58:25,  1.88s/it]
Outer training loop:  35%|###4      | 17484/50000 [10:19:05<16:38:39,  1.84s/it]
Outer training loop:  35%|###4      | 17485/50000 [10:19:07<16:18:27,  1.81s/it]
Outer training loop:  35%|###4      | 17486/50000 [10:19:09<16:09:15,  1.79s/it]
Outer training loop:  35%|###4      | 17487/50000 [10:19:11<16:36:52,  1.84s/it]
Outer training loop:  35%|###4      | 17488/50000 [10:19:12<16:02:21,  1.78s/it]
Outer training loop:  35%|###4      | 17489/50000 [10:19:14<15:46:58,  1.75s/it]
Outer training loop:  35%|###4      | 17490/50000 [10:19:16<16:27:07,  1.82s/it]
Outer training loop:  35%|###4      | 17491/50000 [10:19:18<16:04:43,  1.78s/it]
Outer training loop:  35%|###4      | 17492/50000 [10:19:19<15:41:06,  1.74s/it]
Outer training loop:  35%|###4      | 17493/50000 [10:19:21<16:21:25,  1.81s/it]
Outer training loop:  35%|###4      | 17494/50000 [10:19:23<16:28:43,  1.83s/it]
Outer training loop:  35%|###4      | 17495/50000 [10:19:25<16:13:50,  1.80s/it]
Outer training loop:  35%|###4      | 17496/50000 [10:19:27<16:04:15,  1.78s/it]
Outer training loop:  35%|###4      | 17497/50000 [10:19:28<16:19:17,  1.81s/it]
Outer training loop:  35%|###4      | 17498/50000 [10:19:30<16:02:26,  1.78s/it]
Outer training loop:  35%|###4      | 17499/50000 [10:19:32<15:49:03,  1.75s/it]
Outer training loop:  35%|###5      | 17500/50000 [10:19:34<16:05:02,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:17,  1.68s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:17,  1.70s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:21,  1.77s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:18,  1.76s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.76s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:19,  1.81s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:14,  1.77s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:12,  1.76s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:10,  1.77s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:10,  1.78s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:08,  1.79s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:06,  1.78s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:02,  1.74s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:04,  1.81s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:01,  1.78s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:55,  1.72s/it]

Evaluating policy:  20%|##        | 17/83 [00:29<01:53,  1.72s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:49,  1.69s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:53,  1.77s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:51,  1.76s/it]

Evaluating policy:  25%|##5       | 21/83 [00:36<01:47,  1.73s/it]

Evaluating policy:  27%|##6       | 22/83 [00:38<01:45,  1.73s/it]

Evaluating policy:  28%|##7       | 23/83 [00:40<01:44,  1.74s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:46,  1.80s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:45,  1.81s/it]

Evaluating policy:  31%|###1      | 26/83 [00:45<01:42,  1.79s/it]

Evaluating policy:  33%|###2      | 27/83 [00:47<01:40,  1.80s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:40,  1.83s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:35,  1.81s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:33,  1.81s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:30,  1.78s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:29,  1.80s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:25,  1.75s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:27,  1.83s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:25,  1.81s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:20,  1.75s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:18,  1.74s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:18,  1.78s/it]

Evaluating policy:  48%|####8     | 40/83 [01:10<01:14,  1.73s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:11,  1.71s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:15,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:12,  1.80s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.78s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:05,  1.73s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:02,  1.69s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:01,  1.70s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:24<00:59,  1.71s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<01:01,  1.82s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:57,  1.79s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:55,  1.78s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:52,  1.75s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:49,  1.75s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:50,  1.86s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:44,  1.79s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.75s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.77s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:38,  1.76s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:36,  1.76s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:31,  1.76s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.75s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:27,  1.73s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:26,  1.75s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:23,  1.71s/it]

Evaluating policy:  84%|########4 | 70/83 [02:03<00:22,  1.73s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:20,  1.72s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.80s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:16,  1.78s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:14,  1.77s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.75s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.75s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.78s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.80s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.78s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.91s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.79s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:16,  1.66s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:14,  1.66s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:14,  1.68s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:14,  1.71s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:11,  1.69s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:11,  1.71s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:12,  1.74s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:22,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:17,  1.86s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:13,  1.83s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:11,  1.82s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:11,  1.86s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:05,  1.80s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:02,  1.78s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:00,  1.77s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:03,  1.85s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<02:00,  1.82s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:52,  1.75s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:55,  1.83s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:51,  1.79s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:47,  1.76s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:46,  1.78s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:44,  1.76s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:46,  1.84s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:42,  1.80s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:39,  1.77s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:37,  1.77s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:34,  1.75s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:32,  1.74s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:34,  1.81s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:31,  1.80s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:28,  1.77s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:26,  1.77s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:24,  1.75s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:22,  1.76s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:20,  1.79s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.74s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:15,  1.76s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:13,  1.74s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:11,  1.75s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:11,  1.83s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:08,  1.84s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:05,  1.81s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:01,  1.77s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<00:59,  1.75s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<01:01,  1.85s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:58,  1.84s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:55,  1.80s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:53,  1.77s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:50,  1.73s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:48,  1.74s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:47,  1.75s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:45,  1.75s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:41,  1.80s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:39,  1.78s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:37,  1.77s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:34,  1.74s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:32,  1.72s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:30,  1.71s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:29,  1.85s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:27,  1.81s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:25,  1.80s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:22,  1.76s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:21,  1.75s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:19,  1.73s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.74s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:15,  1.73s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:15,  1.89s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.82s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.80s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.79s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.82s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.81s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.77s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.80s/it]

Outer training loop:  35%|###5      | 17501/50000 [10:24:37<832:01:20, 92.17s/it]
Outer training loop:  35%|###5      | 17502/50000 [10:24:39<589:07:54, 65.26s/it]
Outer training loop:  35%|###5      | 17503/50000 [10:24:41<417:06:09, 46.21s/it]
Outer training loop:  35%|###5      | 17504/50000 [10:24:43<296:49:49, 32.88s/it]
Outer training loop:  35%|###5      | 17505/50000 [10:24:45<212:52:47, 23.58s/it]
Outer training loop:  35%|###5      | 17506/50000 [10:24:47<154:28:07, 17.11s/it]
Outer training loop:  35%|###5      | 17507/50000 [10:24:49<113:16:38, 12.55s/it]
Outer training loop:  35%|###5      | 17508/50000 [10:24:50<83:59:13,  9.31s/it] 
Outer training loop:  35%|###5      | 17509/50000 [10:24:52<63:40:59,  7.06s/it]
Outer training loop:  35%|###5      | 17510/50000 [10:24:54<49:23:13,  5.47s/it]
Outer training loop:  35%|###5      | 17511/50000 [10:24:56<39:11:36,  4.34s/it]
Outer training loop:  35%|###5      | 17512/50000 [10:24:57<32:29:31,  3.60s/it]
Outer training loop:  35%|###5      | 17513/50000 [10:25:00<28:15:01,  3.13s/it]
Outer training loop:  35%|###5      | 17514/50000 [10:25:01<24:42:32,  2.74s/it]
Outer training loop:  35%|###5      | 17515/50000 [10:25:03<21:55:11,  2.43s/it]
Outer training loop:  35%|###5      | 17516/50000 [10:25:05<20:17:20,  2.25s/it]
Outer training loop:  35%|###5      | 17517/50000 [10:25:07<18:44:15,  2.08s/it]
Outer training loop:  35%|###5      | 17518/50000 [10:25:08<18:08:10,  2.01s/it]
Outer training loop:  35%|###5      | 17519/50000 [10:25:10<17:31:08,  1.94s/it]
Outer training loop:  35%|###5      | 17520/50000 [10:25:12<17:18:23,  1.92s/it]
Outer training loop:  35%|###5      | 17521/50000 [10:25:14<17:01:11,  1.89s/it]
Outer training loop:  35%|###5      | 17522/50000 [10:25:16<16:29:14,  1.83s/it]
Outer training loop:  35%|###5      | 17523/50000 [10:25:17<16:38:22,  1.84s/it]
Outer training loop:  35%|###5      | 17524/50000 [10:25:19<16:49:44,  1.87s/it]
Outer training loop:  35%|###5      | 17525/50000 [10:25:21<16:34:15,  1.84s/it]
Outer training loop:  35%|###5      | 17526/50000 [10:25:23<16:12:59,  1.80s/it]
Outer training loop:  35%|###5      | 17527/50000 [10:25:25<16:51:09,  1.87s/it]
Outer training loop:  35%|###5      | 17528/50000 [10:25:27<16:26:38,  1.82s/it]
Outer training loop:  35%|###5      | 17529/50000 [10:25:28<16:19:30,  1.81s/it]
Outer training loop:  35%|###5      | 17530/50000 [10:25:30<16:35:02,  1.84s/it]
Outer training loop:  35%|###5      | 17531/50000 [10:25:32<16:17:13,  1.81s/it]
Outer training loop:  35%|###5      | 17532/50000 [10:25:34<16:05:03,  1.78s/it]
Outer training loop:  35%|###5      | 17533/50000 [10:25:36<16:17:07,  1.81s/it]
Outer training loop:  35%|###5      | 17534/50000 [10:25:38<16:50:58,  1.87s/it]
Outer training loop:  35%|###5      | 17535/50000 [10:25:39<16:28:37,  1.83s/it]
Outer training loop:  35%|###5      | 17536/50000 [10:25:41<16:20:33,  1.81s/it]
Outer training loop:  35%|###5      | 17537/50000 [10:25:43<16:43:12,  1.85s/it]
Outer training loop:  35%|###5      | 17538/50000 [10:25:45<16:19:05,  1.81s/it]
Outer training loop:  35%|###5      | 17539/50000 [10:25:47<16:11:40,  1.80s/it]
Outer training loop:  35%|###5      | 17540/50000 [10:25:49<16:49:20,  1.87s/it]
Outer training loop:  35%|###5      | 17541/50000 [10:25:50<16:38:20,  1.85s/it]
Outer training loop:  35%|###5      | 17542/50000 [10:25:52<16:28:50,  1.83s/it]
Outer training loop:  35%|###5      | 17543/50000 [10:25:54<16:49:18,  1.87s/it]
Outer training loop:  35%|###5      | 17544/50000 [10:25:56<16:31:23,  1.83s/it]
Outer training loop:  35%|###5      | 17545/50000 [10:25:58<16:04:18,  1.78s/it]
Outer training loop:  35%|###5      | 17546/50000 [10:25:59<15:46:58,  1.75s/it]
Outer training loop:  35%|###5      | 17547/50000 [10:26:01<16:15:16,  1.80s/it]
Outer training loop:  35%|###5      | 17548/50000 [10:26:03<15:47:08,  1.75s/it]
Outer training loop:  35%|###5      | 17549/50000 [10:26:05<15:54:05,  1.76s/it]
Outer training loop:  35%|###5      | 17550/50000 [10:26:06<16:13:47,  1.80s/it]
Outer training loop:  35%|###5      | 17551/50000 [10:26:08<16:13:31,  1.80s/it]
Outer training loop:  35%|###5      | 17552/50000 [10:26:10<16:09:30,  1.79s/it]
Outer training loop:  35%|###5      | 17553/50000 [10:26:12<16:16:21,  1.81s/it]
Outer training loop:  35%|###5      | 17554/50000 [10:26:13<15:37:18,  1.73s/it]
Outer training loop:  35%|###5      | 17555/50000 [10:26:15<15:25:44,  1.71s/it]
Outer training loop:  35%|###5      | 17556/50000 [10:26:17<15:41:32,  1.74s/it]
Outer training loop:  35%|###5      | 17557/50000 [10:26:19<15:59:51,  1.78s/it]
Outer training loop:  35%|###5      | 17558/50000 [10:26:21<15:59:02,  1.77s/it]
Outer training loop:  35%|###5      | 17559/50000 [10:26:22<15:51:34,  1.76s/it]
Outer training loop:  35%|###5      | 17560/50000 [10:26:24<16:12:18,  1.80s/it]
Outer training loop:  35%|###5      | 17561/50000 [10:26:26<16:05:51,  1.79s/it]
Outer training loop:  35%|###5      | 17562/50000 [10:26:27<15:31:56,  1.72s/it]
Outer training loop:  35%|###5      | 17563/50000 [10:26:30<16:30:39,  1.83s/it]
Outer training loop:  35%|###5      | 17564/50000 [10:26:31<16:12:09,  1.80s/it]
Outer training loop:  35%|###5      | 17565/50000 [10:26:33<16:14:36,  1.80s/it]
Outer training loop:  35%|###5      | 17566/50000 [10:26:35<16:02:21,  1.78s/it]
Outer training loop:  35%|###5      | 17567/50000 [10:26:37<16:14:49,  1.80s/it]
Outer training loop:  35%|###5      | 17568/50000 [10:26:38<16:04:59,  1.79s/it]
Outer training loop:  35%|###5      | 17569/50000 [10:26:40<16:07:26,  1.79s/it]
Outer training loop:  35%|###5      | 17570/50000 [10:26:42<16:20:01,  1.81s/it]
Outer training loop:  35%|###5      | 17571/50000 [10:26:44<16:01:16,  1.78s/it]
Outer training loop:  35%|###5      | 17572/50000 [10:26:46<15:58:28,  1.77s/it]
Outer training loop:  35%|###5      | 17573/50000 [10:26:47<15:54:48,  1.77s/it]
Outer training loop:  35%|###5      | 17574/50000 [10:26:49<16:16:59,  1.81s/it]
Outer training loop:  35%|###5      | 17575/50000 [10:26:51<16:17:07,  1.81s/it]
Outer training loop:  35%|###5      | 17576/50000 [10:26:53<15:50:29,  1.76s/it]
Outer training loop:  35%|###5      | 17577/50000 [10:26:55<16:20:10,  1.81s/it]
Outer training loop:  35%|###5      | 17578/50000 [10:26:56<16:07:57,  1.79s/it]
Outer training loop:  35%|###5      | 17579/50000 [10:26:58<15:33:30,  1.73s/it]
Outer training loop:  35%|###5      | 17580/50000 [10:27:00<16:01:31,  1.78s/it]
Outer training loop:  35%|###5      | 17581/50000 [10:27:01<15:46:43,  1.75s/it]
Outer training loop:  35%|###5      | 17582/50000 [10:27:03<15:36:50,  1.73s/it]
Outer training loop:  35%|###5      | 17583/50000 [10:27:05<15:31:05,  1.72s/it]
Outer training loop:  35%|###5      | 17584/50000 [10:27:07<15:49:25,  1.76s/it]
Outer training loop:  35%|###5      | 17585/50000 [10:27:09<16:01:39,  1.78s/it]
Outer training loop:  35%|###5      | 17586/50000 [10:27:10<15:51:32,  1.76s/it]
Outer training loop:  35%|###5      | 17587/50000 [10:27:12<16:00:08,  1.78s/it]
Outer training loop:  35%|###5      | 17588/50000 [10:27:14<15:56:36,  1.77s/it]
Outer training loop:  35%|###5      | 17589/50000 [10:27:15<15:30:59,  1.72s/it]
Outer training loop:  35%|###5      | 17590/50000 [10:27:17<15:36:25,  1.73s/it]
Outer training loop:  35%|###5      | 17591/50000 [10:27:19<15:57:48,  1.77s/it]
Outer training loop:  35%|###5      | 17592/50000 [10:27:21<16:07:30,  1.79s/it]
Outer training loop:  35%|###5      | 17593/50000 [10:27:23<15:50:25,  1.76s/it]
Outer training loop:  35%|###5      | 17594/50000 [10:27:24<15:36:26,  1.73s/it]
Outer training loop:  35%|###5      | 17595/50000 [10:27:26<16:12:12,  1.80s/it]
Outer training loop:  35%|###5      | 17596/50000 [10:27:28<15:55:44,  1.77s/it]
Outer training loop:  35%|###5      | 17597/50000 [10:27:30<15:41:30,  1.74s/it]
Outer training loop:  35%|###5      | 17598/50000 [10:27:31<15:57:23,  1.77s/it]
Outer training loop:  35%|###5      | 17599/50000 [10:27:33<15:49:59,  1.76s/it]
Outer training loop:  35%|###5      | 17600/50000 [10:27:35<15:42:00,  1.74s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.68s/it]

Outer training loop:  35%|###5      | 17601/50000 [10:27:42<29:51:49,  3.32s/it]
Outer training loop:  35%|###5      | 17602/50000 [10:27:44<25:58:52,  2.89s/it]
Outer training loop:  35%|###5      | 17603/50000 [10:27:46<22:56:13,  2.55s/it]
Outer training loop:  35%|###5      | 17604/50000 [10:27:47<20:38:19,  2.29s/it]
Outer training loop:  35%|###5      | 17605/50000 [10:27:49<19:42:20,  2.19s/it]
Outer training loop:  35%|###5      | 17606/50000 [10:27:51<18:31:57,  2.06s/it]
Outer training loop:  35%|###5      | 17607/50000 [10:27:53<17:31:30,  1.95s/it]
Outer training loop:  35%|###5      | 17608/50000 [10:27:55<17:42:14,  1.97s/it]
Outer training loop:  35%|###5      | 17609/50000 [10:27:56<16:49:39,  1.87s/it]
Outer training loop:  35%|###5      | 17610/50000 [10:27:58<16:40:44,  1.85s/it]
Outer training loop:  35%|###5      | 17611/50000 [10:28:00<16:59:09,  1.89s/it]
Outer training loop:  35%|###5      | 17612/50000 [10:28:02<16:26:30,  1.83s/it]
Outer training loop:  35%|###5      | 17613/50000 [10:28:03<16:09:00,  1.80s/it]
Outer training loop:  35%|###5      | 17614/50000 [10:28:05<16:04:28,  1.79s/it]
Outer training loop:  35%|###5      | 17615/50000 [10:28:07<16:23:12,  1.82s/it]
Outer training loop:  35%|###5      | 17616/50000 [10:28:09<15:59:33,  1.78s/it]
Outer training loop:  35%|###5      | 17617/50000 [10:28:11<16:08:34,  1.79s/it]
Outer training loop:  35%|###5      | 17618/50000 [10:28:13<16:25:51,  1.83s/it]
Outer training loop:  35%|###5      | 17619/50000 [10:28:14<16:06:13,  1.79s/it]
Outer training loop:  35%|###5      | 17620/50000 [10:28:16<15:55:08,  1.77s/it]
Outer training loop:  35%|###5      | 17621/50000 [10:28:18<15:56:06,  1.77s/it]
Outer training loop:  35%|###5      | 17622/50000 [10:28:20<16:28:14,  1.83s/it]
Outer training loop:  35%|###5      | 17623/50000 [10:28:21<16:09:58,  1.80s/it]
Outer training loop:  35%|###5      | 17624/50000 [10:28:23<16:02:58,  1.78s/it]
Outer training loop:  35%|###5      | 17625/50000 [10:28:25<16:24:42,  1.82s/it]
Outer training loop:  35%|###5      | 17626/50000 [10:28:27<15:51:01,  1.76s/it]
Outer training loop:  35%|###5      | 17627/50000 [10:28:28<15:37:08,  1.74s/it]
Outer training loop:  35%|###5      | 17628/50000 [10:28:30<15:53:01,  1.77s/it]
Outer training loop:  35%|###5      | 17629/50000 [10:28:32<15:41:33,  1.75s/it]
Outer training loop:  35%|###5      | 17630/50000 [10:28:34<15:29:20,  1.72s/it]
Outer training loop:  35%|###5      | 17631/50000 [10:28:35<15:52:43,  1.77s/it]
Outer training loop:  35%|###5      | 17632/50000 [10:28:37<15:51:22,  1.76s/it]
Outer training loop:  35%|###5      | 17633/50000 [10:28:39<15:31:09,  1.73s/it]
Outer training loop:  35%|###5      | 17634/50000 [10:28:41<16:00:14,  1.78s/it]
Outer training loop:  35%|###5      | 17635/50000 [10:28:42<15:53:10,  1.77s/it]
Outer training loop:  35%|###5      | 17636/50000 [10:28:44<15:32:00,  1.73s/it]
Outer training loop:  35%|###5      | 17637/50000 [10:28:46<15:21:12,  1.71s/it]
Outer training loop:  35%|###5      | 17638/50000 [10:28:48<15:50:33,  1.76s/it]
Outer training loop:  35%|###5      | 17639/50000 [10:28:49<15:43:26,  1.75s/it]
Outer training loop:  35%|###5      | 17640/50000 [10:28:51<15:31:50,  1.73s/it]
Outer training loop:  35%|###5      | 17641/50000 [10:28:53<15:59:22,  1.78s/it]
Outer training loop:  35%|###5      | 17642/50000 [10:28:55<15:29:32,  1.72s/it]
Outer training loop:  35%|###5      | 17643/50000 [10:28:56<15:51:18,  1.76s/it]
Outer training loop:  35%|###5      | 17644/50000 [10:28:58<16:13:13,  1.80s/it]
Outer training loop:  35%|###5      | 17645/50000 [10:29:00<16:03:44,  1.79s/it]
Outer training loop:  35%|###5      | 17646/50000 [10:29:02<15:56:45,  1.77s/it]
Outer training loop:  35%|###5      | 17647/50000 [10:29:04<16:08:19,  1.80s/it]
Outer training loop:  35%|###5      | 17648/50000 [10:29:05<16:10:23,  1.80s/it]
Outer training loop:  35%|###5      | 17649/50000 [10:29:07<15:47:03,  1.76s/it]
Outer training loop:  35%|###5      | 17650/50000 [10:29:09<15:38:21,  1.74s/it]
Outer training loop:  35%|###5      | 17651/50000 [10:29:11<16:01:24,  1.78s/it]
Outer training loop:  35%|###5      | 17652/50000 [10:29:12<15:33:14,  1.73s/it]
Outer training loop:  35%|###5      | 17653/50000 [10:29:14<15:19:20,  1.71s/it]
Outer training loop:  35%|###5      | 17654/50000 [10:29:16<15:42:03,  1.75s/it]
Outer training loop:  35%|###5      | 17655/50000 [10:29:17<15:30:46,  1.73s/it]
Outer training loop:  35%|###5      | 17656/50000 [10:29:19<15:24:50,  1.72s/it]
Outer training loop:  35%|###5      | 17657/50000 [10:29:21<15:48:35,  1.76s/it]
Outer training loop:  35%|###5      | 17658/50000 [10:29:23<15:33:38,  1.73s/it]
Outer training loop:  35%|###5      | 17659/50000 [10:29:24<15:30:02,  1.73s/it]
Outer training loop:  35%|###5      | 17660/50000 [10:29:26<15:49:14,  1.76s/it]
Outer training loop:  35%|###5      | 17661/50000 [10:29:28<15:58:16,  1.78s/it]
Outer training loop:  35%|###5      | 17662/50000 [10:29:30<15:52:48,  1.77s/it]
Outer training loop:  35%|###5      | 17663/50000 [10:29:32<16:14:59,  1.81s/it]
Outer training loop:  35%|###5      | 17664/50000 [10:29:34<16:30:18,  1.84s/it]
Outer training loop:  35%|###5      | 17665/50000 [10:29:35<16:18:03,  1.81s/it]
Outer training loop:  35%|###5      | 17666/50000 [10:29:37<15:59:26,  1.78s/it]
Outer training loop:  35%|###5      | 17667/50000 [10:29:39<15:41:47,  1.75s/it]
Outer training loop:  35%|###5      | 17668/50000 [10:29:41<16:04:41,  1.79s/it]
Outer training loop:  35%|###5      | 17669/50000 [10:29:42<16:10:16,  1.80s/it]
Outer training loop:  35%|###5      | 17670/50000 [10:29:44<16:10:02,  1.80s/it]
Outer training loop:  35%|###5      | 17671/50000 [10:29:46<16:18:17,  1.82s/it]
Outer training loop:  35%|###5      | 17672/50000 [10:29:48<16:08:26,  1.80s/it]
Outer training loop:  35%|###5      | 17673/50000 [10:29:50<16:10:50,  1.80s/it]
Outer training loop:  35%|###5      | 17674/50000 [10:29:52<16:30:12,  1.84s/it]
Outer training loop:  35%|###5      | 17675/50000 [10:29:53<16:11:39,  1.80s/it]
Outer training loop:  35%|###5      | 17676/50000 [10:29:55<15:57:48,  1.78s/it]
Outer training loop:  35%|###5      | 17677/50000 [10:29:57<16:06:39,  1.79s/it]
Outer training loop:  35%|###5      | 17678/50000 [10:29:59<16:01:43,  1.79s/it]
Outer training loop:  35%|###5      | 17679/50000 [10:30:00<15:49:54,  1.76s/it]
Outer training loop:  35%|###5      | 17680/50000 [10:30:02<15:44:40,  1.75s/it]
Outer training loop:  35%|###5      | 17681/50000 [10:30:04<16:14:01,  1.81s/it]
Outer training loop:  35%|###5      | 17682/50000 [10:30:06<15:56:44,  1.78s/it]
Outer training loop:  35%|###5      | 17683/50000 [10:30:07<15:52:32,  1.77s/it]
Outer training loop:  35%|###5      | 17684/50000 [10:30:09<16:15:14,  1.81s/it]
Outer training loop:  35%|###5      | 17685/50000 [10:30:11<16:00:51,  1.78s/it]
Outer training loop:  35%|###5      | 17686/50000 [10:30:13<15:45:32,  1.76s/it]
Outer training loop:  35%|###5      | 17687/50000 [10:30:15<16:07:48,  1.80s/it]
Outer training loop:  35%|###5      | 17688/50000 [10:30:16<15:45:20,  1.76s/it]
Outer training loop:  35%|###5      | 17689/50000 [10:30:18<15:41:32,  1.75s/it]
Outer training loop:  35%|###5      | 17690/50000 [10:30:20<15:35:34,  1.74s/it]
Outer training loop:  35%|###5      | 17691/50000 [10:30:22<16:02:33,  1.79s/it]
Outer training loop:  35%|###5      | 17692/50000 [10:30:23<15:59:57,  1.78s/it]
Outer training loop:  35%|###5      | 17693/50000 [10:30:25<15:58:54,  1.78s/it]
Outer training loop:  35%|###5      | 17694/50000 [10:30:27<16:20:46,  1.82s/it]
Outer training loop:  35%|###5      | 17695/50000 [10:30:29<16:07:49,  1.80s/it]
Outer training loop:  35%|###5      | 17696/50000 [10:30:31<15:48:28,  1.76s/it]
Outer training loop:  35%|###5      | 17697/50000 [10:30:33<16:13:24,  1.81s/it]
Outer training loop:  35%|###5      | 17698/50000 [10:30:34<15:58:21,  1.78s/it]
Outer training loop:  35%|###5      | 17699/50000 [10:30:36<16:02:13,  1.79s/it]
Outer training loop:  35%|###5      | 17700/50000 [10:30:38<15:43:33,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  35%|###5      | 17701/50000 [10:30:45<30:16:54,  3.38s/it]
Outer training loop:  35%|###5      | 17702/50000 [10:30:47<25:55:22,  2.89s/it]
Outer training loop:  35%|###5      | 17703/50000 [10:30:48<23:05:38,  2.57s/it]
Outer training loop:  35%|###5      | 17704/50000 [10:30:50<20:58:41,  2.34s/it]
Outer training loop:  35%|###5      | 17705/50000 [10:30:52<19:47:19,  2.21s/it]
Outer training loop:  35%|###5      | 17706/50000 [10:30:54<18:31:12,  2.06s/it]
Outer training loop:  35%|###5      | 17707/50000 [10:30:56<18:11:09,  2.03s/it]
Outer training loop:  35%|###5      | 17708/50000 [10:30:58<17:57:24,  2.00s/it]
Outer training loop:  35%|###5      | 17709/50000 [10:30:59<16:48:23,  1.87s/it]
Outer training loop:  35%|###5      | 17710/50000 [10:31:01<16:07:55,  1.80s/it]
Outer training loop:  35%|###5      | 17711/50000 [10:31:03<16:45:00,  1.87s/it]
Outer training loop:  35%|###5      | 17712/50000 [10:31:05<16:26:25,  1.83s/it]
Outer training loop:  35%|###5      | 17713/50000 [10:31:06<16:09:47,  1.80s/it]
Outer training loop:  35%|###5      | 17714/50000 [10:31:08<15:40:23,  1.75s/it]
Outer training loop:  35%|###5      | 17715/50000 [10:31:10<15:58:09,  1.78s/it]
Outer training loop:  35%|###5      | 17716/50000 [10:31:12<15:52:38,  1.77s/it]
Outer training loop:  35%|###5      | 17717/50000 [10:31:14<16:09:20,  1.80s/it]
Outer training loop:  35%|###5      | 17718/50000 [10:31:15<15:45:25,  1.76s/it]
Outer training loop:  35%|###5      | 17719/50000 [10:31:17<15:32:01,  1.73s/it]
Outer training loop:  35%|###5      | 17720/50000 [10:31:19<15:50:48,  1.77s/it]
Outer training loop:  35%|###5      | 17721/50000 [10:31:21<16:07:19,  1.80s/it]
Outer training loop:  35%|###5      | 17722/50000 [10:31:22<16:05:26,  1.79s/it]
Outer training loop:  35%|###5      | 17723/50000 [10:31:24<15:59:49,  1.78s/it]
Outer training loop:  35%|###5      | 17724/50000 [10:31:26<16:38:25,  1.86s/it]
Outer training loop:  35%|###5      | 17725/50000 [10:31:28<16:24:10,  1.83s/it]
Outer training loop:  35%|###5      | 17726/50000 [10:31:30<16:20:33,  1.82s/it]
Outer training loop:  35%|###5      | 17727/50000 [10:31:32<16:13:49,  1.81s/it]
Outer training loop:  35%|###5      | 17728/50000 [10:31:33<16:34:05,  1.85s/it]
Outer training loop:  35%|###5      | 17729/50000 [10:31:35<16:43:55,  1.87s/it]
Outer training loop:  35%|###5      | 17730/50000 [10:31:37<16:24:27,  1.83s/it]
Outer training loop:  35%|###5      | 17731/50000 [10:31:39<17:01:54,  1.90s/it]
Outer training loop:  35%|###5      | 17732/50000 [10:31:41<16:35:17,  1.85s/it]
Outer training loop:  35%|###5      | 17733/50000 [10:31:43<15:53:36,  1.77s/it]
Outer training loop:  35%|###5      | 17734/50000 [10:31:44<16:17:13,  1.82s/it]
Outer training loop:  35%|###5      | 17735/50000 [10:31:46<15:55:30,  1.78s/it]
Outer training loop:  35%|###5      | 17736/50000 [10:31:48<15:17:40,  1.71s/it]
Outer training loop:  35%|###5      | 17737/50000 [10:31:49<15:19:31,  1.71s/it]
Outer training loop:  35%|###5      | 17738/50000 [10:31:51<15:48:48,  1.76s/it]
Outer training loop:  35%|###5      | 17739/50000 [10:31:53<15:55:16,  1.78s/it]
Outer training loop:  35%|###5      | 17740/50000 [10:31:55<16:01:28,  1.79s/it]
Outer training loop:  35%|###5      | 17741/50000 [10:31:57<16:35:00,  1.85s/it]
Outer training loop:  35%|###5      | 17742/50000 [10:31:59<16:21:55,  1.83s/it]
Outer training loop:  35%|###5      | 17743/50000 [10:32:00<16:00:05,  1.79s/it]
Outer training loop:  35%|###5      | 17744/50000 [10:32:02<16:23:32,  1.83s/it]
Outer training loop:  35%|###5      | 17745/50000 [10:32:04<16:08:45,  1.80s/it]
Outer training loop:  35%|###5      | 17746/50000 [10:32:06<15:54:23,  1.78s/it]
Outer training loop:  35%|###5      | 17747/50000 [10:32:07<15:41:43,  1.75s/it]
Outer training loop:  35%|###5      | 17748/50000 [10:32:09<15:46:46,  1.76s/it]
Outer training loop:  35%|###5      | 17749/50000 [10:32:11<15:33:52,  1.74s/it]
Outer training loop:  36%|###5      | 17750/50000 [10:32:13<15:42:54,  1.75s/it]
Outer training loop:  36%|###5      | 17751/50000 [10:32:15<15:59:07,  1.78s/it]
Outer training loop:  36%|###5      | 17752/50000 [10:32:16<16:02:56,  1.79s/it]
Outer training loop:  36%|###5      | 17753/50000 [10:32:18<15:50:14,  1.77s/it]
Outer training loop:  36%|###5      | 17754/50000 [10:32:20<16:39:02,  1.86s/it]
Outer training loop:  36%|###5      | 17755/50000 [10:32:22<16:12:28,  1.81s/it]
Outer training loop:  36%|###5      | 17756/50000 [10:32:24<16:06:38,  1.80s/it]
Outer training loop:  36%|###5      | 17757/50000 [10:32:26<17:18:34,  1.93s/it]
Outer training loop:  36%|###5      | 17758/50000 [10:32:27<16:26:59,  1.84s/it]
Outer training loop:  36%|###5      | 17759/50000 [10:32:29<16:05:08,  1.80s/it]
Outer training loop:  36%|###5      | 17760/50000 [10:32:31<16:02:24,  1.79s/it]
Outer training loop:  36%|###5      | 17761/50000 [10:32:33<16:30:36,  1.84s/it]
Outer training loop:  36%|###5      | 17762/50000 [10:32:35<16:01:11,  1.79s/it]
Outer training loop:  36%|###5      | 17763/50000 [10:32:36<16:09:16,  1.80s/it]
Outer training loop:  36%|###5      | 17764/50000 [10:32:38<16:35:41,  1.85s/it]
Outer training loop:  36%|###5      | 17765/50000 [10:32:40<16:14:43,  1.81s/it]
Outer training loop:  36%|###5      | 17766/50000 [10:32:42<15:59:10,  1.79s/it]
Outer training loop:  36%|###5      | 17767/50000 [10:32:44<15:48:09,  1.76s/it]
Outer training loop:  36%|###5      | 17768/50000 [10:32:45<16:05:10,  1.80s/it]
Outer training loop:  36%|###5      | 17769/50000 [10:32:47<15:52:58,  1.77s/it]
Outer training loop:  36%|###5      | 17770/50000 [10:32:49<15:44:58,  1.76s/it]
Outer training loop:  36%|###5      | 17771/50000 [10:32:51<16:25:28,  1.83s/it]
Outer training loop:  36%|###5      | 17772/50000 [10:32:53<15:56:59,  1.78s/it]
Outer training loop:  36%|###5      | 17773/50000 [10:32:54<15:23:53,  1.72s/it]
Outer training loop:  36%|###5      | 17774/50000 [10:32:56<15:56:05,  1.78s/it]
Outer training loop:  36%|###5      | 17775/50000 [10:32:58<15:51:25,  1.77s/it]
Outer training loop:  36%|###5      | 17776/50000 [10:32:59<15:39:39,  1.75s/it]
Outer training loop:  36%|###5      | 17777/50000 [10:33:01<16:01:17,  1.79s/it]
Outer training loop:  36%|###5      | 17778/50000 [10:33:03<15:36:03,  1.74s/it]
Outer training loop:  36%|###5      | 17779/50000 [10:33:05<15:47:36,  1.76s/it]
Outer training loop:  36%|###5      | 17780/50000 [10:33:07<15:42:04,  1.75s/it]
Outer training loop:  36%|###5      | 17781/50000 [10:33:08<16:05:22,  1.80s/it]
Outer training loop:  36%|###5      | 17782/50000 [10:33:10<16:08:47,  1.80s/it]
Outer training loop:  36%|###5      | 17783/50000 [10:33:12<15:58:32,  1.79s/it]
Outer training loop:  36%|###5      | 17784/50000 [10:33:14<16:03:31,  1.79s/it]
Outer training loop:  36%|###5      | 17785/50000 [10:33:16<16:01:55,  1.79s/it]
Outer training loop:  36%|###5      | 17786/50000 [10:33:17<15:49:09,  1.77s/it]
Outer training loop:  36%|###5      | 17787/50000 [10:33:19<16:11:49,  1.81s/it]
Outer training loop:  36%|###5      | 17788/50000 [10:33:21<16:00:41,  1.79s/it]
Outer training loop:  36%|###5      | 17789/50000 [10:33:23<15:51:04,  1.77s/it]
Outer training loop:  36%|###5      | 17790/50000 [10:33:25<16:11:55,  1.81s/it]
Outer training loop:  36%|###5      | 17791/50000 [10:33:26<15:46:21,  1.76s/it]
Outer training loop:  36%|###5      | 17792/50000 [10:33:28<15:48:20,  1.77s/it]
Outer training loop:  36%|###5      | 17793/50000 [10:33:30<15:23:19,  1.72s/it]
Outer training loop:  36%|###5      | 17794/50000 [10:33:32<15:53:57,  1.78s/it]
Outer training loop:  36%|###5      | 17795/50000 [10:33:33<15:32:56,  1.74s/it]
Outer training loop:  36%|###5      | 17796/50000 [10:33:35<15:30:34,  1.73s/it]
Outer training loop:  36%|###5      | 17797/50000 [10:33:37<16:06:44,  1.80s/it]
Outer training loop:  36%|###5      | 17798/50000 [10:33:39<15:59:40,  1.79s/it]
Outer training loop:  36%|###5      | 17799/50000 [10:33:40<15:48:00,  1.77s/it]
Outer training loop:  36%|###5      | 17800/50000 [10:33:42<15:56:21,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.64s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.68s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  36%|###5      | 17801/50000 [10:33:49<30:08:18,  3.37s/it]
Outer training loop:  36%|###5      | 17802/50000 [10:33:51<25:52:26,  2.89s/it]
Outer training loop:  36%|###5      | 17803/50000 [10:33:53<22:41:17,  2.54s/it]
Outer training loop:  36%|###5      | 17804/50000 [10:33:55<21:09:32,  2.37s/it]
Outer training loop:  36%|###5      | 17805/50000 [10:33:57<19:42:37,  2.20s/it]
Outer training loop:  36%|###5      | 17806/50000 [10:33:58<18:10:24,  2.03s/it]
Outer training loop:  36%|###5      | 17807/50000 [10:34:00<17:27:40,  1.95s/it]
Outer training loop:  36%|###5      | 17808/50000 [10:34:02<17:25:15,  1.95s/it]
Outer training loop:  36%|###5      | 17809/50000 [10:34:03<16:32:44,  1.85s/it]
Outer training loop:  36%|###5      | 17810/50000 [10:34:05<16:02:27,  1.79s/it]
Outer training loop:  36%|###5      | 17811/50000 [10:34:07<16:01:17,  1.79s/it]
Outer training loop:  36%|###5      | 17812/50000 [10:34:09<16:11:44,  1.81s/it]
Outer training loop:  36%|###5      | 17813/50000 [10:34:10<15:54:31,  1.78s/it]
Outer training loop:  36%|###5      | 17814/50000 [10:34:12<15:35:19,  1.74s/it]
Outer training loop:  36%|###5      | 17815/50000 [10:34:14<16:19:00,  1.83s/it]
Outer training loop:  36%|###5      | 17816/50000 [10:34:16<15:55:24,  1.78s/it]
Outer training loop:  36%|###5      | 17817/50000 [10:34:18<15:35:28,  1.74s/it]
Outer training loop:  36%|###5      | 17818/50000 [10:34:19<16:11:17,  1.81s/it]
Outer training loop:  36%|###5      | 17819/50000 [10:34:21<15:55:12,  1.78s/it]
Outer training loop:  36%|###5      | 17820/50000 [10:34:23<15:36:35,  1.75s/it]
Outer training loop:  36%|###5      | 17821/50000 [10:34:25<15:38:57,  1.75s/it]
Outer training loop:  36%|###5      | 17822/50000 [10:34:26<15:57:52,  1.79s/it]
Outer training loop:  36%|###5      | 17823/50000 [10:34:28<15:47:24,  1.77s/it]
Outer training loop:  36%|###5      | 17824/50000 [10:34:30<15:22:39,  1.72s/it]
Outer training loop:  36%|###5      | 17825/50000 [10:34:32<15:42:08,  1.76s/it]
Outer training loop:  36%|###5      | 17826/50000 [10:34:33<15:40:03,  1.75s/it]
Outer training loop:  36%|###5      | 17827/50000 [10:34:35<15:23:34,  1.72s/it]
Outer training loop:  36%|###5      | 17828/50000 [10:34:37<16:01:55,  1.79s/it]
Outer training loop:  36%|###5      | 17829/50000 [10:34:39<15:53:51,  1.78s/it]
Outer training loop:  36%|###5      | 17830/50000 [10:34:40<15:44:30,  1.76s/it]
Outer training loop:  36%|###5      | 17831/50000 [10:34:42<15:34:04,  1.74s/it]
Outer training loop:  36%|###5      | 17832/50000 [10:34:44<15:48:15,  1.77s/it]
Outer training loop:  36%|###5      | 17833/50000 [10:34:46<15:50:59,  1.77s/it]
Outer training loop:  36%|###5      | 17834/50000 [10:34:47<15:37:22,  1.75s/it]
Outer training loop:  36%|###5      | 17835/50000 [10:34:49<15:49:15,  1.77s/it]
Outer training loop:  36%|###5      | 17836/50000 [10:34:51<15:29:27,  1.73s/it]
Outer training loop:  36%|###5      | 17837/50000 [10:34:53<15:19:23,  1.72s/it]
Outer training loop:  36%|###5      | 17838/50000 [10:34:54<15:23:13,  1.72s/it]
Outer training loop:  36%|###5      | 17839/50000 [10:34:56<15:59:24,  1.79s/it]
Outer training loop:  36%|###5      | 17840/50000 [10:34:58<15:35:35,  1.75s/it]
Outer training loop:  36%|###5      | 17841/50000 [10:35:00<15:17:03,  1.71s/it]
Outer training loop:  36%|###5      | 17842/50000 [10:35:01<15:49:40,  1.77s/it]
Outer training loop:  36%|###5      | 17843/50000 [10:35:03<15:51:00,  1.77s/it]
Outer training loop:  36%|###5      | 17844/50000 [10:35:05<15:46:28,  1.77s/it]
Outer training loop:  36%|###5      | 17845/50000 [10:35:07<16:07:50,  1.81s/it]
Outer training loop:  36%|###5      | 17846/50000 [10:35:09<15:55:36,  1.78s/it]
Outer training loop:  36%|###5      | 17847/50000 [10:35:10<15:50:52,  1.77s/it]
Outer training loop:  36%|###5      | 17848/50000 [10:35:12<16:09:07,  1.81s/it]
Outer training loop:  36%|###5      | 17849/50000 [10:35:14<15:34:39,  1.74s/it]
Outer training loop:  36%|###5      | 17850/50000 [10:35:16<15:41:33,  1.76s/it]
Outer training loop:  36%|###5      | 17851/50000 [10:35:18<15:53:49,  1.78s/it]
Outer training loop:  36%|###5      | 17852/50000 [10:35:19<16:16:42,  1.82s/it]
Outer training loop:  36%|###5      | 17853/50000 [10:35:21<15:53:30,  1.78s/it]
Outer training loop:  36%|###5      | 17854/50000 [10:35:23<15:55:16,  1.78s/it]
Outer training loop:  36%|###5      | 17855/50000 [10:35:25<16:09:31,  1.81s/it]
Outer training loop:  36%|###5      | 17856/50000 [10:35:27<15:57:14,  1.79s/it]
Outer training loop:  36%|###5      | 17857/50000 [10:35:28<15:31:34,  1.74s/it]
Outer training loop:  36%|###5      | 17858/50000 [10:35:30<15:21:52,  1.72s/it]
Outer training loop:  36%|###5      | 17859/50000 [10:35:32<15:43:08,  1.76s/it]
Outer training loop:  36%|###5      | 17860/50000 [10:35:33<15:35:12,  1.75s/it]
Outer training loop:  36%|###5      | 17861/50000 [10:35:35<15:44:38,  1.76s/it]
Outer training loop:  36%|###5      | 17862/50000 [10:35:37<15:49:56,  1.77s/it]
Outer training loop:  36%|###5      | 17863/50000 [10:35:39<15:43:20,  1.76s/it]
Outer training loop:  36%|###5      | 17864/50000 [10:35:40<15:30:43,  1.74s/it]
Outer training loop:  36%|###5      | 17865/50000 [10:35:42<15:44:03,  1.76s/it]
Outer training loop:  36%|###5      | 17866/50000 [10:35:44<16:00:04,  1.79s/it]
Outer training loop:  36%|###5      | 17867/50000 [10:35:46<15:48:29,  1.77s/it]
Outer training loop:  36%|###5      | 17868/50000 [10:35:48<15:52:16,  1.78s/it]
Outer training loop:  36%|###5      | 17869/50000 [10:35:49<16:08:08,  1.81s/it]
Outer training loop:  36%|###5      | 17870/50000 [10:35:51<15:55:17,  1.78s/it]
Outer training loop:  36%|###5      | 17871/50000 [10:35:53<15:48:25,  1.77s/it]
Outer training loop:  36%|###5      | 17872/50000 [10:35:55<16:12:46,  1.82s/it]
Outer training loop:  36%|###5      | 17873/50000 [10:35:57<15:47:17,  1.77s/it]
Outer training loop:  36%|###5      | 17874/50000 [10:35:58<15:45:41,  1.77s/it]
Outer training loop:  36%|###5      | 17875/50000 [10:36:00<15:43:01,  1.76s/it]
Outer training loop:  36%|###5      | 17876/50000 [10:36:02<16:02:41,  1.80s/it]
Outer training loop:  36%|###5      | 17877/50000 [10:36:04<15:40:45,  1.76s/it]
Outer training loop:  36%|###5      | 17878/50000 [10:36:05<15:41:58,  1.76s/it]
Outer training loop:  36%|###5      | 17879/50000 [10:36:07<16:13:02,  1.82s/it]
Outer training loop:  36%|###5      | 17880/50000 [10:36:09<15:44:12,  1.76s/it]
Outer training loop:  36%|###5      | 17881/50000 [10:36:11<15:32:12,  1.74s/it]
Outer training loop:  36%|###5      | 17882/50000 [10:36:12<15:50:46,  1.78s/it]
Outer training loop:  36%|###5      | 17883/50000 [10:36:14<15:43:39,  1.76s/it]
Outer training loop:  36%|###5      | 17884/50000 [10:36:16<15:34:35,  1.75s/it]
Outer training loop:  36%|###5      | 17885/50000 [10:36:18<15:35:08,  1.75s/it]
Outer training loop:  36%|###5      | 17886/50000 [10:36:20<15:59:19,  1.79s/it]
Outer training loop:  36%|###5      | 17887/50000 [10:36:21<16:02:43,  1.80s/it]
Outer training loop:  36%|###5      | 17888/50000 [10:36:23<15:50:46,  1.78s/it]
Outer training loop:  36%|###5      | 17889/50000 [10:36:25<16:20:05,  1.83s/it]
Outer training loop:  36%|###5      | 17890/50000 [10:36:27<15:56:43,  1.79s/it]
Outer training loop:  36%|###5      | 17891/50000 [10:36:28<15:50:36,  1.78s/it]
Outer training loop:  36%|###5      | 17892/50000 [10:36:30<16:14:16,  1.82s/it]
Outer training loop:  36%|###5      | 17893/50000 [10:36:32<15:59:56,  1.79s/it]
Outer training loop:  36%|###5      | 17894/50000 [10:36:34<15:43:34,  1.76s/it]
Outer training loop:  36%|###5      | 17895/50000 [10:36:36<15:58:56,  1.79s/it]
Outer training loop:  36%|###5      | 17896/50000 [10:36:37<15:40:46,  1.76s/it]
Outer training loop:  36%|###5      | 17897/50000 [10:36:39<15:21:52,  1.72s/it]
Outer training loop:  36%|###5      | 17898/50000 [10:36:41<15:30:21,  1.74s/it]
Outer training loop:  36%|###5      | 17899/50000 [10:36:43<15:48:49,  1.77s/it]
Outer training loop:  36%|###5      | 17900/50000 [10:36:44<15:37:58,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.91s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  36%|###5      | 17901/50000 [10:36:51<29:56:24,  3.36s/it]
Outer training loop:  36%|###5      | 17902/50000 [10:36:53<25:22:51,  2.85s/it]
Outer training loop:  36%|###5      | 17903/50000 [10:36:55<22:56:15,  2.57s/it]
Outer training loop:  36%|###5      | 17904/50000 [10:36:57<20:17:06,  2.28s/it]
Outer training loop:  36%|###5      | 17905/50000 [10:36:58<18:54:09,  2.12s/it]
Outer training loop:  36%|###5      | 17906/50000 [10:37:00<18:17:08,  2.05s/it]
Outer training loop:  36%|###5      | 17907/50000 [10:37:02<17:18:52,  1.94s/it]
Outer training loop:  36%|###5      | 17908/50000 [10:37:04<16:43:21,  1.88s/it]
Outer training loop:  36%|###5      | 17909/50000 [10:37:06<17:00:31,  1.91s/it]
Outer training loop:  36%|###5      | 17910/50000 [10:37:07<16:27:37,  1.85s/it]
Outer training loop:  36%|###5      | 17911/50000 [10:37:09<15:49:29,  1.78s/it]
Outer training loop:  36%|###5      | 17912/50000 [10:37:11<16:08:29,  1.81s/it]
Outer training loop:  36%|###5      | 17913/50000 [10:37:13<16:30:03,  1.85s/it]
Outer training loop:  36%|###5      | 17914/50000 [10:37:15<16:14:19,  1.82s/it]
Outer training loop:  36%|###5      | 17915/50000 [10:37:16<16:04:35,  1.80s/it]
Outer training loop:  36%|###5      | 17916/50000 [10:37:18<16:01:14,  1.80s/it]
Outer training loop:  36%|###5      | 17917/50000 [10:37:20<16:21:42,  1.84s/it]
Outer training loop:  36%|###5      | 17918/50000 [10:37:22<15:55:20,  1.79s/it]
Outer training loop:  36%|###5      | 17919/50000 [10:37:23<15:37:45,  1.75s/it]
Outer training loop:  36%|###5      | 17920/50000 [10:37:25<16:03:57,  1.80s/it]
Outer training loop:  36%|###5      | 17921/50000 [10:37:27<15:46:10,  1.77s/it]
Outer training loop:  36%|###5      | 17922/50000 [10:37:29<15:49:09,  1.78s/it]
Outer training loop:  36%|###5      | 17923/50000 [10:37:31<16:02:29,  1.80s/it]
Outer training loop:  36%|###5      | 17924/50000 [10:37:32<15:38:18,  1.76s/it]
Outer training loop:  36%|###5      | 17925/50000 [10:37:34<15:41:14,  1.76s/it]
Outer training loop:  36%|###5      | 17926/50000 [10:37:36<15:45:14,  1.77s/it]
Outer training loop:  36%|###5      | 17927/50000 [10:37:38<15:57:54,  1.79s/it]
Outer training loop:  36%|###5      | 17928/50000 [10:37:39<15:28:34,  1.74s/it]
Outer training loop:  36%|###5      | 17929/50000 [10:37:41<15:29:56,  1.74s/it]
Outer training loop:  36%|###5      | 17930/50000 [10:37:43<15:51:09,  1.78s/it]
Outer training loop:  36%|###5      | 17931/50000 [10:37:45<15:45:08,  1.77s/it]
Outer training loop:  36%|###5      | 17932/50000 [10:37:46<15:40:35,  1.76s/it]
Outer training loop:  36%|###5      | 17933/50000 [10:37:48<16:00:42,  1.80s/it]
Outer training loop:  36%|###5      | 17934/50000 [10:37:50<15:49:49,  1.78s/it]
Outer training loop:  36%|###5      | 17935/50000 [10:37:52<15:38:50,  1.76s/it]
Outer training loop:  36%|###5      | 17936/50000 [10:37:54<15:58:30,  1.79s/it]
Outer training loop:  36%|###5      | 17937/50000 [10:37:55<15:36:42,  1.75s/it]
Outer training loop:  36%|###5      | 17938/50000 [10:37:57<15:34:26,  1.75s/it]
Outer training loop:  36%|###5      | 17939/50000 [10:37:59<15:57:28,  1.79s/it]
Outer training loop:  36%|###5      | 17940/50000 [10:38:01<15:56:42,  1.79s/it]
Outer training loop:  36%|###5      | 17941/50000 [10:38:02<15:41:03,  1.76s/it]
Outer training loop:  36%|###5      | 17942/50000 [10:38:04<15:29:30,  1.74s/it]
Outer training loop:  36%|###5      | 17943/50000 [10:38:06<15:44:34,  1.77s/it]
Outer training loop:  36%|###5      | 17944/50000 [10:38:08<15:44:23,  1.77s/it]
Outer training loop:  36%|###5      | 17945/50000 [10:38:09<15:39:55,  1.76s/it]
Outer training loop:  36%|###5      | 17946/50000 [10:38:11<16:04:43,  1.81s/it]
Outer training loop:  36%|###5      | 17947/50000 [10:38:13<15:43:31,  1.77s/it]
Outer training loop:  36%|###5      | 17948/50000 [10:38:15<15:48:24,  1.78s/it]
Outer training loop:  36%|###5      | 17949/50000 [10:38:16<15:35:53,  1.75s/it]
Outer training loop:  36%|###5      | 17950/50000 [10:38:18<16:06:13,  1.81s/it]
Outer training loop:  36%|###5      | 17951/50000 [10:38:20<16:33:35,  1.86s/it]
Outer training loop:  36%|###5      | 17952/50000 [10:38:22<16:08:39,  1.81s/it]
Outer training loop:  36%|###5      | 17953/50000 [10:38:24<16:18:38,  1.83s/it]
Outer training loop:  36%|###5      | 17954/50000 [10:38:26<15:54:42,  1.79s/it]
Outer training loop:  36%|###5      | 17955/50000 [10:38:27<15:43:18,  1.77s/it]
Outer training loop:  36%|###5      | 17956/50000 [10:38:29<15:41:26,  1.76s/it]
Outer training loop:  36%|###5      | 17957/50000 [10:38:31<16:14:55,  1.83s/it]
Outer training loop:  36%|###5      | 17958/50000 [10:38:33<15:52:40,  1.78s/it]
Outer training loop:  36%|###5      | 17959/50000 [10:38:35<15:38:24,  1.76s/it]
Outer training loop:  36%|###5      | 17960/50000 [10:38:37<16:17:40,  1.83s/it]
Outer training loop:  36%|###5      | 17961/50000 [10:38:38<15:49:44,  1.78s/it]
Outer training loop:  36%|###5      | 17962/50000 [10:38:40<15:43:05,  1.77s/it]
Outer training loop:  36%|###5      | 17963/50000 [10:38:42<16:15:01,  1.83s/it]
Outer training loop:  36%|###5      | 17964/50000 [10:38:44<15:59:49,  1.80s/it]
Outer training loop:  36%|###5      | 17965/50000 [10:38:45<15:53:47,  1.79s/it]
Outer training loop:  36%|###5      | 17966/50000 [10:38:47<15:50:00,  1.78s/it]
Outer training loop:  36%|###5      | 17967/50000 [10:38:49<16:10:04,  1.82s/it]
Outer training loop:  36%|###5      | 17968/50000 [10:38:51<15:50:29,  1.78s/it]
Outer training loop:  36%|###5      | 17969/50000 [10:38:52<15:46:01,  1.77s/it]
Outer training loop:  36%|###5      | 17970/50000 [10:38:54<16:11:20,  1.82s/it]
Outer training loop:  36%|###5      | 17971/50000 [10:38:56<15:48:53,  1.78s/it]
Outer training loop:  36%|###5      | 17972/50000 [10:38:58<15:28:16,  1.74s/it]
Outer training loop:  36%|###5      | 17973/50000 [10:39:00<16:05:18,  1.81s/it]
Outer training loop:  36%|###5      | 17974/50000 [10:39:02<16:10:39,  1.82s/it]
Outer training loop:  36%|###5      | 17975/50000 [10:39:03<15:53:48,  1.79s/it]
Outer training loop:  36%|###5      | 17976/50000 [10:39:05<15:39:47,  1.76s/it]
Outer training loop:  36%|###5      | 17977/50000 [10:39:07<15:59:34,  1.80s/it]
Outer training loop:  36%|###5      | 17978/50000 [10:39:09<15:52:59,  1.79s/it]
Outer training loop:  36%|###5      | 17979/50000 [10:39:10<15:59:58,  1.80s/it]
Outer training loop:  36%|###5      | 17980/50000 [10:39:12<16:15:14,  1.83s/it]
Outer training loop:  36%|###5      | 17981/50000 [10:39:14<16:12:15,  1.82s/it]
Outer training loop:  36%|###5      | 17982/50000 [10:39:16<15:39:37,  1.76s/it]
Outer training loop:  36%|###5      | 17983/50000 [10:39:18<16:06:23,  1.81s/it]
Outer training loop:  36%|###5      | 17984/50000 [10:39:19<15:54:13,  1.79s/it]
Outer training loop:  36%|###5      | 17985/50000 [10:39:21<15:38:57,  1.76s/it]
Outer training loop:  36%|###5      | 17986/50000 [10:39:23<15:20:14,  1.72s/it]
Outer training loop:  36%|###5      | 17987/50000 [10:39:25<16:00:07,  1.80s/it]
Outer training loop:  36%|###5      | 17988/50000 [10:39:26<15:36:24,  1.76s/it]
Outer training loop:  36%|###5      | 17989/50000 [10:39:28<15:20:10,  1.72s/it]
Outer training loop:  36%|###5      | 17990/50000 [10:39:30<15:42:28,  1.77s/it]
Outer training loop:  36%|###5      | 17991/50000 [10:39:32<15:28:58,  1.74s/it]
Outer training loop:  36%|###5      | 17992/50000 [10:39:33<15:28:16,  1.74s/it]
Outer training loop:  36%|###5      | 17993/50000 [10:39:35<16:05:50,  1.81s/it]
Outer training loop:  36%|###5      | 17994/50000 [10:39:37<15:36:38,  1.76s/it]
Outer training loop:  36%|###5      | 17995/50000 [10:39:39<15:20:26,  1.73s/it]
Outer training loop:  36%|###5      | 17996/50000 [10:39:41<15:52:39,  1.79s/it]
Outer training loop:  36%|###5      | 17997/50000 [10:39:42<15:43:04,  1.77s/it]
Outer training loop:  36%|###5      | 17998/50000 [10:39:44<15:41:22,  1.76s/it]
Outer training loop:  36%|###5      | 17999/50000 [10:39:46<15:44:00,  1.77s/it]
Outer training loop:  36%|###6      | 18000/50000 [10:39:48<15:50:20,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.69s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:17,  1.67s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:17,  1.69s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.79s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:18,  1.75s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.76s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:15,  1.76s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:17,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:14,  1.79s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:13,  1.80s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:15,  1.85s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:10,  1.81s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:08,  1.80s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:03,  1.77s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:05,  1.82s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:01,  1.79s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:57,  1.75s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:55,  1.75s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:53,  1.74s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:57,  1.83s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:54,  1.81s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:50,  1.78s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:47,  1.76s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:45,  1.76s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:46,  1.81s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:44,  1.80s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:40,  1.76s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:39,  1.77s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:35,  1.74s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.80s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:35,  1.79s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:33,  1.80s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:30,  1.77s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:28,  1.76s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:24,  1.73s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:26,  1.80s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:23,  1.79s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:20,  1.75s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:17,  1.72s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.73s/it]

Evaluating policy:  48%|####8     | 40/83 [01:10<01:12,  1.70s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:15,  1.80s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:14,  1.81s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:11,  1.79s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:08,  1.74s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:04,  1.71s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:01,  1.66s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:01,  1.71s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:03,  1.81s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<00:59,  1.76s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:57,  1.74s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:55,  1.75s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:54,  1.74s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:52,  1.74s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:50,  1.73s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.72s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:45,  1.80s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.75s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.77s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:38,  1.76s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:36,  1.73s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:34,  1.70s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:32,  1.80s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:29,  1.76s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:27,  1.71s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:25,  1.72s/it]

Evaluating policy:  83%|########3 | 69/83 [02:01<00:23,  1.69s/it]

Evaluating policy:  84%|########4 | 70/83 [02:03<00:22,  1.70s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:20,  1.67s/it]

Evaluating policy:  87%|########6 | 72/83 [02:06<00:18,  1.66s/it]

Evaluating policy:  88%|########7 | 73/83 [02:08<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:16,  1.81s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:14,  1.81s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.78s/it]

Evaluating policy:  93%|#########2| 77/83 [02:15<00:10,  1.75s/it]

Evaluating policy:  94%|#########3| 78/83 [02:17<00:08,  1.75s/it]

Evaluating policy:  95%|#########5| 79/83 [02:19<00:07,  1.79s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.78s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.76s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 83/83 [02:26<00:00,  1.82s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:12,  1.62s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:14,  1.66s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:13,  1.67s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:14,  1.70s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:11,  1.69s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:10,  1.69s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:12,  1.74s/it]

Evaluating policy:  10%|9         | 8/83 [00:13<02:10,  1.74s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:07,  1.73s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:17,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:12,  1.85s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.80s/it]

Evaluating policy:  16%|#5        | 13/83 [00:22<02:02,  1.75s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:05,  1.84s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:03,  1.84s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:57,  1.78s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<02:00,  1.85s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:54,  1.79s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:51,  1.76s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:48,  1.75s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:45,  1.73s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:49,  1.82s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:45,  1.78s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.78s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:41,  1.77s/it]

Evaluating policy:  33%|###2      | 27/83 [00:47<01:37,  1.75s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:40,  1.83s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.78s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.76s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:29,  1.72s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:27,  1.72s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:27,  1.75s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:26,  1.84s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:20,  1.78s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.74s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:19,  1.85s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:15,  1.84s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:12,  1.82s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:10,  1.81s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:10,  1.91s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:07,  1.87s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:03,  1.80s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.78s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:58,  1.77s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:56,  1.76s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:54,  1.75s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:55,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:52,  1.80s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:49,  1.78s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:48,  1.80s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:46,  1.79s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:44,  1.79s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.77s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:43,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:39,  1.82s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:37,  1.78s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.75s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:34,  1.80s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:31,  1.78s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:30,  1.77s/it]

Evaluating policy:  81%|########  | 67/83 [02:00<00:28,  1.76s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:25,  1.73s/it]

Evaluating policy:  83%|########3 | 69/83 [02:04<00:26,  1.92s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:21,  1.80s/it]

Evaluating policy:  87%|########6 | 72/83 [02:09<00:19,  1.75s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.74s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:15,  1.75s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:14,  1.76s/it]

Evaluating policy:  92%|#########1| 76/83 [02:16<00:12,  1.76s/it]

Evaluating policy:  93%|#########2| 77/83 [02:18<00:11,  1.90s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:09,  1.83s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.82s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.80s/it]

Evaluating policy:  98%|#########7| 81/83 [02:25<00:03,  1.77s/it]

Evaluating policy:  99%|#########8| 82/83 [02:27<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.78s/it]

Outer training loop:  36%|###6      | 18001/50000 [10:44:51<819:07:44, 92.15s/it]
Outer training loop:  36%|###6      | 18002/50000 [10:44:52<578:07:59, 65.04s/it]
Outer training loop:  36%|###6      | 18003/50000 [10:44:54<409:04:30, 46.03s/it]
Outer training loop:  36%|###6      | 18004/50000 [10:44:56<290:50:57, 32.72s/it]
Outer training loop:  36%|###6      | 18005/50000 [10:44:58<209:42:48, 23.60s/it]
Outer training loop:  36%|###6      | 18006/50000 [10:45:00<151:29:59, 17.05s/it]
Outer training loop:  36%|###6      | 18007/50000 [10:45:01<110:33:59, 12.44s/it]
Outer training loop:  36%|###6      | 18008/50000 [10:45:03<82:17:54,  9.26s/it] 
Outer training loop:  36%|###6      | 18009/50000 [10:45:05<62:28:26,  7.03s/it]
Outer training loop:  36%|###6      | 18010/50000 [10:45:07<48:03:19,  5.41s/it]
Outer training loop:  36%|###6      | 18011/50000 [10:45:08<37:59:01,  4.27s/it]
Outer training loop:  36%|###6      | 18012/50000 [10:45:10<31:35:19,  3.56s/it]
Outer training loop:  36%|###6      | 18013/50000 [10:45:12<26:55:56,  3.03s/it]
Outer training loop:  36%|###6      | 18014/50000 [10:45:14<23:11:28,  2.61s/it]
Outer training loop:  36%|###6      | 18015/50000 [10:45:16<21:09:45,  2.38s/it]
Outer training loop:  36%|###6      | 18016/50000 [10:45:17<19:28:27,  2.19s/it]
Outer training loop:  36%|###6      | 18017/50000 [10:45:19<18:22:07,  2.07s/it]
Outer training loop:  36%|###6      | 18018/50000 [10:45:21<18:04:56,  2.04s/it]
Outer training loop:  36%|###6      | 18019/50000 [10:45:23<16:56:32,  1.91s/it]
Outer training loop:  36%|###6      | 18020/50000 [10:45:24<16:03:16,  1.81s/it]
Outer training loop:  36%|###6      | 18021/50000 [10:45:26<16:03:20,  1.81s/it]
Outer training loop:  36%|###6      | 18022/50000 [10:45:28<16:14:23,  1.83s/it]
Outer training loop:  36%|###6      | 18023/50000 [10:45:30<15:47:52,  1.78s/it]
Outer training loop:  36%|###6      | 18024/50000 [10:45:31<15:34:15,  1.75s/it]
Outer training loop:  36%|###6      | 18025/50000 [10:45:33<15:52:43,  1.79s/it]
Outer training loop:  36%|###6      | 18026/50000 [10:45:35<15:46:43,  1.78s/it]
Outer training loop:  36%|###6      | 18027/50000 [10:45:37<15:51:11,  1.78s/it]
Outer training loop:  36%|###6      | 18028/50000 [10:45:39<16:22:32,  1.84s/it]
Outer training loop:  36%|###6      | 18029/50000 [10:45:40<16:08:43,  1.82s/it]
Outer training loop:  36%|###6      | 18030/50000 [10:45:42<16:06:23,  1.81s/it]
Outer training loop:  36%|###6      | 18031/50000 [10:45:44<16:03:49,  1.81s/it]
Outer training loop:  36%|###6      | 18032/50000 [10:45:46<16:40:05,  1.88s/it]
Outer training loop:  36%|###6      | 18033/50000 [10:45:48<16:13:33,  1.83s/it]
Outer training loop:  36%|###6      | 18034/50000 [10:45:50<16:04:25,  1.81s/it]
Outer training loop:  36%|###6      | 18035/50000 [10:45:51<16:12:10,  1.82s/it]
Outer training loop:  36%|###6      | 18036/50000 [10:45:53<15:53:17,  1.79s/it]
Outer training loop:  36%|###6      | 18037/50000 [10:45:55<15:55:03,  1.79s/it]
Outer training loop:  36%|###6      | 18038/50000 [10:45:57<16:18:44,  1.84s/it]
Outer training loop:  36%|###6      | 18039/50000 [10:45:58<15:41:09,  1.77s/it]
Outer training loop:  36%|###6      | 18040/50000 [10:46:00<15:29:22,  1.74s/it]
Outer training loop:  36%|###6      | 18041/50000 [10:46:02<15:33:21,  1.75s/it]
Outer training loop:  36%|###6      | 18042/50000 [10:46:04<16:24:41,  1.85s/it]
Outer training loop:  36%|###6      | 18043/50000 [10:46:06<16:15:58,  1.83s/it]
Outer training loop:  36%|###6      | 18044/50000 [10:46:07<15:54:05,  1.79s/it]
Outer training loop:  36%|###6      | 18045/50000 [10:46:09<15:57:06,  1.80s/it]
Outer training loop:  36%|###6      | 18046/50000 [10:46:11<16:12:00,  1.83s/it]
Outer training loop:  36%|###6      | 18047/50000 [10:46:13<15:58:04,  1.80s/it]
Outer training loop:  36%|###6      | 18048/50000 [10:46:15<15:55:21,  1.79s/it]
Outer training loop:  36%|###6      | 18049/50000 [10:46:17<16:20:08,  1.84s/it]
Outer training loop:  36%|###6      | 18050/50000 [10:46:18<16:05:56,  1.81s/it]
Outer training loop:  36%|###6      | 18051/50000 [10:46:20<15:50:48,  1.79s/it]
Outer training loop:  36%|###6      | 18052/50000 [10:46:22<15:58:47,  1.80s/it]
Outer training loop:  36%|###6      | 18053/50000 [10:46:24<15:30:27,  1.75s/it]
Outer training loop:  36%|###6      | 18054/50000 [10:46:25<15:44:03,  1.77s/it]
Outer training loop:  36%|###6      | 18055/50000 [10:46:27<15:35:18,  1.76s/it]
Outer training loop:  36%|###6      | 18056/50000 [10:46:29<16:05:47,  1.81s/it]
Outer training loop:  36%|###6      | 18057/50000 [10:46:31<15:38:26,  1.76s/it]
Outer training loop:  36%|###6      | 18058/50000 [10:46:32<15:35:05,  1.76s/it]
Outer training loop:  36%|###6      | 18059/50000 [10:46:34<16:04:23,  1.81s/it]
Outer training loop:  36%|###6      | 18060/50000 [10:46:36<15:58:38,  1.80s/it]
Outer training loop:  36%|###6      | 18061/50000 [10:46:38<15:45:10,  1.78s/it]
Outer training loop:  36%|###6      | 18062/50000 [10:46:40<15:45:53,  1.78s/it]
Outer training loop:  36%|###6      | 18063/50000 [10:46:42<16:25:06,  1.85s/it]
Outer training loop:  36%|###6      | 18064/50000 [10:46:43<15:51:34,  1.79s/it]
Outer training loop:  36%|###6      | 18065/50000 [10:46:45<15:34:26,  1.76s/it]
Outer training loop:  36%|###6      | 18066/50000 [10:46:47<15:58:34,  1.80s/it]
Outer training loop:  36%|###6      | 18067/50000 [10:46:49<16:03:40,  1.81s/it]
Outer training loop:  36%|###6      | 18068/50000 [10:46:50<15:38:48,  1.76s/it]
Outer training loop:  36%|###6      | 18069/50000 [10:46:52<16:16:31,  1.83s/it]
Outer training loop:  36%|###6      | 18070/50000 [10:46:54<15:49:38,  1.78s/it]
Outer training loop:  36%|###6      | 18071/50000 [10:46:56<15:49:19,  1.78s/it]
Outer training loop:  36%|###6      | 18072/50000 [10:46:58<15:54:52,  1.79s/it]
Outer training loop:  36%|###6      | 18073/50000 [10:46:59<15:28:11,  1.74s/it]
Outer training loop:  36%|###6      | 18074/50000 [10:47:01<15:11:42,  1.71s/it]
Outer training loop:  36%|###6      | 18075/50000 [10:47:03<15:53:28,  1.79s/it]
Outer training loop:  36%|###6      | 18076/50000 [10:47:05<16:00:06,  1.80s/it]
Outer training loop:  36%|###6      | 18077/50000 [10:47:06<15:38:25,  1.76s/it]
Outer training loop:  36%|###6      | 18078/50000 [10:47:08<15:20:24,  1.73s/it]
Outer training loop:  36%|###6      | 18079/50000 [10:47:10<15:46:32,  1.78s/it]
Outer training loop:  36%|###6      | 18080/50000 [10:47:12<15:23:55,  1.74s/it]
Outer training loop:  36%|###6      | 18081/50000 [10:47:13<15:35:46,  1.76s/it]
Outer training loop:  36%|###6      | 18082/50000 [10:47:15<16:22:40,  1.85s/it]
Outer training loop:  36%|###6      | 18083/50000 [10:47:17<16:00:06,  1.80s/it]
Outer training loop:  36%|###6      | 18084/50000 [10:47:19<15:42:58,  1.77s/it]
Outer training loop:  36%|###6      | 18085/50000 [10:47:21<16:13:46,  1.83s/it]
Outer training loop:  36%|###6      | 18086/50000 [10:47:23<15:59:44,  1.80s/it]
Outer training loop:  36%|###6      | 18087/50000 [10:47:24<15:33:02,  1.75s/it]
Outer training loop:  36%|###6      | 18088/50000 [10:47:26<15:40:31,  1.77s/it]
Outer training loop:  36%|###6      | 18089/50000 [10:47:28<16:05:18,  1.81s/it]
Outer training loop:  36%|###6      | 18090/50000 [10:47:30<15:47:23,  1.78s/it]
Outer training loop:  36%|###6      | 18091/50000 [10:47:31<15:48:23,  1.78s/it]
Outer training loop:  36%|###6      | 18092/50000 [10:47:33<16:04:56,  1.81s/it]
Outer training loop:  36%|###6      | 18093/50000 [10:47:35<15:46:22,  1.78s/it]
Outer training loop:  36%|###6      | 18094/50000 [10:47:37<15:25:54,  1.74s/it]
Outer training loop:  36%|###6      | 18095/50000 [10:47:39<15:50:45,  1.79s/it]
Outer training loop:  36%|###6      | 18096/50000 [10:47:40<15:31:04,  1.75s/it]
Outer training loop:  36%|###6      | 18097/50000 [10:47:42<15:28:16,  1.75s/it]
Outer training loop:  36%|###6      | 18098/50000 [10:47:44<15:20:26,  1.73s/it]
Outer training loop:  36%|###6      | 18099/50000 [10:47:46<15:46:36,  1.78s/it]
Outer training loop:  36%|###6      | 18100/50000 [10:47:47<15:41:45,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.91s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  36%|###6      | 18101/50000 [10:47:54<29:36:48,  3.34s/it]
Outer training loop:  36%|###6      | 18102/50000 [10:47:56<25:14:40,  2.85s/it]
Outer training loop:  36%|###6      | 18103/50000 [10:47:58<22:37:49,  2.55s/it]
Outer training loop:  36%|###6      | 18104/50000 [10:48:00<20:24:16,  2.30s/it]
Outer training loop:  36%|###6      | 18105/50000 [10:48:01<18:49:10,  2.12s/it]
Outer training loop:  36%|###6      | 18106/50000 [10:48:03<17:57:43,  2.03s/it]
Outer training loop:  36%|###6      | 18107/50000 [10:48:05<17:07:00,  1.93s/it]
Outer training loop:  36%|###6      | 18108/50000 [10:48:07<16:45:07,  1.89s/it]
Outer training loop:  36%|###6      | 18109/50000 [10:48:09<16:48:29,  1.90s/it]
Outer training loop:  36%|###6      | 18110/50000 [10:48:10<16:15:44,  1.84s/it]
Outer training loop:  36%|###6      | 18111/50000 [10:48:12<16:02:06,  1.81s/it]
Outer training loop:  36%|###6      | 18112/50000 [10:48:14<15:49:34,  1.79s/it]
Outer training loop:  36%|###6      | 18113/50000 [10:48:16<16:02:52,  1.81s/it]
Outer training loop:  36%|###6      | 18114/50000 [10:48:17<15:44:58,  1.78s/it]
Outer training loop:  36%|###6      | 18115/50000 [10:48:19<15:33:42,  1.76s/it]
Outer training loop:  36%|###6      | 18116/50000 [10:48:21<16:02:17,  1.81s/it]
Outer training loop:  36%|###6      | 18117/50000 [10:48:23<15:50:20,  1.79s/it]
Outer training loop:  36%|###6      | 18118/50000 [10:48:24<15:43:09,  1.77s/it]
Outer training loop:  36%|###6      | 18119/50000 [10:48:26<15:55:29,  1.80s/it]
Outer training loop:  36%|###6      | 18120/50000 [10:48:28<15:43:32,  1.78s/it]
Outer training loop:  36%|###6      | 18121/50000 [10:48:30<15:35:29,  1.76s/it]
Outer training loop:  36%|###6      | 18122/50000 [10:48:32<16:03:08,  1.81s/it]
Outer training loop:  36%|###6      | 18123/50000 [10:48:33<15:58:33,  1.80s/it]
Outer training loop:  36%|###6      | 18124/50000 [10:48:35<15:55:59,  1.80s/it]
Outer training loop:  36%|###6      | 18125/50000 [10:48:37<15:43:29,  1.78s/it]
Outer training loop:  36%|###6      | 18126/50000 [10:48:39<16:03:25,  1.81s/it]
Outer training loop:  36%|###6      | 18127/50000 [10:48:40<15:21:35,  1.73s/it]
Outer training loop:  36%|###6      | 18128/50000 [10:48:42<15:25:28,  1.74s/it]
Outer training loop:  36%|###6      | 18129/50000 [10:48:44<16:38:07,  1.88s/it]
Outer training loop:  36%|###6      | 18130/50000 [10:48:46<16:20:45,  1.85s/it]
Outer training loop:  36%|###6      | 18131/50000 [10:48:48<16:07:43,  1.82s/it]
Outer training loop:  36%|###6      | 18132/50000 [10:48:50<16:17:38,  1.84s/it]
Outer training loop:  36%|###6      | 18133/50000 [10:48:51<15:46:04,  1.78s/it]
Outer training loop:  36%|###6      | 18134/50000 [10:48:53<15:42:55,  1.78s/it]
Outer training loop:  36%|###6      | 18135/50000 [10:48:55<15:34:49,  1.76s/it]
Outer training loop:  36%|###6      | 18136/50000 [10:48:57<15:50:57,  1.79s/it]
Outer training loop:  36%|###6      | 18137/50000 [10:48:59<15:44:08,  1.78s/it]
Outer training loop:  36%|###6      | 18138/50000 [10:49:00<15:38:56,  1.77s/it]
Outer training loop:  36%|###6      | 18139/50000 [10:49:02<16:14:12,  1.83s/it]
Outer training loop:  36%|###6      | 18140/50000 [10:49:04<16:06:28,  1.82s/it]
Outer training loop:  36%|###6      | 18141/50000 [10:49:06<15:58:50,  1.81s/it]
Outer training loop:  36%|###6      | 18142/50000 [10:49:08<16:19:26,  1.84s/it]
Outer training loop:  36%|###6      | 18143/50000 [10:49:10<16:07:10,  1.82s/it]
Outer training loop:  36%|###6      | 18144/50000 [10:49:11<16:05:57,  1.82s/it]
Outer training loop:  36%|###6      | 18145/50000 [10:49:13<16:12:17,  1.83s/it]
Outer training loop:  36%|###6      | 18146/50000 [10:49:15<16:06:21,  1.82s/it]
Outer training loop:  36%|###6      | 18147/50000 [10:49:17<15:53:54,  1.80s/it]
Outer training loop:  36%|###6      | 18148/50000 [10:49:19<16:14:58,  1.84s/it]
Outer training loop:  36%|###6      | 18149/50000 [10:49:20<15:57:30,  1.80s/it]
Outer training loop:  36%|###6      | 18150/50000 [10:49:22<15:34:37,  1.76s/it]
Outer training loop:  36%|###6      | 18151/50000 [10:49:24<15:36:51,  1.76s/it]
Outer training loop:  36%|###6      | 18152/50000 [10:49:26<16:03:11,  1.81s/it]
Outer training loop:  36%|###6      | 18153/50000 [10:49:27<15:48:41,  1.79s/it]
Outer training loop:  36%|###6      | 18154/50000 [10:49:29<15:35:40,  1.76s/it]
Outer training loop:  36%|###6      | 18155/50000 [10:49:31<16:00:40,  1.81s/it]
Outer training loop:  36%|###6      | 18156/50000 [10:49:33<15:41:29,  1.77s/it]
Outer training loop:  36%|###6      | 18157/50000 [10:49:34<15:17:14,  1.73s/it]
Outer training loop:  36%|###6      | 18158/50000 [10:49:36<15:25:10,  1.74s/it]
Outer training loop:  36%|###6      | 18159/50000 [10:49:38<15:45:17,  1.78s/it]
Outer training loop:  36%|###6      | 18160/50000 [10:49:40<15:27:22,  1.75s/it]
Outer training loop:  36%|###6      | 18161/50000 [10:49:41<15:20:56,  1.74s/it]
Outer training loop:  36%|###6      | 18162/50000 [10:49:43<15:42:07,  1.78s/it]
Outer training loop:  36%|###6      | 18163/50000 [10:49:45<15:24:40,  1.74s/it]
Outer training loop:  36%|###6      | 18164/50000 [10:49:47<15:25:13,  1.74s/it]
Outer training loop:  36%|###6      | 18165/50000 [10:49:49<15:48:56,  1.79s/it]
Outer training loop:  36%|###6      | 18166/50000 [10:49:50<15:44:13,  1.78s/it]
Outer training loop:  36%|###6      | 18167/50000 [10:49:52<15:25:38,  1.74s/it]
Outer training loop:  36%|###6      | 18168/50000 [10:49:54<15:53:54,  1.80s/it]
Outer training loop:  36%|###6      | 18169/50000 [10:49:56<15:49:46,  1.79s/it]
Outer training loop:  36%|###6      | 18170/50000 [10:49:57<15:48:00,  1.79s/it]
Outer training loop:  36%|###6      | 18171/50000 [10:49:59<16:00:43,  1.81s/it]
Outer training loop:  36%|###6      | 18172/50000 [10:50:01<15:39:42,  1.77s/it]
Outer training loop:  36%|###6      | 18173/50000 [10:50:03<15:02:40,  1.70s/it]
Outer training loop:  36%|###6      | 18174/50000 [10:50:04<15:13:47,  1.72s/it]
Outer training loop:  36%|###6      | 18175/50000 [10:50:06<15:47:55,  1.79s/it]
Outer training loop:  36%|###6      | 18176/50000 [10:50:08<15:38:30,  1.77s/it]
Outer training loop:  36%|###6      | 18177/50000 [10:50:10<15:29:21,  1.75s/it]
Outer training loop:  36%|###6      | 18178/50000 [10:50:12<15:54:36,  1.80s/it]
Outer training loop:  36%|###6      | 18179/50000 [10:50:14<16:21:54,  1.85s/it]
Outer training loop:  36%|###6      | 18180/50000 [10:50:15<16:06:31,  1.82s/it]
Outer training loop:  36%|###6      | 18181/50000 [10:50:17<15:38:39,  1.77s/it]
Outer training loop:  36%|###6      | 18182/50000 [10:50:19<15:51:16,  1.79s/it]
Outer training loop:  36%|###6      | 18183/50000 [10:50:21<15:46:17,  1.78s/it]
Outer training loop:  36%|###6      | 18184/50000 [10:50:22<15:32:33,  1.76s/it]
Outer training loop:  36%|###6      | 18185/50000 [10:50:24<16:12:41,  1.83s/it]
Outer training loop:  36%|###6      | 18186/50000 [10:50:26<15:58:41,  1.81s/it]
Outer training loop:  36%|###6      | 18187/50000 [10:50:28<15:39:41,  1.77s/it]
Outer training loop:  36%|###6      | 18188/50000 [10:50:29<15:30:14,  1.75s/it]
Outer training loop:  36%|###6      | 18189/50000 [10:50:31<15:29:49,  1.75s/it]
Outer training loop:  36%|###6      | 18190/50000 [10:50:33<15:23:40,  1.74s/it]
Outer training loop:  36%|###6      | 18191/50000 [10:50:35<15:37:21,  1.77s/it]
Outer training loop:  36%|###6      | 18192/50000 [10:50:37<16:04:40,  1.82s/it]
Outer training loop:  36%|###6      | 18193/50000 [10:50:38<15:46:32,  1.79s/it]
Outer training loop:  36%|###6      | 18194/50000 [10:50:40<15:56:16,  1.80s/it]
Outer training loop:  36%|###6      | 18195/50000 [10:50:42<15:35:02,  1.76s/it]
Outer training loop:  36%|###6      | 18196/50000 [10:50:44<15:55:20,  1.80s/it]
Outer training loop:  36%|###6      | 18197/50000 [10:50:46<15:39:14,  1.77s/it]
Outer training loop:  36%|###6      | 18198/50000 [10:50:47<15:41:21,  1.78s/it]
Outer training loop:  36%|###6      | 18199/50000 [10:50:49<15:56:16,  1.80s/it]
Outer training loop:  36%|###6      | 18200/50000 [10:50:51<15:25:57,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.94s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  36%|###6      | 18201/50000 [10:50:58<29:19:23,  3.32s/it]
Outer training loop:  36%|###6      | 18202/50000 [10:50:59<24:57:17,  2.83s/it]
Outer training loop:  36%|###6      | 18203/50000 [10:51:01<22:38:40,  2.56s/it]
Outer training loop:  36%|###6      | 18204/50000 [10:51:03<20:22:48,  2.31s/it]
Outer training loop:  36%|###6      | 18205/50000 [10:51:05<19:03:13,  2.16s/it]
Outer training loop:  36%|###6      | 18206/50000 [10:51:07<18:17:50,  2.07s/it]
Outer training loop:  36%|###6      | 18207/50000 [10:51:09<17:17:38,  1.96s/it]
Outer training loop:  36%|###6      | 18208/50000 [10:51:10<16:25:02,  1.86s/it]
Outer training loop:  36%|###6      | 18209/50000 [10:51:12<16:26:07,  1.86s/it]
Outer training loop:  36%|###6      | 18210/50000 [10:51:14<15:58:05,  1.81s/it]
Outer training loop:  36%|###6      | 18211/50000 [10:51:15<15:40:17,  1.77s/it]
Outer training loop:  36%|###6      | 18212/50000 [10:51:17<15:15:27,  1.73s/it]
Outer training loop:  36%|###6      | 18213/50000 [10:51:19<15:42:23,  1.78s/it]
Outer training loop:  36%|###6      | 18214/50000 [10:51:21<15:28:03,  1.75s/it]
Outer training loop:  36%|###6      | 18215/50000 [10:51:22<15:15:11,  1.73s/it]
Outer training loop:  36%|###6      | 18216/50000 [10:51:24<15:49:26,  1.79s/it]
Outer training loop:  36%|###6      | 18217/50000 [10:51:26<15:46:45,  1.79s/it]
Outer training loop:  36%|###6      | 18218/50000 [10:51:28<15:22:04,  1.74s/it]
Outer training loop:  36%|###6      | 18219/50000 [10:51:30<15:51:34,  1.80s/it]
Outer training loop:  36%|###6      | 18220/50000 [10:51:31<15:40:13,  1.78s/it]
Outer training loop:  36%|###6      | 18221/50000 [10:51:33<15:28:59,  1.75s/it]
Outer training loop:  36%|###6      | 18222/50000 [10:51:35<15:58:26,  1.81s/it]
Outer training loop:  36%|###6      | 18223/50000 [10:51:37<15:54:10,  1.80s/it]
Outer training loop:  36%|###6      | 18224/50000 [10:51:38<15:29:12,  1.75s/it]
Outer training loop:  36%|###6      | 18225/50000 [10:51:40<15:48:17,  1.79s/it]
Outer training loop:  36%|###6      | 18226/50000 [10:51:42<16:15:22,  1.84s/it]
Outer training loop:  36%|###6      | 18227/50000 [10:51:44<15:56:56,  1.81s/it]
Outer training loop:  36%|###6      | 18228/50000 [10:51:46<16:01:59,  1.82s/it]
Outer training loop:  36%|###6      | 18229/50000 [10:51:48<16:06:22,  1.82s/it]
Outer training loop:  36%|###6      | 18230/50000 [10:51:49<16:06:39,  1.83s/it]
Outer training loop:  36%|###6      | 18231/50000 [10:51:51<16:00:08,  1.81s/it]
Outer training loop:  36%|###6      | 18232/50000 [10:51:53<16:13:22,  1.84s/it]
Outer training loop:  36%|###6      | 18233/50000 [10:51:55<15:46:44,  1.79s/it]
Outer training loop:  36%|###6      | 18234/50000 [10:51:56<15:30:40,  1.76s/it]
Outer training loop:  36%|###6      | 18235/50000 [10:51:58<15:41:27,  1.78s/it]
Outer training loop:  36%|###6      | 18236/50000 [10:52:00<15:25:31,  1.75s/it]
Outer training loop:  36%|###6      | 18237/50000 [10:52:02<15:46:22,  1.79s/it]
Outer training loop:  36%|###6      | 18238/50000 [10:52:04<15:36:28,  1.77s/it]
Outer training loop:  36%|###6      | 18239/50000 [10:52:06<16:08:04,  1.83s/it]
Outer training loop:  36%|###6      | 18240/50000 [10:52:07<15:41:53,  1.78s/it]
Outer training loop:  36%|###6      | 18241/50000 [10:52:09<15:41:54,  1.78s/it]
Outer training loop:  36%|###6      | 18242/50000 [10:52:11<16:00:10,  1.81s/it]
Outer training loop:  36%|###6      | 18243/50000 [10:52:12<15:30:35,  1.76s/it]
Outer training loop:  36%|###6      | 18244/50000 [10:52:14<15:04:08,  1.71s/it]
Outer training loop:  36%|###6      | 18245/50000 [10:52:16<15:27:26,  1.75s/it]
Outer training loop:  36%|###6      | 18246/50000 [10:52:18<15:20:07,  1.74s/it]
Outer training loop:  36%|###6      | 18247/50000 [10:52:19<15:11:02,  1.72s/it]
Outer training loop:  36%|###6      | 18248/50000 [10:52:21<15:17:02,  1.73s/it]
Outer training loop:  36%|###6      | 18249/50000 [10:52:23<15:23:39,  1.75s/it]
Outer training loop:  36%|###6      | 18250/50000 [10:52:25<15:17:34,  1.73s/it]
Outer training loop:  37%|###6      | 18251/50000 [10:52:26<15:00:11,  1.70s/it]
Outer training loop:  37%|###6      | 18252/50000 [10:52:28<15:36:44,  1.77s/it]
Outer training loop:  37%|###6      | 18253/50000 [10:52:30<15:20:29,  1.74s/it]
Outer training loop:  37%|###6      | 18254/50000 [10:52:31<15:10:58,  1.72s/it]
Outer training loop:  37%|###6      | 18255/50000 [10:52:33<15:34:55,  1.77s/it]
Outer training loop:  37%|###6      | 18256/50000 [10:52:35<15:36:00,  1.77s/it]
Outer training loop:  37%|###6      | 18257/50000 [10:52:37<15:26:02,  1.75s/it]
Outer training loop:  37%|###6      | 18258/50000 [10:52:39<15:25:24,  1.75s/it]
Outer training loop:  37%|###6      | 18259/50000 [10:52:41<15:58:33,  1.81s/it]
Outer training loop:  37%|###6      | 18260/50000 [10:52:42<15:58:29,  1.81s/it]
Outer training loop:  37%|###6      | 18261/50000 [10:52:44<15:35:42,  1.77s/it]
Outer training loop:  37%|###6      | 18262/50000 [10:52:46<16:05:22,  1.83s/it]
Outer training loop:  37%|###6      | 18263/50000 [10:52:48<15:52:17,  1.80s/it]
Outer training loop:  37%|###6      | 18264/50000 [10:52:49<15:29:28,  1.76s/it]
Outer training loop:  37%|###6      | 18265/50000 [10:52:51<15:53:07,  1.80s/it]
Outer training loop:  37%|###6      | 18266/50000 [10:52:53<15:47:28,  1.79s/it]
Outer training loop:  37%|###6      | 18267/50000 [10:52:55<15:55:29,  1.81s/it]
Outer training loop:  37%|###6      | 18268/50000 [10:52:57<15:36:23,  1.77s/it]
Outer training loop:  37%|###6      | 18269/50000 [10:52:58<15:41:22,  1.78s/it]
Outer training loop:  37%|###6      | 18270/50000 [10:53:00<15:16:45,  1.73s/it]
Outer training loop:  37%|###6      | 18271/50000 [10:53:02<15:02:07,  1.71s/it]
Outer training loop:  37%|###6      | 18272/50000 [10:53:03<15:22:57,  1.75s/it]
Outer training loop:  37%|###6      | 18273/50000 [10:53:05<15:19:47,  1.74s/it]
Outer training loop:  37%|###6      | 18274/50000 [10:53:07<15:25:32,  1.75s/it]
Outer training loop:  37%|###6      | 18275/50000 [10:53:09<15:56:42,  1.81s/it]
Outer training loop:  37%|###6      | 18276/50000 [10:53:11<16:25:36,  1.86s/it]
Outer training loop:  37%|###6      | 18277/50000 [10:53:13<16:05:22,  1.83s/it]
Outer training loop:  37%|###6      | 18278/50000 [10:53:15<16:29:38,  1.87s/it]
Outer training loop:  37%|###6      | 18279/50000 [10:53:16<15:47:53,  1.79s/it]
Outer training loop:  37%|###6      | 18280/50000 [10:53:18<15:39:20,  1.78s/it]
Outer training loop:  37%|###6      | 18281/50000 [10:53:20<15:36:46,  1.77s/it]
Outer training loop:  37%|###6      | 18282/50000 [10:53:22<16:03:37,  1.82s/it]
Outer training loop:  37%|###6      | 18283/50000 [10:53:23<15:50:23,  1.80s/it]
Outer training loop:  37%|###6      | 18284/50000 [10:53:25<15:39:19,  1.78s/it]
Outer training loop:  37%|###6      | 18285/50000 [10:53:27<16:17:02,  1.85s/it]
Outer training loop:  37%|###6      | 18286/50000 [10:53:29<16:02:45,  1.82s/it]
Outer training loop:  37%|###6      | 18287/50000 [10:53:31<15:39:28,  1.78s/it]
Outer training loop:  37%|###6      | 18288/50000 [10:53:33<16:10:36,  1.84s/it]
Outer training loop:  37%|###6      | 18289/50000 [10:53:34<15:59:32,  1.82s/it]
Outer training loop:  37%|###6      | 18290/50000 [10:53:36<15:54:36,  1.81s/it]
Outer training loop:  37%|###6      | 18291/50000 [10:53:38<15:45:49,  1.79s/it]
Outer training loop:  37%|###6      | 18292/50000 [10:53:40<16:12:49,  1.84s/it]
Outer training loop:  37%|###6      | 18293/50000 [10:53:42<15:51:33,  1.80s/it]
Outer training loop:  37%|###6      | 18294/50000 [10:53:43<15:36:36,  1.77s/it]
Outer training loop:  37%|###6      | 18295/50000 [10:53:45<16:11:34,  1.84s/it]
Outer training loop:  37%|###6      | 18296/50000 [10:53:47<15:44:12,  1.79s/it]
Outer training loop:  37%|###6      | 18297/50000 [10:53:49<15:22:03,  1.75s/it]
Outer training loop:  37%|###6      | 18298/50000 [10:53:50<15:21:26,  1.74s/it]
Outer training loop:  37%|###6      | 18299/50000 [10:53:52<15:50:02,  1.80s/it]
Outer training loop:  37%|###6      | 18300/50000 [10:53:54<15:38:32,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.87s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  37%|###6      | 18301/50000 [10:54:01<29:29:54,  3.35s/it]
Outer training loop:  37%|###6      | 18302/50000 [10:54:03<25:28:28,  2.89s/it]
Outer training loop:  37%|###6      | 18303/50000 [10:54:05<22:40:27,  2.58s/it]
Outer training loop:  37%|###6      | 18304/50000 [10:54:06<20:37:28,  2.34s/it]
Outer training loop:  37%|###6      | 18305/50000 [10:54:08<19:36:28,  2.23s/it]
Outer training loop:  37%|###6      | 18306/50000 [10:54:10<18:17:58,  2.08s/it]
Outer training loop:  37%|###6      | 18307/50000 [10:54:12<17:25:51,  1.98s/it]
Outer training loop:  37%|###6      | 18308/50000 [10:54:14<16:40:52,  1.89s/it]
Outer training loop:  37%|###6      | 18309/50000 [10:54:15<16:36:56,  1.89s/it]
Outer training loop:  37%|###6      | 18310/50000 [10:54:17<16:19:46,  1.86s/it]
Outer training loop:  37%|###6      | 18311/50000 [10:54:19<16:11:51,  1.84s/it]
Outer training loop:  37%|###6      | 18312/50000 [10:54:21<16:35:27,  1.88s/it]
Outer training loop:  37%|###6      | 18313/50000 [10:54:23<16:07:02,  1.83s/it]
Outer training loop:  37%|###6      | 18314/50000 [10:54:24<15:49:23,  1.80s/it]
Outer training loop:  37%|###6      | 18315/50000 [10:54:26<16:08:16,  1.83s/it]
Outer training loop:  37%|###6      | 18316/50000 [10:54:28<15:51:19,  1.80s/it]
Outer training loop:  37%|###6      | 18317/50000 [10:54:30<15:36:45,  1.77s/it]
Outer training loop:  37%|###6      | 18318/50000 [10:54:31<15:13:44,  1.73s/it]
Outer training loop:  37%|###6      | 18319/50000 [10:54:33<15:31:57,  1.77s/it]
Outer training loop:  37%|###6      | 18320/50000 [10:54:35<15:36:01,  1.77s/it]
Outer training loop:  37%|###6      | 18321/50000 [10:54:37<15:39:12,  1.78s/it]
Outer training loop:  37%|###6      | 18322/50000 [10:54:39<15:52:26,  1.80s/it]
Outer training loop:  37%|###6      | 18323/50000 [10:54:41<16:06:41,  1.83s/it]
Outer training loop:  37%|###6      | 18324/50000 [10:54:42<15:43:45,  1.79s/it]
Outer training loop:  37%|###6      | 18325/50000 [10:54:44<15:11:34,  1.73s/it]
Outer training loop:  37%|###6      | 18326/50000 [10:54:46<15:42:49,  1.79s/it]
Outer training loop:  37%|###6      | 18327/50000 [10:54:48<15:45:25,  1.79s/it]
Outer training loop:  37%|###6      | 18328/50000 [10:54:49<15:35:38,  1.77s/it]
Outer training loop:  37%|###6      | 18329/50000 [10:54:51<16:17:08,  1.85s/it]
Outer training loop:  37%|###6      | 18330/50000 [10:54:53<15:59:43,  1.82s/it]
Outer training loop:  37%|###6      | 18331/50000 [10:54:55<15:45:07,  1.79s/it]
Outer training loop:  37%|###6      | 18332/50000 [10:54:57<16:06:54,  1.83s/it]
Outer training loop:  37%|###6      | 18333/50000 [10:54:59<15:56:59,  1.81s/it]
Outer training loop:  37%|###6      | 18334/50000 [10:55:00<15:34:07,  1.77s/it]
Outer training loop:  37%|###6      | 18335/50000 [10:55:02<16:03:01,  1.82s/it]
Outer training loop:  37%|###6      | 18336/50000 [10:55:04<15:53:41,  1.81s/it]
Outer training loop:  37%|###6      | 18337/50000 [10:55:06<15:49:56,  1.80s/it]
Outer training loop:  37%|###6      | 18338/50000 [10:55:07<15:29:06,  1.76s/it]
Outer training loop:  37%|###6      | 18339/50000 [10:55:09<15:53:23,  1.81s/it]
Outer training loop:  37%|###6      | 18340/50000 [10:55:11<15:29:27,  1.76s/it]
Outer training loop:  37%|###6      | 18341/50000 [10:55:13<15:14:31,  1.73s/it]
Outer training loop:  37%|###6      | 18342/50000 [10:55:14<15:20:50,  1.75s/it]
Outer training loop:  37%|###6      | 18343/50000 [10:55:16<15:52:52,  1.81s/it]
Outer training loop:  37%|###6      | 18344/50000 [10:55:18<15:28:24,  1.76s/it]
Outer training loop:  37%|###6      | 18345/50000 [10:55:20<15:12:11,  1.73s/it]
Outer training loop:  37%|###6      | 18346/50000 [10:55:22<15:44:03,  1.79s/it]
Outer training loop:  37%|###6      | 18347/50000 [10:55:23<15:15:09,  1.73s/it]
Outer training loop:  37%|###6      | 18348/50000 [10:55:25<15:14:19,  1.73s/it]
Outer training loop:  37%|###6      | 18349/50000 [10:55:27<15:19:15,  1.74s/it]
Outer training loop:  37%|###6      | 18350/50000 [10:55:29<15:43:56,  1.79s/it]
Outer training loop:  37%|###6      | 18351/50000 [10:55:30<15:36:38,  1.78s/it]
Outer training loop:  37%|###6      | 18352/50000 [10:55:32<15:25:27,  1.75s/it]
Outer training loop:  37%|###6      | 18353/50000 [10:55:34<15:46:37,  1.79s/it]
Outer training loop:  37%|###6      | 18354/50000 [10:55:36<15:27:47,  1.76s/it]
Outer training loop:  37%|###6      | 18355/50000 [10:55:37<15:16:26,  1.74s/it]
Outer training loop:  37%|###6      | 18356/50000 [10:55:39<15:28:00,  1.76s/it]
Outer training loop:  37%|###6      | 18357/50000 [10:55:41<15:22:29,  1.75s/it]
Outer training loop:  37%|###6      | 18358/50000 [10:55:43<15:18:12,  1.74s/it]
Outer training loop:  37%|###6      | 18359/50000 [10:55:44<15:42:29,  1.79s/it]
Outer training loop:  37%|###6      | 18360/50000 [10:55:46<15:38:50,  1.78s/it]
Outer training loop:  37%|###6      | 18361/50000 [10:55:48<15:47:53,  1.80s/it]
Outer training loop:  37%|###6      | 18362/50000 [10:55:50<15:33:41,  1.77s/it]
Outer training loop:  37%|###6      | 18363/50000 [10:55:52<15:43:58,  1.79s/it]
Outer training loop:  37%|###6      | 18364/50000 [10:55:53<15:32:48,  1.77s/it]
Outer training loop:  37%|###6      | 18365/50000 [10:55:55<15:34:01,  1.77s/it]
Outer training loop:  37%|###6      | 18366/50000 [10:55:57<15:46:51,  1.80s/it]
Outer training loop:  37%|###6      | 18367/50000 [10:55:59<15:31:47,  1.77s/it]
Outer training loop:  37%|###6      | 18368/50000 [10:56:00<15:27:09,  1.76s/it]
Outer training loop:  37%|###6      | 18369/50000 [10:56:02<15:43:57,  1.79s/it]
Outer training loop:  37%|###6      | 18370/50000 [10:56:04<15:33:48,  1.77s/it]
Outer training loop:  37%|###6      | 18371/50000 [10:56:06<15:31:02,  1.77s/it]
Outer training loop:  37%|###6      | 18372/50000 [10:56:08<15:38:00,  1.78s/it]
Outer training loop:  37%|###6      | 18373/50000 [10:56:10<16:14:22,  1.85s/it]
Outer training loop:  37%|###6      | 18374/50000 [10:56:11<16:02:59,  1.83s/it]
Outer training loop:  37%|###6      | 18375/50000 [10:56:13<15:46:53,  1.80s/it]
Outer training loop:  37%|###6      | 18376/50000 [10:56:15<16:06:21,  1.83s/it]
Outer training loop:  37%|###6      | 18377/50000 [10:56:17<15:59:29,  1.82s/it]
Outer training loop:  37%|###6      | 18378/50000 [10:56:19<15:53:13,  1.81s/it]
Outer training loop:  37%|###6      | 18379/50000 [10:56:20<16:13:38,  1.85s/it]
Outer training loop:  37%|###6      | 18380/50000 [10:56:22<16:09:47,  1.84s/it]
Outer training loop:  37%|###6      | 18381/50000 [10:56:24<15:45:01,  1.79s/it]
Outer training loop:  37%|###6      | 18382/50000 [10:56:26<16:22:31,  1.86s/it]
Outer training loop:  37%|###6      | 18383/50000 [10:56:28<16:17:20,  1.85s/it]
Outer training loop:  37%|###6      | 18384/50000 [10:56:30<15:53:39,  1.81s/it]
Outer training loop:  37%|###6      | 18385/50000 [10:56:31<15:38:05,  1.78s/it]
Outer training loop:  37%|###6      | 18386/50000 [10:56:33<16:08:31,  1.84s/it]
Outer training loop:  37%|###6      | 18387/50000 [10:56:35<15:59:01,  1.82s/it]
Outer training loop:  37%|###6      | 18388/50000 [10:56:37<15:43:28,  1.79s/it]
Outer training loop:  37%|###6      | 18389/50000 [10:56:39<16:01:46,  1.83s/it]
Outer training loop:  37%|###6      | 18390/50000 [10:56:40<15:50:14,  1.80s/it]
Outer training loop:  37%|###6      | 18391/50000 [10:56:42<15:52:36,  1.81s/it]
Outer training loop:  37%|###6      | 18392/50000 [10:56:44<16:01:32,  1.83s/it]
Outer training loop:  37%|###6      | 18393/50000 [10:56:46<15:34:00,  1.77s/it]
Outer training loop:  37%|###6      | 18394/50000 [10:56:47<15:14:04,  1.74s/it]
Outer training loop:  37%|###6      | 18395/50000 [10:56:49<15:48:40,  1.80s/it]
Outer training loop:  37%|###6      | 18396/50000 [10:56:51<15:27:52,  1.76s/it]
Outer training loop:  37%|###6      | 18397/50000 [10:56:53<15:39:53,  1.78s/it]
Outer training loop:  37%|###6      | 18398/50000 [10:56:55<15:56:26,  1.82s/it]
Outer training loop:  37%|###6      | 18399/50000 [10:56:56<15:28:23,  1.76s/it]
Outer training loop:  37%|###6      | 18400/50000 [10:56:58<15:36:46,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.79s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  37%|###6      | 18401/50000 [10:57:05<29:12:48,  3.33s/it]
Outer training loop:  37%|###6      | 18402/50000 [10:57:07<25:23:29,  2.89s/it]
Outer training loop:  37%|###6      | 18403/50000 [10:57:09<22:07:13,  2.52s/it]
Outer training loop:  37%|###6      | 18404/50000 [10:57:10<20:09:37,  2.30s/it]
Outer training loop:  37%|###6      | 18405/50000 [10:57:12<19:16:34,  2.20s/it]
Outer training loop:  37%|###6      | 18406/50000 [10:57:14<18:03:46,  2.06s/it]
Outer training loop:  37%|###6      | 18407/50000 [10:57:16<17:12:21,  1.96s/it]
Outer training loop:  37%|###6      | 18408/50000 [10:57:18<16:29:42,  1.88s/it]
Outer training loop:  37%|###6      | 18409/50000 [10:57:20<16:45:11,  1.91s/it]
Outer training loop:  37%|###6      | 18410/50000 [10:57:21<15:53:15,  1.81s/it]
Outer training loop:  37%|###6      | 18411/50000 [10:57:23<15:19:33,  1.75s/it]
Outer training loop:  37%|###6      | 18412/50000 [10:57:25<15:35:50,  1.78s/it]
Outer training loop:  37%|###6      | 18413/50000 [10:57:26<15:48:53,  1.80s/it]
Outer training loop:  37%|###6      | 18414/50000 [10:57:28<15:27:10,  1.76s/it]
Outer training loop:  37%|###6      | 18415/50000 [10:57:30<15:03:49,  1.72s/it]
Outer training loop:  37%|###6      | 18416/50000 [10:57:32<15:35:21,  1.78s/it]
Outer training loop:  37%|###6      | 18417/50000 [10:57:33<15:43:22,  1.79s/it]
Outer training loop:  37%|###6      | 18418/50000 [10:57:35<15:44:13,  1.79s/it]
Outer training loop:  37%|###6      | 18419/50000 [10:57:37<16:02:19,  1.83s/it]
Outer training loop:  37%|###6      | 18420/50000 [10:57:39<16:18:43,  1.86s/it]
Outer training loop:  37%|###6      | 18421/50000 [10:57:41<15:58:29,  1.82s/it]
Outer training loop:  37%|###6      | 18422/50000 [10:57:43<15:47:10,  1.80s/it]
Outer training loop:  37%|###6      | 18423/50000 [10:57:44<15:56:26,  1.82s/it]
Outer training loop:  37%|###6      | 18424/50000 [10:57:46<15:40:50,  1.79s/it]
Outer training loop:  37%|###6      | 18425/50000 [10:57:48<15:24:51,  1.76s/it]
Outer training loop:  37%|###6      | 18426/50000 [10:57:50<15:52:43,  1.81s/it]
Outer training loop:  37%|###6      | 18427/50000 [10:57:51<15:39:56,  1.79s/it]
Outer training loop:  37%|###6      | 18428/50000 [10:57:53<15:16:58,  1.74s/it]
Outer training loop:  37%|###6      | 18429/50000 [10:57:55<15:55:48,  1.82s/it]
Outer training loop:  37%|###6      | 18430/50000 [10:57:57<15:40:44,  1.79s/it]
Outer training loop:  37%|###6      | 18431/50000 [10:57:59<15:38:50,  1.78s/it]
Outer training loop:  37%|###6      | 18432/50000 [10:58:01<16:26:21,  1.87s/it]
Outer training loop:  37%|###6      | 18433/50000 [10:58:02<16:16:32,  1.86s/it]
Outer training loop:  37%|###6      | 18434/50000 [10:58:04<15:55:59,  1.82s/it]
Outer training loop:  37%|###6      | 18435/50000 [10:58:06<15:54:43,  1.81s/it]
Outer training loop:  37%|###6      | 18436/50000 [10:58:08<16:08:55,  1.84s/it]
Outer training loop:  37%|###6      | 18437/50000 [10:58:10<15:50:21,  1.81s/it]
Outer training loop:  37%|###6      | 18438/50000 [10:58:11<15:47:17,  1.80s/it]
Outer training loop:  37%|###6      | 18439/50000 [10:58:13<16:02:00,  1.83s/it]
Outer training loop:  37%|###6      | 18440/50000 [10:58:15<15:55:11,  1.82s/it]
Outer training loop:  37%|###6      | 18441/50000 [10:58:17<15:47:40,  1.80s/it]
Outer training loop:  37%|###6      | 18442/50000 [10:58:19<16:00:29,  1.83s/it]
Outer training loop:  37%|###6      | 18443/50000 [10:58:21<15:45:31,  1.80s/it]
Outer training loop:  37%|###6      | 18444/50000 [10:58:22<15:33:04,  1.77s/it]
Outer training loop:  37%|###6      | 18445/50000 [10:58:24<15:14:22,  1.74s/it]
Outer training loop:  37%|###6      | 18446/50000 [10:58:26<15:51:59,  1.81s/it]
Outer training loop:  37%|###6      | 18447/50000 [10:58:28<15:35:22,  1.78s/it]
Outer training loop:  37%|###6      | 18448/50000 [10:58:29<15:26:57,  1.76s/it]
Outer training loop:  37%|###6      | 18449/50000 [10:58:31<15:30:40,  1.77s/it]
Outer training loop:  37%|###6      | 18450/50000 [10:58:33<15:18:26,  1.75s/it]
Outer training loop:  37%|###6      | 18451/50000 [10:58:35<15:24:21,  1.76s/it]
Outer training loop:  37%|###6      | 18452/50000 [10:58:36<15:16:09,  1.74s/it]
Outer training loop:  37%|###6      | 18453/50000 [10:58:38<15:44:55,  1.80s/it]
Outer training loop:  37%|###6      | 18454/50000 [10:58:40<15:28:17,  1.77s/it]
Outer training loop:  37%|###6      | 18455/50000 [10:58:42<15:14:16,  1.74s/it]
Outer training loop:  37%|###6      | 18456/50000 [10:58:43<15:38:09,  1.78s/it]
Outer training loop:  37%|###6      | 18457/50000 [10:58:45<15:20:20,  1.75s/it]
Outer training loop:  37%|###6      | 18458/50000 [10:58:47<15:09:49,  1.73s/it]
Outer training loop:  37%|###6      | 18459/50000 [10:58:49<15:50:19,  1.81s/it]
Outer training loop:  37%|###6      | 18460/50000 [10:58:51<15:37:46,  1.78s/it]
Outer training loop:  37%|###6      | 18461/50000 [10:58:52<15:25:27,  1.76s/it]
Outer training loop:  37%|###6      | 18462/50000 [10:58:54<15:13:07,  1.74s/it]
Outer training loop:  37%|###6      | 18463/50000 [10:58:56<15:55:42,  1.82s/it]
Outer training loop:  37%|###6      | 18464/50000 [10:58:58<15:35:44,  1.78s/it]
Outer training loop:  37%|###6      | 18465/50000 [10:58:59<15:26:08,  1.76s/it]
Outer training loop:  37%|###6      | 18466/50000 [10:59:01<15:55:10,  1.82s/it]
Outer training loop:  37%|###6      | 18467/50000 [10:59:03<15:38:34,  1.79s/it]
Outer training loop:  37%|###6      | 18468/50000 [10:59:05<15:36:26,  1.78s/it]
Outer training loop:  37%|###6      | 18469/50000 [10:59:07<15:57:19,  1.82s/it]
Outer training loop:  37%|###6      | 18470/50000 [10:59:09<16:24:06,  1.87s/it]
Outer training loop:  37%|###6      | 18471/50000 [10:59:10<16:06:40,  1.84s/it]
Outer training loop:  37%|###6      | 18472/50000 [10:59:12<16:12:16,  1.85s/it]
Outer training loop:  37%|###6      | 18473/50000 [10:59:14<15:43:40,  1.80s/it]
Outer training loop:  37%|###6      | 18474/50000 [10:59:16<15:24:41,  1.76s/it]
Outer training loop:  37%|###6      | 18475/50000 [10:59:18<15:47:15,  1.80s/it]
Outer training loop:  37%|###6      | 18476/50000 [10:59:19<15:36:12,  1.78s/it]
Outer training loop:  37%|###6      | 18477/50000 [10:59:21<15:23:47,  1.76s/it]
Outer training loop:  37%|###6      | 18478/50000 [10:59:23<15:15:09,  1.74s/it]
Outer training loop:  37%|###6      | 18479/50000 [10:59:25<15:59:13,  1.83s/it]
Outer training loop:  37%|###6      | 18480/50000 [10:59:26<15:41:47,  1.79s/it]
Outer training loop:  37%|###6      | 18481/50000 [10:59:28<15:45:12,  1.80s/it]
Outer training loop:  37%|###6      | 18482/50000 [10:59:30<16:10:18,  1.85s/it]
Outer training loop:  37%|###6      | 18483/50000 [10:59:32<15:50:31,  1.81s/it]
Outer training loop:  37%|###6      | 18484/50000 [10:59:34<15:45:21,  1.80s/it]
Outer training loop:  37%|###6      | 18485/50000 [10:59:36<16:03:54,  1.84s/it]
Outer training loop:  37%|###6      | 18486/50000 [10:59:37<15:51:17,  1.81s/it]
Outer training loop:  37%|###6      | 18487/50000 [10:59:39<15:30:46,  1.77s/it]
Outer training loop:  37%|###6      | 18488/50000 [10:59:41<16:05:04,  1.84s/it]
Outer training loop:  37%|###6      | 18489/50000 [10:59:43<15:46:41,  1.80s/it]
Outer training loop:  37%|###6      | 18490/50000 [10:59:45<15:47:18,  1.80s/it]
Outer training loop:  37%|###6      | 18491/50000 [10:59:46<15:36:01,  1.78s/it]
Outer training loop:  37%|###6      | 18492/50000 [10:59:48<15:39:00,  1.79s/it]
Outer training loop:  37%|###6      | 18493/50000 [10:59:50<15:41:42,  1.79s/it]
Outer training loop:  37%|###6      | 18494/50000 [10:59:52<15:30:03,  1.77s/it]
Outer training loop:  37%|###6      | 18495/50000 [10:59:54<16:01:42,  1.83s/it]
Outer training loop:  37%|###6      | 18496/50000 [10:59:55<15:56:16,  1.82s/it]
Outer training loop:  37%|###6      | 18497/50000 [10:59:57<15:22:27,  1.76s/it]
Outer training loop:  37%|###6      | 18498/50000 [10:59:59<15:54:18,  1.82s/it]
Outer training loop:  37%|###6      | 18499/50000 [11:00:01<15:45:02,  1.80s/it]
Outer training loop:  37%|###7      | 18500/50000 [11:00:02<15:14:13,  1.74s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:30,  1.84s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.79s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:22,  1.80s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.77s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.77s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:14,  1.76s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:14,  1.82s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:10,  1.79s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:07,  1.76s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:08,  1.81s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:03,  1.77s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:00,  1.75s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<01:58,  1.74s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:57,  1.76s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<02:01,  1.83s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:53,  1.78s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:52,  1.78s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:53,  1.83s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:50,  1.82s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:48,  1.81s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:44,  1.77s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:42,  1.77s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:44,  1.84s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:41,  1.81s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:37,  1.78s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:33,  1.73s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:32,  1.74s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:32,  1.77s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:30,  1.80s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:25,  1.75s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:22,  1.72s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:21,  1.74s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:18,  1.71s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:20,  1.78s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:17,  1.77s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.73s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:12,  1.72s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:11,  1.74s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:10,  1.77s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:17<01:07,  1.73s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:09,  1.83s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:05,  1.77s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:02,  1.75s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:00,  1.73s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<00:58,  1.71s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:55,  1.70s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:53,  1.68s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:52,  1.75s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:50,  1.73s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.72s/it]

Evaluating policy:  67%|######7   | 56/83 [01:38<00:46,  1.72s/it]

Evaluating policy:  69%|######8   | 57/83 [01:40<00:46,  1.77s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:44,  1.77s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.75s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:43,  1.88s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:37,  1.80s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:35,  1.77s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:33,  1.74s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:54<00:31,  1.72s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:30,  1.78s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:28,  1.76s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:28,  1.88s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:25,  1.80s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:23,  1.79s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:21,  1.76s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:19,  1.74s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.72s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:15,  1.74s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:14,  1.76s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.75s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:11,  1.92s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:09,  1.87s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.84s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.78s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.72s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:13,  1.62s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:13,  1.65s/it]

Evaluating policy:   4%|3         | 3/83 [00:04<02:11,  1.64s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:25,  1.85s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.78s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.77s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:15,  1.78s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:16,  1.82s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:11,  1.78s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:09,  1.77s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:07,  1.77s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.82s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:04,  1.78s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:01,  1.76s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<01:58,  1.74s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:01,  1.81s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:56,  1.77s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:53,  1.75s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:50,  1.72s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:53,  1.80s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:49,  1.77s/it]

Evaluating policy:  27%|##6       | 22/83 [00:38<01:46,  1.74s/it]

Evaluating policy:  28%|##7       | 23/83 [00:40<01:45,  1.75s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:41,  1.73s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:45,  1.82s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:42,  1.80s/it]

Evaluating policy:  33%|###2      | 27/83 [00:47<01:37,  1.74s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:35,  1.74s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:33,  1.73s/it]

Evaluating policy:  36%|###6      | 30/83 [00:52<01:30,  1.71s/it]

Evaluating policy:  37%|###7      | 31/83 [00:54<01:33,  1.79s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:29,  1.76s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:31,  1.82s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:27,  1.79s/it]

Evaluating policy:  42%|####2     | 35/83 [01:01<01:25,  1.78s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:28,  1.89s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:20,  1.79s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.75s/it]

Evaluating policy:  48%|####8     | 40/83 [01:10<01:15,  1.75s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:13,  1.76s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:12,  1.77s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:11,  1.84s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:10,  1.85s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:06,  1.80s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:04,  1.78s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:01,  1.75s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<00:59,  1.74s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:55,  1.80s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:53,  1.77s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.74s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:47,  1.75s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:46,  1.85s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:43,  1.81s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.77s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:38,  1.76s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:36,  1.73s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:34,  1.70s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:32,  1.69s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:32,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:30,  1.80s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:28,  1.79s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:26,  1.77s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.76s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.73s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:20,  1.73s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:18,  1.68s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:16,  1.68s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.83s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.78s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.77s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.73s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:06,  1.73s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.73s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.70s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.70s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.94s/it]

Outer training loop:  37%|###7      | 18501/50000 [11:05:05<804:11:12, 91.91s/it]
Outer training loop:  37%|###7      | 18502/50000 [11:05:06<567:29:37, 64.86s/it]
Outer training loop:  37%|###7      | 18503/50000 [11:05:08<401:53:05, 45.93s/it]
Outer training loop:  37%|###7      | 18504/50000 [11:05:10<285:49:27, 32.67s/it]
Outer training loop:  37%|###7      | 18505/50000 [11:05:12<204:44:54, 23.40s/it]
Outer training loop:  37%|###7      | 18506/50000 [11:05:13<147:43:39, 16.89s/it]
Outer training loop:  37%|###7      | 18507/50000 [11:05:15<108:15:21, 12.37s/it]
Outer training loop:  37%|###7      | 18508/50000 [11:05:17<80:09:30,  9.16s/it] 
Outer training loop:  37%|###7      | 18509/50000 [11:05:19<60:35:34,  6.93s/it]
Outer training loop:  37%|###7      | 18510/50000 [11:05:20<47:08:39,  5.39s/it]
Outer training loop:  37%|###7      | 18511/50000 [11:05:23<38:51:11,  4.44s/it]
Outer training loop:  37%|###7      | 18512/50000 [11:05:24<31:47:22,  3.63s/it]
Outer training loop:  37%|###7      | 18513/50000 [11:05:26<26:50:26,  3.07s/it]
Outer training loop:  37%|###7      | 18514/50000 [11:05:28<23:33:44,  2.69s/it]
Outer training loop:  37%|###7      | 18515/50000 [11:05:30<21:30:20,  2.46s/it]
Outer training loop:  37%|###7      | 18516/50000 [11:05:32<19:36:39,  2.24s/it]
Outer training loop:  37%|###7      | 18517/50000 [11:05:33<17:57:35,  2.05s/it]
Outer training loop:  37%|###7      | 18518/50000 [11:05:35<17:52:18,  2.04s/it]
Outer training loop:  37%|###7      | 18519/50000 [11:05:37<16:57:24,  1.94s/it]
Outer training loop:  37%|###7      | 18520/50000 [11:05:39<16:08:54,  1.85s/it]
Outer training loop:  37%|###7      | 18521/50000 [11:05:40<15:52:27,  1.82s/it]
Outer training loop:  37%|###7      | 18522/50000 [11:05:42<16:05:54,  1.84s/it]
Outer training loop:  37%|###7      | 18523/50000 [11:05:44<16:02:04,  1.83s/it]
Outer training loop:  37%|###7      | 18524/50000 [11:05:46<15:24:19,  1.76s/it]
Outer training loop:  37%|###7      | 18525/50000 [11:05:47<15:38:35,  1.79s/it]
Outer training loop:  37%|###7      | 18526/50000 [11:05:49<15:38:39,  1.79s/it]
Outer training loop:  37%|###7      | 18527/50000 [11:05:51<15:23:01,  1.76s/it]
Outer training loop:  37%|###7      | 18528/50000 [11:05:53<15:26:28,  1.77s/it]
Outer training loop:  37%|###7      | 18529/50000 [11:05:55<15:53:53,  1.82s/it]
Outer training loop:  37%|###7      | 18530/50000 [11:05:56<15:28:40,  1.77s/it]
Outer training loop:  37%|###7      | 18531/50000 [11:05:58<15:12:49,  1.74s/it]
Outer training loop:  37%|###7      | 18532/50000 [11:06:00<15:33:05,  1.78s/it]
Outer training loop:  37%|###7      | 18533/50000 [11:06:02<15:32:41,  1.78s/it]
Outer training loop:  37%|###7      | 18534/50000 [11:06:03<15:21:05,  1.76s/it]
Outer training loop:  37%|###7      | 18535/50000 [11:06:05<15:02:09,  1.72s/it]
Outer training loop:  37%|###7      | 18536/50000 [11:06:07<15:36:36,  1.79s/it]
Outer training loop:  37%|###7      | 18537/50000 [11:06:09<15:30:42,  1.77s/it]
Outer training loop:  37%|###7      | 18538/50000 [11:06:10<15:24:07,  1.76s/it]
Outer training loop:  37%|###7      | 18539/50000 [11:06:12<15:38:27,  1.79s/it]
Outer training loop:  37%|###7      | 18540/50000 [11:06:14<15:31:40,  1.78s/it]
Outer training loop:  37%|###7      | 18541/50000 [11:06:16<15:17:40,  1.75s/it]
Outer training loop:  37%|###7      | 18542/50000 [11:06:18<15:41:57,  1.80s/it]
Outer training loop:  37%|###7      | 18543/50000 [11:06:19<15:42:58,  1.80s/it]
Outer training loop:  37%|###7      | 18544/50000 [11:06:21<15:40:39,  1.79s/it]
Outer training loop:  37%|###7      | 18545/50000 [11:06:23<15:37:22,  1.79s/it]
Outer training loop:  37%|###7      | 18546/50000 [11:06:25<16:13:48,  1.86s/it]
Outer training loop:  37%|###7      | 18547/50000 [11:06:27<15:43:15,  1.80s/it]
Outer training loop:  37%|###7      | 18548/50000 [11:06:28<15:08:16,  1.73s/it]
Outer training loop:  37%|###7      | 18549/50000 [11:06:30<15:31:43,  1.78s/it]
Outer training loop:  37%|###7      | 18550/50000 [11:06:32<15:25:56,  1.77s/it]
Outer training loop:  37%|###7      | 18551/50000 [11:06:34<15:54:21,  1.82s/it]
Outer training loop:  37%|###7      | 18552/50000 [11:06:36<16:18:58,  1.87s/it]
Outer training loop:  37%|###7      | 18553/50000 [11:06:37<15:46:27,  1.81s/it]
Outer training loop:  37%|###7      | 18554/50000 [11:06:39<15:39:16,  1.79s/it]
Outer training loop:  37%|###7      | 18555/50000 [11:06:41<15:41:34,  1.80s/it]
Outer training loop:  37%|###7      | 18556/50000 [11:06:43<16:02:15,  1.84s/it]
Outer training loop:  37%|###7      | 18557/50000 [11:06:45<15:59:52,  1.83s/it]
Outer training loop:  37%|###7      | 18558/50000 [11:06:46<15:30:32,  1.78s/it]
Outer training loop:  37%|###7      | 18559/50000 [11:06:48<15:43:10,  1.80s/it]
Outer training loop:  37%|###7      | 18560/50000 [11:06:50<15:38:44,  1.79s/it]
Outer training loop:  37%|###7      | 18561/50000 [11:06:52<15:31:10,  1.78s/it]
Outer training loop:  37%|###7      | 18562/50000 [11:06:53<15:22:59,  1.76s/it]
Outer training loop:  37%|###7      | 18563/50000 [11:06:55<15:48:17,  1.81s/it]
Outer training loop:  37%|###7      | 18564/50000 [11:06:57<15:44:30,  1.80s/it]
Outer training loop:  37%|###7      | 18565/50000 [11:06:59<15:27:10,  1.77s/it]
Outer training loop:  37%|###7      | 18566/50000 [11:07:01<16:10:58,  1.85s/it]
Outer training loop:  37%|###7      | 18567/50000 [11:07:03<16:02:27,  1.84s/it]
Outer training loop:  37%|###7      | 18568/50000 [11:07:04<15:32:46,  1.78s/it]
Outer training loop:  37%|###7      | 18569/50000 [11:07:06<15:36:51,  1.79s/it]
Outer training loop:  37%|###7      | 18570/50000 [11:07:08<16:06:18,  1.84s/it]
Outer training loop:  37%|###7      | 18571/50000 [11:07:10<15:34:31,  1.78s/it]
Outer training loop:  37%|###7      | 18572/50000 [11:07:11<15:22:22,  1.76s/it]
Outer training loop:  37%|###7      | 18573/50000 [11:07:13<15:40:43,  1.80s/it]
Outer training loop:  37%|###7      | 18574/50000 [11:07:15<15:41:58,  1.80s/it]
Outer training loop:  37%|###7      | 18575/50000 [11:07:17<15:12:53,  1.74s/it]
Outer training loop:  37%|###7      | 18576/50000 [11:07:18<14:44:30,  1.69s/it]
Outer training loop:  37%|###7      | 18577/50000 [11:07:20<15:22:56,  1.76s/it]
Outer training loop:  37%|###7      | 18578/50000 [11:07:22<15:11:56,  1.74s/it]
Outer training loop:  37%|###7      | 18579/50000 [11:07:24<15:14:47,  1.75s/it]
Outer training loop:  37%|###7      | 18580/50000 [11:07:26<16:03:58,  1.84s/it]
Outer training loop:  37%|###7      | 18581/50000 [11:07:27<15:39:29,  1.79s/it]
Outer training loop:  37%|###7      | 18582/50000 [11:07:29<15:16:01,  1.75s/it]
Outer training loop:  37%|###7      | 18583/50000 [11:07:31<15:03:15,  1.73s/it]
Outer training loop:  37%|###7      | 18584/50000 [11:07:33<15:28:28,  1.77s/it]
Outer training loop:  37%|###7      | 18585/50000 [11:07:34<15:33:05,  1.78s/it]
Outer training loop:  37%|###7      | 18586/50000 [11:07:36<15:24:52,  1.77s/it]
Outer training loop:  37%|###7      | 18587/50000 [11:07:38<15:42:27,  1.80s/it]
Outer training loop:  37%|###7      | 18588/50000 [11:07:40<15:25:15,  1.77s/it]
Outer training loop:  37%|###7      | 18589/50000 [11:07:42<15:31:01,  1.78s/it]
Outer training loop:  37%|###7      | 18590/50000 [11:07:44<16:21:18,  1.87s/it]
Outer training loop:  37%|###7      | 18591/50000 [11:07:45<16:03:47,  1.84s/it]
Outer training loop:  37%|###7      | 18592/50000 [11:07:47<15:55:18,  1.82s/it]
Outer training loop:  37%|###7      | 18593/50000 [11:07:49<15:38:43,  1.79s/it]
Outer training loop:  37%|###7      | 18594/50000 [11:07:51<16:08:54,  1.85s/it]
Outer training loop:  37%|###7      | 18595/50000 [11:07:53<15:42:26,  1.80s/it]
Outer training loop:  37%|###7      | 18596/50000 [11:07:54<15:20:06,  1.76s/it]
Outer training loop:  37%|###7      | 18597/50000 [11:07:56<15:43:46,  1.80s/it]
Outer training loop:  37%|###7      | 18598/50000 [11:07:58<15:35:22,  1.79s/it]
Outer training loop:  37%|###7      | 18599/50000 [11:08:00<15:31:06,  1.78s/it]
Outer training loop:  37%|###7      | 18600/50000 [11:08:02<15:43:25,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  37%|###7      | 18601/50000 [11:08:09<29:51:19,  3.42s/it]
Outer training loop:  37%|###7      | 18602/50000 [11:08:11<25:39:06,  2.94s/it]
Outer training loop:  37%|###7      | 18603/50000 [11:08:12<22:18:23,  2.56s/it]
Outer training loop:  37%|###7      | 18604/50000 [11:08:14<20:29:55,  2.35s/it]
Outer training loop:  37%|###7      | 18605/50000 [11:08:16<19:03:18,  2.19s/it]
Outer training loop:  37%|###7      | 18606/50000 [11:08:18<17:55:26,  2.06s/it]
Outer training loop:  37%|###7      | 18607/50000 [11:08:20<17:38:44,  2.02s/it]
Outer training loop:  37%|###7      | 18608/50000 [11:08:21<16:49:35,  1.93s/it]
Outer training loop:  37%|###7      | 18609/50000 [11:08:23<16:09:17,  1.85s/it]
Outer training loop:  37%|###7      | 18610/50000 [11:08:25<16:10:21,  1.85s/it]
Outer training loop:  37%|###7      | 18611/50000 [11:08:27<15:56:08,  1.83s/it]
Outer training loop:  37%|###7      | 18612/50000 [11:08:28<15:45:20,  1.81s/it]
Outer training loop:  37%|###7      | 18613/50000 [11:08:30<16:16:26,  1.87s/it]
Outer training loop:  37%|###7      | 18614/50000 [11:08:32<15:43:07,  1.80s/it]
Outer training loop:  37%|###7      | 18615/50000 [11:08:34<15:14:00,  1.75s/it]
Outer training loop:  37%|###7      | 18616/50000 [11:08:35<15:20:26,  1.76s/it]
Outer training loop:  37%|###7      | 18617/50000 [11:08:37<15:37:20,  1.79s/it]
Outer training loop:  37%|###7      | 18618/50000 [11:08:39<15:17:31,  1.75s/it]
Outer training loop:  37%|###7      | 18619/50000 [11:08:41<15:17:51,  1.75s/it]
Outer training loop:  37%|###7      | 18620/50000 [11:08:43<15:43:08,  1.80s/it]
Outer training loop:  37%|###7      | 18621/50000 [11:08:44<15:44:48,  1.81s/it]
Outer training loop:  37%|###7      | 18622/50000 [11:08:46<15:29:18,  1.78s/it]
Outer training loop:  37%|###7      | 18623/50000 [11:08:48<15:08:41,  1.74s/it]
Outer training loop:  37%|###7      | 18624/50000 [11:08:50<15:43:59,  1.81s/it]
Outer training loop:  37%|###7      | 18625/50000 [11:08:51<15:22:56,  1.76s/it]
Outer training loop:  37%|###7      | 18626/50000 [11:08:53<15:37:18,  1.79s/it]
Outer training loop:  37%|###7      | 18627/50000 [11:08:55<15:47:16,  1.81s/it]
Outer training loop:  37%|###7      | 18628/50000 [11:08:57<15:33:02,  1.78s/it]
Outer training loop:  37%|###7      | 18629/50000 [11:08:59<15:14:40,  1.75s/it]
Outer training loop:  37%|###7      | 18630/50000 [11:09:00<15:44:49,  1.81s/it]
Outer training loop:  37%|###7      | 18631/50000 [11:09:02<15:28:01,  1.78s/it]
Outer training loop:  37%|###7      | 18632/50000 [11:09:04<15:22:29,  1.76s/it]
Outer training loop:  37%|###7      | 18633/50000 [11:09:06<15:16:23,  1.75s/it]
Outer training loop:  37%|###7      | 18634/50000 [11:09:08<15:42:44,  1.80s/it]
Outer training loop:  37%|###7      | 18635/50000 [11:09:09<15:23:59,  1.77s/it]
Outer training loop:  37%|###7      | 18636/50000 [11:09:11<15:21:06,  1.76s/it]
Outer training loop:  37%|###7      | 18637/50000 [11:09:13<15:51:00,  1.82s/it]
Outer training loop:  37%|###7      | 18638/50000 [11:09:15<15:27:41,  1.77s/it]
Outer training loop:  37%|###7      | 18639/50000 [11:09:16<15:19:47,  1.76s/it]
Outer training loop:  37%|###7      | 18640/50000 [11:09:18<15:41:34,  1.80s/it]
Outer training loop:  37%|###7      | 18641/50000 [11:09:20<15:39:27,  1.80s/it]
Outer training loop:  37%|###7      | 18642/50000 [11:09:22<15:18:00,  1.76s/it]
Outer training loop:  37%|###7      | 18643/50000 [11:09:23<14:57:18,  1.72s/it]
Outer training loop:  37%|###7      | 18644/50000 [11:09:25<15:23:19,  1.77s/it]
Outer training loop:  37%|###7      | 18645/50000 [11:09:27<15:27:10,  1.77s/it]
Outer training loop:  37%|###7      | 18646/50000 [11:09:29<15:08:56,  1.74s/it]
Outer training loop:  37%|###7      | 18647/50000 [11:09:31<15:26:13,  1.77s/it]
Outer training loop:  37%|###7      | 18648/50000 [11:09:32<15:41:59,  1.80s/it]
Outer training loop:  37%|###7      | 18649/50000 [11:09:34<15:21:30,  1.76s/it]
Outer training loop:  37%|###7      | 18650/50000 [11:09:36<15:39:53,  1.80s/it]
Outer training loop:  37%|###7      | 18651/50000 [11:09:38<15:19:32,  1.76s/it]
Outer training loop:  37%|###7      | 18652/50000 [11:09:39<15:16:40,  1.75s/it]
Outer training loop:  37%|###7      | 18653/50000 [11:09:41<15:17:03,  1.76s/it]
Outer training loop:  37%|###7      | 18654/50000 [11:09:43<15:42:00,  1.80s/it]
Outer training loop:  37%|###7      | 18655/50000 [11:09:45<15:21:15,  1.76s/it]
Outer training loop:  37%|###7      | 18656/50000 [11:09:46<15:02:35,  1.73s/it]
Outer training loop:  37%|###7      | 18657/50000 [11:09:48<15:32:01,  1.78s/it]
Outer training loop:  37%|###7      | 18658/50000 [11:09:50<15:22:41,  1.77s/it]
Outer training loop:  37%|###7      | 18659/50000 [11:09:52<15:08:09,  1.74s/it]
Outer training loop:  37%|###7      | 18660/50000 [11:09:53<14:58:55,  1.72s/it]
Outer training loop:  37%|###7      | 18661/50000 [11:09:55<15:38:41,  1.80s/it]
Outer training loop:  37%|###7      | 18662/50000 [11:09:57<15:17:01,  1.76s/it]
Outer training loop:  37%|###7      | 18663/50000 [11:09:59<15:22:43,  1.77s/it]
Outer training loop:  37%|###7      | 18664/50000 [11:10:01<15:49:31,  1.82s/it]
Outer training loop:  37%|###7      | 18665/50000 [11:10:02<15:42:52,  1.81s/it]
Outer training loop:  37%|###7      | 18666/50000 [11:10:04<15:24:43,  1.77s/it]
Outer training loop:  37%|###7      | 18667/50000 [11:10:06<15:38:35,  1.80s/it]
Outer training loop:  37%|###7      | 18668/50000 [11:10:08<15:30:16,  1.78s/it]
Outer training loop:  37%|###7      | 18669/50000 [11:10:09<15:17:12,  1.76s/it]
Outer training loop:  37%|###7      | 18670/50000 [11:10:11<14:54:09,  1.71s/it]
Outer training loop:  37%|###7      | 18671/50000 [11:10:13<15:25:55,  1.77s/it]
Outer training loop:  37%|###7      | 18672/50000 [11:10:15<15:18:17,  1.76s/it]
Outer training loop:  37%|###7      | 18673/50000 [11:10:16<14:59:34,  1.72s/it]
Outer training loop:  37%|###7      | 18674/50000 [11:10:18<15:03:25,  1.73s/it]
Outer training loop:  37%|###7      | 18675/50000 [11:10:20<15:11:46,  1.75s/it]
Outer training loop:  37%|###7      | 18676/50000 [11:10:22<15:00:20,  1.72s/it]
Outer training loop:  37%|###7      | 18677/50000 [11:10:23<15:08:51,  1.74s/it]
Outer training loop:  37%|###7      | 18678/50000 [11:10:25<15:54:12,  1.83s/it]
Outer training loop:  37%|###7      | 18679/50000 [11:10:27<15:37:32,  1.80s/it]
Outer training loop:  37%|###7      | 18680/50000 [11:10:29<15:26:56,  1.78s/it]
Outer training loop:  37%|###7      | 18681/50000 [11:10:31<15:57:26,  1.83s/it]
Outer training loop:  37%|###7      | 18682/50000 [11:10:32<15:32:07,  1.79s/it]
Outer training loop:  37%|###7      | 18683/50000 [11:10:34<15:16:53,  1.76s/it]
Outer training loop:  37%|###7      | 18684/50000 [11:10:36<14:52:48,  1.71s/it]
Outer training loop:  37%|###7      | 18685/50000 [11:10:38<15:25:05,  1.77s/it]
Outer training loop:  37%|###7      | 18686/50000 [11:10:39<15:10:23,  1.74s/it]
Outer training loop:  37%|###7      | 18687/50000 [11:10:41<14:48:38,  1.70s/it]
Outer training loop:  37%|###7      | 18688/50000 [11:10:43<15:26:19,  1.78s/it]
Outer training loop:  37%|###7      | 18689/50000 [11:10:45<15:23:01,  1.77s/it]
Outer training loop:  37%|###7      | 18690/50000 [11:10:46<15:02:56,  1.73s/it]
Outer training loop:  37%|###7      | 18691/50000 [11:10:48<15:09:40,  1.74s/it]
Outer training loop:  37%|###7      | 18692/50000 [11:10:50<15:48:09,  1.82s/it]
Outer training loop:  37%|###7      | 18693/50000 [11:10:52<15:27:01,  1.78s/it]
Outer training loop:  37%|###7      | 18694/50000 [11:10:53<15:20:53,  1.76s/it]
Outer training loop:  37%|###7      | 18695/50000 [11:10:55<15:47:29,  1.82s/it]
Outer training loop:  37%|###7      | 18696/50000 [11:10:57<15:38:25,  1.80s/it]
Outer training loop:  37%|###7      | 18697/50000 [11:10:59<15:12:49,  1.75s/it]
Outer training loop:  37%|###7      | 18698/50000 [11:11:01<15:25:37,  1.77s/it]
Outer training loop:  37%|###7      | 18699/50000 [11:11:02<15:33:45,  1.79s/it]
Outer training loop:  37%|###7      | 18700/50000 [11:11:04<15:12:32,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  37%|###7      | 18701/50000 [11:11:11<28:43:22,  3.30s/it]
Outer training loop:  37%|###7      | 18702/50000 [11:11:13<24:41:56,  2.84s/it]
Outer training loop:  37%|###7      | 18703/50000 [11:11:15<22:16:57,  2.56s/it]
Outer training loop:  37%|###7      | 18704/50000 [11:11:16<19:57:32,  2.30s/it]
Outer training loop:  37%|###7      | 18705/50000 [11:11:18<18:19:40,  2.11s/it]
Outer training loop:  37%|###7      | 18706/50000 [11:11:20<17:51:38,  2.05s/it]
Outer training loop:  37%|###7      | 18707/50000 [11:11:22<17:07:03,  1.97s/it]
Outer training loop:  37%|###7      | 18708/50000 [11:11:24<17:05:13,  1.97s/it]
Outer training loop:  37%|###7      | 18709/50000 [11:11:26<16:57:27,  1.95s/it]
Outer training loop:  37%|###7      | 18710/50000 [11:11:27<16:18:29,  1.88s/it]
Outer training loop:  37%|###7      | 18711/50000 [11:11:29<15:41:57,  1.81s/it]
Outer training loop:  37%|###7      | 18712/50000 [11:11:31<15:22:42,  1.77s/it]
Outer training loop:  37%|###7      | 18713/50000 [11:11:33<15:43:19,  1.81s/it]
Outer training loop:  37%|###7      | 18714/50000 [11:11:34<15:32:38,  1.79s/it]
Outer training loop:  37%|###7      | 18715/50000 [11:11:36<15:27:01,  1.78s/it]
Outer training loop:  37%|###7      | 18716/50000 [11:11:38<15:41:35,  1.81s/it]
Outer training loop:  37%|###7      | 18717/50000 [11:11:40<15:36:18,  1.80s/it]
Outer training loop:  37%|###7      | 18718/50000 [11:11:41<15:09:38,  1.74s/it]
Outer training loop:  37%|###7      | 18719/50000 [11:11:43<15:31:44,  1.79s/it]
Outer training loop:  37%|###7      | 18720/50000 [11:11:45<15:21:56,  1.77s/it]
Outer training loop:  37%|###7      | 18721/50000 [11:11:47<15:17:09,  1.76s/it]
Outer training loop:  37%|###7      | 18722/50000 [11:11:48<15:17:56,  1.76s/it]
Outer training loop:  37%|###7      | 18723/50000 [11:11:50<15:41:45,  1.81s/it]
Outer training loop:  37%|###7      | 18724/50000 [11:11:52<15:46:42,  1.82s/it]
Outer training loop:  37%|###7      | 18725/50000 [11:11:54<15:36:57,  1.80s/it]
Outer training loop:  37%|###7      | 18726/50000 [11:11:56<16:00:23,  1.84s/it]
Outer training loop:  37%|###7      | 18727/50000 [11:11:58<15:36:32,  1.80s/it]
Outer training loop:  37%|###7      | 18728/50000 [11:11:59<15:19:28,  1.76s/it]
Outer training loop:  37%|###7      | 18729/50000 [11:12:01<15:38:46,  1.80s/it]
Outer training loop:  37%|###7      | 18730/50000 [11:12:03<15:20:50,  1.77s/it]
Outer training loop:  37%|###7      | 18731/50000 [11:12:05<15:19:36,  1.76s/it]
Outer training loop:  37%|###7      | 18732/50000 [11:12:06<15:17:52,  1.76s/it]
Outer training loop:  37%|###7      | 18733/50000 [11:12:08<15:45:25,  1.81s/it]
Outer training loop:  37%|###7      | 18734/50000 [11:12:10<15:21:54,  1.77s/it]
Outer training loop:  37%|###7      | 18735/50000 [11:12:12<15:13:00,  1.75s/it]
Outer training loop:  37%|###7      | 18736/50000 [11:12:14<15:44:09,  1.81s/it]
Outer training loop:  37%|###7      | 18737/50000 [11:12:15<15:14:14,  1.75s/it]
Outer training loop:  37%|###7      | 18738/50000 [11:12:17<15:07:00,  1.74s/it]
Outer training loop:  37%|###7      | 18739/50000 [11:12:19<15:22:37,  1.77s/it]
Outer training loop:  37%|###7      | 18740/50000 [11:12:21<15:51:40,  1.83s/it]
Outer training loop:  37%|###7      | 18741/50000 [11:12:22<15:39:41,  1.80s/it]
Outer training loop:  37%|###7      | 18742/50000 [11:12:24<15:09:07,  1.75s/it]
Outer training loop:  37%|###7      | 18743/50000 [11:12:26<15:31:43,  1.79s/it]
Outer training loop:  37%|###7      | 18744/50000 [11:12:28<15:12:31,  1.75s/it]
Outer training loop:  37%|###7      | 18745/50000 [11:12:29<15:22:50,  1.77s/it]
Outer training loop:  37%|###7      | 18746/50000 [11:12:31<15:49:17,  1.82s/it]
Outer training loop:  37%|###7      | 18747/50000 [11:12:33<15:38:25,  1.80s/it]
Outer training loop:  37%|###7      | 18748/50000 [11:12:35<15:50:21,  1.82s/it]
Outer training loop:  37%|###7      | 18749/50000 [11:12:37<15:33:17,  1.79s/it]
Outer training loop:  38%|###7      | 18750/50000 [11:12:39<15:57:47,  1.84s/it]
Outer training loop:  38%|###7      | 18751/50000 [11:12:40<15:38:04,  1.80s/it]
Outer training loop:  38%|###7      | 18752/50000 [11:12:42<15:18:48,  1.76s/it]
Outer training loop:  38%|###7      | 18753/50000 [11:12:44<15:17:30,  1.76s/it]
Outer training loop:  38%|###7      | 18754/50000 [11:12:46<15:50:08,  1.82s/it]
Outer training loop:  38%|###7      | 18755/50000 [11:12:48<15:45:09,  1.82s/it]
Outer training loop:  38%|###7      | 18756/50000 [11:12:49<15:37:39,  1.80s/it]
Outer training loop:  38%|###7      | 18757/50000 [11:12:51<16:14:43,  1.87s/it]
Outer training loop:  38%|###7      | 18758/50000 [11:12:53<15:57:59,  1.84s/it]
Outer training loop:  38%|###7      | 18759/50000 [11:12:55<15:53:45,  1.83s/it]
Outer training loop:  38%|###7      | 18760/50000 [11:12:57<16:05:12,  1.85s/it]
Outer training loop:  38%|###7      | 18761/50000 [11:12:59<15:43:01,  1.81s/it]
Outer training loop:  38%|###7      | 18762/50000 [11:13:00<15:24:41,  1.78s/it]
Outer training loop:  38%|###7      | 18763/50000 [11:13:02<15:41:07,  1.81s/it]
Outer training loop:  38%|###7      | 18764/50000 [11:13:04<15:33:52,  1.79s/it]
Outer training loop:  38%|###7      | 18765/50000 [11:13:06<15:41:13,  1.81s/it]
Outer training loop:  38%|###7      | 18766/50000 [11:13:08<15:26:54,  1.78s/it]
Outer training loop:  38%|###7      | 18767/50000 [11:13:09<15:53:05,  1.83s/it]
Outer training loop:  38%|###7      | 18768/50000 [11:13:11<15:33:39,  1.79s/it]
Outer training loop:  38%|###7      | 18769/50000 [11:13:13<15:03:04,  1.73s/it]
Outer training loop:  38%|###7      | 18770/50000 [11:13:15<15:17:31,  1.76s/it]
Outer training loop:  38%|###7      | 18771/50000 [11:13:16<15:28:55,  1.78s/it]
Outer training loop:  38%|###7      | 18772/50000 [11:13:18<15:09:52,  1.75s/it]
Outer training loop:  38%|###7      | 18773/50000 [11:13:20<15:17:25,  1.76s/it]
Outer training loop:  38%|###7      | 18774/50000 [11:13:22<15:40:22,  1.81s/it]
Outer training loop:  38%|###7      | 18775/50000 [11:13:24<15:25:53,  1.78s/it]
Outer training loop:  38%|###7      | 18776/50000 [11:13:25<15:28:54,  1.79s/it]
Outer training loop:  38%|###7      | 18777/50000 [11:13:27<15:33:47,  1.79s/it]
Outer training loop:  38%|###7      | 18778/50000 [11:13:29<15:19:49,  1.77s/it]
Outer training loop:  38%|###7      | 18779/50000 [11:13:31<15:18:57,  1.77s/it]
Outer training loop:  38%|###7      | 18780/50000 [11:13:33<15:46:11,  1.82s/it]
Outer training loop:  38%|###7      | 18781/50000 [11:13:34<15:39:32,  1.81s/it]
Outer training loop:  38%|###7      | 18782/50000 [11:13:36<15:37:50,  1.80s/it]
Outer training loop:  38%|###7      | 18783/50000 [11:13:38<15:19:36,  1.77s/it]
Outer training loop:  38%|###7      | 18784/50000 [11:13:40<15:48:27,  1.82s/it]
Outer training loop:  38%|###7      | 18785/50000 [11:13:42<15:37:32,  1.80s/it]
Outer training loop:  38%|###7      | 18786/50000 [11:13:43<15:18:42,  1.77s/it]
Outer training loop:  38%|###7      | 18787/50000 [11:13:45<15:50:27,  1.83s/it]
Outer training loop:  38%|###7      | 18788/50000 [11:13:47<15:24:03,  1.78s/it]
Outer training loop:  38%|###7      | 18789/50000 [11:13:49<15:21:12,  1.77s/it]
Outer training loop:  38%|###7      | 18790/50000 [11:13:50<15:45:10,  1.82s/it]
Outer training loop:  38%|###7      | 18791/50000 [11:13:52<15:48:37,  1.82s/it]
Outer training loop:  38%|###7      | 18792/50000 [11:13:54<15:22:49,  1.77s/it]
Outer training loop:  38%|###7      | 18793/50000 [11:13:56<15:03:45,  1.74s/it]
Outer training loop:  38%|###7      | 18794/50000 [11:13:57<15:16:06,  1.76s/it]
Outer training loop:  38%|###7      | 18795/50000 [11:13:59<15:32:21,  1.79s/it]
Outer training loop:  38%|###7      | 18796/50000 [11:14:01<15:26:30,  1.78s/it]
Outer training loop:  38%|###7      | 18797/50000 [11:14:03<15:36:03,  1.80s/it]
Outer training loop:  38%|###7      | 18798/50000 [11:14:05<15:29:36,  1.79s/it]
Outer training loop:  38%|###7      | 18799/50000 [11:14:06<15:25:10,  1.78s/it]
Outer training loop:  38%|###7      | 18800/50000 [11:14:08<16:02:22,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  38%|###7      | 18801/50000 [11:14:15<29:18:26,  3.38s/it]
Outer training loop:  38%|###7      | 18802/50000 [11:14:17<24:56:18,  2.88s/it]
Outer training loop:  38%|###7      | 18803/50000 [11:14:19<21:57:48,  2.53s/it]
Outer training loop:  38%|###7      | 18804/50000 [11:14:21<19:56:24,  2.30s/it]
Outer training loop:  38%|###7      | 18805/50000 [11:14:23<19:16:20,  2.22s/it]
Outer training loop:  38%|###7      | 18806/50000 [11:14:24<17:58:24,  2.07s/it]
Outer training loop:  38%|###7      | 18807/50000 [11:14:26<17:10:58,  1.98s/it]
Outer training loop:  38%|###7      | 18808/50000 [11:14:28<16:34:38,  1.91s/it]
Outer training loop:  38%|###7      | 18809/50000 [11:14:30<15:51:11,  1.83s/it]
Outer training loop:  38%|###7      | 18810/50000 [11:14:31<15:28:37,  1.79s/it]
Outer training loop:  38%|###7      | 18811/50000 [11:14:33<15:13:22,  1.76s/it]
Outer training loop:  38%|###7      | 18812/50000 [11:14:35<15:34:41,  1.80s/it]
Outer training loop:  38%|###7      | 18813/50000 [11:14:37<15:20:14,  1.77s/it]
Outer training loop:  38%|###7      | 18814/50000 [11:14:38<15:22:18,  1.77s/it]
Outer training loop:  38%|###7      | 18815/50000 [11:14:40<16:02:38,  1.85s/it]
Outer training loop:  38%|###7      | 18816/50000 [11:14:42<15:58:39,  1.84s/it]
Outer training loop:  38%|###7      | 18817/50000 [11:14:44<15:41:26,  1.81s/it]
Outer training loop:  38%|###7      | 18818/50000 [11:14:46<16:10:40,  1.87s/it]
Outer training loop:  38%|###7      | 18819/50000 [11:14:48<15:57:11,  1.84s/it]
Outer training loop:  38%|###7      | 18820/50000 [11:14:49<15:35:34,  1.80s/it]
Outer training loop:  38%|###7      | 18821/50000 [11:14:51<15:37:17,  1.80s/it]
Outer training loop:  38%|###7      | 18822/50000 [11:14:53<15:19:45,  1.77s/it]
Outer training loop:  38%|###7      | 18823/50000 [11:14:55<15:24:21,  1.78s/it]
Outer training loop:  38%|###7      | 18824/50000 [11:14:57<15:42:43,  1.81s/it]
Outer training loop:  38%|###7      | 18825/50000 [11:14:58<15:32:19,  1.79s/it]
Outer training loop:  38%|###7      | 18826/50000 [11:15:00<15:03:01,  1.74s/it]
Outer training loop:  38%|###7      | 18827/50000 [11:15:02<14:50:20,  1.71s/it]
Outer training loop:  38%|###7      | 18828/50000 [11:15:04<15:36:40,  1.80s/it]
Outer training loop:  38%|###7      | 18829/50000 [11:15:05<15:23:12,  1.78s/it]
Outer training loop:  38%|###7      | 18830/50000 [11:15:07<15:01:18,  1.73s/it]
Outer training loop:  38%|###7      | 18831/50000 [11:15:09<15:30:17,  1.79s/it]
Outer training loop:  38%|###7      | 18832/50000 [11:15:11<15:36:48,  1.80s/it]
Outer training loop:  38%|###7      | 18833/50000 [11:15:12<15:18:21,  1.77s/it]
Outer training loop:  38%|###7      | 18834/50000 [11:15:14<15:02:29,  1.74s/it]
Outer training loop:  38%|###7      | 18835/50000 [11:15:16<15:20:36,  1.77s/it]
Outer training loop:  38%|###7      | 18836/50000 [11:15:18<15:29:48,  1.79s/it]
Outer training loop:  38%|###7      | 18837/50000 [11:15:20<15:46:13,  1.82s/it]
Outer training loop:  38%|###7      | 18838/50000 [11:15:22<15:54:51,  1.84s/it]
Outer training loop:  38%|###7      | 18839/50000 [11:15:23<15:38:45,  1.81s/it]
Outer training loop:  38%|###7      | 18840/50000 [11:15:25<15:35:45,  1.80s/it]
Outer training loop:  38%|###7      | 18841/50000 [11:15:27<15:21:49,  1.78s/it]
Outer training loop:  38%|###7      | 18842/50000 [11:15:29<15:58:21,  1.85s/it]
Outer training loop:  38%|###7      | 18843/50000 [11:15:31<15:43:55,  1.82s/it]
Outer training loop:  38%|###7      | 18844/50000 [11:15:32<15:30:46,  1.79s/it]
Outer training loop:  38%|###7      | 18845/50000 [11:15:34<15:49:15,  1.83s/it]
Outer training loop:  38%|###7      | 18846/50000 [11:15:36<15:57:27,  1.84s/it]
Outer training loop:  38%|###7      | 18847/50000 [11:15:38<15:49:08,  1.83s/it]
Outer training loop:  38%|###7      | 18848/50000 [11:15:40<16:11:23,  1.87s/it]
Outer training loop:  38%|###7      | 18849/50000 [11:15:42<15:52:23,  1.83s/it]
Outer training loop:  38%|###7      | 18850/50000 [11:15:43<15:37:38,  1.81s/it]
Outer training loop:  38%|###7      | 18851/50000 [11:15:45<15:12:53,  1.76s/it]
Outer training loop:  38%|###7      | 18852/50000 [11:15:47<15:26:33,  1.78s/it]
Outer training loop:  38%|###7      | 18853/50000 [11:15:49<15:18:59,  1.77s/it]
Outer training loop:  38%|###7      | 18854/50000 [11:15:50<15:28:41,  1.79s/it]
Outer training loop:  38%|###7      | 18855/50000 [11:15:52<16:00:18,  1.85s/it]
Outer training loop:  38%|###7      | 18856/50000 [11:15:54<16:00:08,  1.85s/it]
Outer training loop:  38%|###7      | 18857/50000 [11:15:56<15:39:52,  1.81s/it]
Outer training loop:  38%|###7      | 18858/50000 [11:15:58<16:00:30,  1.85s/it]
Outer training loop:  38%|###7      | 18859/50000 [11:16:00<15:41:30,  1.81s/it]
Outer training loop:  38%|###7      | 18860/50000 [11:16:01<15:10:24,  1.75s/it]
Outer training loop:  38%|###7      | 18861/50000 [11:16:03<14:59:30,  1.73s/it]
Outer training loop:  38%|###7      | 18862/50000 [11:16:05<15:37:57,  1.81s/it]
Outer training loop:  38%|###7      | 18863/50000 [11:16:07<15:24:54,  1.78s/it]
Outer training loop:  38%|###7      | 18864/50000 [11:16:08<15:25:09,  1.78s/it]
Outer training loop:  38%|###7      | 18865/50000 [11:16:10<15:58:16,  1.85s/it]
Outer training loop:  38%|###7      | 18866/50000 [11:16:12<16:00:30,  1.85s/it]
Outer training loop:  38%|###7      | 18867/50000 [11:16:14<15:41:34,  1.81s/it]
Outer training loop:  38%|###7      | 18868/50000 [11:16:16<16:02:33,  1.86s/it]
Outer training loop:  38%|###7      | 18869/50000 [11:16:18<15:58:20,  1.85s/it]
Outer training loop:  38%|###7      | 18870/50000 [11:16:19<15:36:54,  1.81s/it]
Outer training loop:  38%|###7      | 18871/50000 [11:16:21<15:32:38,  1.80s/it]
Outer training loop:  38%|###7      | 18872/50000 [11:16:23<15:57:03,  1.84s/it]
Outer training loop:  38%|###7      | 18873/50000 [11:16:25<15:30:54,  1.79s/it]
Outer training loop:  38%|###7      | 18874/50000 [11:16:27<15:42:23,  1.82s/it]
Outer training loop:  38%|###7      | 18875/50000 [11:16:29<15:53:08,  1.84s/it]
Outer training loop:  38%|###7      | 18876/50000 [11:16:30<15:30:56,  1.79s/it]
Outer training loop:  38%|###7      | 18877/50000 [11:16:32<15:20:57,  1.78s/it]
Outer training loop:  38%|###7      | 18878/50000 [11:16:34<15:28:40,  1.79s/it]
Outer training loop:  38%|###7      | 18879/50000 [11:16:36<15:42:38,  1.82s/it]
Outer training loop:  38%|###7      | 18880/50000 [11:16:37<15:24:47,  1.78s/it]
Outer training loop:  38%|###7      | 18881/50000 [11:16:39<15:25:51,  1.79s/it]
Outer training loop:  38%|###7      | 18882/50000 [11:16:41<15:36:17,  1.81s/it]
Outer training loop:  38%|###7      | 18883/50000 [11:16:43<15:12:25,  1.76s/it]
Outer training loop:  38%|###7      | 18884/50000 [11:16:45<15:22:29,  1.78s/it]
Outer training loop:  38%|###7      | 18885/50000 [11:16:47<15:50:37,  1.83s/it]
Outer training loop:  38%|###7      | 18886/50000 [11:16:48<15:42:58,  1.82s/it]
Outer training loop:  38%|###7      | 18887/50000 [11:16:50<15:28:53,  1.79s/it]
Outer training loop:  38%|###7      | 18888/50000 [11:16:52<15:16:16,  1.77s/it]
Outer training loop:  38%|###7      | 18889/50000 [11:16:54<15:43:34,  1.82s/it]
Outer training loop:  38%|###7      | 18890/50000 [11:16:55<15:30:50,  1.80s/it]
Outer training loop:  38%|###7      | 18891/50000 [11:16:57<15:21:16,  1.78s/it]
Outer training loop:  38%|###7      | 18892/50000 [11:16:59<16:03:56,  1.86s/it]
Outer training loop:  38%|###7      | 18893/50000 [11:17:01<15:24:22,  1.78s/it]
Outer training loop:  38%|###7      | 18894/50000 [11:17:03<15:12:49,  1.76s/it]
Outer training loop:  38%|###7      | 18895/50000 [11:17:04<15:09:55,  1.76s/it]
Outer training loop:  38%|###7      | 18896/50000 [11:17:06<15:35:18,  1.80s/it]
Outer training loop:  38%|###7      | 18897/50000 [11:17:08<15:18:10,  1.77s/it]
Outer training loop:  38%|###7      | 18898/50000 [11:17:10<15:09:40,  1.75s/it]
Outer training loop:  38%|###7      | 18899/50000 [11:17:12<15:32:48,  1.80s/it]
Outer training loop:  38%|###7      | 18900/50000 [11:17:13<15:18:19,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]

Outer training loop:  38%|###7      | 18901/50000 [11:17:20<28:37:39,  3.31s/it]
Outer training loop:  38%|###7      | 18902/50000 [11:17:22<24:29:22,  2.83s/it]
Outer training loop:  38%|###7      | 18903/50000 [11:17:24<22:08:12,  2.56s/it]
Outer training loop:  38%|###7      | 18904/50000 [11:17:25<19:49:59,  2.30s/it]
Outer training loop:  38%|###7      | 18905/50000 [11:17:27<18:17:48,  2.12s/it]
Outer training loop:  38%|###7      | 18906/50000 [11:17:29<17:14:29,  2.00s/it]
Outer training loop:  38%|###7      | 18907/50000 [11:17:31<16:58:27,  1.97s/it]
Outer training loop:  38%|###7      | 18908/50000 [11:17:32<16:15:28,  1.88s/it]
Outer training loop:  38%|###7      | 18909/50000 [11:17:34<15:49:32,  1.83s/it]
Outer training loop:  38%|###7      | 18910/50000 [11:17:36<15:36:53,  1.81s/it]
Outer training loop:  38%|###7      | 18911/50000 [11:17:38<15:58:28,  1.85s/it]
Outer training loop:  38%|###7      | 18912/50000 [11:17:40<15:41:13,  1.82s/it]
Outer training loop:  38%|###7      | 18913/50000 [11:17:41<15:18:56,  1.77s/it]
Outer training loop:  38%|###7      | 18914/50000 [11:17:43<15:29:10,  1.79s/it]
Outer training loop:  38%|###7      | 18915/50000 [11:17:45<15:08:21,  1.75s/it]
Outer training loop:  38%|###7      | 18916/50000 [11:17:46<15:03:52,  1.74s/it]
Outer training loop:  38%|###7      | 18917/50000 [11:17:48<15:40:02,  1.81s/it]
Outer training loop:  38%|###7      | 18918/50000 [11:17:50<15:56:46,  1.85s/it]
Outer training loop:  38%|###7      | 18919/50000 [11:17:52<15:47:35,  1.83s/it]
Outer training loop:  38%|###7      | 18920/50000 [11:17:54<15:33:51,  1.80s/it]
Outer training loop:  38%|###7      | 18921/50000 [11:17:56<15:40:09,  1.82s/it]
Outer training loop:  38%|###7      | 18922/50000 [11:17:57<15:08:19,  1.75s/it]
Outer training loop:  38%|###7      | 18923/50000 [11:17:59<15:03:28,  1.74s/it]
Outer training loop:  38%|###7      | 18924/50000 [11:18:01<15:21:22,  1.78s/it]
Outer training loop:  38%|###7      | 18925/50000 [11:18:03<15:08:52,  1.75s/it]
Outer training loop:  38%|###7      | 18926/50000 [11:18:04<15:05:10,  1.75s/it]
Outer training loop:  38%|###7      | 18927/50000 [11:18:06<15:16:10,  1.77s/it]
Outer training loop:  38%|###7      | 18928/50000 [11:18:08<15:39:36,  1.81s/it]
Outer training loop:  38%|###7      | 18929/50000 [11:18:10<15:35:36,  1.81s/it]
Outer training loop:  38%|###7      | 18930/50000 [11:18:12<15:26:07,  1.79s/it]
Outer training loop:  38%|###7      | 18931/50000 [11:18:13<15:27:19,  1.79s/it]
Outer training loop:  38%|###7      | 18932/50000 [11:18:15<15:16:19,  1.77s/it]
Outer training loop:  38%|###7      | 18933/50000 [11:18:17<15:15:02,  1.77s/it]
Outer training loop:  38%|###7      | 18934/50000 [11:18:19<15:30:31,  1.80s/it]
Outer training loop:  38%|###7      | 18935/50000 [11:18:21<15:22:44,  1.78s/it]
Outer training loop:  38%|###7      | 18936/50000 [11:18:22<15:13:13,  1.76s/it]
Outer training loop:  38%|###7      | 18937/50000 [11:18:24<15:43:30,  1.82s/it]
Outer training loop:  38%|###7      | 18938/50000 [11:18:26<15:34:00,  1.80s/it]
Outer training loop:  38%|###7      | 18939/50000 [11:18:28<15:49:03,  1.83s/it]
Outer training loop:  38%|###7      | 18940/50000 [11:18:30<15:39:38,  1.82s/it]
Outer training loop:  38%|###7      | 18941/50000 [11:18:31<15:37:08,  1.81s/it]
Outer training loop:  38%|###7      | 18942/50000 [11:18:33<15:24:48,  1.79s/it]
Outer training loop:  38%|###7      | 18943/50000 [11:18:35<15:43:29,  1.82s/it]
Outer training loop:  38%|###7      | 18944/50000 [11:18:37<15:44:14,  1.82s/it]
Outer training loop:  38%|###7      | 18945/50000 [11:18:39<15:31:12,  1.80s/it]
Outer training loop:  38%|###7      | 18946/50000 [11:18:40<15:15:52,  1.77s/it]
Outer training loop:  38%|###7      | 18947/50000 [11:18:42<15:38:57,  1.81s/it]
Outer training loop:  38%|###7      | 18948/50000 [11:18:44<15:30:14,  1.80s/it]
Outer training loop:  38%|###7      | 18949/50000 [11:18:46<15:05:37,  1.75s/it]
Outer training loop:  38%|###7      | 18950/50000 [11:18:47<15:12:42,  1.76s/it]
Outer training loop:  38%|###7      | 18951/50000 [11:18:50<15:56:01,  1.85s/it]
Outer training loop:  38%|###7      | 18952/50000 [11:18:51<15:45:57,  1.83s/it]
Outer training loop:  38%|###7      | 18953/50000 [11:18:53<15:37:30,  1.81s/it]
Outer training loop:  38%|###7      | 18954/50000 [11:18:55<16:00:17,  1.86s/it]
Outer training loop:  38%|###7      | 18955/50000 [11:18:57<15:29:53,  1.80s/it]
Outer training loop:  38%|###7      | 18956/50000 [11:18:58<15:23:43,  1.79s/it]
Outer training loop:  38%|###7      | 18957/50000 [11:19:00<15:13:09,  1.76s/it]
Outer training loop:  38%|###7      | 18958/50000 [11:19:02<15:22:13,  1.78s/it]
Outer training loop:  38%|###7      | 18959/50000 [11:19:04<15:00:30,  1.74s/it]
Outer training loop:  38%|###7      | 18960/50000 [11:19:05<15:04:28,  1.75s/it]
Outer training loop:  38%|###7      | 18961/50000 [11:19:07<15:40:11,  1.82s/it]
Outer training loop:  38%|###7      | 18962/50000 [11:19:09<15:20:35,  1.78s/it]
Outer training loop:  38%|###7      | 18963/50000 [11:19:11<15:09:08,  1.76s/it]
Outer training loop:  38%|###7      | 18964/50000 [11:19:13<15:02:41,  1.75s/it]
Outer training loop:  38%|###7      | 18965/50000 [11:19:14<15:18:36,  1.78s/it]
Outer training loop:  38%|###7      | 18966/50000 [11:19:16<14:55:50,  1.73s/it]
Outer training loop:  38%|###7      | 18967/50000 [11:19:18<15:03:02,  1.75s/it]
Outer training loop:  38%|###7      | 18968/50000 [11:19:20<15:25:15,  1.79s/it]
Outer training loop:  38%|###7      | 18969/50000 [11:19:21<15:04:12,  1.75s/it]
Outer training loop:  38%|###7      | 18970/50000 [11:19:23<14:48:57,  1.72s/it]
Outer training loop:  38%|###7      | 18971/50000 [11:19:25<14:37:01,  1.70s/it]
Outer training loop:  38%|###7      | 18972/50000 [11:19:26<14:51:38,  1.72s/it]
Outer training loop:  38%|###7      | 18973/50000 [11:19:28<14:51:27,  1.72s/it]
Outer training loop:  38%|###7      | 18974/50000 [11:19:30<14:48:42,  1.72s/it]
Outer training loop:  38%|###7      | 18975/50000 [11:19:32<15:14:06,  1.77s/it]
Outer training loop:  38%|###7      | 18976/50000 [11:19:34<15:21:18,  1.78s/it]
Outer training loop:  38%|###7      | 18977/50000 [11:19:35<15:14:01,  1.77s/it]
Outer training loop:  38%|###7      | 18978/50000 [11:19:37<14:55:06,  1.73s/it]
Outer training loop:  38%|###7      | 18979/50000 [11:19:39<15:21:11,  1.78s/it]
Outer training loop:  38%|###7      | 18980/50000 [11:19:41<15:32:03,  1.80s/it]
Outer training loop:  38%|###7      | 18981/50000 [11:19:42<15:08:44,  1.76s/it]
Outer training loop:  38%|###7      | 18982/50000 [11:19:44<15:07:59,  1.76s/it]
Outer training loop:  38%|###7      | 18983/50000 [11:19:46<15:20:35,  1.78s/it]
Outer training loop:  38%|###7      | 18984/50000 [11:19:48<15:09:08,  1.76s/it]
Outer training loop:  38%|###7      | 18985/50000 [11:19:49<15:05:22,  1.75s/it]
Outer training loop:  38%|###7      | 18986/50000 [11:19:51<15:23:15,  1.79s/it]
Outer training loop:  38%|###7      | 18987/50000 [11:19:53<15:07:46,  1.76s/it]
Outer training loop:  38%|###7      | 18988/50000 [11:19:55<14:48:44,  1.72s/it]
Outer training loop:  38%|###7      | 18989/50000 [11:19:57<15:30:06,  1.80s/it]
Outer training loop:  38%|###7      | 18990/50000 [11:19:58<15:19:13,  1.78s/it]
Outer training loop:  38%|###7      | 18991/50000 [11:20:00<15:10:20,  1.76s/it]
Outer training loop:  38%|###7      | 18992/50000 [11:20:02<15:22:18,  1.78s/it]
Outer training loop:  38%|###7      | 18993/50000 [11:20:04<15:11:35,  1.76s/it]
Outer training loop:  38%|###7      | 18994/50000 [11:20:05<14:55:26,  1.73s/it]
Outer training loop:  38%|###7      | 18995/50000 [11:20:07<14:59:55,  1.74s/it]
Outer training loop:  38%|###7      | 18996/50000 [11:20:09<15:33:26,  1.81s/it]
Outer training loop:  38%|###7      | 18997/50000 [11:20:11<15:27:52,  1.80s/it]
Outer training loop:  38%|###7      | 18998/50000 [11:20:12<15:14:36,  1.77s/it]
Outer training loop:  38%|###7      | 18999/50000 [11:20:14<15:09:20,  1.76s/it]
Outer training loop:  38%|###8      | 19000/50000 [11:20:16<15:38:01,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.71s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.73s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.75s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:24,  1.81s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:19,  1.77s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.76s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:19,  1.81s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:15,  1.79s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:13,  1.78s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:12,  1.79s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:13,  1.82s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:10,  1.81s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:06,  1.79s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:02,  1.75s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:04,  1.81s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:02,  1.79s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:56,  1.75s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:54,  1.73s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:52,  1.73s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:55,  1.81s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:52,  1.79s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:48,  1.75s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:45,  1.73s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:48,  1.81s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:45,  1.79s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:44,  1.79s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:39,  1.75s/it]

Evaluating policy:  33%|###2      | 27/83 [00:47<01:37,  1.75s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:39,  1.81s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:34,  1.75s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:32,  1.74s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.76s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:29,  1.75s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:26,  1.73s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:27,  1.79s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:24,  1.80s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:20,  1.75s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:17,  1.72s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:15,  1.72s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:17,  1.81s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:15,  1.80s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:13,  1.80s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:11,  1.78s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:17<01:07,  1.73s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:04,  1.70s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:06,  1.81s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:04,  1.78s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:01,  1.77s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<00:59,  1.76s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:57,  1.73s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:54,  1.72s/it]

Evaluating policy:  63%|######2   | 52/83 [01:31<00:53,  1.73s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:51,  1.73s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:50,  1.79s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:47,  1.77s/it]

Evaluating policy:  69%|######8   | 57/83 [01:40<00:46,  1.78s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:44,  1.77s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:42,  1.75s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.76s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:47<00:38,  1.76s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:35,  1.80s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:33,  1.76s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:31,  1.74s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:29,  1.72s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:27,  1.70s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:25,  1.72s/it]

Evaluating policy:  83%|########3 | 69/83 [02:01<00:23,  1.71s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:21,  1.80s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:19,  1.77s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.74s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:15,  1.75s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:14,  1.77s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.75s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.74s/it]

Evaluating policy:  94%|#########3| 78/83 [02:17<00:08,  1.73s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.92s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.81s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.75s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:11,  1.60s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:15,  1.67s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:14,  1.68s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:15,  1.71s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:10,  1.68s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:22,  1.85s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.80s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:12,  1.79s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:13,  1.83s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:11,  1.82s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.79s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:04,  1.77s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:06,  1.83s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:03,  1.81s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:01,  1.81s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:57,  1.78s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:55,  1.78s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:57,  1.83s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:54,  1.81s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:49,  1.77s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:45,  1.74s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:49,  1.82s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:45,  1.78s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.79s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:40,  1.76s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:36,  1.73s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:40,  1.82s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:36,  1.78s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:32,  1.75s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:29,  1.73s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:27,  1.72s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:26,  1.74s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:29,  1.83s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:26,  1.79s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:26,  1.83s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:22,  1.80s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:19,  1.76s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.73s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:19,  1.85s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:16,  1.82s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:14,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:11,  1.79s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.78s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:07,  1.78s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:04,  1.74s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:07,  1.86s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:03,  1.81s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.78s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:58,  1.76s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:55,  1.75s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:55,  1.79s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:57,  1.91s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:50,  1.81s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:48,  1.80s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:46,  1.77s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:44,  1.78s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.77s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:40,  1.76s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:38,  1.82s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.79s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:33,  1.78s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:31,  1.77s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:29,  1.74s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:28,  1.77s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:25,  1.72s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:24,  1.75s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:24,  1.89s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:19,  1.76s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.74s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:15,  1.74s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.75s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.74s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.73s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.71s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.89s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.79s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.76s/it]

Outer training loop:  38%|###8      | 19001/50000 [11:25:19<792:28:09, 92.03s/it]
Outer training loop:  38%|###8      | 19002/50000 [11:25:20<559:14:49, 64.95s/it]
Outer training loop:  38%|###8      | 19003/50000 [11:25:22<396:00:05, 45.99s/it]
Outer training loop:  38%|###8      | 19004/50000 [11:25:24<281:37:08, 32.71s/it]
Outer training loop:  38%|###8      | 19005/50000 [11:25:26<201:43:19, 23.43s/it]
Outer training loop:  38%|###8      | 19006/50000 [11:25:27<145:34:00, 16.91s/it]
Outer training loop:  38%|###8      | 19007/50000 [11:25:30<107:45:30, 12.52s/it]
Outer training loop:  38%|###8      | 19008/50000 [11:25:31<79:52:49,  9.28s/it] 
Outer training loop:  38%|###8      | 19009/50000 [11:25:33<60:23:58,  7.02s/it]
Outer training loop:  38%|###8      | 19010/50000 [11:25:35<47:18:05,  5.49s/it]
Outer training loop:  38%|###8      | 19011/50000 [11:25:37<37:27:01,  4.35s/it]
Outer training loop:  38%|###8      | 19012/50000 [11:25:38<30:32:17,  3.55s/it]
Outer training loop:  38%|###8      | 19013/50000 [11:25:40<25:48:05,  3.00s/it]
Outer training loop:  38%|###8      | 19014/50000 [11:25:42<23:10:13,  2.69s/it]
Outer training loop:  38%|###8      | 19015/50000 [11:25:44<20:36:27,  2.39s/it]
Outer training loop:  38%|###8      | 19016/50000 [11:25:45<18:54:53,  2.20s/it]
Outer training loop:  38%|###8      | 19017/50000 [11:25:47<18:22:20,  2.13s/it]
Outer training loop:  38%|###8      | 19018/50000 [11:25:49<17:12:57,  2.00s/it]
Outer training loop:  38%|###8      | 19019/50000 [11:25:51<16:43:43,  1.94s/it]
Outer training loop:  38%|###8      | 19020/50000 [11:25:53<16:37:09,  1.93s/it]
Outer training loop:  38%|###8      | 19021/50000 [11:25:55<16:35:48,  1.93s/it]
Outer training loop:  38%|###8      | 19022/50000 [11:25:57<16:07:33,  1.87s/it]
Outer training loop:  38%|###8      | 19023/50000 [11:25:58<15:44:06,  1.83s/it]
Outer training loop:  38%|###8      | 19024/50000 [11:26:00<15:18:02,  1.78s/it]
Outer training loop:  38%|###8      | 19025/50000 [11:26:02<15:36:44,  1.81s/it]
Outer training loop:  38%|###8      | 19026/50000 [11:26:04<15:39:54,  1.82s/it]
Outer training loop:  38%|###8      | 19027/50000 [11:26:05<15:21:56,  1.79s/it]
Outer training loop:  38%|###8      | 19028/50000 [11:26:07<15:29:56,  1.80s/it]
Outer training loop:  38%|###8      | 19029/50000 [11:26:09<15:24:54,  1.79s/it]
Outer training loop:  38%|###8      | 19030/50000 [11:26:11<15:15:47,  1.77s/it]
Outer training loop:  38%|###8      | 19031/50000 [11:26:13<15:29:39,  1.80s/it]
Outer training loop:  38%|###8      | 19032/50000 [11:26:14<15:21:16,  1.78s/it]
Outer training loop:  38%|###8      | 19033/50000 [11:26:16<14:57:50,  1.74s/it]
Outer training loop:  38%|###8      | 19034/50000 [11:26:18<14:51:18,  1.73s/it]
Outer training loop:  38%|###8      | 19035/50000 [11:26:20<15:29:11,  1.80s/it]
Outer training loop:  38%|###8      | 19036/50000 [11:26:21<15:16:56,  1.78s/it]
Outer training loop:  38%|###8      | 19037/50000 [11:26:23<15:18:25,  1.78s/it]
Outer training loop:  38%|###8      | 19038/50000 [11:26:25<15:45:29,  1.83s/it]
Outer training loop:  38%|###8      | 19039/50000 [11:26:27<15:22:50,  1.79s/it]
Outer training loop:  38%|###8      | 19040/50000 [11:26:28<15:13:40,  1.77s/it]
Outer training loop:  38%|###8      | 19041/50000 [11:26:30<15:08:04,  1.76s/it]
Outer training loop:  38%|###8      | 19042/50000 [11:26:32<15:17:34,  1.78s/it]
Outer training loop:  38%|###8      | 19043/50000 [11:26:34<15:02:30,  1.75s/it]
Outer training loop:  38%|###8      | 19044/50000 [11:26:35<15:02:26,  1.75s/it]
Outer training loop:  38%|###8      | 19045/50000 [11:26:37<15:34:15,  1.81s/it]
Outer training loop:  38%|###8      | 19046/50000 [11:26:39<15:25:12,  1.79s/it]
Outer training loop:  38%|###8      | 19047/50000 [11:26:41<15:16:15,  1.78s/it]
Outer training loop:  38%|###8      | 19048/50000 [11:26:43<15:45:56,  1.83s/it]
Outer training loop:  38%|###8      | 19049/50000 [11:26:45<15:27:36,  1.80s/it]
Outer training loop:  38%|###8      | 19050/50000 [11:26:46<15:25:12,  1.79s/it]
Outer training loop:  38%|###8      | 19051/50000 [11:26:48<15:36:13,  1.82s/it]
Outer training loop:  38%|###8      | 19052/50000 [11:26:50<15:30:00,  1.80s/it]
Outer training loop:  38%|###8      | 19053/50000 [11:26:52<15:18:53,  1.78s/it]
Outer training loop:  38%|###8      | 19054/50000 [11:26:53<15:02:14,  1.75s/it]
Outer training loop:  38%|###8      | 19055/50000 [11:26:55<15:30:52,  1.80s/it]
Outer training loop:  38%|###8      | 19056/50000 [11:26:57<15:14:38,  1.77s/it]
Outer training loop:  38%|###8      | 19057/50000 [11:26:59<15:05:38,  1.76s/it]
Outer training loop:  38%|###8      | 19058/50000 [11:27:01<15:39:40,  1.82s/it]
Outer training loop:  38%|###8      | 19059/50000 [11:27:02<15:17:34,  1.78s/it]
Outer training loop:  38%|###8      | 19060/50000 [11:27:04<15:10:24,  1.77s/it]
Outer training loop:  38%|###8      | 19061/50000 [11:27:06<15:26:00,  1.80s/it]
Outer training loop:  38%|###8      | 19062/50000 [11:27:08<15:02:20,  1.75s/it]
Outer training loop:  38%|###8      | 19063/50000 [11:27:09<15:15:30,  1.78s/it]
Outer training loop:  38%|###8      | 19064/50000 [11:27:11<15:12:27,  1.77s/it]
Outer training loop:  38%|###8      | 19065/50000 [11:27:13<15:29:24,  1.80s/it]
Outer training loop:  38%|###8      | 19066/50000 [11:27:15<15:08:21,  1.76s/it]
Outer training loop:  38%|###8      | 19067/50000 [11:27:16<14:50:02,  1.73s/it]
Outer training loop:  38%|###8      | 19068/50000 [11:27:18<14:42:52,  1.71s/it]
Outer training loop:  38%|###8      | 19069/50000 [11:27:20<15:13:35,  1.77s/it]
Outer training loop:  38%|###8      | 19070/50000 [11:27:22<15:27:21,  1.80s/it]
Outer training loop:  38%|###8      | 19071/50000 [11:27:24<15:13:50,  1.77s/it]
Outer training loop:  38%|###8      | 19072/50000 [11:27:25<15:30:56,  1.81s/it]
Outer training loop:  38%|###8      | 19073/50000 [11:27:27<15:10:09,  1.77s/it]
Outer training loop:  38%|###8      | 19074/50000 [11:27:29<14:58:23,  1.74s/it]
Outer training loop:  38%|###8      | 19075/50000 [11:27:31<15:23:31,  1.79s/it]
Outer training loop:  38%|###8      | 19076/50000 [11:27:32<15:11:54,  1.77s/it]
Outer training loop:  38%|###8      | 19077/50000 [11:27:34<15:13:57,  1.77s/it]
Outer training loop:  38%|###8      | 19078/50000 [11:27:36<15:41:58,  1.83s/it]
Outer training loop:  38%|###8      | 19079/50000 [11:27:38<15:24:46,  1.79s/it]
Outer training loop:  38%|###8      | 19080/50000 [11:27:40<15:11:37,  1.77s/it]
Outer training loop:  38%|###8      | 19081/50000 [11:27:42<15:43:11,  1.83s/it]
Outer training loop:  38%|###8      | 19082/50000 [11:27:43<15:43:47,  1.83s/it]
Outer training loop:  38%|###8      | 19083/50000 [11:27:45<15:25:41,  1.80s/it]
Outer training loop:  38%|###8      | 19084/50000 [11:27:47<15:22:45,  1.79s/it]
Outer training loop:  38%|###8      | 19085/50000 [11:27:49<15:32:04,  1.81s/it]
Outer training loop:  38%|###8      | 19086/50000 [11:27:51<15:32:01,  1.81s/it]
Outer training loop:  38%|###8      | 19087/50000 [11:27:52<15:07:49,  1.76s/it]
Outer training loop:  38%|###8      | 19088/50000 [11:27:54<15:17:08,  1.78s/it]
Outer training loop:  38%|###8      | 19089/50000 [11:27:56<15:07:53,  1.76s/it]
Outer training loop:  38%|###8      | 19090/50000 [11:27:57<14:55:45,  1.74s/it]
Outer training loop:  38%|###8      | 19091/50000 [11:27:59<15:36:00,  1.82s/it]
Outer training loop:  38%|###8      | 19092/50000 [11:28:01<15:22:22,  1.79s/it]
Outer training loop:  38%|###8      | 19093/50000 [11:28:03<15:22:37,  1.79s/it]
Outer training loop:  38%|###8      | 19094/50000 [11:28:05<15:08:35,  1.76s/it]
Outer training loop:  38%|###8      | 19095/50000 [11:28:07<15:28:57,  1.80s/it]
Outer training loop:  38%|###8      | 19096/50000 [11:28:08<15:10:52,  1.77s/it]
Outer training loop:  38%|###8      | 19097/50000 [11:28:10<15:16:58,  1.78s/it]
Outer training loop:  38%|###8      | 19098/50000 [11:28:12<15:38:22,  1.82s/it]
Outer training loop:  38%|###8      | 19099/50000 [11:28:14<15:19:16,  1.78s/it]
Outer training loop:  38%|###8      | 19100/50000 [11:28:15<15:00:56,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.65s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.70s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  38%|###8      | 19101/50000 [11:28:22<28:49:52,  3.36s/it]
Outer training loop:  38%|###8      | 19102/50000 [11:28:24<24:22:55,  2.84s/it]
Outer training loop:  38%|###8      | 19103/50000 [11:28:26<21:42:02,  2.53s/it]
Outer training loop:  38%|###8      | 19104/50000 [11:28:28<19:35:40,  2.28s/it]
Outer training loop:  38%|###8      | 19105/50000 [11:28:29<18:33:08,  2.16s/it]
Outer training loop:  38%|###8      | 19106/50000 [11:28:31<17:16:12,  2.01s/it]
Outer training loop:  38%|###8      | 19107/50000 [11:28:33<16:18:56,  1.90s/it]
Outer training loop:  38%|###8      | 19108/50000 [11:28:35<16:14:00,  1.89s/it]
Outer training loop:  38%|###8      | 19109/50000 [11:28:36<15:33:04,  1.81s/it]
Outer training loop:  38%|###8      | 19110/50000 [11:28:38<15:08:10,  1.76s/it]
Outer training loop:  38%|###8      | 19111/50000 [11:28:40<15:36:56,  1.82s/it]
Outer training loop:  38%|###8      | 19112/50000 [11:28:42<15:15:31,  1.78s/it]
Outer training loop:  38%|###8      | 19113/50000 [11:28:43<15:19:53,  1.79s/it]
Outer training loop:  38%|###8      | 19114/50000 [11:28:45<15:23:27,  1.79s/it]
Outer training loop:  38%|###8      | 19115/50000 [11:28:47<14:59:33,  1.75s/it]
Outer training loop:  38%|###8      | 19116/50000 [11:28:49<14:48:28,  1.73s/it]
Outer training loop:  38%|###8      | 19117/50000 [11:28:50<15:13:39,  1.78s/it]
Outer training loop:  38%|###8      | 19118/50000 [11:28:52<15:44:17,  1.83s/it]
Outer training loop:  38%|###8      | 19119/50000 [11:28:54<15:20:54,  1.79s/it]
Outer training loop:  38%|###8      | 19120/50000 [11:28:56<15:09:01,  1.77s/it]
Outer training loop:  38%|###8      | 19121/50000 [11:28:58<15:27:25,  1.80s/it]
Outer training loop:  38%|###8      | 19122/50000 [11:28:59<15:15:18,  1.78s/it]
Outer training loop:  38%|###8      | 19123/50000 [11:29:01<15:15:22,  1.78s/it]
Outer training loop:  38%|###8      | 19124/50000 [11:29:03<15:36:43,  1.82s/it]
Outer training loop:  38%|###8      | 19125/50000 [11:29:05<15:37:18,  1.82s/it]
Outer training loop:  38%|###8      | 19126/50000 [11:29:07<15:35:31,  1.82s/it]
Outer training loop:  38%|###8      | 19127/50000 [11:29:08<15:24:00,  1.80s/it]
Outer training loop:  38%|###8      | 19128/50000 [11:29:10<15:33:58,  1.82s/it]
Outer training loop:  38%|###8      | 19129/50000 [11:29:12<14:57:47,  1.74s/it]
Outer training loop:  38%|###8      | 19130/50000 [11:29:14<15:02:02,  1.75s/it]
Outer training loop:  38%|###8      | 19131/50000 [11:29:16<15:24:47,  1.80s/it]
Outer training loop:  38%|###8      | 19132/50000 [11:29:17<15:10:16,  1.77s/it]
Outer training loop:  38%|###8      | 19133/50000 [11:29:19<14:55:40,  1.74s/it]
Outer training loop:  38%|###8      | 19134/50000 [11:29:21<14:48:40,  1.73s/it]
Outer training loop:  38%|###8      | 19135/50000 [11:29:23<15:10:50,  1.77s/it]
Outer training loop:  38%|###8      | 19136/50000 [11:29:24<15:07:13,  1.76s/it]
Outer training loop:  38%|###8      | 19137/50000 [11:29:26<15:01:23,  1.75s/it]
Outer training loop:  38%|###8      | 19138/50000 [11:29:28<15:32:19,  1.81s/it]
Outer training loop:  38%|###8      | 19139/50000 [11:29:30<15:18:58,  1.79s/it]
Outer training loop:  38%|###8      | 19140/50000 [11:29:31<15:08:20,  1.77s/it]
Outer training loop:  38%|###8      | 19141/50000 [11:29:33<15:44:55,  1.84s/it]
Outer training loop:  38%|###8      | 19142/50000 [11:29:35<15:27:00,  1.80s/it]
Outer training loop:  38%|###8      | 19143/50000 [11:29:37<15:31:02,  1.81s/it]
Outer training loop:  38%|###8      | 19144/50000 [11:29:39<15:08:59,  1.77s/it]
Outer training loop:  38%|###8      | 19145/50000 [11:29:41<15:44:42,  1.84s/it]
Outer training loop:  38%|###8      | 19146/50000 [11:29:42<15:29:57,  1.81s/it]
Outer training loop:  38%|###8      | 19147/50000 [11:29:44<15:34:23,  1.82s/it]
Outer training loop:  38%|###8      | 19148/50000 [11:29:46<16:07:14,  1.88s/it]
Outer training loop:  38%|###8      | 19149/50000 [11:29:48<15:39:26,  1.83s/it]
Outer training loop:  38%|###8      | 19150/50000 [11:29:50<15:12:58,  1.78s/it]
Outer training loop:  38%|###8      | 19151/50000 [11:29:51<15:04:34,  1.76s/it]
Outer training loop:  38%|###8      | 19152/50000 [11:29:53<15:29:22,  1.81s/it]
Outer training loop:  38%|###8      | 19153/50000 [11:29:55<15:25:01,  1.80s/it]
Outer training loop:  38%|###8      | 19154/50000 [11:29:57<14:49:15,  1.73s/it]
Outer training loop:  38%|###8      | 19155/50000 [11:29:58<15:11:26,  1.77s/it]
Outer training loop:  38%|###8      | 19156/50000 [11:30:00<14:59:50,  1.75s/it]
Outer training loop:  38%|###8      | 19157/50000 [11:30:02<14:53:09,  1.74s/it]
Outer training loop:  38%|###8      | 19158/50000 [11:30:04<14:53:13,  1.74s/it]
Outer training loop:  38%|###8      | 19159/50000 [11:30:06<15:26:47,  1.80s/it]
Outer training loop:  38%|###8      | 19160/50000 [11:30:07<15:31:31,  1.81s/it]
Outer training loop:  38%|###8      | 19161/50000 [11:30:09<15:13:05,  1.78s/it]
Outer training loop:  38%|###8      | 19162/50000 [11:30:11<15:40:00,  1.83s/it]
Outer training loop:  38%|###8      | 19163/50000 [11:30:13<15:29:07,  1.81s/it]
Outer training loop:  38%|###8      | 19164/50000 [11:30:15<15:29:22,  1.81s/it]
Outer training loop:  38%|###8      | 19165/50000 [11:30:16<14:58:39,  1.75s/it]
Outer training loop:  38%|###8      | 19166/50000 [11:30:18<15:32:31,  1.81s/it]
Outer training loop:  38%|###8      | 19167/50000 [11:30:20<15:42:15,  1.83s/it]
Outer training loop:  38%|###8      | 19168/50000 [11:30:22<15:33:04,  1.82s/it]
Outer training loop:  38%|###8      | 19169/50000 [11:30:24<15:38:49,  1.83s/it]
Outer training loop:  38%|###8      | 19170/50000 [11:30:25<15:28:35,  1.81s/it]
Outer training loop:  38%|###8      | 19171/50000 [11:30:27<14:57:48,  1.75s/it]
Outer training loop:  38%|###8      | 19172/50000 [11:30:29<14:58:27,  1.75s/it]
Outer training loop:  38%|###8      | 19173/50000 [11:30:31<15:26:04,  1.80s/it]
Outer training loop:  38%|###8      | 19174/50000 [11:30:32<15:21:04,  1.79s/it]
Outer training loop:  38%|###8      | 19175/50000 [11:30:34<15:18:59,  1.79s/it]
Outer training loop:  38%|###8      | 19176/50000 [11:30:36<15:40:07,  1.83s/it]
Outer training loop:  38%|###8      | 19177/50000 [11:30:38<15:26:49,  1.80s/it]
Outer training loop:  38%|###8      | 19178/50000 [11:30:40<15:24:00,  1.80s/it]
Outer training loop:  38%|###8      | 19179/50000 [11:30:42<15:31:27,  1.81s/it]
Outer training loop:  38%|###8      | 19180/50000 [11:30:43<15:22:18,  1.80s/it]
Outer training loop:  38%|###8      | 19181/50000 [11:30:45<15:01:22,  1.75s/it]
Outer training loop:  38%|###8      | 19182/50000 [11:30:47<15:30:18,  1.81s/it]
Outer training loop:  38%|###8      | 19183/50000 [11:30:49<15:06:31,  1.76s/it]
Outer training loop:  38%|###8      | 19184/50000 [11:30:50<14:51:50,  1.74s/it]
Outer training loop:  38%|###8      | 19185/50000 [11:30:52<15:19:05,  1.79s/it]
Outer training loop:  38%|###8      | 19186/50000 [11:30:54<15:06:25,  1.76s/it]
Outer training loop:  38%|###8      | 19187/50000 [11:30:56<15:11:43,  1.78s/it]
Outer training loop:  38%|###8      | 19188/50000 [11:30:57<14:52:20,  1.74s/it]
Outer training loop:  38%|###8      | 19189/50000 [11:30:59<15:29:39,  1.81s/it]
Outer training loop:  38%|###8      | 19190/50000 [11:31:01<15:30:51,  1.81s/it]
Outer training loop:  38%|###8      | 19191/50000 [11:31:03<15:17:11,  1.79s/it]
Outer training loop:  38%|###8      | 19192/50000 [11:31:04<14:53:39,  1.74s/it]
Outer training loop:  38%|###8      | 19193/50000 [11:31:06<15:17:23,  1.79s/it]
Outer training loop:  38%|###8      | 19194/50000 [11:31:08<15:09:28,  1.77s/it]
Outer training loop:  38%|###8      | 19195/50000 [11:31:10<15:08:18,  1.77s/it]
Outer training loop:  38%|###8      | 19196/50000 [11:31:12<15:18:39,  1.79s/it]
Outer training loop:  38%|###8      | 19197/50000 [11:31:13<15:13:00,  1.78s/it]
Outer training loop:  38%|###8      | 19198/50000 [11:31:15<15:09:09,  1.77s/it]
Outer training loop:  38%|###8      | 19199/50000 [11:31:17<15:41:46,  1.83s/it]
Outer training loop:  38%|###8      | 19200/50000 [11:31:19<15:27:44,  1.81s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  38%|###8      | 19201/50000 [11:31:26<28:31:19,  3.33s/it]
Outer training loop:  38%|###8      | 19202/50000 [11:31:28<24:36:58,  2.88s/it]
Outer training loop:  38%|###8      | 19203/50000 [11:31:30<22:03:03,  2.58s/it]
Outer training loop:  38%|###8      | 19204/50000 [11:31:31<19:47:07,  2.31s/it]
Outer training loop:  38%|###8      | 19205/50000 [11:31:33<18:05:18,  2.11s/it]
Outer training loop:  38%|###8      | 19206/50000 [11:31:35<17:15:08,  2.02s/it]
Outer training loop:  38%|###8      | 19207/50000 [11:31:37<16:47:40,  1.96s/it]
Outer training loop:  38%|###8      | 19208/50000 [11:31:38<16:08:21,  1.89s/it]
Outer training loop:  38%|###8      | 19209/50000 [11:31:40<15:26:10,  1.80s/it]
Outer training loop:  38%|###8      | 19210/50000 [11:31:42<15:38:23,  1.83s/it]
Outer training loop:  38%|###8      | 19211/50000 [11:31:43<15:08:13,  1.77s/it]
Outer training loop:  38%|###8      | 19212/50000 [11:31:45<14:56:09,  1.75s/it]
Outer training loop:  38%|###8      | 19213/50000 [11:31:47<14:52:20,  1.74s/it]
Outer training loop:  38%|###8      | 19214/50000 [11:31:49<15:42:28,  1.84s/it]
Outer training loop:  38%|###8      | 19215/50000 [11:31:51<15:20:59,  1.80s/it]
Outer training loop:  38%|###8      | 19216/50000 [11:31:52<15:04:33,  1.76s/it]
Outer training loop:  38%|###8      | 19217/50000 [11:31:54<15:31:43,  1.82s/it]
Outer training loop:  38%|###8      | 19218/50000 [11:31:56<15:17:04,  1.79s/it]
Outer training loop:  38%|###8      | 19219/50000 [11:31:58<15:18:03,  1.79s/it]
Outer training loop:  38%|###8      | 19220/50000 [11:32:00<15:39:18,  1.83s/it]
Outer training loop:  38%|###8      | 19221/50000 [11:32:01<15:41:05,  1.83s/it]
Outer training loop:  38%|###8      | 19222/50000 [11:32:03<15:25:06,  1.80s/it]
Outer training loop:  38%|###8      | 19223/50000 [11:32:05<15:43:21,  1.84s/it]
Outer training loop:  38%|###8      | 19224/50000 [11:32:07<15:23:37,  1.80s/it]
Outer training loop:  38%|###8      | 19225/50000 [11:32:09<15:26:29,  1.81s/it]
Outer training loop:  38%|###8      | 19226/50000 [11:32:10<15:15:00,  1.78s/it]
Outer training loop:  38%|###8      | 19227/50000 [11:32:12<15:30:16,  1.81s/it]
Outer training loop:  38%|###8      | 19228/50000 [11:32:14<15:11:54,  1.78s/it]
Outer training loop:  38%|###8      | 19229/50000 [11:32:16<14:54:33,  1.74s/it]
Outer training loop:  38%|###8      | 19230/50000 [11:32:17<15:02:44,  1.76s/it]
Outer training loop:  38%|###8      | 19231/50000 [11:32:19<14:52:29,  1.74s/it]
Outer training loop:  38%|###8      | 19232/50000 [11:32:21<14:40:48,  1.72s/it]
Outer training loop:  38%|###8      | 19233/50000 [11:32:23<14:56:08,  1.75s/it]
Outer training loop:  38%|###8      | 19234/50000 [11:32:24<15:02:17,  1.76s/it]
Outer training loop:  38%|###8      | 19235/50000 [11:32:26<15:04:38,  1.76s/it]
Outer training loop:  38%|###8      | 19236/50000 [11:32:28<15:25:46,  1.81s/it]
Outer training loop:  38%|###8      | 19237/50000 [11:32:30<15:00:07,  1.76s/it]
Outer training loop:  38%|###8      | 19238/50000 [11:32:31<14:55:28,  1.75s/it]
Outer training loop:  38%|###8      | 19239/50000 [11:32:33<14:59:15,  1.75s/it]
Outer training loop:  38%|###8      | 19240/50000 [11:32:35<15:21:15,  1.80s/it]
Outer training loop:  38%|###8      | 19241/50000 [11:32:37<15:33:04,  1.82s/it]
Outer training loop:  38%|###8      | 19242/50000 [11:32:39<15:19:12,  1.79s/it]
Outer training loop:  38%|###8      | 19243/50000 [11:32:41<15:42:43,  1.84s/it]
Outer training loop:  38%|###8      | 19244/50000 [11:32:42<15:29:27,  1.81s/it]
Outer training loop:  38%|###8      | 19245/50000 [11:32:44<15:24:23,  1.80s/it]
Outer training loop:  38%|###8      | 19246/50000 [11:32:46<15:34:23,  1.82s/it]
Outer training loop:  38%|###8      | 19247/50000 [11:32:48<15:22:34,  1.80s/it]
Outer training loop:  38%|###8      | 19248/50000 [11:32:49<15:00:10,  1.76s/it]
Outer training loop:  38%|###8      | 19249/50000 [11:32:51<15:05:33,  1.77s/it]
Outer training loop:  38%|###8      | 19250/50000 [11:32:53<15:31:35,  1.82s/it]
Outer training loop:  39%|###8      | 19251/50000 [11:32:55<15:15:11,  1.79s/it]
Outer training loop:  39%|###8      | 19252/50000 [11:32:57<15:12:05,  1.78s/it]
Outer training loop:  39%|###8      | 19253/50000 [11:32:58<15:25:43,  1.81s/it]
Outer training loop:  39%|###8      | 19254/50000 [11:33:00<15:24:27,  1.80s/it]
Outer training loop:  39%|###8      | 19255/50000 [11:33:02<15:08:09,  1.77s/it]
Outer training loop:  39%|###8      | 19256/50000 [11:33:04<15:36:32,  1.83s/it]
Outer training loop:  39%|###8      | 19257/50000 [11:33:06<15:22:43,  1.80s/it]
Outer training loop:  39%|###8      | 19258/50000 [11:33:07<14:56:42,  1.75s/it]
Outer training loop:  39%|###8      | 19259/50000 [11:33:09<15:00:45,  1.76s/it]
Outer training loop:  39%|###8      | 19260/50000 [11:33:11<14:50:12,  1.74s/it]
Outer training loop:  39%|###8      | 19261/50000 [11:33:13<14:52:37,  1.74s/it]
Outer training loop:  39%|###8      | 19262/50000 [11:33:14<14:38:38,  1.72s/it]
Outer training loop:  39%|###8      | 19263/50000 [11:33:16<15:06:24,  1.77s/it]
Outer training loop:  39%|###8      | 19264/50000 [11:33:18<15:39:14,  1.83s/it]
Outer training loop:  39%|###8      | 19265/50000 [11:33:20<15:34:10,  1.82s/it]
Outer training loop:  39%|###8      | 19266/50000 [11:33:22<15:13:30,  1.78s/it]
Outer training loop:  39%|###8      | 19267/50000 [11:33:23<15:22:39,  1.80s/it]
Outer training loop:  39%|###8      | 19268/50000 [11:33:25<15:14:35,  1.79s/it]
Outer training loop:  39%|###8      | 19269/50000 [11:33:27<14:52:08,  1.74s/it]
Outer training loop:  39%|###8      | 19270/50000 [11:33:29<15:37:12,  1.83s/it]
Outer training loop:  39%|###8      | 19271/50000 [11:33:31<15:29:45,  1.82s/it]
Outer training loop:  39%|###8      | 19272/50000 [11:33:32<15:17:33,  1.79s/it]
Outer training loop:  39%|###8      | 19273/50000 [11:33:34<15:28:20,  1.81s/it]
Outer training loop:  39%|###8      | 19274/50000 [11:33:36<15:17:55,  1.79s/it]
Outer training loop:  39%|###8      | 19275/50000 [11:33:38<15:19:30,  1.80s/it]
Outer training loop:  39%|###8      | 19276/50000 [11:33:40<15:32:55,  1.82s/it]
Outer training loop:  39%|###8      | 19277/50000 [11:33:41<15:23:51,  1.80s/it]
Outer training loop:  39%|###8      | 19278/50000 [11:33:43<15:19:18,  1.80s/it]
Outer training loop:  39%|###8      | 19279/50000 [11:33:45<15:42:15,  1.84s/it]
Outer training loop:  39%|###8      | 19280/50000 [11:33:47<15:04:38,  1.77s/it]
Outer training loop:  39%|###8      | 19281/50000 [11:33:49<15:20:11,  1.80s/it]
Outer training loop:  39%|###8      | 19282/50000 [11:33:50<15:10:59,  1.78s/it]
Outer training loop:  39%|###8      | 19283/50000 [11:33:52<15:18:48,  1.79s/it]
Outer training loop:  39%|###8      | 19284/50000 [11:33:54<15:09:23,  1.78s/it]
Outer training loop:  39%|###8      | 19285/50000 [11:33:56<15:12:12,  1.78s/it]
Outer training loop:  39%|###8      | 19286/50000 [11:33:58<15:27:20,  1.81s/it]
Outer training loop:  39%|###8      | 19287/50000 [11:33:59<15:04:34,  1.77s/it]
Outer training loop:  39%|###8      | 19288/50000 [11:34:01<15:09:26,  1.78s/it]
Outer training loop:  39%|###8      | 19289/50000 [11:34:03<15:31:04,  1.82s/it]
Outer training loop:  39%|###8      | 19290/50000 [11:34:05<15:14:19,  1.79s/it]
Outer training loop:  39%|###8      | 19291/50000 [11:34:06<15:01:33,  1.76s/it]
Outer training loop:  39%|###8      | 19292/50000 [11:34:08<15:11:14,  1.78s/it]
Outer training loop:  39%|###8      | 19293/50000 [11:34:10<14:44:35,  1.73s/it]
Outer training loop:  39%|###8      | 19294/50000 [11:34:11<14:41:46,  1.72s/it]
Outer training loop:  39%|###8      | 19295/50000 [11:34:13<14:29:44,  1.70s/it]
Outer training loop:  39%|###8      | 19296/50000 [11:34:15<14:54:25,  1.75s/it]
Outer training loop:  39%|###8      | 19297/50000 [11:34:17<14:45:52,  1.73s/it]
Outer training loop:  39%|###8      | 19298/50000 [11:34:18<14:38:50,  1.72s/it]
Outer training loop:  39%|###8      | 19299/50000 [11:34:20<15:01:53,  1.76s/it]
Outer training loop:  39%|###8      | 19300/50000 [11:34:22<15:00:47,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.66s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  39%|###8      | 19301/50000 [11:34:29<28:13:36,  3.31s/it]
Outer training loop:  39%|###8      | 19302/50000 [11:34:31<24:11:06,  2.84s/it]
Outer training loop:  39%|###8      | 19303/50000 [11:34:32<21:35:53,  2.53s/it]
Outer training loop:  39%|###8      | 19304/50000 [11:34:34<19:20:21,  2.27s/it]
Outer training loop:  39%|###8      | 19305/50000 [11:34:36<17:49:10,  2.09s/it]
Outer training loop:  39%|###8      | 19306/50000 [11:34:38<16:49:17,  1.97s/it]
Outer training loop:  39%|###8      | 19307/50000 [11:34:39<16:33:37,  1.94s/it]
Outer training loop:  39%|###8      | 19308/50000 [11:34:41<16:12:47,  1.90s/it]
Outer training loop:  39%|###8      | 19309/50000 [11:34:43<15:25:49,  1.81s/it]
Outer training loop:  39%|###8      | 19310/50000 [11:34:45<15:41:59,  1.84s/it]
Outer training loop:  39%|###8      | 19311/50000 [11:34:47<16:13:09,  1.90s/it]
Outer training loop:  39%|###8      | 19312/50000 [11:34:48<15:34:42,  1.83s/it]
Outer training loop:  39%|###8      | 19313/50000 [11:34:50<15:46:39,  1.85s/it]
Outer training loop:  39%|###8      | 19314/50000 [11:34:52<15:30:01,  1.82s/it]
Outer training loop:  39%|###8      | 19315/50000 [11:34:54<15:13:18,  1.79s/it]
Outer training loop:  39%|###8      | 19316/50000 [11:34:55<14:56:39,  1.75s/it]
Outer training loop:  39%|###8      | 19317/50000 [11:34:57<15:33:02,  1.82s/it]
Outer training loop:  39%|###8      | 19318/50000 [11:34:59<15:13:14,  1.79s/it]
Outer training loop:  39%|###8      | 19319/50000 [11:35:01<15:25:18,  1.81s/it]
Outer training loop:  39%|###8      | 19320/50000 [11:35:03<15:35:01,  1.83s/it]
Outer training loop:  39%|###8      | 19321/50000 [11:35:05<15:15:20,  1.79s/it]
Outer training loop:  39%|###8      | 19322/50000 [11:35:06<14:50:31,  1.74s/it]
Outer training loop:  39%|###8      | 19323/50000 [11:35:08<15:08:06,  1.78s/it]
Outer training loop:  39%|###8      | 19324/50000 [11:35:10<14:58:53,  1.76s/it]
Outer training loop:  39%|###8      | 19325/50000 [11:35:11<14:53:51,  1.75s/it]
Outer training loop:  39%|###8      | 19326/50000 [11:35:13<14:52:22,  1.75s/it]
Outer training loop:  39%|###8      | 19327/50000 [11:35:15<15:03:49,  1.77s/it]
Outer training loop:  39%|###8      | 19328/50000 [11:35:17<14:40:26,  1.72s/it]
Outer training loop:  39%|###8      | 19329/50000 [11:35:18<14:28:37,  1.70s/it]
Outer training loop:  39%|###8      | 19330/50000 [11:35:20<14:57:33,  1.76s/it]
Outer training loop:  39%|###8      | 19331/50000 [11:35:22<14:44:11,  1.73s/it]
Outer training loop:  39%|###8      | 19332/50000 [11:35:23<14:26:40,  1.70s/it]
Outer training loop:  39%|###8      | 19333/50000 [11:35:25<14:27:04,  1.70s/it]
Outer training loop:  39%|###8      | 19334/50000 [11:35:27<14:56:12,  1.75s/it]
Outer training loop:  39%|###8      | 19335/50000 [11:35:29<14:58:26,  1.76s/it]
Outer training loop:  39%|###8      | 19336/50000 [11:35:30<14:42:22,  1.73s/it]
Outer training loop:  39%|###8      | 19337/50000 [11:35:32<14:57:26,  1.76s/it]
Outer training loop:  39%|###8      | 19338/50000 [11:35:34<14:52:44,  1.75s/it]
Outer training loop:  39%|###8      | 19339/50000 [11:35:36<14:55:09,  1.75s/it]
Outer training loop:  39%|###8      | 19340/50000 [11:35:37<14:47:46,  1.74s/it]
Outer training loop:  39%|###8      | 19341/50000 [11:35:39<15:15:44,  1.79s/it]
Outer training loop:  39%|###8      | 19342/50000 [11:35:41<15:17:05,  1.79s/it]
Outer training loop:  39%|###8      | 19343/50000 [11:35:43<14:49:57,  1.74s/it]
Outer training loop:  39%|###8      | 19344/50000 [11:35:45<15:21:47,  1.80s/it]
Outer training loop:  39%|###8      | 19345/50000 [11:35:46<14:56:50,  1.76s/it]
Outer training loop:  39%|###8      | 19346/50000 [11:35:48<14:45:40,  1.73s/it]
Outer training loop:  39%|###8      | 19347/50000 [11:35:50<14:48:01,  1.74s/it]
Outer training loop:  39%|###8      | 19348/50000 [11:35:52<15:07:27,  1.78s/it]
Outer training loop:  39%|###8      | 19349/50000 [11:35:54<15:08:03,  1.78s/it]
Outer training loop:  39%|###8      | 19350/50000 [11:35:55<14:54:14,  1.75s/it]
Outer training loop:  39%|###8      | 19351/50000 [11:35:57<15:01:23,  1.76s/it]
Outer training loop:  39%|###8      | 19352/50000 [11:35:59<14:37:34,  1.72s/it]
Outer training loop:  39%|###8      | 19353/50000 [11:36:00<14:30:50,  1.70s/it]
Outer training loop:  39%|###8      | 19354/50000 [11:36:02<15:07:56,  1.78s/it]
Outer training loop:  39%|###8      | 19355/50000 [11:36:04<15:05:52,  1.77s/it]
Outer training loop:  39%|###8      | 19356/50000 [11:36:06<15:07:22,  1.78s/it]
Outer training loop:  39%|###8      | 19357/50000 [11:36:08<15:06:48,  1.78s/it]
Outer training loop:  39%|###8      | 19358/50000 [11:36:10<15:40:20,  1.84s/it]
Outer training loop:  39%|###8      | 19359/50000 [11:36:11<15:22:07,  1.81s/it]
Outer training loop:  39%|###8      | 19360/50000 [11:36:13<15:07:06,  1.78s/it]
Outer training loop:  39%|###8      | 19361/50000 [11:36:15<15:44:32,  1.85s/it]
Outer training loop:  39%|###8      | 19362/50000 [11:36:17<15:36:41,  1.83s/it]
Outer training loop:  39%|###8      | 19363/50000 [11:36:18<15:05:48,  1.77s/it]
Outer training loop:  39%|###8      | 19364/50000 [11:36:20<15:04:26,  1.77s/it]
Outer training loop:  39%|###8      | 19365/50000 [11:36:22<15:16:35,  1.80s/it]
Outer training loop:  39%|###8      | 19366/50000 [11:36:24<14:52:03,  1.75s/it]
Outer training loop:  39%|###8      | 19367/50000 [11:36:25<14:44:49,  1.73s/it]
Outer training loop:  39%|###8      | 19368/50000 [11:36:27<15:11:29,  1.79s/it]
Outer training loop:  39%|###8      | 19369/50000 [11:36:29<14:53:06,  1.75s/it]
Outer training loop:  39%|###8      | 19370/50000 [11:36:31<15:02:04,  1.77s/it]
Outer training loop:  39%|###8      | 19371/50000 [11:36:33<15:25:40,  1.81s/it]
Outer training loop:  39%|###8      | 19372/50000 [11:36:34<15:22:27,  1.81s/it]
Outer training loop:  39%|###8      | 19373/50000 [11:36:36<15:13:32,  1.79s/it]
Outer training loop:  39%|###8      | 19374/50000 [11:36:38<15:09:19,  1.78s/it]
Outer training loop:  39%|###8      | 19375/50000 [11:36:40<15:49:06,  1.86s/it]
Outer training loop:  39%|###8      | 19376/50000 [11:36:42<15:34:20,  1.83s/it]
Outer training loop:  39%|###8      | 19377/50000 [11:36:43<15:16:09,  1.80s/it]
Outer training loop:  39%|###8      | 19378/50000 [11:36:45<15:08:09,  1.78s/it]
Outer training loop:  39%|###8      | 19379/50000 [11:36:47<15:01:05,  1.77s/it]
Outer training loop:  39%|###8      | 19380/50000 [11:36:49<14:44:44,  1.73s/it]
Outer training loop:  39%|###8      | 19381/50000 [11:36:50<14:47:10,  1.74s/it]
Outer training loop:  39%|###8      | 19382/50000 [11:36:52<14:53:34,  1.75s/it]
Outer training loop:  39%|###8      | 19383/50000 [11:36:54<14:34:36,  1.71s/it]
Outer training loop:  39%|###8      | 19384/50000 [11:36:56<14:47:03,  1.74s/it]
Outer training loop:  39%|###8      | 19385/50000 [11:36:58<15:17:07,  1.80s/it]
Outer training loop:  39%|###8      | 19386/50000 [11:36:59<15:05:16,  1.77s/it]
Outer training loop:  39%|###8      | 19387/50000 [11:37:01<14:44:41,  1.73s/it]
Outer training loop:  39%|###8      | 19388/50000 [11:37:03<15:12:42,  1.79s/it]
Outer training loop:  39%|###8      | 19389/50000 [11:37:04<14:58:06,  1.76s/it]
Outer training loop:  39%|###8      | 19390/50000 [11:37:06<14:39:07,  1.72s/it]
Outer training loop:  39%|###8      | 19391/50000 [11:37:08<14:53:53,  1.75s/it]
Outer training loop:  39%|###8      | 19392/50000 [11:37:10<14:46:02,  1.74s/it]
Outer training loop:  39%|###8      | 19393/50000 [11:37:11<14:37:01,  1.72s/it]
Outer training loop:  39%|###8      | 19394/50000 [11:37:13<14:43:22,  1.73s/it]
Outer training loop:  39%|###8      | 19395/50000 [11:37:15<15:26:49,  1.82s/it]
Outer training loop:  39%|###8      | 19396/50000 [11:37:17<15:05:41,  1.78s/it]
Outer training loop:  39%|###8      | 19397/50000 [11:37:18<14:53:40,  1.75s/it]
Outer training loop:  39%|###8      | 19398/50000 [11:37:20<15:17:55,  1.80s/it]
Outer training loop:  39%|###8      | 19399/50000 [11:37:22<15:07:20,  1.78s/it]
Outer training loop:  39%|###8      | 19400/50000 [11:37:24<14:52:29,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.61s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.64s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.60s/it]

Outer training loop:  39%|###8      | 19401/50000 [11:37:31<27:53:24,  3.28s/it]
Outer training loop:  39%|###8      | 19402/50000 [11:37:33<24:26:02,  2.87s/it]
Outer training loop:  39%|###8      | 19403/50000 [11:37:34<21:27:54,  2.53s/it]
Outer training loop:  39%|###8      | 19404/50000 [11:37:36<19:37:18,  2.31s/it]
Outer training loop:  39%|###8      | 19405/50000 [11:37:38<18:52:26,  2.22s/it]
Outer training loop:  39%|###8      | 19406/50000 [11:37:40<17:48:41,  2.10s/it]
Outer training loop:  39%|###8      | 19407/50000 [11:37:42<16:48:22,  1.98s/it]
Outer training loop:  39%|###8      | 19408/50000 [11:37:43<16:14:24,  1.91s/it]
Outer training loop:  39%|###8      | 19409/50000 [11:37:45<16:12:58,  1.91s/it]
Outer training loop:  39%|###8      | 19410/50000 [11:37:47<15:51:23,  1.87s/it]
Outer training loop:  39%|###8      | 19411/50000 [11:37:49<15:10:20,  1.79s/it]
Outer training loop:  39%|###8      | 19412/50000 [11:37:50<15:19:55,  1.80s/it]
Outer training loop:  39%|###8      | 19413/50000 [11:37:52<15:02:21,  1.77s/it]
Outer training loop:  39%|###8      | 19414/50000 [11:37:54<14:59:52,  1.77s/it]
Outer training loop:  39%|###8      | 19415/50000 [11:37:56<15:11:18,  1.79s/it]
Outer training loop:  39%|###8      | 19416/50000 [11:37:57<14:53:12,  1.75s/it]
Outer training loop:  39%|###8      | 19417/50000 [11:37:59<14:35:12,  1.72s/it]
Outer training loop:  39%|###8      | 19418/50000 [11:38:01<14:34:54,  1.72s/it]
Outer training loop:  39%|###8      | 19419/50000 [11:38:03<14:58:02,  1.76s/it]
Outer training loop:  39%|###8      | 19420/50000 [11:38:04<14:48:37,  1.74s/it]
Outer training loop:  39%|###8      | 19421/50000 [11:38:06<14:50:52,  1.75s/it]
Outer training loop:  39%|###8      | 19422/50000 [11:38:08<15:24:09,  1.81s/it]
Outer training loop:  39%|###8      | 19423/50000 [11:38:10<15:10:04,  1.79s/it]
Outer training loop:  39%|###8      | 19424/50000 [11:38:12<14:56:41,  1.76s/it]
Outer training loop:  39%|###8      | 19425/50000 [11:38:13<15:06:35,  1.78s/it]
Outer training loop:  39%|###8      | 19426/50000 [11:38:15<14:48:28,  1.74s/it]
Outer training loop:  39%|###8      | 19427/50000 [11:38:17<14:52:54,  1.75s/it]
Outer training loop:  39%|###8      | 19428/50000 [11:38:18<14:46:42,  1.74s/it]
Outer training loop:  39%|###8      | 19429/50000 [11:38:20<15:17:56,  1.80s/it]
Outer training loop:  39%|###8      | 19430/50000 [11:38:22<15:01:24,  1.77s/it]
Outer training loop:  39%|###8      | 19431/50000 [11:38:24<15:14:28,  1.79s/it]
Outer training loop:  39%|###8      | 19432/50000 [11:38:26<15:27:57,  1.82s/it]
Outer training loop:  39%|###8      | 19433/50000 [11:38:28<15:18:37,  1.80s/it]
Outer training loop:  39%|###8      | 19434/50000 [11:38:29<14:47:52,  1.74s/it]
Outer training loop:  39%|###8      | 19435/50000 [11:38:31<15:01:03,  1.77s/it]
Outer training loop:  39%|###8      | 19436/50000 [11:38:33<14:56:18,  1.76s/it]
Outer training loop:  39%|###8      | 19437/50000 [11:38:35<15:00:23,  1.77s/it]
Outer training loop:  39%|###8      | 19438/50000 [11:38:36<14:48:42,  1.74s/it]
Outer training loop:  39%|###8      | 19439/50000 [11:38:38<15:20:13,  1.81s/it]
Outer training loop:  39%|###8      | 19440/50000 [11:38:40<14:58:56,  1.76s/it]
Outer training loop:  39%|###8      | 19441/50000 [11:38:42<14:48:39,  1.74s/it]
Outer training loop:  39%|###8      | 19442/50000 [11:38:43<15:13:05,  1.79s/it]
Outer training loop:  39%|###8      | 19443/50000 [11:38:45<14:57:31,  1.76s/it]
Outer training loop:  39%|###8      | 19444/50000 [11:38:47<14:50:06,  1.75s/it]
Outer training loop:  39%|###8      | 19445/50000 [11:38:49<15:30:11,  1.83s/it]
Outer training loop:  39%|###8      | 19446/50000 [11:38:51<14:57:20,  1.76s/it]
Outer training loop:  39%|###8      | 19447/50000 [11:38:52<14:42:51,  1.73s/it]
Outer training loop:  39%|###8      | 19448/50000 [11:38:54<14:42:24,  1.73s/it]
Outer training loop:  39%|###8      | 19449/50000 [11:38:56<15:08:38,  1.78s/it]
Outer training loop:  39%|###8      | 19450/50000 [11:38:58<14:56:36,  1.76s/it]
Outer training loop:  39%|###8      | 19451/50000 [11:38:59<14:57:16,  1.76s/it]
Outer training loop:  39%|###8      | 19452/50000 [11:39:01<15:19:28,  1.81s/it]
Outer training loop:  39%|###8      | 19453/50000 [11:39:03<14:52:33,  1.75s/it]
Outer training loop:  39%|###8      | 19454/50000 [11:39:04<14:31:18,  1.71s/it]
Outer training loop:  39%|###8      | 19455/50000 [11:39:06<14:58:37,  1.77s/it]
Outer training loop:  39%|###8      | 19456/50000 [11:39:08<14:55:44,  1.76s/it]
Outer training loop:  39%|###8      | 19457/50000 [11:39:10<14:50:38,  1.75s/it]
Outer training loop:  39%|###8      | 19458/50000 [11:39:12<15:09:31,  1.79s/it]
Outer training loop:  39%|###8      | 19459/50000 [11:39:13<15:00:42,  1.77s/it]
Outer training loop:  39%|###8      | 19460/50000 [11:39:15<14:57:16,  1.76s/it]
Outer training loop:  39%|###8      | 19461/50000 [11:39:17<14:42:41,  1.73s/it]
Outer training loop:  39%|###8      | 19462/50000 [11:39:19<15:01:56,  1.77s/it]
Outer training loop:  39%|###8      | 19463/50000 [11:39:20<15:06:24,  1.78s/it]
Outer training loop:  39%|###8      | 19464/50000 [11:39:22<14:48:49,  1.75s/it]
Outer training loop:  39%|###8      | 19465/50000 [11:39:24<14:47:01,  1.74s/it]
Outer training loop:  39%|###8      | 19466/50000 [11:39:26<15:20:08,  1.81s/it]
Outer training loop:  39%|###8      | 19467/50000 [11:39:27<14:50:06,  1.75s/it]
Outer training loop:  39%|###8      | 19468/50000 [11:39:29<14:39:04,  1.73s/it]
Outer training loop:  39%|###8      | 19469/50000 [11:39:31<15:05:50,  1.78s/it]
Outer training loop:  39%|###8      | 19470/50000 [11:39:33<15:08:56,  1.79s/it]
Outer training loop:  39%|###8      | 19471/50000 [11:39:35<14:58:26,  1.77s/it]
Outer training loop:  39%|###8      | 19472/50000 [11:39:36<14:46:41,  1.74s/it]
Outer training loop:  39%|###8      | 19473/50000 [11:39:38<15:13:06,  1.79s/it]
Outer training loop:  39%|###8      | 19474/50000 [11:39:40<14:58:56,  1.77s/it]
Outer training loop:  39%|###8      | 19475/50000 [11:39:42<14:44:31,  1.74s/it]
Outer training loop:  39%|###8      | 19476/50000 [11:39:43<15:12:15,  1.79s/it]
Outer training loop:  39%|###8      | 19477/50000 [11:39:45<14:53:56,  1.76s/it]
Outer training loop:  39%|###8      | 19478/50000 [11:39:47<14:37:57,  1.73s/it]
Outer training loop:  39%|###8      | 19479/50000 [11:39:48<14:30:52,  1.71s/it]
Outer training loop:  39%|###8      | 19480/50000 [11:39:50<14:49:43,  1.75s/it]
Outer training loop:  39%|###8      | 19481/50000 [11:39:52<14:47:55,  1.75s/it]
Outer training loop:  39%|###8      | 19482/50000 [11:39:54<14:41:49,  1.73s/it]
Outer training loop:  39%|###8      | 19483/50000 [11:39:56<15:08:38,  1.79s/it]
Outer training loop:  39%|###8      | 19484/50000 [11:39:57<14:52:40,  1.76s/it]
Outer training loop:  39%|###8      | 19485/50000 [11:39:59<14:47:25,  1.74s/it]
Outer training loop:  39%|###8      | 19486/50000 [11:40:01<15:02:49,  1.78s/it]
Outer training loop:  39%|###8      | 19487/50000 [11:40:03<15:19:18,  1.81s/it]
Outer training loop:  39%|###8      | 19488/50000 [11:40:04<14:49:36,  1.75s/it]
Outer training loop:  39%|###8      | 19489/50000 [11:40:06<14:57:21,  1.76s/it]
Outer training loop:  39%|###8      | 19490/50000 [11:40:08<15:24:18,  1.82s/it]
Outer training loop:  39%|###8      | 19491/50000 [11:40:10<15:03:53,  1.78s/it]
Outer training loop:  39%|###8      | 19492/50000 [11:40:12<14:54:48,  1.76s/it]
Outer training loop:  39%|###8      | 19493/50000 [11:40:13<14:44:58,  1.74s/it]
Outer training loop:  39%|###8      | 19494/50000 [11:40:15<14:58:22,  1.77s/it]
Outer training loop:  39%|###8      | 19495/50000 [11:40:17<14:49:15,  1.75s/it]
Outer training loop:  39%|###8      | 19496/50000 [11:40:18<14:41:21,  1.73s/it]
Outer training loop:  39%|###8      | 19497/50000 [11:40:20<15:16:11,  1.80s/it]
Outer training loop:  39%|###8      | 19498/50000 [11:40:22<14:58:39,  1.77s/it]
Outer training loop:  39%|###8      | 19499/50000 [11:40:24<14:56:09,  1.76s/it]
Outer training loop:  39%|###9      | 19500/50000 [11:40:26<15:11:07,  1.79s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.67s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.69s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.71s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:18,  1.69s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:18,  1.71s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:21,  1.77s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:20,  1.78s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.78s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.77s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:14,  1.82s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:09,  1.78s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:12,  1.85s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.82s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:04,  1.78s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:01,  1.76s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:02,  1.81s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:59,  1.78s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.79s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:56,  1.79s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:55,  1.84s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:52,  1.81s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:48,  1.78s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:46,  1.78s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:45,  1.81s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:42,  1.79s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:41,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:38,  1.79s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:54<01:35,  1.79s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:32,  1.78s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:29,  1.76s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:27,  1.75s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:29,  1.83s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:25,  1.78s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:23,  1.77s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:20,  1.74s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:17,  1.72s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:16,  1.75s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.72s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:14,  1.81s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:11,  1.79s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.77s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:05,  1.73s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:03,  1.72s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:01,  1.71s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:03,  1.81s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<00:59,  1.75s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:56,  1.72s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:55,  1.72s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:53,  1.72s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:51,  1.71s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:51,  1.76s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:47,  1.81s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:44,  1.79s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.76s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:40,  1.78s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:39,  1.78s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:36,  1.76s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:34,  1.80s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:31,  1.76s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.74s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.73s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:26,  1.76s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.74s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.74s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:20,  1.73s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:20,  1.88s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:14,  1.82s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.79s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.81s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.78s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.79s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.93s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.75s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:14,  1.64s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:13,  1.64s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:14,  1.68s/it]

Evaluating policy:   5%|4         | 4/83 [00:06<02:16,  1.72s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:13,  1.71s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:10,  1.69s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:23,  1.89s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:17,  1.84s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:14,  1.82s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:12,  1.81s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:14,  1.87s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:12,  1.86s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:06,  1.81s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:02,  1.78s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:04,  1.83s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:01,  1.81s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:58,  1.79s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:55,  1.77s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:57,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:54,  1.81s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:53,  1.83s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:49,  1.79s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:47,  1.80s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:44,  1.81s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:42,  1.80s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:38,  1.76s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:35,  1.74s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:33,  1.72s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:35,  1.81s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.76s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:28,  1.74s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:26,  1.74s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:25,  1.75s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:22,  1.73s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:28,  1.88s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:20,  1.80s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:17,  1.77s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.74s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:12,  1.72s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:12,  1.80s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:11,  1.82s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:09,  1.82s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:06,  1.79s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:04,  1.79s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:01,  1.75s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:28<01:04,  1.89s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:55,  1.79s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:53,  1.79s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:50,  1.75s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:49,  1.75s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:50,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:44<00:45,  1.81s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.77s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:40,  1.75s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:38,  1.74s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:36,  1.73s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:34,  1.71s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:32,  1.78s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:29,  1.74s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.74s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:25,  1.72s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:24,  1.75s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.72s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:21,  1.77s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:20,  1.88s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:16,  1.78s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:14,  1.77s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.76s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.73s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.71s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:06,  1.73s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.73s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.74s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.87s/it]

Outer training loop:  39%|###9      | 19501/50000 [11:45:30<782:51:54, 92.41s/it]
Outer training loop:  39%|###9      | 19502/50000 [11:45:31<552:14:17, 65.19s/it]
Outer training loop:  39%|###9      | 19503/50000 [11:45:33<390:39:08, 46.11s/it]
Outer training loop:  39%|###9      | 19504/50000 [11:45:35<277:54:04, 32.81s/it]
Outer training loop:  39%|###9      | 19505/50000 [11:45:36<198:57:06, 23.49s/it]
Outer training loop:  39%|###9      | 19506/50000 [11:45:38<143:22:35, 16.93s/it]
Outer training loop:  39%|###9      | 19507/50000 [11:45:40<104:51:16, 12.38s/it]
Outer training loop:  39%|###9      | 19508/50000 [11:45:41<77:44:43,  9.18s/it] 
Outer training loop:  39%|###9      | 19509/50000 [11:45:43<58:37:51,  6.92s/it]
Outer training loop:  39%|###9      | 19510/50000 [11:45:45<45:21:34,  5.36s/it]
Outer training loop:  39%|###9      | 19511/50000 [11:45:47<37:43:47,  4.45s/it]
Outer training loop:  39%|###9      | 19512/50000 [11:45:49<30:59:58,  3.66s/it]
Outer training loop:  39%|###9      | 19513/50000 [11:45:51<26:12:49,  3.10s/it]
Outer training loop:  39%|###9      | 19514/50000 [11:45:53<23:13:25,  2.74s/it]
Outer training loop:  39%|###9      | 19515/50000 [11:45:54<20:50:25,  2.46s/it]
Outer training loop:  39%|###9      | 19516/50000 [11:45:56<18:52:20,  2.23s/it]
Outer training loop:  39%|###9      | 19517/50000 [11:45:58<17:50:11,  2.11s/it]
Outer training loop:  39%|###9      | 19518/50000 [11:46:00<16:48:29,  1.99s/it]
Outer training loop:  39%|###9      | 19519/50000 [11:46:01<16:13:24,  1.92s/it]
Outer training loop:  39%|###9      | 19520/50000 [11:46:03<16:22:14,  1.93s/it]
Outer training loop:  39%|###9      | 19521/50000 [11:46:05<15:57:03,  1.88s/it]
Outer training loop:  39%|###9      | 19522/50000 [11:46:07<15:36:04,  1.84s/it]
Outer training loop:  39%|###9      | 19523/50000 [11:46:09<15:39:29,  1.85s/it]
Outer training loop:  39%|###9      | 19524/50000 [11:46:11<15:23:36,  1.82s/it]
Outer training loop:  39%|###9      | 19525/50000 [11:46:12<15:09:42,  1.79s/it]
Outer training loop:  39%|###9      | 19526/50000 [11:46:14<15:03:41,  1.78s/it]
Outer training loop:  39%|###9      | 19527/50000 [11:46:16<15:23:55,  1.82s/it]
Outer training loop:  39%|###9      | 19528/50000 [11:46:18<15:09:07,  1.79s/it]
Outer training loop:  39%|###9      | 19529/50000 [11:46:19<15:02:57,  1.78s/it]
Outer training loop:  39%|###9      | 19530/50000 [11:46:21<15:13:21,  1.80s/it]
Outer training loop:  39%|###9      | 19531/50000 [11:46:23<14:59:22,  1.77s/it]
Outer training loop:  39%|###9      | 19532/50000 [11:46:25<14:50:04,  1.75s/it]
Outer training loop:  39%|###9      | 19533/50000 [11:46:27<15:09:35,  1.79s/it]
Outer training loop:  39%|###9      | 19534/50000 [11:46:28<14:59:26,  1.77s/it]
Outer training loop:  39%|###9      | 19535/50000 [11:46:30<14:43:21,  1.74s/it]
Outer training loop:  39%|###9      | 19536/50000 [11:46:32<14:24:14,  1.70s/it]
Outer training loop:  39%|###9      | 19537/50000 [11:46:34<15:04:39,  1.78s/it]
Outer training loop:  39%|###9      | 19538/50000 [11:46:35<14:49:46,  1.75s/it]
Outer training loop:  39%|###9      | 19539/50000 [11:46:37<15:12:30,  1.80s/it]
Outer training loop:  39%|###9      | 19540/50000 [11:46:39<15:26:03,  1.82s/it]
Outer training loop:  39%|###9      | 19541/50000 [11:46:41<14:59:10,  1.77s/it]
Outer training loop:  39%|###9      | 19542/50000 [11:46:42<14:48:50,  1.75s/it]
Outer training loop:  39%|###9      | 19543/50000 [11:46:44<14:38:29,  1.73s/it]
Outer training loop:  39%|###9      | 19544/50000 [11:46:46<14:55:06,  1.76s/it]
Outer training loop:  39%|###9      | 19545/50000 [11:46:48<14:42:07,  1.74s/it]
Outer training loop:  39%|###9      | 19546/50000 [11:46:49<14:37:45,  1.73s/it]
Outer training loop:  39%|###9      | 19547/50000 [11:46:51<14:31:24,  1.72s/it]
Outer training loop:  39%|###9      | 19548/50000 [11:46:53<14:39:53,  1.73s/it]
Outer training loop:  39%|###9      | 19549/50000 [11:46:55<15:05:01,  1.78s/it]
Outer training loop:  39%|###9      | 19550/50000 [11:46:56<14:45:33,  1.74s/it]
Outer training loop:  39%|###9      | 19551/50000 [11:46:58<15:19:42,  1.81s/it]
Outer training loop:  39%|###9      | 19552/50000 [11:47:00<15:12:38,  1.80s/it]
Outer training loop:  39%|###9      | 19553/50000 [11:47:02<15:08:34,  1.79s/it]
Outer training loop:  39%|###9      | 19554/50000 [11:47:04<15:26:34,  1.83s/it]
Outer training loop:  39%|###9      | 19555/50000 [11:47:05<15:19:13,  1.81s/it]
Outer training loop:  39%|###9      | 19556/50000 [11:47:07<14:59:38,  1.77s/it]
Outer training loop:  39%|###9      | 19557/50000 [11:47:09<14:46:17,  1.75s/it]
Outer training loop:  39%|###9      | 19558/50000 [11:47:11<15:03:42,  1.78s/it]
Outer training loop:  39%|###9      | 19559/50000 [11:47:12<14:47:17,  1.75s/it]
Outer training loop:  39%|###9      | 19560/50000 [11:47:14<14:39:45,  1.73s/it]
Outer training loop:  39%|###9      | 19561/50000 [11:47:16<15:14:08,  1.80s/it]
Outer training loop:  39%|###9      | 19562/50000 [11:47:18<14:54:10,  1.76s/it]
Outer training loop:  39%|###9      | 19563/50000 [11:47:19<14:57:05,  1.77s/it]
Outer training loop:  39%|###9      | 19564/50000 [11:47:21<15:15:40,  1.81s/it]
Outer training loop:  39%|###9      | 19565/50000 [11:47:23<15:04:15,  1.78s/it]
Outer training loop:  39%|###9      | 19566/50000 [11:47:25<14:51:16,  1.76s/it]
Outer training loop:  39%|###9      | 19567/50000 [11:47:26<14:43:59,  1.74s/it]
Outer training loop:  39%|###9      | 19568/50000 [11:47:28<15:08:16,  1.79s/it]
Outer training loop:  39%|###9      | 19569/50000 [11:47:30<14:58:35,  1.77s/it]
Outer training loop:  39%|###9      | 19570/50000 [11:47:32<14:38:12,  1.73s/it]
Outer training loop:  39%|###9      | 19571/50000 [11:47:34<15:05:38,  1.79s/it]
Outer training loop:  39%|###9      | 19572/50000 [11:47:35<14:56:40,  1.77s/it]
Outer training loop:  39%|###9      | 19573/50000 [11:47:37<14:48:15,  1.75s/it]
Outer training loop:  39%|###9      | 19574/50000 [11:47:39<14:36:46,  1.73s/it]
Outer training loop:  39%|###9      | 19575/50000 [11:47:41<15:02:49,  1.78s/it]
Outer training loop:  39%|###9      | 19576/50000 [11:47:42<14:57:05,  1.77s/it]
Outer training loop:  39%|###9      | 19577/50000 [11:47:44<14:47:17,  1.75s/it]
Outer training loop:  39%|###9      | 19578/50000 [11:47:46<15:11:54,  1.80s/it]
Outer training loop:  39%|###9      | 19579/50000 [11:47:48<15:01:34,  1.78s/it]
Outer training loop:  39%|###9      | 19580/50000 [11:47:49<14:50:19,  1.76s/it]
Outer training loop:  39%|###9      | 19581/50000 [11:47:51<14:38:27,  1.73s/it]
Outer training loop:  39%|###9      | 19582/50000 [11:47:53<14:57:25,  1.77s/it]
Outer training loop:  39%|###9      | 19583/50000 [11:47:55<15:18:48,  1.81s/it]
Outer training loop:  39%|###9      | 19584/50000 [11:47:57<15:15:53,  1.81s/it]
Outer training loop:  39%|###9      | 19585/50000 [11:47:59<15:53:17,  1.88s/it]
Outer training loop:  39%|###9      | 19586/50000 [11:48:00<15:22:17,  1.82s/it]
Outer training loop:  39%|###9      | 19587/50000 [11:48:02<15:14:05,  1.80s/it]
Outer training loop:  39%|###9      | 19588/50000 [11:48:04<14:51:22,  1.76s/it]
Outer training loop:  39%|###9      | 19589/50000 [11:48:06<15:57:13,  1.89s/it]
Outer training loop:  39%|###9      | 19590/50000 [11:48:08<15:21:56,  1.82s/it]
Outer training loop:  39%|###9      | 19591/50000 [11:48:09<15:12:41,  1.80s/it]
Outer training loop:  39%|###9      | 19592/50000 [11:48:11<15:26:54,  1.83s/it]
Outer training loop:  39%|###9      | 19593/50000 [11:48:13<15:02:51,  1.78s/it]
Outer training loop:  39%|###9      | 19594/50000 [11:48:15<14:38:51,  1.73s/it]
Outer training loop:  39%|###9      | 19595/50000 [11:48:17<15:01:12,  1.78s/it]
Outer training loop:  39%|###9      | 19596/50000 [11:48:18<14:56:44,  1.77s/it]
Outer training loop:  39%|###9      | 19597/50000 [11:48:20<14:47:12,  1.75s/it]
Outer training loop:  39%|###9      | 19598/50000 [11:48:22<14:35:50,  1.73s/it]
Outer training loop:  39%|###9      | 19599/50000 [11:48:24<15:00:37,  1.78s/it]
Outer training loop:  39%|###9      | 19600/50000 [11:48:25<14:55:34,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]

Outer training loop:  39%|###9      | 19601/50000 [11:48:32<28:05:43,  3.33s/it]
Outer training loop:  39%|###9      | 19602/50000 [11:48:34<24:35:25,  2.91s/it]
Outer training loop:  39%|###9      | 19603/50000 [11:48:36<21:19:20,  2.53s/it]
Outer training loop:  39%|###9      | 19604/50000 [11:48:38<19:12:20,  2.27s/it]
Outer training loop:  39%|###9      | 19605/50000 [11:48:40<18:27:33,  2.19s/it]
Outer training loop:  39%|###9      | 19606/50000 [11:48:41<17:08:26,  2.03s/it]
Outer training loop:  39%|###9      | 19607/50000 [11:48:43<16:27:59,  1.95s/it]
Outer training loop:  39%|###9      | 19608/50000 [11:48:45<16:07:44,  1.91s/it]
Outer training loop:  39%|###9      | 19609/50000 [11:48:47<16:12:20,  1.92s/it]
Outer training loop:  39%|###9      | 19610/50000 [11:48:48<15:34:09,  1.84s/it]
Outer training loop:  39%|###9      | 19611/50000 [11:48:50<15:18:56,  1.81s/it]
Outer training loop:  39%|###9      | 19612/50000 [11:48:52<15:31:41,  1.84s/it]
Outer training loop:  39%|###9      | 19613/50000 [11:48:54<15:23:16,  1.82s/it]
Outer training loop:  39%|###9      | 19614/50000 [11:48:55<14:59:59,  1.78s/it]
Outer training loop:  39%|###9      | 19615/50000 [11:48:57<15:04:43,  1.79s/it]
Outer training loop:  39%|###9      | 19616/50000 [11:48:59<14:50:43,  1.76s/it]
Outer training loop:  39%|###9      | 19617/50000 [11:49:01<14:37:12,  1.73s/it]
Outer training loop:  39%|###9      | 19618/50000 [11:49:02<14:22:48,  1.70s/it]
Outer training loop:  39%|###9      | 19619/50000 [11:49:04<14:53:38,  1.76s/it]
Outer training loop:  39%|###9      | 19620/50000 [11:49:06<14:32:59,  1.72s/it]
Outer training loop:  39%|###9      | 19621/50000 [11:49:08<14:42:51,  1.74s/it]
Outer training loop:  39%|###9      | 19622/50000 [11:49:09<15:02:44,  1.78s/it]
Outer training loop:  39%|###9      | 19623/50000 [11:49:11<14:52:53,  1.76s/it]
Outer training loop:  39%|###9      | 19624/50000 [11:49:13<14:57:23,  1.77s/it]
Outer training loop:  39%|###9      | 19625/50000 [11:49:15<15:19:24,  1.82s/it]
Outer training loop:  39%|###9      | 19626/50000 [11:49:17<15:02:17,  1.78s/it]
Outer training loop:  39%|###9      | 19627/50000 [11:49:18<14:51:54,  1.76s/it]
Outer training loop:  39%|###9      | 19628/50000 [11:49:20<15:04:23,  1.79s/it]
Outer training loop:  39%|###9      | 19629/50000 [11:49:22<15:10:01,  1.80s/it]
Outer training loop:  39%|###9      | 19630/50000 [11:49:24<14:51:46,  1.76s/it]
Outer training loop:  39%|###9      | 19631/50000 [11:49:25<14:49:42,  1.76s/it]
Outer training loop:  39%|###9      | 19632/50000 [11:49:27<15:11:11,  1.80s/it]
Outer training loop:  39%|###9      | 19633/50000 [11:49:29<14:54:25,  1.77s/it]
Outer training loop:  39%|###9      | 19634/50000 [11:49:31<14:35:36,  1.73s/it]
Outer training loop:  39%|###9      | 19635/50000 [11:49:33<15:05:09,  1.79s/it]
Outer training loop:  39%|###9      | 19636/50000 [11:49:35<15:33:48,  1.85s/it]
Outer training loop:  39%|###9      | 19637/50000 [11:49:36<15:15:01,  1.81s/it]
Outer training loop:  39%|###9      | 19638/50000 [11:49:38<14:57:30,  1.77s/it]
Outer training loop:  39%|###9      | 19639/50000 [11:49:40<15:12:22,  1.80s/it]
Outer training loop:  39%|###9      | 19640/50000 [11:49:41<14:47:06,  1.75s/it]
Outer training loop:  39%|###9      | 19641/50000 [11:49:43<14:43:36,  1.75s/it]
Outer training loop:  39%|###9      | 19642/50000 [11:49:45<15:01:35,  1.78s/it]
Outer training loop:  39%|###9      | 19643/50000 [11:49:47<14:31:13,  1.72s/it]
Outer training loop:  39%|###9      | 19644/50000 [11:49:48<14:27:14,  1.71s/it]
Outer training loop:  39%|###9      | 19645/50000 [11:49:50<14:18:04,  1.70s/it]
Outer training loop:  39%|###9      | 19646/50000 [11:49:52<14:51:11,  1.76s/it]
Outer training loop:  39%|###9      | 19647/50000 [11:49:54<14:51:56,  1.76s/it]
Outer training loop:  39%|###9      | 19648/50000 [11:49:55<14:44:22,  1.75s/it]
Outer training loop:  39%|###9      | 19649/50000 [11:49:57<14:42:38,  1.74s/it]
Outer training loop:  39%|###9      | 19650/50000 [11:49:59<15:03:30,  1.79s/it]
Outer training loop:  39%|###9      | 19651/50000 [11:50:01<15:04:01,  1.79s/it]
Outer training loop:  39%|###9      | 19652/50000 [11:50:03<14:50:20,  1.76s/it]
Outer training loop:  39%|###9      | 19653/50000 [11:50:04<15:05:20,  1.79s/it]
Outer training loop:  39%|###9      | 19654/50000 [11:50:06<14:48:10,  1.76s/it]
Outer training loop:  39%|###9      | 19655/50000 [11:50:08<14:38:05,  1.74s/it]
Outer training loop:  39%|###9      | 19656/50000 [11:50:10<14:53:55,  1.77s/it]
Outer training loop:  39%|###9      | 19657/50000 [11:50:11<14:35:12,  1.73s/it]
Outer training loop:  39%|###9      | 19658/50000 [11:50:13<14:35:42,  1.73s/it]
Outer training loop:  39%|###9      | 19659/50000 [11:50:15<14:25:38,  1.71s/it]
Outer training loop:  39%|###9      | 19660/50000 [11:50:17<14:49:27,  1.76s/it]
Outer training loop:  39%|###9      | 19661/50000 [11:50:18<14:45:27,  1.75s/it]
Outer training loop:  39%|###9      | 19662/50000 [11:50:20<14:54:57,  1.77s/it]
Outer training loop:  39%|###9      | 19663/50000 [11:50:22<15:17:56,  1.82s/it]
Outer training loop:  39%|###9      | 19664/50000 [11:50:24<15:02:26,  1.78s/it]
Outer training loop:  39%|###9      | 19665/50000 [11:50:26<15:14:10,  1.81s/it]
Outer training loop:  39%|###9      | 19666/50000 [11:50:27<15:24:24,  1.83s/it]
Outer training loop:  39%|###9      | 19667/50000 [11:50:29<14:48:46,  1.76s/it]
Outer training loop:  39%|###9      | 19668/50000 [11:50:31<14:31:27,  1.72s/it]
Outer training loop:  39%|###9      | 19669/50000 [11:50:32<14:38:15,  1.74s/it]
Outer training loop:  39%|###9      | 19670/50000 [11:50:34<15:00:13,  1.78s/it]
Outer training loop:  39%|###9      | 19671/50000 [11:50:36<14:46:25,  1.75s/it]
Outer training loop:  39%|###9      | 19672/50000 [11:50:38<14:29:21,  1.72s/it]
Outer training loop:  39%|###9      | 19673/50000 [11:50:40<15:12:18,  1.80s/it]
Outer training loop:  39%|###9      | 19674/50000 [11:50:41<14:50:25,  1.76s/it]
Outer training loop:  39%|###9      | 19675/50000 [11:50:43<14:17:11,  1.70s/it]
Outer training loop:  39%|###9      | 19676/50000 [11:50:45<14:47:03,  1.76s/it]
Outer training loop:  39%|###9      | 19677/50000 [11:50:47<14:52:25,  1.77s/it]
Outer training loop:  39%|###9      | 19678/50000 [11:50:48<14:53:39,  1.77s/it]
Outer training loop:  39%|###9      | 19679/50000 [11:50:50<15:33:13,  1.85s/it]
Outer training loop:  39%|###9      | 19680/50000 [11:50:52<15:38:20,  1.86s/it]
Outer training loop:  39%|###9      | 19681/50000 [11:50:54<15:10:06,  1.80s/it]
Outer training loop:  39%|###9      | 19682/50000 [11:50:56<15:10:22,  1.80s/it]
Outer training loop:  39%|###9      | 19683/50000 [11:50:58<15:37:45,  1.86s/it]
Outer training loop:  39%|###9      | 19684/50000 [11:50:59<15:22:37,  1.83s/it]
Outer training loop:  39%|###9      | 19685/50000 [11:51:01<15:09:09,  1.80s/it]
Outer training loop:  39%|###9      | 19686/50000 [11:51:03<15:51:37,  1.88s/it]
Outer training loop:  39%|###9      | 19687/50000 [11:51:05<15:35:27,  1.85s/it]
Outer training loop:  39%|###9      | 19688/50000 [11:51:07<15:21:48,  1.82s/it]
Outer training loop:  39%|###9      | 19689/50000 [11:51:08<14:56:09,  1.77s/it]
Outer training loop:  39%|###9      | 19690/50000 [11:51:10<15:27:16,  1.84s/it]
Outer training loop:  39%|###9      | 19691/50000 [11:51:12<14:54:38,  1.77s/it]
Outer training loop:  39%|###9      | 19692/50000 [11:51:14<14:37:07,  1.74s/it]
Outer training loop:  39%|###9      | 19693/50000 [11:51:15<14:36:16,  1.73s/it]
Outer training loop:  39%|###9      | 19694/50000 [11:51:17<15:02:05,  1.79s/it]
Outer training loop:  39%|###9      | 19695/50000 [11:51:19<14:57:48,  1.78s/it]
Outer training loop:  39%|###9      | 19696/50000 [11:51:21<15:00:34,  1.78s/it]
Outer training loop:  39%|###9      | 19697/50000 [11:51:23<15:05:11,  1.79s/it]
Outer training loop:  39%|###9      | 19698/50000 [11:51:24<14:50:42,  1.76s/it]
Outer training loop:  39%|###9      | 19699/50000 [11:51:26<14:50:44,  1.76s/it]
Outer training loop:  39%|###9      | 19700/50000 [11:51:28<15:07:36,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.63s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.65s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.69s/it]

Outer training loop:  39%|###9      | 19701/50000 [11:51:35<27:47:58,  3.30s/it]
Outer training loop:  39%|###9      | 19702/50000 [11:51:37<23:42:47,  2.82s/it]
Outer training loop:  39%|###9      | 19703/50000 [11:51:38<20:55:02,  2.49s/it]
Outer training loop:  39%|###9      | 19704/50000 [11:51:40<19:39:46,  2.34s/it]
Outer training loop:  39%|###9      | 19705/50000 [11:51:42<17:54:13,  2.13s/it]
Outer training loop:  39%|###9      | 19706/50000 [11:51:44<16:48:06,  2.00s/it]
Outer training loop:  39%|###9      | 19707/50000 [11:51:45<16:22:17,  1.95s/it]
Outer training loop:  39%|###9      | 19708/50000 [11:51:47<15:46:18,  1.87s/it]
Outer training loop:  39%|###9      | 19709/50000 [11:51:49<15:29:33,  1.84s/it]
Outer training loop:  39%|###9      | 19710/50000 [11:51:51<15:17:02,  1.82s/it]
Outer training loop:  39%|###9      | 19711/50000 [11:51:53<15:37:21,  1.86s/it]
Outer training loop:  39%|###9      | 19712/50000 [11:51:54<15:21:23,  1.83s/it]
Outer training loop:  39%|###9      | 19713/50000 [11:51:56<15:02:52,  1.79s/it]
Outer training loop:  39%|###9      | 19714/50000 [11:51:58<15:22:00,  1.83s/it]
Outer training loop:  39%|###9      | 19715/50000 [11:52:00<15:26:07,  1.83s/it]
Outer training loop:  39%|###9      | 19716/50000 [11:52:01<15:04:54,  1.79s/it]
Outer training loop:  39%|###9      | 19717/50000 [11:52:03<15:22:26,  1.83s/it]
Outer training loop:  39%|###9      | 19718/50000 [11:52:05<15:11:20,  1.81s/it]
Outer training loop:  39%|###9      | 19719/50000 [11:52:07<14:50:30,  1.76s/it]
Outer training loop:  39%|###9      | 19720/50000 [11:52:09<15:28:48,  1.84s/it]
Outer training loop:  39%|###9      | 19721/50000 [11:52:11<15:09:38,  1.80s/it]
Outer training loop:  39%|###9      | 19722/50000 [11:52:12<14:52:20,  1.77s/it]
Outer training loop:  39%|###9      | 19723/50000 [11:52:14<14:40:44,  1.75s/it]
Outer training loop:  39%|###9      | 19724/50000 [11:52:16<15:05:31,  1.79s/it]
Outer training loop:  39%|###9      | 19725/50000 [11:52:18<14:46:14,  1.76s/it]
Outer training loop:  39%|###9      | 19726/50000 [11:52:19<14:45:11,  1.75s/it]
Outer training loop:  39%|###9      | 19727/50000 [11:52:21<15:07:53,  1.80s/it]
Outer training loop:  39%|###9      | 19728/50000 [11:52:23<14:52:39,  1.77s/it]
Outer training loop:  39%|###9      | 19729/50000 [11:52:25<14:50:26,  1.76s/it]
Outer training loop:  39%|###9      | 19730/50000 [11:52:26<15:04:49,  1.79s/it]
Outer training loop:  39%|###9      | 19731/50000 [11:52:28<15:01:42,  1.79s/it]
Outer training loop:  39%|###9      | 19732/50000 [11:52:30<14:44:45,  1.75s/it]
Outer training loop:  39%|###9      | 19733/50000 [11:52:32<15:11:20,  1.81s/it]
Outer training loop:  39%|###9      | 19734/50000 [11:52:34<15:32:09,  1.85s/it]
Outer training loop:  39%|###9      | 19735/50000 [11:52:36<15:18:32,  1.82s/it]
Outer training loop:  39%|###9      | 19736/50000 [11:52:37<15:04:39,  1.79s/it]
Outer training loop:  39%|###9      | 19737/50000 [11:52:39<15:22:26,  1.83s/it]
Outer training loop:  39%|###9      | 19738/50000 [11:52:41<15:05:22,  1.80s/it]
Outer training loop:  39%|###9      | 19739/50000 [11:52:43<15:05:48,  1.80s/it]
Outer training loop:  39%|###9      | 19740/50000 [11:52:44<14:47:11,  1.76s/it]
Outer training loop:  39%|###9      | 19741/50000 [11:52:46<15:24:21,  1.83s/it]
Outer training loop:  39%|###9      | 19742/50000 [11:52:48<14:59:06,  1.78s/it]
Outer training loop:  39%|###9      | 19743/50000 [11:52:50<14:58:15,  1.78s/it]
Outer training loop:  39%|###9      | 19744/50000 [11:52:52<15:17:53,  1.82s/it]
Outer training loop:  39%|###9      | 19745/50000 [11:52:54<15:23:36,  1.83s/it]
Outer training loop:  39%|###9      | 19746/50000 [11:52:55<15:02:48,  1.79s/it]
Outer training loop:  39%|###9      | 19747/50000 [11:52:57<14:56:01,  1.78s/it]
Outer training loop:  39%|###9      | 19748/50000 [11:52:59<15:09:32,  1.80s/it]
Outer training loop:  39%|###9      | 19749/50000 [11:53:01<15:00:07,  1.79s/it]
Outer training loop:  40%|###9      | 19750/50000 [11:53:02<14:50:38,  1.77s/it]
Outer training loop:  40%|###9      | 19751/50000 [11:53:04<15:10:01,  1.81s/it]
Outer training loop:  40%|###9      | 19752/50000 [11:53:06<15:09:31,  1.80s/it]
Outer training loop:  40%|###9      | 19753/50000 [11:53:08<15:08:55,  1.80s/it]
Outer training loop:  40%|###9      | 19754/50000 [11:53:10<15:20:23,  1.83s/it]
Outer training loop:  40%|###9      | 19755/50000 [11:53:12<15:16:13,  1.82s/it]
Outer training loop:  40%|###9      | 19756/50000 [11:53:13<15:11:46,  1.81s/it]
Outer training loop:  40%|###9      | 19757/50000 [11:53:15<15:20:05,  1.83s/it]
Outer training loop:  40%|###9      | 19758/50000 [11:53:17<14:56:40,  1.78s/it]
Outer training loop:  40%|###9      | 19759/50000 [11:53:19<14:40:10,  1.75s/it]
Outer training loop:  40%|###9      | 19760/50000 [11:53:20<15:08:14,  1.80s/it]
Outer training loop:  40%|###9      | 19761/50000 [11:53:22<14:42:18,  1.75s/it]
Outer training loop:  40%|###9      | 19762/50000 [11:53:24<14:33:31,  1.73s/it]
Outer training loop:  40%|###9      | 19763/50000 [11:53:26<14:42:21,  1.75s/it]
Outer training loop:  40%|###9      | 19764/50000 [11:53:28<15:08:02,  1.80s/it]
Outer training loop:  40%|###9      | 19765/50000 [11:53:29<14:58:38,  1.78s/it]
Outer training loop:  40%|###9      | 19766/50000 [11:53:31<14:42:00,  1.75s/it]
Outer training loop:  40%|###9      | 19767/50000 [11:53:33<15:06:56,  1.80s/it]
Outer training loop:  40%|###9      | 19768/50000 [11:53:35<15:11:43,  1.81s/it]
Outer training loop:  40%|###9      | 19769/50000 [11:53:36<14:46:47,  1.76s/it]
Outer training loop:  40%|###9      | 19770/50000 [11:53:38<14:37:09,  1.74s/it]
Outer training loop:  40%|###9      | 19771/50000 [11:53:40<14:52:13,  1.77s/it]
Outer training loop:  40%|###9      | 19772/50000 [11:53:42<14:45:02,  1.76s/it]
Outer training loop:  40%|###9      | 19773/50000 [11:53:43<14:42:47,  1.75s/it]
Outer training loop:  40%|###9      | 19774/50000 [11:53:45<15:13:09,  1.81s/it]
Outer training loop:  40%|###9      | 19775/50000 [11:53:47<14:52:16,  1.77s/it]
Outer training loop:  40%|###9      | 19776/50000 [11:53:49<14:35:41,  1.74s/it]
Outer training loop:  40%|###9      | 19777/50000 [11:53:51<15:00:22,  1.79s/it]
Outer training loop:  40%|###9      | 19778/50000 [11:53:52<14:49:28,  1.77s/it]
Outer training loop:  40%|###9      | 19779/50000 [11:53:54<14:53:33,  1.77s/it]
Outer training loop:  40%|###9      | 19780/50000 [11:53:56<14:49:47,  1.77s/it]
Outer training loop:  40%|###9      | 19781/50000 [11:53:58<15:24:54,  1.84s/it]
Outer training loop:  40%|###9      | 19782/50000 [11:53:59<14:54:53,  1.78s/it]
Outer training loop:  40%|###9      | 19783/50000 [11:54:01<15:11:16,  1.81s/it]
Outer training loop:  40%|###9      | 19784/50000 [11:54:03<15:37:30,  1.86s/it]
Outer training loop:  40%|###9      | 19785/50000 [11:54:05<15:19:10,  1.83s/it]
Outer training loop:  40%|###9      | 19786/50000 [11:54:07<15:06:01,  1.80s/it]
Outer training loop:  40%|###9      | 19787/50000 [11:54:09<15:18:06,  1.82s/it]
Outer training loop:  40%|###9      | 19788/50000 [11:54:10<15:11:55,  1.81s/it]
Outer training loop:  40%|###9      | 19789/50000 [11:54:12<15:05:52,  1.80s/it]
Outer training loop:  40%|###9      | 19790/50000 [11:54:14<14:52:33,  1.77s/it]
Outer training loop:  40%|###9      | 19791/50000 [11:54:16<15:08:33,  1.80s/it]
Outer training loop:  40%|###9      | 19792/50000 [11:54:17<14:42:10,  1.75s/it]
Outer training loop:  40%|###9      | 19793/50000 [11:54:19<14:41:48,  1.75s/it]
Outer training loop:  40%|###9      | 19794/50000 [11:54:21<15:04:07,  1.80s/it]
Outer training loop:  40%|###9      | 19795/50000 [11:54:23<14:50:31,  1.77s/it]
Outer training loop:  40%|###9      | 19796/50000 [11:54:25<14:59:32,  1.79s/it]
Outer training loop:  40%|###9      | 19797/50000 [11:54:26<14:35:35,  1.74s/it]
Outer training loop:  40%|###9      | 19798/50000 [11:54:28<14:46:07,  1.76s/it]
Outer training loop:  40%|###9      | 19799/50000 [11:54:30<14:44:11,  1.76s/it]
Outer training loop:  40%|###9      | 19800/50000 [11:54:32<14:40:34,  1.75s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.72s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.66s/it]

Outer training loop:  40%|###9      | 19801/50000 [11:54:38<27:48:04,  3.31s/it]
Outer training loop:  40%|###9      | 19802/50000 [11:54:40<24:18:56,  2.90s/it]
Outer training loop:  40%|###9      | 19803/50000 [11:54:42<21:29:08,  2.56s/it]
Outer training loop:  40%|###9      | 19804/50000 [11:54:44<19:33:45,  2.33s/it]
Outer training loop:  40%|###9      | 19805/50000 [11:54:46<18:38:35,  2.22s/it]
Outer training loop:  40%|###9      | 19806/50000 [11:54:48<17:24:18,  2.08s/it]
Outer training loop:  40%|###9      | 19807/50000 [11:54:49<16:32:28,  1.97s/it]
Outer training loop:  40%|###9      | 19808/50000 [11:54:51<16:23:44,  1.95s/it]
Outer training loop:  40%|###9      | 19809/50000 [11:54:53<15:58:13,  1.90s/it]
Outer training loop:  40%|###9      | 19810/50000 [11:54:55<15:32:30,  1.85s/it]
Outer training loop:  40%|###9      | 19811/50000 [11:54:57<15:07:27,  1.80s/it]
Outer training loop:  40%|###9      | 19812/50000 [11:54:58<15:23:16,  1.84s/it]
Outer training loop:  40%|###9      | 19813/50000 [11:55:00<14:48:00,  1.76s/it]
Outer training loop:  40%|###9      | 19814/50000 [11:55:02<14:39:58,  1.75s/it]
Outer training loop:  40%|###9      | 19815/50000 [11:55:04<14:47:06,  1.76s/it]
Outer training loop:  40%|###9      | 19816/50000 [11:55:05<15:02:19,  1.79s/it]
Outer training loop:  40%|###9      | 19817/50000 [11:55:07<14:55:47,  1.78s/it]
Outer training loop:  40%|###9      | 19818/50000 [11:55:09<14:38:16,  1.75s/it]
Outer training loop:  40%|###9      | 19819/50000 [11:55:11<15:02:14,  1.79s/it]
Outer training loop:  40%|###9      | 19820/50000 [11:55:12<14:56:17,  1.78s/it]
Outer training loop:  40%|###9      | 19821/50000 [11:55:14<14:30:38,  1.73s/it]
Outer training loop:  40%|###9      | 19822/50000 [11:55:16<14:16:28,  1.70s/it]
Outer training loop:  40%|###9      | 19823/50000 [11:55:18<14:55:27,  1.78s/it]
Outer training loop:  40%|###9      | 19824/50000 [11:55:19<14:50:14,  1.77s/it]
Outer training loop:  40%|###9      | 19825/50000 [11:55:21<14:49:19,  1.77s/it]
Outer training loop:  40%|###9      | 19826/50000 [11:55:23<15:21:11,  1.83s/it]
Outer training loop:  40%|###9      | 19827/50000 [11:55:25<15:01:43,  1.79s/it]
Outer training loop:  40%|###9      | 19828/50000 [11:55:27<14:54:43,  1.78s/it]
Outer training loop:  40%|###9      | 19829/50000 [11:55:29<15:11:10,  1.81s/it]
Outer training loop:  40%|###9      | 19830/50000 [11:55:30<15:08:32,  1.81s/it]
Outer training loop:  40%|###9      | 19831/50000 [11:55:32<15:06:35,  1.80s/it]
Outer training loop:  40%|###9      | 19832/50000 [11:55:34<14:54:49,  1.78s/it]
Outer training loop:  40%|###9      | 19833/50000 [11:55:36<15:03:39,  1.80s/it]
Outer training loop:  40%|###9      | 19834/50000 [11:55:37<14:45:12,  1.76s/it]
Outer training loop:  40%|###9      | 19835/50000 [11:55:39<14:32:22,  1.74s/it]
Outer training loop:  40%|###9      | 19836/50000 [11:55:41<15:07:42,  1.81s/it]
Outer training loop:  40%|###9      | 19837/50000 [11:55:43<14:59:06,  1.79s/it]
Outer training loop:  40%|###9      | 19838/50000 [11:55:44<14:46:06,  1.76s/it]
Outer training loop:  40%|###9      | 19839/50000 [11:55:46<14:25:35,  1.72s/it]
Outer training loop:  40%|###9      | 19840/50000 [11:55:48<14:34:18,  1.74s/it]
Outer training loop:  40%|###9      | 19841/50000 [11:55:50<14:48:25,  1.77s/it]
Outer training loop:  40%|###9      | 19842/50000 [11:55:51<14:36:46,  1.74s/it]
Outer training loop:  40%|###9      | 19843/50000 [11:55:53<14:52:01,  1.77s/it]
Outer training loop:  40%|###9      | 19844/50000 [11:55:55<14:44:57,  1.76s/it]
Outer training loop:  40%|###9      | 19845/50000 [11:55:57<14:47:41,  1.77s/it]
Outer training loop:  40%|###9      | 19846/50000 [11:55:58<14:33:38,  1.74s/it]
Outer training loop:  40%|###9      | 19847/50000 [11:56:00<14:43:31,  1.76s/it]
Outer training loop:  40%|###9      | 19848/50000 [11:56:02<14:53:20,  1.78s/it]
Outer training loop:  40%|###9      | 19849/50000 [11:56:04<14:49:35,  1.77s/it]
Outer training loop:  40%|###9      | 19850/50000 [11:56:06<15:17:39,  1.83s/it]
Outer training loop:  40%|###9      | 19851/50000 [11:56:07<15:05:06,  1.80s/it]
Outer training loop:  40%|###9      | 19852/50000 [11:56:09<14:36:21,  1.74s/it]
Outer training loop:  40%|###9      | 19853/50000 [11:56:11<14:58:57,  1.79s/it]
Outer training loop:  40%|###9      | 19854/50000 [11:56:13<14:44:50,  1.76s/it]
Outer training loop:  40%|###9      | 19855/50000 [11:56:14<14:39:16,  1.75s/it]
Outer training loop:  40%|###9      | 19856/50000 [11:56:16<14:45:33,  1.76s/it]
Outer training loop:  40%|###9      | 19857/50000 [11:56:18<14:30:43,  1.73s/it]
Outer training loop:  40%|###9      | 19858/50000 [11:56:20<14:28:08,  1.73s/it]
Outer training loop:  40%|###9      | 19859/50000 [11:56:21<14:26:14,  1.72s/it]
Outer training loop:  40%|###9      | 19860/50000 [11:56:23<14:41:48,  1.76s/it]
Outer training loop:  40%|###9      | 19861/50000 [11:56:25<14:19:06,  1.71s/it]
Outer training loop:  40%|###9      | 19862/50000 [11:56:26<14:12:06,  1.70s/it]
Outer training loop:  40%|###9      | 19863/50000 [11:56:28<14:01:01,  1.67s/it]
Outer training loop:  40%|###9      | 19864/50000 [11:56:30<14:47:05,  1.77s/it]
Outer training loop:  40%|###9      | 19865/50000 [11:56:32<15:00:58,  1.79s/it]
Outer training loop:  40%|###9      | 19866/50000 [11:56:34<14:47:39,  1.77s/it]
Outer training loop:  40%|###9      | 19867/50000 [11:56:35<14:59:27,  1.79s/it]
Outer training loop:  40%|###9      | 19868/50000 [11:56:37<14:42:22,  1.76s/it]
Outer training loop:  40%|###9      | 19869/50000 [11:56:39<14:38:18,  1.75s/it]
Outer training loop:  40%|###9      | 19870/50000 [11:56:41<15:01:34,  1.80s/it]
Outer training loop:  40%|###9      | 19871/50000 [11:56:42<14:43:45,  1.76s/it]
Outer training loop:  40%|###9      | 19872/50000 [11:56:44<14:43:19,  1.76s/it]
Outer training loop:  40%|###9      | 19873/50000 [11:56:46<15:04:32,  1.80s/it]
Outer training loop:  40%|###9      | 19874/50000 [11:56:48<15:02:43,  1.80s/it]
Outer training loop:  40%|###9      | 19875/50000 [11:56:50<14:49:08,  1.77s/it]
Outer training loop:  40%|###9      | 19876/50000 [11:56:52<15:13:50,  1.82s/it]
Outer training loop:  40%|###9      | 19877/50000 [11:56:53<15:01:42,  1.80s/it]
Outer training loop:  40%|###9      | 19878/50000 [11:56:55<14:51:05,  1.77s/it]
Outer training loop:  40%|###9      | 19879/50000 [11:56:57<14:33:31,  1.74s/it]
Outer training loop:  40%|###9      | 19880/50000 [11:56:59<15:30:08,  1.85s/it]
Outer training loop:  40%|###9      | 19881/50000 [11:57:00<15:12:10,  1.82s/it]
Outer training loop:  40%|###9      | 19882/50000 [11:57:02<14:52:48,  1.78s/it]
Outer training loop:  40%|###9      | 19883/50000 [11:57:04<15:09:16,  1.81s/it]
Outer training loop:  40%|###9      | 19884/50000 [11:57:06<14:56:12,  1.79s/it]
Outer training loop:  40%|###9      | 19885/50000 [11:57:07<14:41:45,  1.76s/it]
Outer training loop:  40%|###9      | 19886/50000 [11:57:09<14:49:13,  1.77s/it]
Outer training loop:  40%|###9      | 19887/50000 [11:57:11<14:43:50,  1.76s/it]
Outer training loop:  40%|###9      | 19888/50000 [11:57:13<14:53:14,  1.78s/it]
Outer training loop:  40%|###9      | 19889/50000 [11:57:15<14:53:44,  1.78s/it]
Outer training loop:  40%|###9      | 19890/50000 [11:57:17<15:11:27,  1.82s/it]
Outer training loop:  40%|###9      | 19891/50000 [11:57:18<15:07:36,  1.81s/it]
Outer training loop:  40%|###9      | 19892/50000 [11:57:20<14:53:59,  1.78s/it]
Outer training loop:  40%|###9      | 19893/50000 [11:57:22<15:09:13,  1.81s/it]
Outer training loop:  40%|###9      | 19894/50000 [11:57:24<14:58:42,  1.79s/it]
Outer training loop:  40%|###9      | 19895/50000 [11:57:25<14:57:07,  1.79s/it]
Outer training loop:  40%|###9      | 19896/50000 [11:57:27<14:48:17,  1.77s/it]
Outer training loop:  40%|###9      | 19897/50000 [11:57:29<15:08:03,  1.81s/it]
Outer training loop:  40%|###9      | 19898/50000 [11:57:31<14:58:13,  1.79s/it]
Outer training loop:  40%|###9      | 19899/50000 [11:57:33<14:47:41,  1.77s/it]
Outer training loop:  40%|###9      | 19900/50000 [11:57:35<15:24:11,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.67s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.71s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]

Outer training loop:  40%|###9      | 19901/50000 [11:57:42<28:16:42,  3.38s/it]
Outer training loop:  40%|###9      | 19902/50000 [11:57:43<23:59:53,  2.87s/it]
Outer training loop:  40%|###9      | 19903/50000 [11:57:45<21:14:43,  2.54s/it]
Outer training loop:  40%|###9      | 19904/50000 [11:57:47<19:43:32,  2.36s/it]
Outer training loop:  40%|###9      | 19905/50000 [11:57:49<18:07:33,  2.17s/it]
Outer training loop:  40%|###9      | 19906/50000 [11:57:50<16:58:40,  2.03s/it]
Outer training loop:  40%|###9      | 19907/50000 [11:57:52<16:36:26,  1.99s/it]
Outer training loop:  40%|###9      | 19908/50000 [11:57:54<15:57:52,  1.91s/it]
Outer training loop:  40%|###9      | 19909/50000 [11:57:56<15:41:33,  1.88s/it]
Outer training loop:  40%|###9      | 19910/50000 [11:57:58<15:23:34,  1.84s/it]
Outer training loop:  40%|###9      | 19911/50000 [11:57:59<15:39:25,  1.87s/it]
Outer training loop:  40%|###9      | 19912/50000 [11:58:01<15:00:39,  1.80s/it]
Outer training loop:  40%|###9      | 19913/50000 [11:58:03<14:43:50,  1.76s/it]
Outer training loop:  40%|###9      | 19914/50000 [11:58:05<14:41:26,  1.76s/it]
Outer training loop:  40%|###9      | 19915/50000 [11:58:06<14:54:13,  1.78s/it]
Outer training loop:  40%|###9      | 19916/50000 [11:58:08<14:50:03,  1.78s/it]
Outer training loop:  40%|###9      | 19917/50000 [11:58:10<15:00:22,  1.80s/it]
Outer training loop:  40%|###9      | 19918/50000 [11:58:12<15:15:37,  1.83s/it]
Outer training loop:  40%|###9      | 19919/50000 [11:58:14<14:55:28,  1.79s/it]
Outer training loop:  40%|###9      | 19920/50000 [11:58:15<14:55:58,  1.79s/it]
Outer training loop:  40%|###9      | 19921/50000 [11:58:17<15:20:08,  1.84s/it]
Outer training loop:  40%|###9      | 19922/50000 [11:58:19<14:54:48,  1.78s/it]
Outer training loop:  40%|###9      | 19923/50000 [11:58:21<14:36:22,  1.75s/it]
Outer training loop:  40%|###9      | 19924/50000 [11:58:22<14:49:35,  1.77s/it]
Outer training loop:  40%|###9      | 19925/50000 [11:58:24<14:35:30,  1.75s/it]
Outer training loop:  40%|###9      | 19926/50000 [11:58:26<14:36:24,  1.75s/it]
Outer training loop:  40%|###9      | 19927/50000 [11:58:28<14:50:22,  1.78s/it]
Outer training loop:  40%|###9      | 19928/50000 [11:58:30<14:57:24,  1.79s/it]
Outer training loop:  40%|###9      | 19929/50000 [11:58:31<14:58:22,  1.79s/it]
Outer training loop:  40%|###9      | 19930/50000 [11:58:33<14:48:35,  1.77s/it]
Outer training loop:  40%|###9      | 19931/50000 [11:58:35<15:09:14,  1.81s/it]
Outer training loop:  40%|###9      | 19932/50000 [11:58:37<14:58:00,  1.79s/it]
Outer training loop:  40%|###9      | 19933/50000 [11:58:38<14:43:31,  1.76s/it]
Outer training loop:  40%|###9      | 19934/50000 [11:58:40<15:10:08,  1.82s/it]
Outer training loop:  40%|###9      | 19935/50000 [11:58:42<15:10:45,  1.82s/it]
Outer training loop:  40%|###9      | 19936/50000 [11:58:44<14:43:14,  1.76s/it]
Outer training loop:  40%|###9      | 19937/50000 [11:58:46<15:03:39,  1.80s/it]
Outer training loop:  40%|###9      | 19938/50000 [11:58:47<14:56:28,  1.79s/it]
Outer training loop:  40%|###9      | 19939/50000 [11:58:49<14:53:23,  1.78s/it]
Outer training loop:  40%|###9      | 19940/50000 [11:58:51<14:59:03,  1.79s/it]
Outer training loop:  40%|###9      | 19941/50000 [11:58:53<15:37:53,  1.87s/it]
Outer training loop:  40%|###9      | 19942/50000 [11:58:55<15:30:05,  1.86s/it]
Outer training loop:  40%|###9      | 19943/50000 [11:58:57<15:17:30,  1.83s/it]
Outer training loop:  40%|###9      | 19944/50000 [11:58:59<15:28:17,  1.85s/it]
Outer training loop:  40%|###9      | 19945/50000 [11:59:00<14:55:32,  1.79s/it]
Outer training loop:  40%|###9      | 19946/50000 [11:59:02<14:45:05,  1.77s/it]
Outer training loop:  40%|###9      | 19947/50000 [11:59:04<14:33:18,  1.74s/it]
Outer training loop:  40%|###9      | 19948/50000 [11:59:06<14:56:42,  1.79s/it]
Outer training loop:  40%|###9      | 19949/50000 [11:59:07<14:46:41,  1.77s/it]
Outer training loop:  40%|###9      | 19950/50000 [11:59:09<14:42:08,  1.76s/it]
Outer training loop:  40%|###9      | 19951/50000 [11:59:11<14:33:35,  1.74s/it]
Outer training loop:  40%|###9      | 19952/50000 [11:59:13<14:52:59,  1.78s/it]
Outer training loop:  40%|###9      | 19953/50000 [11:59:14<14:46:57,  1.77s/it]
Outer training loop:  40%|###9      | 19954/50000 [11:59:16<14:28:20,  1.73s/it]
Outer training loop:  40%|###9      | 19955/50000 [11:59:18<14:50:50,  1.78s/it]
Outer training loop:  40%|###9      | 19956/50000 [11:59:20<14:32:52,  1.74s/it]
Outer training loop:  40%|###9      | 19957/50000 [11:59:21<14:32:58,  1.74s/it]
Outer training loop:  40%|###9      | 19958/50000 [11:59:23<15:00:04,  1.80s/it]
Outer training loop:  40%|###9      | 19959/50000 [11:59:25<14:49:44,  1.78s/it]
Outer training loop:  40%|###9      | 19960/50000 [11:59:27<14:34:40,  1.75s/it]
Outer training loop:  40%|###9      | 19961/50000 [11:59:29<14:59:02,  1.80s/it]
Outer training loop:  40%|###9      | 19962/50000 [11:59:30<14:55:50,  1.79s/it]
Outer training loop:  40%|###9      | 19963/50000 [11:59:32<14:47:45,  1.77s/it]
Outer training loop:  40%|###9      | 19964/50000 [11:59:34<15:15:14,  1.83s/it]
Outer training loop:  40%|###9      | 19965/50000 [11:59:36<14:41:21,  1.76s/it]
Outer training loop:  40%|###9      | 19966/50000 [11:59:37<14:39:59,  1.76s/it]
Outer training loop:  40%|###9      | 19967/50000 [11:59:39<14:25:58,  1.73s/it]
Outer training loop:  40%|###9      | 19968/50000 [11:59:41<14:56:28,  1.79s/it]
Outer training loop:  40%|###9      | 19969/50000 [11:59:43<15:04:50,  1.81s/it]
Outer training loop:  40%|###9      | 19970/50000 [11:59:44<14:51:34,  1.78s/it]
Outer training loop:  40%|###9      | 19971/50000 [11:59:46<15:11:25,  1.82s/it]
Outer training loop:  40%|###9      | 19972/50000 [11:59:48<15:01:22,  1.80s/it]
Outer training loop:  40%|###9      | 19973/50000 [11:59:50<14:53:38,  1.79s/it]
Outer training loop:  40%|###9      | 19974/50000 [11:59:52<15:16:53,  1.83s/it]
Outer training loop:  40%|###9      | 19975/50000 [11:59:54<14:51:29,  1.78s/it]
Outer training loop:  40%|###9      | 19976/50000 [11:59:55<15:12:38,  1.82s/it]
Outer training loop:  40%|###9      | 19977/50000 [11:59:57<15:15:48,  1.83s/it]
Outer training loop:  40%|###9      | 19978/50000 [11:59:59<15:25:16,  1.85s/it]
Outer training loop:  40%|###9      | 19979/50000 [12:00:01<15:05:52,  1.81s/it]
Outer training loop:  40%|###9      | 19980/50000 [12:00:03<14:57:21,  1.79s/it]
Outer training loop:  40%|###9      | 19981/50000 [12:00:05<15:30:26,  1.86s/it]
Outer training loop:  40%|###9      | 19982/50000 [12:00:06<15:13:45,  1.83s/it]
Outer training loop:  40%|###9      | 19983/50000 [12:00:08<15:29:33,  1.86s/it]
Outer training loop:  40%|###9      | 19984/50000 [12:00:10<15:50:25,  1.90s/it]
Outer training loop:  40%|###9      | 19985/50000 [12:00:12<15:32:32,  1.86s/it]
Outer training loop:  40%|###9      | 19986/50000 [12:00:14<15:11:11,  1.82s/it]
Outer training loop:  40%|###9      | 19987/50000 [12:00:16<14:50:34,  1.78s/it]
Outer training loop:  40%|###9      | 19988/50000 [12:00:17<15:01:53,  1.80s/it]
Outer training loop:  40%|###9      | 19989/50000 [12:00:19<14:47:10,  1.77s/it]
Outer training loop:  40%|###9      | 19990/50000 [12:00:21<14:47:32,  1.77s/it]
Outer training loop:  40%|###9      | 19991/50000 [12:00:23<15:09:38,  1.82s/it]
Outer training loop:  40%|###9      | 19992/50000 [12:00:25<14:59:22,  1.80s/it]
Outer training loop:  40%|###9      | 19993/50000 [12:00:26<14:55:19,  1.79s/it]
Outer training loop:  40%|###9      | 19994/50000 [12:00:28<15:32:58,  1.87s/it]
Outer training loop:  40%|###9      | 19995/50000 [12:00:30<15:18:25,  1.84s/it]
Outer training loop:  40%|###9      | 19996/50000 [12:00:32<15:01:54,  1.80s/it]
Outer training loop:  40%|###9      | 19997/50000 [12:00:34<15:29:18,  1.86s/it]
Outer training loop:  40%|###9      | 19998/50000 [12:00:36<15:05:50,  1.81s/it]
Outer training loop:  40%|###9      | 19999/50000 [12:00:37<14:44:45,  1.77s/it]
Outer training loop:  40%|####      | 20000/50000 [12:00:39<14:39:28,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.68s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.70s/it]

Evaluating policy: 100%|##########| 3/3 [00:04<00:00,  1.65s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:32,  1.87s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.82s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.79s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.81s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.79s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:14,  1.75s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:12,  1.74s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:14,  1.80s/it]

Evaluating policy:  11%|#         | 9/83 [00:15<02:11,  1.78s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:07,  1.75s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:05,  1.75s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.79s/it]

Evaluating policy:  16%|#5        | 13/83 [00:22<02:03,  1.77s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:01,  1.76s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<01:59,  1.75s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:59,  1.79s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:57,  1.78s/it]

Evaluating policy:  22%|##1       | 18/83 [00:31<01:54,  1.76s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:52,  1.76s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:51,  1.77s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:53,  1.83s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:49,  1.79s/it]

Evaluating policy:  28%|##7       | 23/83 [00:40<01:49,  1.83s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:46,  1.80s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.78s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:44,  1.84s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:38,  1.80s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:34,  1.75s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.76s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:37,  1.87s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:29,  1.80s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:25,  1.75s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:23,  1.73s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:22,  1.76s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:20,  1.79s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:18,  1.79s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.74s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:12,  1.72s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:10,  1.73s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:08,  1.71s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:10,  1.80s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:07,  1.77s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:04,  1.73s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:01,  1.71s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:24<00:59,  1.70s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<00:57,  1.68s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:55,  1.68s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:55,  1.79s/it]

Evaluating policy:  64%|######3   | 53/83 [01:33<00:52,  1.76s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:50,  1.74s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:48,  1.73s/it]

Evaluating policy:  67%|######7   | 56/83 [01:38<00:46,  1.73s/it]

Evaluating policy:  69%|######8   | 57/83 [01:40<00:45,  1.73s/it]

Evaluating policy:  70%|######9   | 58/83 [01:42<00:43,  1.74s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:44,  1.83s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:41,  1.82s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:39,  1.80s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:49<00:37,  1.77s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:34,  1.75s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:32,  1.72s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:54<00:30,  1.70s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:56<00:28,  1.69s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:28,  1.81s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:27,  1.80s/it]

Evaluating policy:  83%|########3 | 69/83 [02:01<00:24,  1.75s/it]

Evaluating policy:  84%|########4 | 70/83 [02:03<00:22,  1.76s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:20,  1.73s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:18,  1.71s/it]

Evaluating policy:  88%|########7 | 73/83 [02:08<00:17,  1.75s/it]

Evaluating policy:  89%|########9 | 74/83 [02:10<00:15,  1.76s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:14,  1.77s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:11,  1.85s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:09,  1.83s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.86s/it]

Evaluating policy:  96%|#########6| 80/83 [02:21<00:05,  1.83s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.78s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 83/83 [02:26<00:00,  1.72s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:12,  1.62s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:26,  1.81s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.79s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.79s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:15,  1.73s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:16,  1.78s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:15,  1.78s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:11,  1.76s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:07,  1.72s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:08,  1.76s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:06,  1.75s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:03,  1.74s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:01,  1.74s/it]

Evaluating policy:  17%|#6        | 14/83 [00:24<02:03,  1.80s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:00,  1.78s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:59,  1.78s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:55,  1.76s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:57,  1.81s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:53,  1.77s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:51,  1.76s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:48,  1.75s/it]

Evaluating policy:  27%|##6       | 22/83 [00:38<01:45,  1.73s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:49,  1.82s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:46,  1.80s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:43,  1.78s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:40,  1.77s/it]

Evaluating policy:  33%|###2      | 27/83 [00:47<01:37,  1.73s/it]

Evaluating policy:  34%|###3      | 28/83 [00:49<01:39,  1.81s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:35,  1.77s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.76s/it]

Evaluating policy:  37%|###7      | 31/83 [00:54<01:30,  1.75s/it]

Evaluating policy:  39%|###8      | 32/83 [00:56<01:29,  1.75s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:30,  1.81s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:26,  1.77s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:24,  1.75s/it]

Evaluating policy:  43%|####3     | 36/83 [01:03<01:21,  1.74s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:20,  1.74s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:17,  1.72s/it]

Evaluating policy:  47%|####6     | 39/83 [01:08<01:14,  1.70s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:20,  1.88s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:14,  1.81s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:12,  1.80s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:09,  1.78s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:19<01:07,  1.78s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:10,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:02,  1.80s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.78s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:57,  1.75s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:56,  1.76s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:54,  1.74s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:51,  1.78s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:49,  1.78s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:47,  1.76s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:45,  1.74s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:43,  1.76s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:41,  1.74s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:39,  1.72s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:40,  1.84s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:37,  1.79s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.77s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:33,  1.75s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:31,  1.74s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.72s/it]

Evaluating policy:  81%|########  | 67/83 [01:58<00:27,  1.72s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:25,  1.70s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:26,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:23,  1.82s/it]

Evaluating policy:  86%|########5 | 71/83 [02:06<00:21,  1.79s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:19,  1.73s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.72s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:15,  1.71s/it]

Evaluating policy:  90%|######### | 75/83 [02:12<00:13,  1.74s/it]

Evaluating policy:  92%|#########1| 76/83 [02:14<00:12,  1.74s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.71s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.70s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.90s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.79s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.78s/it]

Outer training loop:  40%|####      | 20001/50000 [12:05:40<763:17:24, 91.60s/it]
Outer training loop:  40%|####      | 20002/50000 [12:05:42<538:25:14, 64.61s/it]
Outer training loop:  40%|####      | 20003/50000 [12:05:44<381:16:22, 45.76s/it]
Outer training loop:  40%|####      | 20004/50000 [12:05:45<271:27:10, 32.58s/it]
Outer training loop:  40%|####      | 20005/50000 [12:05:47<194:05:53, 23.30s/it]
Outer training loop:  40%|####      | 20006/50000 [12:05:49<140:07:58, 16.82s/it]
Outer training loop:  40%|####      | 20007/50000 [12:05:51<104:10:15, 12.50s/it]
Outer training loop:  40%|####      | 20008/50000 [12:05:53<77:43:18,  9.33s/it] 
Outer training loop:  40%|####      | 20009/50000 [12:05:55<58:49:24,  7.06s/it]
Outer training loop:  40%|####      | 20010/50000 [12:05:57<46:01:25,  5.52s/it]
Outer training loop:  40%|####      | 20011/50000 [12:05:58<36:21:50,  4.37s/it]
Outer training loop:  40%|####      | 20012/50000 [12:06:00<29:49:27,  3.58s/it]
Outer training loop:  40%|####      | 20013/50000 [12:06:02<25:37:05,  3.08s/it]
Outer training loop:  40%|####      | 20014/50000 [12:06:04<22:21:30,  2.68s/it]
Outer training loop:  40%|####      | 20015/50000 [12:06:06<20:05:41,  2.41s/it]
Outer training loop:  40%|####      | 20016/50000 [12:06:08<18:52:21,  2.27s/it]
Outer training loop:  40%|####      | 20017/50000 [12:06:09<17:34:11,  2.11s/it]
Outer training loop:  40%|####      | 20018/50000 [12:06:11<16:37:40,  2.00s/it]
Outer training loop:  40%|####      | 20019/50000 [12:06:13<15:48:24,  1.90s/it]
Outer training loop:  40%|####      | 20020/50000 [12:06:15<15:45:33,  1.89s/it]
Outer training loop:  40%|####      | 20021/50000 [12:06:16<15:28:54,  1.86s/it]
Outer training loop:  40%|####      | 20022/50000 [12:06:18<15:07:53,  1.82s/it]
Outer training loop:  40%|####      | 20023/50000 [12:06:20<15:18:24,  1.84s/it]
Outer training loop:  40%|####      | 20024/50000 [12:06:22<15:11:05,  1.82s/it]
Outer training loop:  40%|####      | 20025/50000 [12:06:23<14:41:55,  1.77s/it]
Outer training loop:  40%|####      | 20026/50000 [12:06:25<14:42:26,  1.77s/it]
Outer training loop:  40%|####      | 20027/50000 [12:06:27<15:08:54,  1.82s/it]
Outer training loop:  40%|####      | 20028/50000 [12:06:29<15:03:28,  1.81s/it]
Outer training loop:  40%|####      | 20029/50000 [12:06:31<14:55:38,  1.79s/it]
Outer training loop:  40%|####      | 20030/50000 [12:06:33<15:06:48,  1.82s/it]
Outer training loop:  40%|####      | 20031/50000 [12:06:34<15:18:50,  1.84s/it]
Outer training loop:  40%|####      | 20032/50000 [12:06:36<15:08:04,  1.82s/it]
Outer training loop:  40%|####      | 20033/50000 [12:06:38<15:23:27,  1.85s/it]
Outer training loop:  40%|####      | 20034/50000 [12:06:40<15:05:15,  1.81s/it]
Outer training loop:  40%|####      | 20035/50000 [12:06:41<14:35:54,  1.75s/it]
Outer training loop:  40%|####      | 20036/50000 [12:06:43<14:47:41,  1.78s/it]
Outer training loop:  40%|####      | 20037/50000 [12:06:45<14:36:11,  1.75s/it]
Outer training loop:  40%|####      | 20038/50000 [12:06:47<14:37:43,  1.76s/it]
Outer training loop:  40%|####      | 20039/50000 [12:06:48<14:32:57,  1.75s/it]
Outer training loop:  40%|####      | 20040/50000 [12:06:50<15:05:40,  1.81s/it]
Outer training loop:  40%|####      | 20041/50000 [12:06:52<14:42:20,  1.77s/it]
Outer training loop:  40%|####      | 20042/50000 [12:06:54<14:42:05,  1.77s/it]
Outer training loop:  40%|####      | 20043/50000 [12:06:56<15:06:07,  1.81s/it]
Outer training loop:  40%|####      | 20044/50000 [12:06:57<14:33:06,  1.75s/it]
Outer training loop:  40%|####      | 20045/50000 [12:06:59<14:39:18,  1.76s/it]
Outer training loop:  40%|####      | 20046/50000 [12:07:01<14:30:38,  1.74s/it]
Outer training loop:  40%|####      | 20047/50000 [12:07:03<14:48:29,  1.78s/it]
Outer training loop:  40%|####      | 20048/50000 [12:07:04<14:33:13,  1.75s/it]
Outer training loop:  40%|####      | 20049/50000 [12:07:06<14:31:43,  1.75s/it]
Outer training loop:  40%|####      | 20050/50000 [12:07:08<14:54:18,  1.79s/it]
Outer training loop:  40%|####      | 20051/50000 [12:07:10<14:55:43,  1.79s/it]
Outer training loop:  40%|####      | 20052/50000 [12:07:12<14:36:12,  1.76s/it]
Outer training loop:  40%|####      | 20053/50000 [12:07:13<14:35:20,  1.75s/it]
Outer training loop:  40%|####      | 20054/50000 [12:07:15<15:13:26,  1.83s/it]
Outer training loop:  40%|####      | 20055/50000 [12:07:17<15:02:35,  1.81s/it]
Outer training loop:  40%|####      | 20056/50000 [12:07:19<14:52:46,  1.79s/it]
Outer training loop:  40%|####      | 20057/50000 [12:07:21<15:14:57,  1.83s/it]
Outer training loop:  40%|####      | 20058/50000 [12:07:23<15:35:28,  1.87s/it]
Outer training loop:  40%|####      | 20059/50000 [12:07:24<15:13:00,  1.83s/it]
Outer training loop:  40%|####      | 20060/50000 [12:07:26<15:19:21,  1.84s/it]
Outer training loop:  40%|####      | 20061/50000 [12:07:28<14:49:23,  1.78s/it]
Outer training loop:  40%|####      | 20062/50000 [12:07:30<14:52:19,  1.79s/it]
Outer training loop:  40%|####      | 20063/50000 [12:07:31<14:44:42,  1.77s/it]
Outer training loop:  40%|####      | 20064/50000 [12:07:33<15:02:27,  1.81s/it]
Outer training loop:  40%|####      | 20065/50000 [12:07:35<14:44:26,  1.77s/it]
Outer training loop:  40%|####      | 20066/50000 [12:07:37<14:31:49,  1.75s/it]
Outer training loop:  40%|####      | 20067/50000 [12:07:39<15:09:17,  1.82s/it]
Outer training loop:  40%|####      | 20068/50000 [12:07:41<15:04:37,  1.81s/it]
Outer training loop:  40%|####      | 20069/50000 [12:07:42<14:52:16,  1.79s/it]
Outer training loop:  40%|####      | 20070/50000 [12:07:44<15:01:40,  1.81s/it]
Outer training loop:  40%|####      | 20071/50000 [12:07:46<14:46:27,  1.78s/it]
Outer training loop:  40%|####      | 20072/50000 [12:07:48<14:37:09,  1.76s/it]
Outer training loop:  40%|####      | 20073/50000 [12:07:49<14:26:51,  1.74s/it]
Outer training loop:  40%|####      | 20074/50000 [12:07:51<15:04:19,  1.81s/it]
Outer training loop:  40%|####      | 20075/50000 [12:07:53<14:52:19,  1.79s/it]
Outer training loop:  40%|####      | 20076/50000 [12:07:55<15:13:35,  1.83s/it]
Outer training loop:  40%|####      | 20077/50000 [12:07:57<14:52:43,  1.79s/it]
Outer training loop:  40%|####      | 20078/50000 [12:07:58<14:39:54,  1.76s/it]
Outer training loop:  40%|####      | 20079/50000 [12:08:00<14:28:41,  1.74s/it]
Outer training loop:  40%|####      | 20080/50000 [12:08:02<15:04:07,  1.81s/it]
Outer training loop:  40%|####      | 20081/50000 [12:08:04<15:11:19,  1.83s/it]
Outer training loop:  40%|####      | 20082/50000 [12:08:05<14:35:14,  1.76s/it]
Outer training loop:  40%|####      | 20083/50000 [12:08:07<14:57:03,  1.80s/it]
Outer training loop:  40%|####      | 20084/50000 [12:08:09<14:43:17,  1.77s/it]
Outer training loop:  40%|####      | 20085/50000 [12:08:11<14:39:23,  1.76s/it]
Outer training loop:  40%|####      | 20086/50000 [12:08:13<15:16:56,  1.84s/it]
Outer training loop:  40%|####      | 20087/50000 [12:08:14<14:49:31,  1.78s/it]
Outer training loop:  40%|####      | 20088/50000 [12:08:16<14:45:09,  1.78s/it]
Outer training loop:  40%|####      | 20089/50000 [12:08:18<14:33:47,  1.75s/it]
Outer training loop:  40%|####      | 20090/50000 [12:08:20<15:17:51,  1.84s/it]
Outer training loop:  40%|####      | 20091/50000 [12:08:22<15:01:11,  1.81s/it]
Outer training loop:  40%|####      | 20092/50000 [12:08:23<14:33:22,  1.75s/it]
Outer training loop:  40%|####      | 20093/50000 [12:08:25<14:39:13,  1.76s/it]
Outer training loop:  40%|####      | 20094/50000 [12:08:27<14:44:15,  1.77s/it]
Outer training loop:  40%|####      | 20095/50000 [12:08:29<14:39:30,  1.76s/it]
Outer training loop:  40%|####      | 20096/50000 [12:08:30<14:51:54,  1.79s/it]
Outer training loop:  40%|####      | 20097/50000 [12:08:32<15:01:05,  1.81s/it]
Outer training loop:  40%|####      | 20098/50000 [12:08:34<14:51:54,  1.79s/it]
Outer training loop:  40%|####      | 20099/50000 [12:08:36<14:47:45,  1.78s/it]
Outer training loop:  40%|####      | 20100/50000 [12:08:38<14:44:03,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.66s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.68s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.69s/it]

Outer training loop:  40%|####      | 20101/50000 [12:08:45<27:40:09,  3.33s/it]
Outer training loop:  40%|####      | 20102/50000 [12:08:46<23:39:17,  2.85s/it]
Outer training loop:  40%|####      | 20103/50000 [12:08:48<20:52:09,  2.51s/it]
Outer training loop:  40%|####      | 20104/50000 [12:08:50<19:14:40,  2.32s/it]
Outer training loop:  40%|####      | 20105/50000 [12:08:52<18:36:38,  2.24s/it]
Outer training loop:  40%|####      | 20106/50000 [12:08:54<17:16:00,  2.08s/it]
Outer training loop:  40%|####      | 20107/50000 [12:08:55<16:27:29,  1.98s/it]
Outer training loop:  40%|####      | 20108/50000 [12:08:57<16:19:56,  1.97s/it]
Outer training loop:  40%|####      | 20109/50000 [12:08:59<15:32:56,  1.87s/it]
Outer training loop:  40%|####      | 20110/50000 [12:09:01<15:03:02,  1.81s/it]
Outer training loop:  40%|####      | 20111/50000 [12:09:03<15:31:58,  1.87s/it]
Outer training loop:  40%|####      | 20112/50000 [12:09:04<15:18:25,  1.84s/it]
Outer training loop:  40%|####      | 20113/50000 [12:09:06<14:52:25,  1.79s/it]
Outer training loop:  40%|####      | 20114/50000 [12:09:08<14:43:05,  1.77s/it]
Outer training loop:  40%|####      | 20115/50000 [12:09:10<15:21:09,  1.85s/it]
Outer training loop:  40%|####      | 20116/50000 [12:09:12<15:08:50,  1.82s/it]
Outer training loop:  40%|####      | 20117/50000 [12:09:13<14:54:03,  1.80s/it]
Outer training loop:  40%|####      | 20118/50000 [12:09:15<15:10:16,  1.83s/it]
Outer training loop:  40%|####      | 20119/50000 [12:09:17<14:58:24,  1.80s/it]
Outer training loop:  40%|####      | 20120/50000 [12:09:19<14:35:22,  1.76s/it]
Outer training loop:  40%|####      | 20121/50000 [12:09:21<15:07:35,  1.82s/it]
Outer training loop:  40%|####      | 20122/50000 [12:09:22<14:47:20,  1.78s/it]
Outer training loop:  40%|####      | 20123/50000 [12:09:24<14:33:29,  1.75s/it]
Outer training loop:  40%|####      | 20124/50000 [12:09:26<14:39:32,  1.77s/it]
Outer training loop:  40%|####      | 20125/50000 [12:09:28<15:08:16,  1.82s/it]
Outer training loop:  40%|####      | 20126/50000 [12:09:30<15:11:00,  1.83s/it]
Outer training loop:  40%|####      | 20127/50000 [12:09:31<14:37:06,  1.76s/it]
Outer training loop:  40%|####      | 20128/50000 [12:09:33<14:51:21,  1.79s/it]
Outer training loop:  40%|####      | 20129/50000 [12:09:35<14:50:37,  1.79s/it]
Outer training loop:  40%|####      | 20130/50000 [12:09:36<14:30:03,  1.75s/it]
Outer training loop:  40%|####      | 20131/50000 [12:09:38<14:26:18,  1.74s/it]
Outer training loop:  40%|####      | 20132/50000 [12:09:40<14:55:41,  1.80s/it]
Outer training loop:  40%|####      | 20133/50000 [12:09:42<14:59:11,  1.81s/it]
Outer training loop:  40%|####      | 20134/50000 [12:09:44<14:40:37,  1.77s/it]
Outer training loop:  40%|####      | 20135/50000 [12:09:45<14:21:47,  1.73s/it]
Outer training loop:  40%|####      | 20136/50000 [12:09:47<14:52:15,  1.79s/it]
Outer training loop:  40%|####      | 20137/50000 [12:09:49<14:36:42,  1.76s/it]
Outer training loop:  40%|####      | 20138/50000 [12:09:51<14:41:27,  1.77s/it]
Outer training loop:  40%|####      | 20139/50000 [12:09:53<14:56:56,  1.80s/it]
Outer training loop:  40%|####      | 20140/50000 [12:09:54<14:52:36,  1.79s/it]
Outer training loop:  40%|####      | 20141/50000 [12:09:56<14:29:54,  1.75s/it]
Outer training loop:  40%|####      | 20142/50000 [12:09:58<14:42:16,  1.77s/it]
Outer training loop:  40%|####      | 20143/50000 [12:09:59<14:23:55,  1.74s/it]
Outer training loop:  40%|####      | 20144/50000 [12:10:01<14:13:12,  1.71s/it]
Outer training loop:  40%|####      | 20145/50000 [12:10:03<14:19:47,  1.73s/it]
Outer training loop:  40%|####      | 20146/50000 [12:10:05<14:54:50,  1.80s/it]
Outer training loop:  40%|####      | 20147/50000 [12:10:07<14:40:01,  1.77s/it]
Outer training loop:  40%|####      | 20148/50000 [12:10:08<14:31:57,  1.75s/it]
Outer training loop:  40%|####      | 20149/50000 [12:10:10<14:54:17,  1.80s/it]
Outer training loop:  40%|####      | 20150/50000 [12:10:12<14:38:06,  1.77s/it]
Outer training loop:  40%|####      | 20151/50000 [12:10:14<14:33:08,  1.76s/it]
Outer training loop:  40%|####      | 20152/50000 [12:10:15<14:38:28,  1.77s/it]
Outer training loop:  40%|####      | 20153/50000 [12:10:17<14:41:39,  1.77s/it]
Outer training loop:  40%|####      | 20154/50000 [12:10:19<14:37:46,  1.76s/it]
Outer training loop:  40%|####      | 20155/50000 [12:10:21<15:28:17,  1.87s/it]
Outer training loop:  40%|####      | 20156/50000 [12:10:23<14:52:40,  1.79s/it]
Outer training loop:  40%|####      | 20157/50000 [12:10:24<14:35:48,  1.76s/it]
Outer training loop:  40%|####      | 20158/50000 [12:10:26<14:29:06,  1.75s/it]
Outer training loop:  40%|####      | 20159/50000 [12:10:28<15:02:54,  1.82s/it]
Outer training loop:  40%|####      | 20160/50000 [12:10:30<15:17:19,  1.84s/it]
Outer training loop:  40%|####      | 20161/50000 [12:10:32<15:01:00,  1.81s/it]
Outer training loop:  40%|####      | 20162/50000 [12:10:34<15:16:44,  1.84s/it]
Outer training loop:  40%|####      | 20163/50000 [12:10:35<14:59:04,  1.81s/it]
Outer training loop:  40%|####      | 20164/50000 [12:10:37<15:03:28,  1.82s/it]
Outer training loop:  40%|####      | 20165/50000 [12:10:39<15:20:38,  1.85s/it]
Outer training loop:  40%|####      | 20166/50000 [12:10:41<15:05:17,  1.82s/it]
Outer training loop:  40%|####      | 20167/50000 [12:10:43<14:49:21,  1.79s/it]
Outer training loop:  40%|####      | 20168/50000 [12:10:44<14:58:05,  1.81s/it]
Outer training loop:  40%|####      | 20169/50000 [12:10:46<15:29:02,  1.87s/it]
Outer training loop:  40%|####      | 20170/50000 [12:10:48<15:03:04,  1.82s/it]
Outer training loop:  40%|####      | 20171/50000 [12:10:50<15:13:07,  1.84s/it]
Outer training loop:  40%|####      | 20172/50000 [12:10:52<15:03:15,  1.82s/it]
Outer training loop:  40%|####      | 20173/50000 [12:10:54<15:00:46,  1.81s/it]
Outer training loop:  40%|####      | 20174/50000 [12:10:55<14:44:54,  1.78s/it]
Outer training loop:  40%|####      | 20175/50000 [12:10:57<14:57:16,  1.81s/it]
Outer training loop:  40%|####      | 20176/50000 [12:10:59<14:32:20,  1.75s/it]
Outer training loop:  40%|####      | 20177/50000 [12:11:01<14:53:52,  1.80s/it]
Outer training loop:  40%|####      | 20178/50000 [12:11:02<14:45:29,  1.78s/it]
Outer training loop:  40%|####      | 20179/50000 [12:11:04<15:01:49,  1.81s/it]
Outer training loop:  40%|####      | 20180/50000 [12:11:06<14:52:26,  1.80s/it]
Outer training loop:  40%|####      | 20181/50000 [12:11:08<14:32:10,  1.75s/it]
Outer training loop:  40%|####      | 20182/50000 [12:11:10<14:42:16,  1.78s/it]
Outer training loop:  40%|####      | 20183/50000 [12:11:11<14:22:36,  1.74s/it]
Outer training loop:  40%|####      | 20184/50000 [12:11:13<14:08:15,  1.71s/it]
Outer training loop:  40%|####      | 20185/50000 [12:11:15<14:10:44,  1.71s/it]
Outer training loop:  40%|####      | 20186/50000 [12:11:16<14:23:54,  1.74s/it]
Outer training loop:  40%|####      | 20187/50000 [12:11:18<14:10:45,  1.71s/it]
Outer training loop:  40%|####      | 20188/50000 [12:11:20<14:19:40,  1.73s/it]
Outer training loop:  40%|####      | 20189/50000 [12:11:22<14:43:30,  1.78s/it]
Outer training loop:  40%|####      | 20190/50000 [12:11:23<14:37:27,  1.77s/it]
Outer training loop:  40%|####      | 20191/50000 [12:11:25<14:52:54,  1.80s/it]
Outer training loop:  40%|####      | 20192/50000 [12:11:27<14:31:50,  1.75s/it]
Outer training loop:  40%|####      | 20193/50000 [12:11:29<15:02:23,  1.82s/it]
Outer training loop:  40%|####      | 20194/50000 [12:11:31<14:42:17,  1.78s/it]
Outer training loop:  40%|####      | 20195/50000 [12:11:32<14:36:38,  1.76s/it]
Outer training loop:  40%|####      | 20196/50000 [12:11:34<14:53:46,  1.80s/it]
Outer training loop:  40%|####      | 20197/50000 [12:11:36<14:58:31,  1.81s/it]
Outer training loop:  40%|####      | 20198/50000 [12:11:38<14:32:13,  1.76s/it]
Outer training loop:  40%|####      | 20199/50000 [12:11:40<14:56:08,  1.80s/it]
Outer training loop:  40%|####      | 20200/50000 [12:11:41<14:44:03,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  40%|####      | 20201/50000 [12:11:48<27:36:53,  3.34s/it]
Outer training loop:  40%|####      | 20202/50000 [12:11:50<23:58:56,  2.90s/it]
Outer training loop:  40%|####      | 20203/50000 [12:11:52<21:15:01,  2.57s/it]
Outer training loop:  40%|####      | 20204/50000 [12:11:54<19:04:49,  2.31s/it]
Outer training loop:  40%|####      | 20205/50000 [12:11:55<17:37:09,  2.13s/it]
Outer training loop:  40%|####      | 20206/50000 [12:11:57<16:48:02,  2.03s/it]
Outer training loop:  40%|####      | 20207/50000 [12:11:59<15:50:11,  1.91s/it]
Outer training loop:  40%|####      | 20208/50000 [12:12:01<15:31:38,  1.88s/it]
Outer training loop:  40%|####      | 20209/50000 [12:12:03<15:46:53,  1.91s/it]
Outer training loop:  40%|####      | 20210/50000 [12:12:04<15:10:07,  1.83s/it]
Outer training loop:  40%|####      | 20211/50000 [12:12:06<14:57:22,  1.81s/it]
Outer training loop:  40%|####      | 20212/50000 [12:12:08<15:05:48,  1.82s/it]
Outer training loop:  40%|####      | 20213/50000 [12:12:10<15:11:39,  1.84s/it]
Outer training loop:  40%|####      | 20214/50000 [12:12:11<14:49:37,  1.79s/it]
Outer training loop:  40%|####      | 20215/50000 [12:12:13<14:48:57,  1.79s/it]
Outer training loop:  40%|####      | 20216/50000 [12:12:15<14:41:10,  1.78s/it]
Outer training loop:  40%|####      | 20217/50000 [12:12:17<14:49:20,  1.79s/it]
Outer training loop:  40%|####      | 20218/50000 [12:12:19<14:47:25,  1.79s/it]
Outer training loop:  40%|####      | 20219/50000 [12:12:20<14:32:18,  1.76s/it]
Outer training loop:  40%|####      | 20220/50000 [12:12:22<14:47:36,  1.79s/it]
Outer training loop:  40%|####      | 20221/50000 [12:12:24<14:29:58,  1.75s/it]
Outer training loop:  40%|####      | 20222/50000 [12:12:25<14:31:13,  1.76s/it]
Outer training loop:  40%|####      | 20223/50000 [12:12:27<14:57:19,  1.81s/it]
Outer training loop:  40%|####      | 20224/50000 [12:12:29<14:55:16,  1.80s/it]
Outer training loop:  40%|####      | 20225/50000 [12:12:31<15:00:04,  1.81s/it]
Outer training loop:  40%|####      | 20226/50000 [12:12:33<15:15:07,  1.84s/it]
Outer training loop:  40%|####      | 20227/50000 [12:12:35<15:13:22,  1.84s/it]
Outer training loop:  40%|####      | 20228/50000 [12:12:36<14:47:18,  1.79s/it]
Outer training loop:  40%|####      | 20229/50000 [12:12:38<14:30:27,  1.75s/it]
Outer training loop:  40%|####      | 20230/50000 [12:12:40<14:55:38,  1.81s/it]
Outer training loop:  40%|####      | 20231/50000 [12:12:42<14:52:17,  1.80s/it]
Outer training loop:  40%|####      | 20232/50000 [12:12:44<14:33:42,  1.76s/it]
Outer training loop:  40%|####      | 20233/50000 [12:12:45<14:40:49,  1.78s/it]
Outer training loop:  40%|####      | 20234/50000 [12:12:47<14:37:20,  1.77s/it]
Outer training loop:  40%|####      | 20235/50000 [12:12:49<14:43:25,  1.78s/it]
Outer training loop:  40%|####      | 20236/50000 [12:12:51<15:03:50,  1.82s/it]
Outer training loop:  40%|####      | 20237/50000 [12:12:53<14:43:55,  1.78s/it]
Outer training loop:  40%|####      | 20238/50000 [12:12:54<14:51:08,  1.80s/it]
Outer training loop:  40%|####      | 20239/50000 [12:12:56<14:35:18,  1.76s/it]
Outer training loop:  40%|####      | 20240/50000 [12:12:58<15:05:11,  1.82s/it]
Outer training loop:  40%|####      | 20241/50000 [12:13:00<14:44:59,  1.78s/it]
Outer training loop:  40%|####      | 20242/50000 [12:13:01<14:28:59,  1.75s/it]
Outer training loop:  40%|####      | 20243/50000 [12:13:03<14:49:27,  1.79s/it]
Outer training loop:  40%|####      | 20244/50000 [12:13:05<14:42:46,  1.78s/it]
Outer training loop:  40%|####      | 20245/50000 [12:13:07<14:38:15,  1.77s/it]
Outer training loop:  40%|####      | 20246/50000 [12:13:09<14:51:07,  1.80s/it]
Outer training loop:  40%|####      | 20247/50000 [12:13:10<14:34:44,  1.76s/it]
Outer training loop:  40%|####      | 20248/50000 [12:13:12<14:18:30,  1.73s/it]
Outer training loop:  40%|####      | 20249/50000 [12:13:14<14:24:49,  1.74s/it]
Outer training loop:  40%|####      | 20250/50000 [12:13:16<14:47:19,  1.79s/it]
Outer training loop:  41%|####      | 20251/50000 [12:13:17<14:42:17,  1.78s/it]
Outer training loop:  41%|####      | 20252/50000 [12:13:19<15:02:17,  1.82s/it]
Outer training loop:  41%|####      | 20253/50000 [12:13:21<15:04:09,  1.82s/it]
Outer training loop:  41%|####      | 20254/50000 [12:13:23<14:59:44,  1.81s/it]
Outer training loop:  41%|####      | 20255/50000 [12:13:25<14:55:31,  1.81s/it]
Outer training loop:  41%|####      | 20256/50000 [12:13:27<15:16:21,  1.85s/it]
Outer training loop:  41%|####      | 20257/50000 [12:13:28<15:04:07,  1.82s/it]
Outer training loop:  41%|####      | 20258/50000 [12:13:30<14:49:16,  1.79s/it]
Outer training loop:  41%|####      | 20259/50000 [12:13:32<14:26:02,  1.75s/it]
Outer training loop:  41%|####      | 20260/50000 [12:13:34<14:48:25,  1.79s/it]
Outer training loop:  41%|####      | 20261/50000 [12:13:35<14:33:47,  1.76s/it]
Outer training loop:  41%|####      | 20262/50000 [12:13:37<14:28:58,  1.75s/it]
Outer training loop:  41%|####      | 20263/50000 [12:13:39<14:47:40,  1.79s/it]
Outer training loop:  41%|####      | 20264/50000 [12:13:41<14:42:21,  1.78s/it]
Outer training loop:  41%|####      | 20265/50000 [12:13:43<14:42:07,  1.78s/it]
Outer training loop:  41%|####      | 20266/50000 [12:13:44<14:51:24,  1.80s/it]
Outer training loop:  41%|####      | 20267/50000 [12:13:46<14:48:57,  1.79s/it]
Outer training loop:  41%|####      | 20268/50000 [12:13:48<14:39:40,  1.78s/it]
Outer training loop:  41%|####      | 20269/50000 [12:13:50<14:32:06,  1.76s/it]
Outer training loop:  41%|####      | 20270/50000 [12:13:51<14:52:26,  1.80s/it]
Outer training loop:  41%|####      | 20271/50000 [12:13:53<14:36:56,  1.77s/it]
Outer training loop:  41%|####      | 20272/50000 [12:13:55<14:28:21,  1.75s/it]
Outer training loop:  41%|####      | 20273/50000 [12:13:57<15:00:30,  1.82s/it]
Outer training loop:  41%|####      | 20274/50000 [12:13:59<14:36:33,  1.77s/it]
Outer training loop:  41%|####      | 20275/50000 [12:14:00<14:46:16,  1.79s/it]
Outer training loop:  41%|####      | 20276/50000 [12:14:02<14:58:15,  1.81s/it]
Outer training loop:  41%|####      | 20277/50000 [12:14:04<14:48:18,  1.79s/it]
Outer training loop:  41%|####      | 20278/50000 [12:14:06<14:29:30,  1.76s/it]
Outer training loop:  41%|####      | 20279/50000 [12:14:07<14:12:29,  1.72s/it]
Outer training loop:  41%|####      | 20280/50000 [12:14:09<14:32:41,  1.76s/it]
Outer training loop:  41%|####      | 20281/50000 [12:14:11<14:17:34,  1.73s/it]
Outer training loop:  41%|####      | 20282/50000 [12:14:12<14:08:44,  1.71s/it]
Outer training loop:  41%|####      | 20283/50000 [12:14:14<14:32:55,  1.76s/it]
Outer training loop:  41%|####      | 20284/50000 [12:14:16<14:28:07,  1.75s/it]
Outer training loop:  41%|####      | 20285/50000 [12:14:18<14:22:06,  1.74s/it]
Outer training loop:  41%|####      | 20286/50000 [12:14:20<14:53:32,  1.80s/it]
Outer training loop:  41%|####      | 20287/50000 [12:14:21<14:34:51,  1.77s/it]
Outer training loop:  41%|####      | 20288/50000 [12:14:23<14:30:24,  1.76s/it]
Outer training loop:  41%|####      | 20289/50000 [12:14:25<15:02:03,  1.82s/it]
Outer training loop:  41%|####      | 20290/50000 [12:14:27<14:42:30,  1.78s/it]
Outer training loop:  41%|####      | 20291/50000 [12:14:29<14:35:47,  1.77s/it]
Outer training loop:  41%|####      | 20292/50000 [12:14:30<14:32:19,  1.76s/it]
Outer training loop:  41%|####      | 20293/50000 [12:14:32<14:50:20,  1.80s/it]
Outer training loop:  41%|####      | 20294/50000 [12:14:34<14:59:51,  1.82s/it]
Outer training loop:  41%|####      | 20295/50000 [12:14:36<14:40:44,  1.78s/it]
Outer training loop:  41%|####      | 20296/50000 [12:14:38<14:56:12,  1.81s/it]
Outer training loop:  41%|####      | 20297/50000 [12:14:39<14:39:59,  1.78s/it]
Outer training loop:  41%|####      | 20298/50000 [12:14:41<14:37:00,  1.77s/it]
Outer training loop:  41%|####      | 20299/50000 [12:14:43<14:41:07,  1.78s/it]
Outer training loop:  41%|####      | 20300/50000 [12:14:45<14:29:17,  1.76s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  41%|####      | 20301/50000 [12:14:52<27:58:13,  3.39s/it]
Outer training loop:  41%|####      | 20302/50000 [12:14:54<24:02:12,  2.91s/it]
Outer training loop:  41%|####      | 20303/50000 [12:14:56<21:37:06,  2.62s/it]
Outer training loop:  41%|####      | 20304/50000 [12:14:57<19:04:51,  2.31s/it]
Outer training loop:  41%|####      | 20305/50000 [12:14:59<17:32:03,  2.13s/it]
Outer training loop:  41%|####      | 20306/50000 [12:15:01<16:32:10,  2.00s/it]
Outer training loop:  41%|####      | 20307/50000 [12:15:02<16:12:54,  1.97s/it]
Outer training loop:  41%|####      | 20308/50000 [12:15:04<15:43:45,  1.91s/it]
Outer training loop:  41%|####      | 20309/50000 [12:15:06<15:17:17,  1.85s/it]
Outer training loop:  41%|####      | 20310/50000 [12:15:08<15:44:11,  1.91s/it]
Outer training loop:  41%|####      | 20311/50000 [12:15:10<15:33:32,  1.89s/it]
Outer training loop:  41%|####      | 20312/50000 [12:15:12<15:10:43,  1.84s/it]
Outer training loop:  41%|####      | 20313/50000 [12:15:13<15:23:05,  1.87s/it]
Outer training loop:  41%|####      | 20314/50000 [12:15:15<15:08:59,  1.84s/it]
Outer training loop:  41%|####      | 20315/50000 [12:15:17<14:44:30,  1.79s/it]
Outer training loop:  41%|####      | 20316/50000 [12:15:19<15:03:02,  1.83s/it]
Outer training loop:  41%|####      | 20317/50000 [12:15:21<14:51:22,  1.80s/it]
Outer training loop:  41%|####      | 20318/50000 [12:15:22<14:52:35,  1.80s/it]
Outer training loop:  41%|####      | 20319/50000 [12:15:24<14:57:22,  1.81s/it]
Outer training loop:  41%|####      | 20320/50000 [12:15:26<14:54:45,  1.81s/it]
Outer training loop:  41%|####      | 20321/50000 [12:15:28<14:46:37,  1.79s/it]
Outer training loop:  41%|####      | 20322/50000 [12:15:29<14:33:05,  1.77s/it]
Outer training loop:  41%|####      | 20323/50000 [12:15:31<14:43:22,  1.79s/it]
Outer training loop:  41%|####      | 20324/50000 [12:15:33<14:51:07,  1.80s/it]
Outer training loop:  41%|####      | 20325/50000 [12:15:35<14:43:11,  1.79s/it]
Outer training loop:  41%|####      | 20326/50000 [12:15:37<14:49:26,  1.80s/it]
Outer training loop:  41%|####      | 20327/50000 [12:15:38<14:40:15,  1.78s/it]
Outer training loop:  41%|####      | 20328/50000 [12:15:40<14:41:24,  1.78s/it]
Outer training loop:  41%|####      | 20329/50000 [12:15:42<14:43:52,  1.79s/it]
Outer training loop:  41%|####      | 20330/50000 [12:15:44<14:55:23,  1.81s/it]
Outer training loop:  41%|####      | 20331/50000 [12:15:46<14:39:25,  1.78s/it]
Outer training loop:  41%|####      | 20332/50000 [12:15:47<14:40:57,  1.78s/it]
Outer training loop:  41%|####      | 20333/50000 [12:15:49<14:49:41,  1.80s/it]
Outer training loop:  41%|####      | 20334/50000 [12:15:51<14:45:52,  1.79s/it]
Outer training loop:  41%|####      | 20335/50000 [12:15:53<14:32:15,  1.76s/it]
Outer training loop:  41%|####      | 20336/50000 [12:15:55<15:16:06,  1.85s/it]
Outer training loop:  41%|####      | 20337/50000 [12:15:56<15:01:07,  1.82s/it]
Outer training loop:  41%|####      | 20338/50000 [12:15:58<14:27:09,  1.75s/it]
Outer training loop:  41%|####      | 20339/50000 [12:16:00<14:18:09,  1.74s/it]
Outer training loop:  41%|####      | 20340/50000 [12:16:02<14:34:11,  1.77s/it]
Outer training loop:  41%|####      | 20341/50000 [12:16:03<14:37:08,  1.77s/it]
Outer training loop:  41%|####      | 20342/50000 [12:16:05<14:19:51,  1.74s/it]
Outer training loop:  41%|####      | 20343/50000 [12:16:07<14:34:56,  1.77s/it]
Outer training loop:  41%|####      | 20344/50000 [12:16:09<14:25:26,  1.75s/it]
Outer training loop:  41%|####      | 20345/50000 [12:16:10<14:11:59,  1.72s/it]
Outer training loop:  41%|####      | 20346/50000 [12:16:12<14:20:32,  1.74s/it]
Outer training loop:  41%|####      | 20347/50000 [12:16:14<14:36:23,  1.77s/it]
Outer training loop:  41%|####      | 20348/50000 [12:16:16<14:25:52,  1.75s/it]
Outer training loop:  41%|####      | 20349/50000 [12:16:17<14:32:02,  1.76s/it]
Outer training loop:  41%|####      | 20350/50000 [12:16:19<15:00:54,  1.82s/it]
Outer training loop:  41%|####      | 20351/50000 [12:16:21<14:36:13,  1.77s/it]
Outer training loop:  41%|####      | 20352/50000 [12:16:23<14:41:24,  1.78s/it]
Outer training loop:  41%|####      | 20353/50000 [12:16:25<14:44:36,  1.79s/it]
Outer training loop:  41%|####      | 20354/50000 [12:16:26<14:32:33,  1.77s/it]
Outer training loop:  41%|####      | 20355/50000 [12:16:28<14:29:21,  1.76s/it]
Outer training loop:  41%|####      | 20356/50000 [12:16:30<14:56:18,  1.81s/it]
Outer training loop:  41%|####      | 20357/50000 [12:16:32<14:33:12,  1.77s/it]
Outer training loop:  41%|####      | 20358/50000 [12:16:33<14:28:52,  1.76s/it]
Outer training loop:  41%|####      | 20359/50000 [12:16:35<14:25:23,  1.75s/it]
Outer training loop:  41%|####      | 20360/50000 [12:16:37<14:56:04,  1.81s/it]
Outer training loop:  41%|####      | 20361/50000 [12:16:39<14:28:18,  1.76s/it]
Outer training loop:  41%|####      | 20362/50000 [12:16:40<14:24:27,  1.75s/it]
Outer training loop:  41%|####      | 20363/50000 [12:16:42<14:55:00,  1.81s/it]
Outer training loop:  41%|####      | 20364/50000 [12:16:44<14:40:34,  1.78s/it]
Outer training loop:  41%|####      | 20365/50000 [12:16:46<14:25:21,  1.75s/it]
Outer training loop:  41%|####      | 20366/50000 [12:16:48<14:52:39,  1.81s/it]
Outer training loop:  41%|####      | 20367/50000 [12:16:50<14:51:37,  1.81s/it]
Outer training loop:  41%|####      | 20368/50000 [12:16:51<14:28:15,  1.76s/it]
Outer training loop:  41%|####      | 20369/50000 [12:16:53<14:22:34,  1.75s/it]
Outer training loop:  41%|####      | 20370/50000 [12:16:55<14:34:16,  1.77s/it]
Outer training loop:  41%|####      | 20371/50000 [12:16:56<14:16:09,  1.73s/it]
Outer training loop:  41%|####      | 20372/50000 [12:16:58<14:12:13,  1.73s/it]
Outer training loop:  41%|####      | 20373/50000 [12:17:00<14:38:32,  1.78s/it]
Outer training loop:  41%|####      | 20374/50000 [12:17:02<14:14:25,  1.73s/it]
Outer training loop:  41%|####      | 20375/50000 [12:17:03<14:17:20,  1.74s/it]
Outer training loop:  41%|####      | 20376/50000 [12:17:05<14:45:20,  1.79s/it]
Outer training loop:  41%|####      | 20377/50000 [12:17:07<14:40:38,  1.78s/it]
Outer training loop:  41%|####      | 20378/50000 [12:17:09<14:36:09,  1.77s/it]
Outer training loop:  41%|####      | 20379/50000 [12:17:11<14:27:30,  1.76s/it]
Outer training loop:  41%|####      | 20380/50000 [12:17:12<14:49:11,  1.80s/it]
Outer training loop:  41%|####      | 20381/50000 [12:17:14<14:38:42,  1.78s/it]
Outer training loop:  41%|####      | 20382/50000 [12:17:16<14:24:57,  1.75s/it]
Outer training loop:  41%|####      | 20383/50000 [12:17:18<14:40:06,  1.78s/it]
Outer training loop:  41%|####      | 20384/50000 [12:17:19<14:36:58,  1.78s/it]
Outer training loop:  41%|####      | 20385/50000 [12:17:21<14:36:18,  1.78s/it]
Outer training loop:  41%|####      | 20386/50000 [12:17:23<14:34:28,  1.77s/it]
Outer training loop:  41%|####      | 20387/50000 [12:17:25<14:57:54,  1.82s/it]
Outer training loop:  41%|####      | 20388/50000 [12:17:27<14:36:37,  1.78s/it]
Outer training loop:  41%|####      | 20389/50000 [12:17:28<14:24:32,  1.75s/it]
Outer training loop:  41%|####      | 20390/50000 [12:17:30<14:41:41,  1.79s/it]
Outer training loop:  41%|####      | 20391/50000 [12:17:32<14:37:45,  1.78s/it]
Outer training loop:  41%|####      | 20392/50000 [12:17:34<14:23:32,  1.75s/it]
Outer training loop:  41%|####      | 20393/50000 [12:17:36<15:01:19,  1.83s/it]
Outer training loop:  41%|####      | 20394/50000 [12:17:37<14:47:35,  1.80s/it]
Outer training loop:  41%|####      | 20395/50000 [12:17:39<14:44:22,  1.79s/it]
Outer training loop:  41%|####      | 20396/50000 [12:17:41<15:01:33,  1.83s/it]
Outer training loop:  41%|####      | 20397/50000 [12:17:43<14:33:44,  1.77s/it]
Outer training loop:  41%|####      | 20398/50000 [12:17:44<14:22:53,  1.75s/it]
Outer training loop:  41%|####      | 20399/50000 [12:17:46<14:20:54,  1.75s/it]
Outer training loop:  41%|####      | 20400/50000 [12:17:48<14:47:02,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]

Outer training loop:  41%|####      | 20401/50000 [12:17:55<27:49:02,  3.38s/it]
Outer training loop:  41%|####      | 20402/50000 [12:17:57<23:49:34,  2.90s/it]
Outer training loop:  41%|####      | 20403/50000 [12:17:59<20:51:02,  2.54s/it]
Outer training loop:  41%|####      | 20404/50000 [12:18:00<19:15:35,  2.34s/it]
Outer training loop:  41%|####      | 20405/50000 [12:18:02<17:51:30,  2.17s/it]
Outer training loop:  41%|####      | 20406/50000 [12:18:04<16:47:31,  2.04s/it]
Outer training loop:  41%|####      | 20407/50000 [12:18:06<16:25:00,  2.00s/it]
Outer training loop:  41%|####      | 20408/50000 [12:18:08<15:34:39,  1.90s/it]
Outer training loop:  41%|####      | 20409/50000 [12:18:09<15:10:15,  1.85s/it]
Outer training loop:  41%|####      | 20410/50000 [12:18:11<14:51:28,  1.81s/it]
Outer training loop:  41%|####      | 20411/50000 [12:18:13<15:07:46,  1.84s/it]
Outer training loop:  41%|####      | 20412/50000 [12:18:15<14:39:00,  1.78s/it]
Outer training loop:  41%|####      | 20413/50000 [12:18:16<14:50:50,  1.81s/it]
Outer training loop:  41%|####      | 20414/50000 [12:18:18<15:10:45,  1.85s/it]
Outer training loop:  41%|####      | 20415/50000 [12:18:20<14:53:23,  1.81s/it]
Outer training loop:  41%|####      | 20416/50000 [12:18:22<14:37:47,  1.78s/it]
Outer training loop:  41%|####      | 20417/50000 [12:18:24<14:53:07,  1.81s/it]
Outer training loop:  41%|####      | 20418/50000 [12:18:26<14:55:49,  1.82s/it]
Outer training loop:  41%|####      | 20419/50000 [12:18:27<14:28:41,  1.76s/it]
Outer training loop:  41%|####      | 20420/50000 [12:18:29<14:43:57,  1.79s/it]
Outer training loop:  41%|####      | 20421/50000 [12:18:31<14:44:35,  1.79s/it]
Outer training loop:  41%|####      | 20422/50000 [12:18:33<14:28:43,  1.76s/it]
Outer training loop:  41%|####      | 20423/50000 [12:18:34<14:54:25,  1.81s/it]
Outer training loop:  41%|####      | 20424/50000 [12:18:36<14:39:44,  1.78s/it]
Outer training loop:  41%|####      | 20425/50000 [12:18:38<14:31:09,  1.77s/it]
Outer training loop:  41%|####      | 20426/50000 [12:18:40<14:15:14,  1.74s/it]
Outer training loop:  41%|####      | 20427/50000 [12:18:41<14:42:33,  1.79s/it]
Outer training loop:  41%|####      | 20428/50000 [12:18:43<14:35:06,  1.78s/it]
Outer training loop:  41%|####      | 20429/50000 [12:18:45<14:15:52,  1.74s/it]
Outer training loop:  41%|####      | 20430/50000 [12:18:46<13:53:28,  1.69s/it]
Outer training loop:  41%|####      | 20431/50000 [12:18:48<14:22:24,  1.75s/it]
Outer training loop:  41%|####      | 20432/50000 [12:18:50<14:26:56,  1.76s/it]
Outer training loop:  41%|####      | 20433/50000 [12:18:52<14:10:24,  1.73s/it]
Outer training loop:  41%|####      | 20434/50000 [12:18:54<14:22:48,  1.75s/it]
Outer training loop:  41%|####      | 20435/50000 [12:18:55<14:22:31,  1.75s/it]
Outer training loop:  41%|####      | 20436/50000 [12:18:57<14:17:54,  1.74s/it]
Outer training loop:  41%|####      | 20437/50000 [12:18:59<14:36:03,  1.78s/it]
Outer training loop:  41%|####      | 20438/50000 [12:19:01<14:23:24,  1.75s/it]
Outer training loop:  41%|####      | 20439/50000 [12:19:02<14:12:07,  1.73s/it]
Outer training loop:  41%|####      | 20440/50000 [12:19:04<14:20:51,  1.75s/it]
Outer training loop:  41%|####      | 20441/50000 [12:19:06<14:25:28,  1.76s/it]
Outer training loop:  41%|####      | 20442/50000 [12:19:08<14:28:12,  1.76s/it]
Outer training loop:  41%|####      | 20443/50000 [12:19:09<14:26:09,  1.76s/it]
Outer training loop:  41%|####      | 20444/50000 [12:19:11<14:51:11,  1.81s/it]
Outer training loop:  41%|####      | 20445/50000 [12:19:13<14:30:22,  1.77s/it]
Outer training loop:  41%|####      | 20446/50000 [12:19:15<14:47:20,  1.80s/it]
Outer training loop:  41%|####      | 20447/50000 [12:19:17<14:46:16,  1.80s/it]
Outer training loop:  41%|####      | 20448/50000 [12:19:18<14:35:44,  1.78s/it]
Outer training loop:  41%|####      | 20449/50000 [12:19:20<14:38:10,  1.78s/it]
Outer training loop:  41%|####      | 20450/50000 [12:19:22<14:31:28,  1.77s/it]
Outer training loop:  41%|####      | 20451/50000 [12:19:24<14:56:58,  1.82s/it]
Outer training loop:  41%|####      | 20452/50000 [12:19:26<14:48:28,  1.80s/it]
Outer training loop:  41%|####      | 20453/50000 [12:19:27<14:42:24,  1.79s/it]
Outer training loop:  41%|####      | 20454/50000 [12:19:29<14:50:35,  1.81s/it]
Outer training loop:  41%|####      | 20455/50000 [12:19:31<14:41:31,  1.79s/it]
Outer training loop:  41%|####      | 20456/50000 [12:19:33<14:43:39,  1.79s/it]
Outer training loop:  41%|####      | 20457/50000 [12:19:35<15:15:40,  1.86s/it]
Outer training loop:  41%|####      | 20458/50000 [12:19:37<14:58:20,  1.82s/it]
Outer training loop:  41%|####      | 20459/50000 [12:19:38<14:35:29,  1.78s/it]
Outer training loop:  41%|####      | 20460/50000 [12:19:40<14:27:54,  1.76s/it]
Outer training loop:  41%|####      | 20461/50000 [12:19:42<15:07:43,  1.84s/it]
Outer training loop:  41%|####      | 20462/50000 [12:19:44<14:47:18,  1.80s/it]
Outer training loop:  41%|####      | 20463/50000 [12:19:45<14:11:54,  1.73s/it]
Outer training loop:  41%|####      | 20464/50000 [12:19:47<14:28:08,  1.76s/it]
Outer training loop:  41%|####      | 20465/50000 [12:19:49<14:22:21,  1.75s/it]
Outer training loop:  41%|####      | 20466/50000 [12:19:50<14:12:37,  1.73s/it]
Outer training loop:  41%|####      | 20467/50000 [12:19:52<14:04:13,  1.72s/it]
Outer training loop:  41%|####      | 20468/50000 [12:19:54<14:33:34,  1.77s/it]
Outer training loop:  41%|####      | 20469/50000 [12:19:56<14:18:10,  1.74s/it]
Outer training loop:  41%|####      | 20470/50000 [12:19:57<14:16:16,  1.74s/it]
Outer training loop:  41%|####      | 20471/50000 [12:19:59<14:48:10,  1.80s/it]
Outer training loop:  41%|####      | 20472/50000 [12:20:01<14:33:54,  1.78s/it]
Outer training loop:  41%|####      | 20473/50000 [12:20:03<14:27:52,  1.76s/it]
Outer training loop:  41%|####      | 20474/50000 [12:20:05<14:28:46,  1.77s/it]
Outer training loop:  41%|####      | 20475/50000 [12:20:06<14:44:35,  1.80s/it]
Outer training loop:  41%|####      | 20476/50000 [12:20:08<14:21:02,  1.75s/it]
Outer training loop:  41%|####      | 20477/50000 [12:20:10<14:19:28,  1.75s/it]
Outer training loop:  41%|####      | 20478/50000 [12:20:12<14:48:38,  1.81s/it]
Outer training loop:  41%|####      | 20479/50000 [12:20:14<14:35:54,  1.78s/it]
Outer training loop:  41%|####      | 20480/50000 [12:20:15<14:24:53,  1.76s/it]
Outer training loop:  41%|####      | 20481/50000 [12:20:17<14:51:41,  1.81s/it]
Outer training loop:  41%|####      | 20482/50000 [12:20:19<14:30:18,  1.77s/it]
Outer training loop:  41%|####      | 20483/50000 [12:20:21<14:43:20,  1.80s/it]
Outer training loop:  41%|####      | 20484/50000 [12:20:22<14:22:57,  1.75s/it]
Outer training loop:  41%|####      | 20485/50000 [12:20:24<14:45:56,  1.80s/it]
Outer training loop:  41%|####      | 20486/50000 [12:20:26<14:35:55,  1.78s/it]
Outer training loop:  41%|####      | 20487/50000 [12:20:28<14:32:12,  1.77s/it]
Outer training loop:  41%|####      | 20488/50000 [12:20:30<14:38:25,  1.79s/it]
Outer training loop:  41%|####      | 20489/50000 [12:20:31<14:22:06,  1.75s/it]
Outer training loop:  41%|####      | 20490/50000 [12:20:33<14:18:02,  1.74s/it]
Outer training loop:  41%|####      | 20491/50000 [12:20:35<14:37:38,  1.78s/it]
Outer training loop:  41%|####      | 20492/50000 [12:20:37<14:38:11,  1.79s/it]
Outer training loop:  41%|####      | 20493/50000 [12:20:38<14:30:53,  1.77s/it]
Outer training loop:  41%|####      | 20494/50000 [12:20:40<14:28:35,  1.77s/it]
Outer training loop:  41%|####      | 20495/50000 [12:20:42<14:42:06,  1.79s/it]
Outer training loop:  41%|####      | 20496/50000 [12:20:44<14:56:41,  1.82s/it]
Outer training loop:  41%|####      | 20497/50000 [12:20:46<14:44:58,  1.80s/it]
Outer training loop:  41%|####      | 20498/50000 [12:20:47<14:51:38,  1.81s/it]
Outer training loop:  41%|####      | 20499/50000 [12:20:49<14:29:42,  1.77s/it]
Outer training loop:  41%|####1     | 20500/50000 [12:20:51<14:17:51,  1.74s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.94s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:35,  1.89s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:29,  1.84s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:25,  1.82s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.79s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:23,  1.84s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:19,  1.81s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:16,  1.80s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:12,  1.77s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:15,  1.83s/it]

Evaluating policy:  12%|#2        | 10/83 [00:17<02:09,  1.78s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:07,  1.77s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:09,  1.83s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:06,  1.81s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:03,  1.79s/it]

Evaluating policy:  18%|#8        | 15/83 [00:26<02:01,  1.79s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:57,  1.75s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:59,  1.81s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:54,  1.76s/it]

Evaluating policy:  23%|##2       | 19/83 [00:33<01:52,  1.76s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:50,  1.75s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:52,  1.81s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:48,  1.78s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:45,  1.76s/it]

Evaluating policy:  29%|##8       | 24/83 [00:42<01:42,  1.74s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:42,  1.77s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:44,  1.83s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:38,  1.78s/it]

Evaluating policy:  35%|###4      | 29/83 [00:51<01:34,  1.75s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:32,  1.75s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.76s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [00:58<01:29,  1.79s/it]

Evaluating policy:  41%|####      | 34/83 [01:00<01:25,  1.74s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:22,  1.71s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:21,  1.74s/it]

Evaluating policy:  45%|####4     | 37/83 [01:05<01:18,  1.70s/it]

Evaluating policy:  46%|####5     | 38/83 [01:07<01:20,  1.79s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:18,  1.78s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:14,  1.74s/it]

Evaluating policy:  49%|####9     | 41/83 [01:12<01:12,  1.73s/it]

Evaluating policy:  51%|#####     | 42/83 [01:14<01:14,  1.81s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:16<01:11,  1.78s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:07,  1.79s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:21<01:04,  1.75s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:23<01:03,  1.77s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:25<01:01,  1.75s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:26<00:58,  1.73s/it]

Evaluating policy:  60%|######    | 50/83 [01:28<00:56,  1.71s/it]

Evaluating policy:  61%|######1   | 51/83 [01:30<00:58,  1.83s/it]

Evaluating policy:  63%|######2   | 52/83 [01:32<00:56,  1.81s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:52,  1.77s/it]

Evaluating policy:  65%|######5   | 54/83 [01:35<00:51,  1.76s/it]

Evaluating policy:  66%|######6   | 55/83 [01:37<00:49,  1.76s/it]

Evaluating policy:  67%|######7   | 56/83 [01:39<00:47,  1.77s/it]

Evaluating policy:  69%|######8   | 57/83 [01:41<00:45,  1.77s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:44<00:43,  1.79s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:46<00:40,  1.78s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:48<00:38,  1.76s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:50<00:36,  1.74s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:51<00:34,  1.73s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:53<00:32,  1.72s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:55<00:30,  1.72s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:28,  1.80s/it]

Evaluating policy:  82%|########1 | 68/83 [02:00<00:26,  1.79s/it]

Evaluating policy:  83%|########3 | 69/83 [02:02<00:24,  1.76s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.75s/it]

Evaluating policy:  86%|########5 | 71/83 [02:05<00:20,  1.72s/it]

Evaluating policy:  87%|########6 | 72/83 [02:07<00:18,  1.71s/it]

Evaluating policy:  88%|########7 | 73/83 [02:09<00:17,  1.71s/it]

Evaluating policy:  89%|########9 | 74/83 [02:11<00:15,  1.74s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:15,  1.91s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:13,  1.86s/it]

Evaluating policy:  93%|#########2| 77/83 [02:16<00:10,  1.82s/it]

Evaluating policy:  94%|#########3| 78/83 [02:18<00:08,  1.79s/it]

Evaluating policy:  95%|#########5| 79/83 [02:20<00:07,  1.81s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.79s/it]

Evaluating policy:  98%|#########7| 81/83 [02:23<00:03,  1.75s/it]

Evaluating policy:  99%|#########8| 82/83 [02:25<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 83/83 [02:27<00:00,  1.72s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:58,  2.18s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:44,  2.03s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:33,  1.92s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.89s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.88s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:20,  1.82s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:17,  1.81s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:13,  1.78s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:12,  1.82s/it]

Evaluating policy:  13%|#3        | 11/83 [00:19<02:10,  1.81s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:06,  1.78s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:07,  1.82s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:04,  1.81s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:02,  1.80s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<02:00,  1.80s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:59,  1.83s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:53,  1.78s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:51,  1.76s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:48,  1.75s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:50,  1.82s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:48,  1.81s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:46,  1.80s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:43,  1.79s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:40,  1.76s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:41,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:39,  1.81s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:35,  1.77s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:33,  1.76s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:31,  1.76s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:29,  1.79s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:26,  1.77s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:24,  1.77s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:25,  1.81s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:21,  1.78s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:10<01:19,  1.80s/it]

Evaluating policy:  48%|####8     | 40/83 [01:11<01:16,  1.79s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:13,  1.75s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:12,  1.76s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:10,  1.76s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:18<01:08,  1.76s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:21<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:07,  1.82s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:04,  1.79s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:01,  1.76s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<00:59,  1.76s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:57,  1.75s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:55,  1.75s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:35<00:54,  1.81s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:51,  1.76s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:48,  1.74s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:47,  1.76s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:44,  1.73s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:43,  1.74s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.79s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:40,  1.82s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:37,  1.79s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.77s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:34,  1.79s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:31,  1.76s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:58<00:29,  1.73s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.75s/it]

Evaluating policy:  82%|########1 | 68/83 [02:02<00:27,  1.86s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:25,  1.82s/it]

Evaluating policy:  84%|########4 | 70/83 [02:05<00:23,  1.79s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:21,  1.76s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:18,  1.73s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.74s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:15,  1.72s/it]

Evaluating policy:  90%|######### | 75/83 [02:13<00:13,  1.71s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:11,  1.71s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:11,  1.86s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.79s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.78s/it]

Evaluating policy:  96%|#########6| 80/83 [02:22<00:05,  1.76s/it]

Evaluating policy:  98%|#########7| 81/83 [02:24<00:03,  1.74s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.76s/it]

Outer training loop:  41%|####1     | 20501/50000 [12:25:53<753:48:59, 91.99s/it]
Outer training loop:  41%|####1     | 20502/50000 [12:25:55<531:36:50, 64.88s/it]
Outer training loop:  41%|####1     | 20503/50000 [12:25:57<376:23:17, 45.94s/it]
Outer training loop:  41%|####1     | 20504/50000 [12:25:59<269:01:27, 32.83s/it]
Outer training loop:  41%|####1     | 20505/50000 [12:26:01<192:33:56, 23.50s/it]
Outer training loop:  41%|####1     | 20506/50000 [12:26:02<139:00:07, 16.97s/it]
Outer training loop:  41%|####1     | 20507/50000 [12:26:04<101:42:35, 12.41s/it]
Outer training loop:  41%|####1     | 20508/50000 [12:26:06<75:50:54,  9.26s/it] 
Outer training loop:  41%|####1     | 20509/50000 [12:26:08<57:12:29,  6.98s/it]
Outer training loop:  41%|####1     | 20510/50000 [12:26:10<44:14:51,  5.40s/it]
Outer training loop:  41%|####1     | 20511/50000 [12:26:11<35:12:51,  4.30s/it]
Outer training loop:  41%|####1     | 20512/50000 [12:26:13<29:14:08,  3.57s/it]
Outer training loop:  41%|####1     | 20513/50000 [12:26:15<24:29:55,  2.99s/it]
Outer training loop:  41%|####1     | 20514/50000 [12:26:16<21:10:22,  2.59s/it]
Outer training loop:  41%|####1     | 20515/50000 [12:26:18<19:14:28,  2.35s/it]
Outer training loop:  41%|####1     | 20516/50000 [12:26:20<17:44:53,  2.17s/it]
Outer training loop:  41%|####1     | 20517/50000 [12:26:22<16:40:52,  2.04s/it]
Outer training loop:  41%|####1     | 20518/50000 [12:26:24<16:16:31,  1.99s/it]
Outer training loop:  41%|####1     | 20519/50000 [12:26:25<15:20:30,  1.87s/it]
Outer training loop:  41%|####1     | 20520/50000 [12:26:27<14:55:26,  1.82s/it]
Outer training loop:  41%|####1     | 20521/50000 [12:26:29<14:35:39,  1.78s/it]
Outer training loop:  41%|####1     | 20522/50000 [12:26:30<14:44:10,  1.80s/it]
Outer training loop:  41%|####1     | 20523/50000 [12:26:32<14:30:56,  1.77s/it]
Outer training loop:  41%|####1     | 20524/50000 [12:26:34<14:18:00,  1.75s/it]
Outer training loop:  41%|####1     | 20525/50000 [12:26:36<14:29:47,  1.77s/it]
Outer training loop:  41%|####1     | 20526/50000 [12:26:37<14:03:56,  1.72s/it]
Outer training loop:  41%|####1     | 20527/50000 [12:26:39<14:43:28,  1.80s/it]
Outer training loop:  41%|####1     | 20528/50000 [12:26:41<15:07:51,  1.85s/it]
Outer training loop:  41%|####1     | 20529/50000 [12:26:43<14:56:59,  1.83s/it]
Outer training loop:  41%|####1     | 20530/50000 [12:26:45<14:52:16,  1.82s/it]
Outer training loop:  41%|####1     | 20531/50000 [12:26:46<14:37:31,  1.79s/it]
Outer training loop:  41%|####1     | 20532/50000 [12:26:48<14:54:07,  1.82s/it]
Outer training loop:  41%|####1     | 20533/50000 [12:26:50<14:42:44,  1.80s/it]
Outer training loop:  41%|####1     | 20534/50000 [12:26:52<14:32:51,  1.78s/it]
Outer training loop:  41%|####1     | 20535/50000 [12:26:54<14:46:43,  1.81s/it]
Outer training loop:  41%|####1     | 20536/50000 [12:26:55<14:45:50,  1.80s/it]
Outer training loop:  41%|####1     | 20537/50000 [12:26:57<14:23:55,  1.76s/it]
Outer training loop:  41%|####1     | 20538/50000 [12:26:59<14:56:12,  1.83s/it]
Outer training loop:  41%|####1     | 20539/50000 [12:27:01<14:42:09,  1.80s/it]
Outer training loop:  41%|####1     | 20540/50000 [12:27:03<14:36:27,  1.79s/it]
Outer training loop:  41%|####1     | 20541/50000 [12:27:04<14:24:03,  1.76s/it]
Outer training loop:  41%|####1     | 20542/50000 [12:27:06<14:36:14,  1.78s/it]
Outer training loop:  41%|####1     | 20543/50000 [12:27:08<14:32:04,  1.78s/it]
Outer training loop:  41%|####1     | 20544/50000 [12:27:10<14:21:55,  1.76s/it]
Outer training loop:  41%|####1     | 20545/50000 [12:27:12<14:43:00,  1.80s/it]
Outer training loop:  41%|####1     | 20546/50000 [12:27:13<14:56:39,  1.83s/it]
Outer training loop:  41%|####1     | 20547/50000 [12:27:15<14:38:46,  1.79s/it]
Outer training loop:  41%|####1     | 20548/50000 [12:27:17<15:01:07,  1.84s/it]
Outer training loop:  41%|####1     | 20549/50000 [12:27:19<14:43:09,  1.80s/it]
Outer training loop:  41%|####1     | 20550/50000 [12:27:21<14:33:35,  1.78s/it]
Outer training loop:  41%|####1     | 20551/50000 [12:27:22<14:35:44,  1.78s/it]
Outer training loop:  41%|####1     | 20552/50000 [12:27:24<14:53:05,  1.82s/it]
Outer training loop:  41%|####1     | 20553/50000 [12:27:26<14:39:12,  1.79s/it]
Outer training loop:  41%|####1     | 20554/50000 [12:27:28<14:20:54,  1.75s/it]
Outer training loop:  41%|####1     | 20555/50000 [12:27:29<14:33:34,  1.78s/it]
Outer training loop:  41%|####1     | 20556/50000 [12:27:31<14:24:20,  1.76s/it]
Outer training loop:  41%|####1     | 20557/50000 [12:27:33<14:39:28,  1.79s/it]
Outer training loop:  41%|####1     | 20558/50000 [12:27:35<15:13:37,  1.86s/it]
Outer training loop:  41%|####1     | 20559/50000 [12:27:37<14:36:39,  1.79s/it]
Outer training loop:  41%|####1     | 20560/50000 [12:27:38<14:35:49,  1.78s/it]
Outer training loop:  41%|####1     | 20561/50000 [12:27:40<14:54:42,  1.82s/it]
Outer training loop:  41%|####1     | 20562/50000 [12:27:42<14:43:27,  1.80s/it]
Outer training loop:  41%|####1     | 20563/50000 [12:27:44<14:35:00,  1.78s/it]
Outer training loop:  41%|####1     | 20564/50000 [12:27:45<14:08:27,  1.73s/it]
Outer training loop:  41%|####1     | 20565/50000 [12:27:47<14:09:22,  1.73s/it]
Outer training loop:  41%|####1     | 20566/50000 [12:27:49<14:17:28,  1.75s/it]
Outer training loop:  41%|####1     | 20567/50000 [12:27:51<14:04:10,  1.72s/it]
Outer training loop:  41%|####1     | 20568/50000 [12:27:52<14:25:50,  1.77s/it]
Outer training loop:  41%|####1     | 20569/50000 [12:27:54<14:06:17,  1.73s/it]
Outer training loop:  41%|####1     | 20570/50000 [12:27:56<14:02:53,  1.72s/it]
Outer training loop:  41%|####1     | 20571/50000 [12:27:57<13:47:10,  1.69s/it]
Outer training loop:  41%|####1     | 20572/50000 [12:27:59<14:14:55,  1.74s/it]
Outer training loop:  41%|####1     | 20573/50000 [12:28:01<14:04:28,  1.72s/it]
Outer training loop:  41%|####1     | 20574/50000 [12:28:03<14:01:37,  1.72s/it]
Outer training loop:  41%|####1     | 20575/50000 [12:28:05<14:33:37,  1.78s/it]
Outer training loop:  41%|####1     | 20576/50000 [12:28:06<14:40:01,  1.79s/it]
Outer training loop:  41%|####1     | 20577/50000 [12:28:08<14:41:12,  1.80s/it]
Outer training loop:  41%|####1     | 20578/50000 [12:28:10<14:44:11,  1.80s/it]
Outer training loop:  41%|####1     | 20579/50000 [12:28:12<14:51:50,  1.82s/it]
Outer training loop:  41%|####1     | 20580/50000 [12:28:14<14:32:47,  1.78s/it]
Outer training loop:  41%|####1     | 20581/50000 [12:28:15<14:18:58,  1.75s/it]
Outer training loop:  41%|####1     | 20582/50000 [12:28:17<14:43:40,  1.80s/it]
Outer training loop:  41%|####1     | 20583/50000 [12:28:19<14:34:23,  1.78s/it]
Outer training loop:  41%|####1     | 20584/50000 [12:28:21<14:33:39,  1.78s/it]
Outer training loop:  41%|####1     | 20585/50000 [12:28:23<14:56:55,  1.83s/it]
Outer training loop:  41%|####1     | 20586/50000 [12:28:24<14:43:46,  1.80s/it]
Outer training loop:  41%|####1     | 20587/50000 [12:28:26<14:14:26,  1.74s/it]
Outer training loop:  41%|####1     | 20588/50000 [12:28:28<14:04:26,  1.72s/it]
Outer training loop:  41%|####1     | 20589/50000 [12:28:30<14:26:22,  1.77s/it]
Outer training loop:  41%|####1     | 20590/50000 [12:28:31<14:15:52,  1.75s/it]
Outer training loop:  41%|####1     | 20591/50000 [12:28:33<14:17:54,  1.75s/it]
Outer training loop:  41%|####1     | 20592/50000 [12:28:35<14:43:22,  1.80s/it]
Outer training loop:  41%|####1     | 20593/50000 [12:28:37<14:36:25,  1.79s/it]
Outer training loop:  41%|####1     | 20594/50000 [12:28:38<14:26:46,  1.77s/it]
Outer training loop:  41%|####1     | 20595/50000 [12:28:40<14:57:45,  1.83s/it]
Outer training loop:  41%|####1     | 20596/50000 [12:28:42<14:26:40,  1.77s/it]
Outer training loop:  41%|####1     | 20597/50000 [12:28:44<14:10:05,  1.73s/it]
Outer training loop:  41%|####1     | 20598/50000 [12:28:45<14:07:52,  1.73s/it]
Outer training loop:  41%|####1     | 20599/50000 [12:28:47<14:30:48,  1.78s/it]
Outer training loop:  41%|####1     | 20600/50000 [12:28:49<14:24:54,  1.77s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.67s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.70s/it]

Outer training loop:  41%|####1     | 20601/50000 [12:28:56<26:45:39,  3.28s/it]
Outer training loop:  41%|####1     | 20602/50000 [12:28:58<22:50:24,  2.80s/it]
Outer training loop:  41%|####1     | 20603/50000 [12:28:59<20:31:44,  2.51s/it]
Outer training loop:  41%|####1     | 20604/50000 [12:29:01<18:33:17,  2.27s/it]
Outer training loop:  41%|####1     | 20605/50000 [12:29:03<17:02:48,  2.09s/it]
Outer training loop:  41%|####1     | 20606/50000 [12:29:05<16:24:38,  2.01s/it]
Outer training loop:  41%|####1     | 20607/50000 [12:29:06<15:43:51,  1.93s/it]
Outer training loop:  41%|####1     | 20608/50000 [12:29:08<15:13:46,  1.87s/it]
Outer training loop:  41%|####1     | 20609/50000 [12:29:10<14:45:33,  1.81s/it]
Outer training loop:  41%|####1     | 20610/50000 [12:29:12<14:50:11,  1.82s/it]
Outer training loop:  41%|####1     | 20611/50000 [12:29:13<14:23:45,  1.76s/it]
Outer training loop:  41%|####1     | 20612/50000 [12:29:15<14:15:17,  1.75s/it]
Outer training loop:  41%|####1     | 20613/50000 [12:29:17<14:54:17,  1.83s/it]
Outer training loop:  41%|####1     | 20614/50000 [12:29:19<14:45:01,  1.81s/it]
Outer training loop:  41%|####1     | 20615/50000 [12:29:20<14:26:17,  1.77s/it]
Outer training loop:  41%|####1     | 20616/50000 [12:29:22<14:07:47,  1.73s/it]
Outer training loop:  41%|####1     | 20617/50000 [12:29:24<14:37:35,  1.79s/it]
Outer training loop:  41%|####1     | 20618/50000 [12:29:26<14:36:27,  1.79s/it]
Outer training loop:  41%|####1     | 20619/50000 [12:29:27<14:27:09,  1.77s/it]
Outer training loop:  41%|####1     | 20620/50000 [12:29:29<14:47:24,  1.81s/it]
Outer training loop:  41%|####1     | 20621/50000 [12:29:31<14:30:46,  1.78s/it]
Outer training loop:  41%|####1     | 20622/50000 [12:29:33<14:14:50,  1.75s/it]
Outer training loop:  41%|####1     | 20623/50000 [12:29:35<14:37:41,  1.79s/it]
Outer training loop:  41%|####1     | 20624/50000 [12:29:36<14:48:07,  1.81s/it]
Outer training loop:  41%|####1     | 20625/50000 [12:29:38<14:52:59,  1.82s/it]
Outer training loop:  41%|####1     | 20626/50000 [12:29:40<14:32:40,  1.78s/it]
Outer training loop:  41%|####1     | 20627/50000 [12:29:42<14:39:56,  1.80s/it]
Outer training loop:  41%|####1     | 20628/50000 [12:29:43<14:21:40,  1.76s/it]
Outer training loop:  41%|####1     | 20629/50000 [12:29:45<14:06:10,  1.73s/it]
Outer training loop:  41%|####1     | 20630/50000 [12:29:47<14:31:30,  1.78s/it]
Outer training loop:  41%|####1     | 20631/50000 [12:29:49<14:26:29,  1.77s/it]
Outer training loop:  41%|####1     | 20632/50000 [12:29:51<14:21:21,  1.76s/it]
Outer training loop:  41%|####1     | 20633/50000 [12:29:52<14:33:21,  1.78s/it]
Outer training loop:  41%|####1     | 20634/50000 [12:29:54<14:22:00,  1.76s/it]
Outer training loop:  41%|####1     | 20635/50000 [12:29:56<14:22:20,  1.76s/it]
Outer training loop:  41%|####1     | 20636/50000 [12:29:58<14:12:49,  1.74s/it]
Outer training loop:  41%|####1     | 20637/50000 [12:29:59<14:35:06,  1.79s/it]
Outer training loop:  41%|####1     | 20638/50000 [12:30:01<14:21:38,  1.76s/it]
Outer training loop:  41%|####1     | 20639/50000 [12:30:03<14:17:11,  1.75s/it]
Outer training loop:  41%|####1     | 20640/50000 [12:30:05<14:29:46,  1.78s/it]
Outer training loop:  41%|####1     | 20641/50000 [12:30:06<14:26:50,  1.77s/it]
Outer training loop:  41%|####1     | 20642/50000 [12:30:08<14:15:34,  1.75s/it]
Outer training loop:  41%|####1     | 20643/50000 [12:30:10<14:32:33,  1.78s/it]
Outer training loop:  41%|####1     | 20644/50000 [12:30:12<14:37:34,  1.79s/it]
Outer training loop:  41%|####1     | 20645/50000 [12:30:14<14:27:19,  1.77s/it]
Outer training loop:  41%|####1     | 20646/50000 [12:30:15<14:22:30,  1.76s/it]
Outer training loop:  41%|####1     | 20647/50000 [12:30:17<14:56:03,  1.83s/it]
Outer training loop:  41%|####1     | 20648/50000 [12:30:19<14:33:21,  1.79s/it]
Outer training loop:  41%|####1     | 20649/50000 [12:30:21<14:33:37,  1.79s/it]
Outer training loop:  41%|####1     | 20650/50000 [12:30:23<14:38:35,  1.80s/it]
Outer training loop:  41%|####1     | 20651/50000 [12:30:24<14:55:00,  1.83s/it]
Outer training loop:  41%|####1     | 20652/50000 [12:30:26<14:38:56,  1.80s/it]
Outer training loop:  41%|####1     | 20653/50000 [12:30:28<14:27:04,  1.77s/it]
Outer training loop:  41%|####1     | 20654/50000 [12:30:30<14:36:06,  1.79s/it]
Outer training loop:  41%|####1     | 20655/50000 [12:30:32<14:34:43,  1.79s/it]
Outer training loop:  41%|####1     | 20656/50000 [12:30:33<14:16:06,  1.75s/it]
Outer training loop:  41%|####1     | 20657/50000 [12:30:35<14:37:32,  1.79s/it]
Outer training loop:  41%|####1     | 20658/50000 [12:30:37<14:27:29,  1.77s/it]
Outer training loop:  41%|####1     | 20659/50000 [12:30:39<14:26:08,  1.77s/it]
Outer training loop:  41%|####1     | 20660/50000 [12:30:40<14:30:23,  1.78s/it]
Outer training loop:  41%|####1     | 20661/50000 [12:30:42<14:05:20,  1.73s/it]
Outer training loop:  41%|####1     | 20662/50000 [12:30:44<13:54:33,  1.71s/it]
Outer training loop:  41%|####1     | 20663/50000 [12:30:45<13:35:47,  1.67s/it]
Outer training loop:  41%|####1     | 20664/50000 [12:30:47<14:15:38,  1.75s/it]
Outer training loop:  41%|####1     | 20665/50000 [12:30:49<14:14:32,  1.75s/it]
Outer training loop:  41%|####1     | 20666/50000 [12:30:51<14:13:50,  1.75s/it]
Outer training loop:  41%|####1     | 20667/50000 [12:30:53<14:33:39,  1.79s/it]
Outer training loop:  41%|####1     | 20668/50000 [12:30:54<14:14:36,  1.75s/it]
Outer training loop:  41%|####1     | 20669/50000 [12:30:56<14:13:45,  1.75s/it]
Outer training loop:  41%|####1     | 20670/50000 [12:30:58<14:00:41,  1.72s/it]
Outer training loop:  41%|####1     | 20671/50000 [12:30:59<14:21:27,  1.76s/it]
Outer training loop:  41%|####1     | 20672/50000 [12:31:01<13:54:18,  1.71s/it]
Outer training loop:  41%|####1     | 20673/50000 [12:31:03<13:47:31,  1.69s/it]
Outer training loop:  41%|####1     | 20674/50000 [12:31:05<15:06:52,  1.86s/it]
Outer training loop:  41%|####1     | 20675/50000 [12:31:07<14:32:01,  1.78s/it]
Outer training loop:  41%|####1     | 20676/50000 [12:31:08<14:18:03,  1.76s/it]
Outer training loop:  41%|####1     | 20677/50000 [12:31:10<14:31:09,  1.78s/it]
Outer training loop:  41%|####1     | 20678/50000 [12:31:12<14:25:17,  1.77s/it]
Outer training loop:  41%|####1     | 20679/50000 [12:31:14<14:29:48,  1.78s/it]
Outer training loop:  41%|####1     | 20680/50000 [12:31:15<14:18:46,  1.76s/it]
Outer training loop:  41%|####1     | 20681/50000 [12:31:17<14:50:25,  1.82s/it]
Outer training loop:  41%|####1     | 20682/50000 [12:31:19<14:37:08,  1.80s/it]
Outer training loop:  41%|####1     | 20683/50000 [12:31:21<14:29:55,  1.78s/it]
Outer training loop:  41%|####1     | 20684/50000 [12:31:23<14:58:48,  1.84s/it]
Outer training loop:  41%|####1     | 20685/50000 [12:31:25<14:47:34,  1.82s/it]
Outer training loop:  41%|####1     | 20686/50000 [12:31:26<14:27:05,  1.77s/it]
Outer training loop:  41%|####1     | 20687/50000 [12:31:28<14:25:16,  1.77s/it]
Outer training loop:  41%|####1     | 20688/50000 [12:31:30<14:47:05,  1.82s/it]
Outer training loop:  41%|####1     | 20689/50000 [12:31:32<14:26:49,  1.77s/it]
Outer training loop:  41%|####1     | 20690/50000 [12:31:33<14:20:02,  1.76s/it]
Outer training loop:  41%|####1     | 20691/50000 [12:31:35<14:43:45,  1.81s/it]
Outer training loop:  41%|####1     | 20692/50000 [12:31:37<14:40:50,  1.80s/it]
Outer training loop:  41%|####1     | 20693/50000 [12:31:39<14:23:22,  1.77s/it]
Outer training loop:  41%|####1     | 20694/50000 [12:31:41<14:39:08,  1.80s/it]
Outer training loop:  41%|####1     | 20695/50000 [12:31:42<14:24:41,  1.77s/it]
Outer training loop:  41%|####1     | 20696/50000 [12:31:44<14:00:16,  1.72s/it]
Outer training loop:  41%|####1     | 20697/50000 [12:31:46<13:52:35,  1.70s/it]
Outer training loop:  41%|####1     | 20698/50000 [12:31:47<14:19:45,  1.76s/it]
Outer training loop:  41%|####1     | 20699/50000 [12:31:49<14:21:04,  1.76s/it]
Outer training loop:  41%|####1     | 20700/50000 [12:31:51<14:07:46,  1.74s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.86s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  41%|####1     | 20701/50000 [12:31:58<27:11:52,  3.34s/it]
Outer training loop:  41%|####1     | 20702/50000 [12:32:00<23:30:38,  2.89s/it]
Outer training loop:  41%|####1     | 20703/50000 [12:32:02<20:41:34,  2.54s/it]
Outer training loop:  41%|####1     | 20704/50000 [12:32:03<18:35:42,  2.29s/it]
Outer training loop:  41%|####1     | 20705/50000 [12:32:05<17:15:50,  2.12s/it]
Outer training loop:  41%|####1     | 20706/50000 [12:32:07<16:38:01,  2.04s/it]
Outer training loop:  41%|####1     | 20707/50000 [12:32:09<15:50:18,  1.95s/it]
Outer training loop:  41%|####1     | 20708/50000 [12:32:10<15:26:37,  1.90s/it]
Outer training loop:  41%|####1     | 20709/50000 [12:32:12<15:40:18,  1.93s/it]
Outer training loop:  41%|####1     | 20710/50000 [12:32:14<15:07:01,  1.86s/it]
Outer training loop:  41%|####1     | 20711/50000 [12:32:16<15:00:26,  1.84s/it]
Outer training loop:  41%|####1     | 20712/50000 [12:32:18<14:54:35,  1.83s/it]
Outer training loop:  41%|####1     | 20713/50000 [12:32:20<15:04:04,  1.85s/it]
Outer training loop:  41%|####1     | 20714/50000 [12:32:21<14:40:54,  1.80s/it]
Outer training loop:  41%|####1     | 20715/50000 [12:32:23<14:19:07,  1.76s/it]
Outer training loop:  41%|####1     | 20716/50000 [12:32:25<15:01:55,  1.85s/it]
Outer training loop:  41%|####1     | 20717/50000 [12:32:27<14:44:48,  1.81s/it]
Outer training loop:  41%|####1     | 20718/50000 [12:32:28<14:41:14,  1.81s/it]
Outer training loop:  41%|####1     | 20719/50000 [12:32:30<14:25:16,  1.77s/it]
Outer training loop:  41%|####1     | 20720/50000 [12:32:32<14:26:04,  1.77s/it]
Outer training loop:  41%|####1     | 20721/50000 [12:32:34<14:49:21,  1.82s/it]
Outer training loop:  41%|####1     | 20722/50000 [12:32:36<14:40:10,  1.80s/it]
Outer training loop:  41%|####1     | 20723/50000 [12:32:37<14:50:07,  1.82s/it]
Outer training loop:  41%|####1     | 20724/50000 [12:32:39<14:35:09,  1.79s/it]
Outer training loop:  41%|####1     | 20725/50000 [12:32:41<14:14:30,  1.75s/it]
Outer training loop:  41%|####1     | 20726/50000 [12:32:43<14:49:52,  1.82s/it]
Outer training loop:  41%|####1     | 20727/50000 [12:32:45<14:28:18,  1.78s/it]
Outer training loop:  41%|####1     | 20728/50000 [12:32:46<14:23:23,  1.77s/it]
Outer training loop:  41%|####1     | 20729/50000 [12:32:48<14:49:21,  1.82s/it]
Outer training loop:  41%|####1     | 20730/50000 [12:32:50<14:37:02,  1.80s/it]
Outer training loop:  41%|####1     | 20731/50000 [12:32:52<14:22:56,  1.77s/it]
Outer training loop:  41%|####1     | 20732/50000 [12:32:54<14:32:38,  1.79s/it]
Outer training loop:  41%|####1     | 20733/50000 [12:32:55<14:33:35,  1.79s/it]
Outer training loop:  41%|####1     | 20734/50000 [12:32:57<14:29:34,  1.78s/it]
Outer training loop:  41%|####1     | 20735/50000 [12:32:59<14:27:39,  1.78s/it]
Outer training loop:  41%|####1     | 20736/50000 [12:33:01<14:48:39,  1.82s/it]
Outer training loop:  41%|####1     | 20737/50000 [12:33:02<14:21:03,  1.77s/it]
Outer training loop:  41%|####1     | 20738/50000 [12:33:04<14:16:22,  1.76s/it]
Outer training loop:  41%|####1     | 20739/50000 [12:33:06<14:40:08,  1.80s/it]
Outer training loop:  41%|####1     | 20740/50000 [12:33:08<14:24:49,  1.77s/it]
Outer training loop:  41%|####1     | 20741/50000 [12:33:10<14:44:13,  1.81s/it]
Outer training loop:  41%|####1     | 20742/50000 [12:33:11<14:39:03,  1.80s/it]
Outer training loop:  41%|####1     | 20743/50000 [12:33:13<14:57:52,  1.84s/it]
Outer training loop:  41%|####1     | 20744/50000 [12:33:15<14:37:16,  1.80s/it]
Outer training loop:  41%|####1     | 20745/50000 [12:33:17<14:29:46,  1.78s/it]
Outer training loop:  41%|####1     | 20746/50000 [12:33:19<15:00:08,  1.85s/it]
Outer training loop:  41%|####1     | 20747/50000 [12:33:21<14:42:07,  1.81s/it]
Outer training loop:  41%|####1     | 20748/50000 [12:33:22<14:25:33,  1.78s/it]
Outer training loop:  41%|####1     | 20749/50000 [12:33:24<14:22:41,  1.77s/it]
Outer training loop:  42%|####1     | 20750/50000 [12:33:26<14:34:27,  1.79s/it]
Outer training loop:  42%|####1     | 20751/50000 [12:33:28<14:36:47,  1.80s/it]
Outer training loop:  42%|####1     | 20752/50000 [12:33:29<14:26:08,  1.78s/it]
Outer training loop:  42%|####1     | 20753/50000 [12:33:31<14:16:34,  1.76s/it]
Outer training loop:  42%|####1     | 20754/50000 [12:33:33<14:05:08,  1.73s/it]
Outer training loop:  42%|####1     | 20755/50000 [12:33:35<14:09:17,  1.74s/it]
Outer training loop:  42%|####1     | 20756/50000 [12:33:36<14:25:33,  1.78s/it]
Outer training loop:  42%|####1     | 20757/50000 [12:33:38<14:00:50,  1.73s/it]
Outer training loop:  42%|####1     | 20758/50000 [12:33:40<14:02:22,  1.73s/it]
Outer training loop:  42%|####1     | 20759/50000 [12:33:41<13:45:56,  1.69s/it]
Outer training loop:  42%|####1     | 20760/50000 [12:33:43<14:05:07,  1.73s/it]
Outer training loop:  42%|####1     | 20761/50000 [12:33:45<13:53:17,  1.71s/it]
Outer training loop:  42%|####1     | 20762/50000 [12:33:46<13:45:22,  1.69s/it]
Outer training loop:  42%|####1     | 20763/50000 [12:33:48<14:09:42,  1.74s/it]
Outer training loop:  42%|####1     | 20764/50000 [12:33:50<14:03:34,  1.73s/it]
Outer training loop:  42%|####1     | 20765/50000 [12:33:52<14:04:44,  1.73s/it]
Outer training loop:  42%|####1     | 20766/50000 [12:33:53<13:48:25,  1.70s/it]
Outer training loop:  42%|####1     | 20767/50000 [12:33:55<14:09:45,  1.74s/it]
Outer training loop:  42%|####1     | 20768/50000 [12:33:57<14:09:15,  1.74s/it]
Outer training loop:  42%|####1     | 20769/50000 [12:33:59<13:57:34,  1.72s/it]
Outer training loop:  42%|####1     | 20770/50000 [12:34:00<14:11:05,  1.75s/it]
Outer training loop:  42%|####1     | 20771/50000 [12:34:02<14:26:24,  1.78s/it]
Outer training loop:  42%|####1     | 20772/50000 [12:34:04<14:11:34,  1.75s/it]
Outer training loop:  42%|####1     | 20773/50000 [12:34:06<14:03:29,  1.73s/it]
Outer training loop:  42%|####1     | 20774/50000 [12:34:08<14:29:17,  1.78s/it]
Outer training loop:  42%|####1     | 20775/50000 [12:34:09<14:19:54,  1.77s/it]
Outer training loop:  42%|####1     | 20776/50000 [12:34:11<14:07:15,  1.74s/it]
Outer training loop:  42%|####1     | 20777/50000 [12:34:13<14:24:55,  1.78s/it]
Outer training loop:  42%|####1     | 20778/50000 [12:34:15<14:14:40,  1.75s/it]
Outer training loop:  42%|####1     | 20779/50000 [12:34:16<14:17:59,  1.76s/it]
Outer training loop:  42%|####1     | 20780/50000 [12:34:18<14:17:41,  1.76s/it]
Outer training loop:  42%|####1     | 20781/50000 [12:34:20<14:41:10,  1.81s/it]
Outer training loop:  42%|####1     | 20782/50000 [12:34:22<14:33:05,  1.79s/it]
Outer training loop:  42%|####1     | 20783/50000 [12:34:23<14:21:05,  1.77s/it]
Outer training loop:  42%|####1     | 20784/50000 [12:34:25<14:54:44,  1.84s/it]
Outer training loop:  42%|####1     | 20785/50000 [12:34:27<14:28:56,  1.78s/it]
Outer training loop:  42%|####1     | 20786/50000 [12:34:29<14:19:49,  1.77s/it]
Outer training loop:  42%|####1     | 20787/50000 [12:34:31<14:39:26,  1.81s/it]
Outer training loop:  42%|####1     | 20788/50000 [12:34:32<14:22:25,  1.77s/it]
Outer training loop:  42%|####1     | 20789/50000 [12:34:34<14:25:00,  1.78s/it]
Outer training loop:  42%|####1     | 20790/50000 [12:34:36<14:27:03,  1.78s/it]
Outer training loop:  42%|####1     | 20791/50000 [12:34:38<14:38:48,  1.81s/it]
Outer training loop:  42%|####1     | 20792/50000 [12:34:40<14:25:34,  1.78s/it]
Outer training loop:  42%|####1     | 20793/50000 [12:34:41<14:35:28,  1.80s/it]
Outer training loop:  42%|####1     | 20794/50000 [12:34:43<14:38:16,  1.80s/it]
Outer training loop:  42%|####1     | 20795/50000 [12:34:45<14:32:05,  1.79s/it]
Outer training loop:  42%|####1     | 20796/50000 [12:34:47<14:20:54,  1.77s/it]
Outer training loop:  42%|####1     | 20797/50000 [12:34:49<14:41:00,  1.81s/it]
Outer training loop:  42%|####1     | 20798/50000 [12:34:50<14:28:54,  1.79s/it]
Outer training loop:  42%|####1     | 20799/50000 [12:34:52<14:13:08,  1.75s/it]
Outer training loop:  42%|####1     | 20800/50000 [12:34:54<13:52:01,  1.71s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.72s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.72s/it]

Outer training loop:  42%|####1     | 20801/50000 [12:35:01<26:55:21,  3.32s/it]
Outer training loop:  42%|####1     | 20802/50000 [12:35:03<23:17:57,  2.87s/it]
Outer training loop:  42%|####1     | 20803/50000 [12:35:04<20:31:02,  2.53s/it]
Outer training loop:  42%|####1     | 20804/50000 [12:35:06<18:31:15,  2.28s/it]
Outer training loop:  42%|####1     | 20805/50000 [12:35:08<17:37:44,  2.17s/it]
Outer training loop:  42%|####1     | 20806/50000 [12:35:10<16:20:55,  2.02s/it]
Outer training loop:  42%|####1     | 20807/50000 [12:35:11<15:47:19,  1.95s/it]
Outer training loop:  42%|####1     | 20808/50000 [12:35:13<15:33:28,  1.92s/it]
Outer training loop:  42%|####1     | 20809/50000 [12:35:15<15:10:25,  1.87s/it]
Outer training loop:  42%|####1     | 20810/50000 [12:35:17<14:46:16,  1.82s/it]
Outer training loop:  42%|####1     | 20811/50000 [12:35:19<14:56:00,  1.84s/it]
Outer training loop:  42%|####1     | 20812/50000 [12:35:20<14:42:25,  1.81s/it]
Outer training loop:  42%|####1     | 20813/50000 [12:35:22<14:20:08,  1.77s/it]
Outer training loop:  42%|####1     | 20814/50000 [12:35:24<14:28:16,  1.78s/it]
Outer training loop:  42%|####1     | 20815/50000 [12:35:26<14:37:51,  1.80s/it]
Outer training loop:  42%|####1     | 20816/50000 [12:35:27<14:11:00,  1.75s/it]
Outer training loop:  42%|####1     | 20817/50000 [12:35:29<14:26:43,  1.78s/it]
Outer training loop:  42%|####1     | 20818/50000 [12:35:31<14:42:00,  1.81s/it]
Outer training loop:  42%|####1     | 20819/50000 [12:35:33<14:32:37,  1.79s/it]
Outer training loop:  42%|####1     | 20820/50000 [12:35:35<14:38:11,  1.81s/it]
Outer training loop:  42%|####1     | 20821/50000 [12:35:37<14:57:25,  1.85s/it]
Outer training loop:  42%|####1     | 20822/50000 [12:35:38<14:21:31,  1.77s/it]
Outer training loop:  42%|####1     | 20823/50000 [12:35:40<14:28:07,  1.79s/it]
Outer training loop:  42%|####1     | 20824/50000 [12:35:42<14:55:19,  1.84s/it]
Outer training loop:  42%|####1     | 20825/50000 [12:35:44<14:44:57,  1.82s/it]
Outer training loop:  42%|####1     | 20826/50000 [12:35:45<14:22:05,  1.77s/it]
Outer training loop:  42%|####1     | 20827/50000 [12:35:47<14:10:01,  1.75s/it]
Outer training loop:  42%|####1     | 20828/50000 [12:35:49<14:47:55,  1.83s/it]
Outer training loop:  42%|####1     | 20829/50000 [12:35:51<14:49:32,  1.83s/it]
Outer training loop:  42%|####1     | 20830/50000 [12:35:53<14:29:49,  1.79s/it]
Outer training loop:  42%|####1     | 20831/50000 [12:35:55<15:00:01,  1.85s/it]
Outer training loop:  42%|####1     | 20832/50000 [12:35:56<14:38:35,  1.81s/it]
Outer training loop:  42%|####1     | 20833/50000 [12:35:58<14:28:17,  1.79s/it]
Outer training loop:  42%|####1     | 20834/50000 [12:36:00<14:38:01,  1.81s/it]
Outer training loop:  42%|####1     | 20835/50000 [12:36:02<14:29:08,  1.79s/it]
Outer training loop:  42%|####1     | 20836/50000 [12:36:03<14:11:19,  1.75s/it]
Outer training loop:  42%|####1     | 20837/50000 [12:36:05<13:59:32,  1.73s/it]
Outer training loop:  42%|####1     | 20838/50000 [12:36:07<14:24:19,  1.78s/it]
Outer training loop:  42%|####1     | 20839/50000 [12:36:09<14:23:52,  1.78s/it]
Outer training loop:  42%|####1     | 20840/50000 [12:36:11<14:38:29,  1.81s/it]
Outer training loop:  42%|####1     | 20841/50000 [12:36:12<14:51:35,  1.83s/it]
Outer training loop:  42%|####1     | 20842/50000 [12:36:14<14:23:44,  1.78s/it]
Outer training loop:  42%|####1     | 20843/50000 [12:36:16<14:06:19,  1.74s/it]
Outer training loop:  42%|####1     | 20844/50000 [12:36:17<14:06:28,  1.74s/it]
Outer training loop:  42%|####1     | 20845/50000 [12:36:19<14:40:11,  1.81s/it]
Outer training loop:  42%|####1     | 20846/50000 [12:36:21<14:45:10,  1.82s/it]
Outer training loop:  42%|####1     | 20847/50000 [12:36:23<14:22:17,  1.77s/it]
Outer training loop:  42%|####1     | 20848/50000 [12:36:25<14:43:04,  1.82s/it]
Outer training loop:  42%|####1     | 20849/50000 [12:36:27<14:26:53,  1.78s/it]
Outer training loop:  42%|####1     | 20850/50000 [12:36:28<14:14:50,  1.76s/it]
Outer training loop:  42%|####1     | 20851/50000 [12:36:30<14:08:24,  1.75s/it]
Outer training loop:  42%|####1     | 20852/50000 [12:36:32<14:38:13,  1.81s/it]
Outer training loop:  42%|####1     | 20853/50000 [12:36:34<14:21:04,  1.77s/it]
Outer training loop:  42%|####1     | 20854/50000 [12:36:35<14:13:57,  1.76s/it]
Outer training loop:  42%|####1     | 20855/50000 [12:36:37<14:36:44,  1.80s/it]
Outer training loop:  42%|####1     | 20856/50000 [12:36:39<14:05:21,  1.74s/it]
Outer training loop:  42%|####1     | 20857/50000 [12:36:41<14:09:31,  1.75s/it]
Outer training loop:  42%|####1     | 20858/50000 [12:36:42<14:04:24,  1.74s/it]
Outer training loop:  42%|####1     | 20859/50000 [12:36:44<14:27:55,  1.79s/it]
Outer training loop:  42%|####1     | 20860/50000 [12:36:46<14:09:50,  1.75s/it]
Outer training loop:  42%|####1     | 20861/50000 [12:36:48<14:06:42,  1.74s/it]
Outer training loop:  42%|####1     | 20862/50000 [12:36:50<14:37:54,  1.81s/it]
Outer training loop:  42%|####1     | 20863/50000 [12:36:51<14:20:36,  1.77s/it]
Outer training loop:  42%|####1     | 20864/50000 [12:36:53<14:08:31,  1.75s/it]
Outer training loop:  42%|####1     | 20865/50000 [12:36:55<14:31:47,  1.80s/it]
Outer training loop:  42%|####1     | 20866/50000 [12:36:57<14:17:47,  1.77s/it]
Outer training loop:  42%|####1     | 20867/50000 [12:36:58<14:05:13,  1.74s/it]
Outer training loop:  42%|####1     | 20868/50000 [12:37:00<14:23:49,  1.78s/it]
Outer training loop:  42%|####1     | 20869/50000 [12:37:02<14:25:56,  1.78s/it]
Outer training loop:  42%|####1     | 20870/50000 [12:37:04<14:25:48,  1.78s/it]
Outer training loop:  42%|####1     | 20871/50000 [12:37:05<14:23:32,  1.78s/it]
Outer training loop:  42%|####1     | 20872/50000 [12:37:08<15:14:57,  1.88s/it]
Outer training loop:  42%|####1     | 20873/50000 [12:37:10<15:34:29,  1.93s/it]
Outer training loop:  42%|####1     | 20874/50000 [12:37:12<15:35:06,  1.93s/it]
Outer training loop:  42%|####1     | 20875/50000 [12:37:14<15:40:40,  1.94s/it]
Outer training loop:  42%|####1     | 20876/50000 [12:37:15<15:23:20,  1.90s/it]
Outer training loop:  42%|####1     | 20877/50000 [12:37:17<15:10:38,  1.88s/it]
Outer training loop:  42%|####1     | 20878/50000 [12:37:19<15:17:51,  1.89s/it]
Outer training loop:  42%|####1     | 20879/50000 [12:37:21<15:47:47,  1.95s/it]
Outer training loop:  42%|####1     | 20880/50000 [12:37:23<15:24:56,  1.91s/it]
Outer training loop:  42%|####1     | 20881/50000 [12:37:25<15:34:11,  1.92s/it]
Outer training loop:  42%|####1     | 20882/50000 [12:37:27<15:45:54,  1.95s/it]
Outer training loop:  42%|####1     | 20883/50000 [12:37:29<15:32:29,  1.92s/it]
Outer training loop:  42%|####1     | 20884/50000 [12:37:30<14:52:49,  1.84s/it]
Outer training loop:  42%|####1     | 20885/50000 [12:37:32<14:59:52,  1.85s/it]
Outer training loop:  42%|####1     | 20886/50000 [12:37:34<14:58:10,  1.85s/it]
Outer training loop:  42%|####1     | 20887/50000 [12:37:36<15:00:59,  1.86s/it]
Outer training loop:  42%|####1     | 20888/50000 [12:37:38<15:26:22,  1.91s/it]
Outer training loop:  42%|####1     | 20889/50000 [12:37:40<16:11:43,  2.00s/it]
Outer training loop:  42%|####1     | 20890/50000 [12:37:42<15:50:06,  1.96s/it]
Outer training loop:  42%|####1     | 20891/50000 [12:37:44<15:24:58,  1.91s/it]
Outer training loop:  42%|####1     | 20892/50000 [12:37:46<14:57:55,  1.85s/it]
Outer training loop:  42%|####1     | 20893/50000 [12:37:48<15:21:21,  1.90s/it]
Outer training loop:  42%|####1     | 20894/50000 [12:37:49<14:56:07,  1.85s/it]
Outer training loop:  42%|####1     | 20895/50000 [12:37:51<15:05:58,  1.87s/it]
Outer training loop:  42%|####1     | 20896/50000 [12:37:53<15:30:53,  1.92s/it]
Outer training loop:  42%|####1     | 20897/50000 [12:37:55<15:08:12,  1.87s/it]
Outer training loop:  42%|####1     | 20898/50000 [12:37:57<14:53:01,  1.84s/it]
Outer training loop:  42%|####1     | 20899/50000 [12:37:59<14:53:17,  1.84s/it]
Outer training loop:  42%|####1     | 20900/50000 [12:38:01<14:57:28,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.70s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.71s/it]

Outer training loop:  42%|####1     | 20901/50000 [12:38:08<27:37:58,  3.42s/it]
Outer training loop:  42%|####1     | 20902/50000 [12:38:09<23:30:04,  2.91s/it]
Outer training loop:  42%|####1     | 20903/50000 [12:38:11<21:17:05,  2.63s/it]
Outer training loop:  42%|####1     | 20904/50000 [12:38:13<19:51:35,  2.46s/it]
Outer training loop:  42%|####1     | 20905/50000 [12:38:15<18:15:06,  2.26s/it]
Outer training loop:  42%|####1     | 20906/50000 [12:38:17<17:21:53,  2.15s/it]
Outer training loop:  42%|####1     | 20907/50000 [12:38:19<16:48:12,  2.08s/it]
Outer training loop:  42%|####1     | 20908/50000 [12:38:21<15:51:28,  1.96s/it]
Outer training loop:  42%|####1     | 20909/50000 [12:38:22<15:21:46,  1.90s/it]
Outer training loop:  42%|####1     | 20910/50000 [12:38:25<15:43:33,  1.95s/it]
Outer training loop:  42%|####1     | 20911/50000 [12:38:26<15:29:50,  1.92s/it]
Outer training loop:  42%|####1     | 20912/50000 [12:38:28<15:08:28,  1.87s/it]
Outer training loop:  42%|####1     | 20913/50000 [12:38:30<14:50:56,  1.84s/it]
Outer training loop:  42%|####1     | 20914/50000 [12:38:32<14:57:38,  1.85s/it]
Outer training loop:  42%|####1     | 20915/50000 [12:38:34<15:01:19,  1.86s/it]
Outer training loop:  42%|####1     | 20916/50000 [12:38:35<14:53:06,  1.84s/it]
Outer training loop:  42%|####1     | 20917/50000 [12:38:38<15:24:33,  1.91s/it]
Outer training loop:  42%|####1     | 20918/50000 [12:38:39<15:12:37,  1.88s/it]
Outer training loop:  42%|####1     | 20919/50000 [12:38:41<15:15:01,  1.89s/it]
Outer training loop:  42%|####1     | 20920/50000 [12:38:43<15:36:00,  1.93s/it]
Outer training loop:  42%|####1     | 20921/50000 [12:38:45<15:05:57,  1.87s/it]
Outer training loop:  42%|####1     | 20922/50000 [12:38:47<14:48:06,  1.83s/it]
Outer training loop:  42%|####1     | 20923/50000 [12:38:49<14:56:33,  1.85s/it]
Outer training loop:  42%|####1     | 20924/50000 [12:38:51<15:08:24,  1.87s/it]
Outer training loop:  42%|####1     | 20925/50000 [12:38:52<14:52:28,  1.84s/it]
Outer training loop:  42%|####1     | 20926/50000 [12:38:54<14:36:20,  1.81s/it]
Outer training loop:  42%|####1     | 20927/50000 [12:38:56<15:14:44,  1.89s/it]
Outer training loop:  42%|####1     | 20928/50000 [12:38:58<15:04:19,  1.87s/it]
Outer training loop:  42%|####1     | 20929/50000 [12:39:00<14:57:56,  1.85s/it]
Outer training loop:  42%|####1     | 20930/50000 [12:39:02<14:45:03,  1.83s/it]
Outer training loop:  42%|####1     | 20931/50000 [12:39:03<15:00:31,  1.86s/it]
Outer training loop:  42%|####1     | 20932/50000 [12:39:05<14:58:44,  1.86s/it]
Outer training loop:  42%|####1     | 20933/50000 [12:39:07<14:53:40,  1.84s/it]
Outer training loop:  42%|####1     | 20934/50000 [12:39:09<15:01:13,  1.86s/it]
Outer training loop:  42%|####1     | 20935/50000 [12:39:11<14:53:18,  1.84s/it]
Outer training loop:  42%|####1     | 20936/50000 [12:39:13<14:31:25,  1.80s/it]
Outer training loop:  42%|####1     | 20937/50000 [12:39:15<14:53:53,  1.85s/it]
Outer training loop:  42%|####1     | 20938/50000 [12:39:16<14:44:50,  1.83s/it]
Outer training loop:  42%|####1     | 20939/50000 [12:39:18<14:26:59,  1.79s/it]
Outer training loop:  42%|####1     | 20940/50000 [12:39:20<14:33:05,  1.80s/it]
Outer training loop:  42%|####1     | 20941/50000 [12:39:22<14:52:56,  1.84s/it]
Outer training loop:  42%|####1     | 20942/50000 [12:39:24<14:59:38,  1.86s/it]
Outer training loop:  42%|####1     | 20943/50000 [12:39:25<14:56:19,  1.85s/it]
Outer training loop:  42%|####1     | 20944/50000 [12:39:27<15:05:11,  1.87s/it]
Outer training loop:  42%|####1     | 20945/50000 [12:39:29<15:03:24,  1.87s/it]
Outer training loop:  42%|####1     | 20946/50000 [12:39:31<14:54:33,  1.85s/it]
Outer training loop:  42%|####1     | 20947/50000 [12:39:33<15:14:00,  1.89s/it]
Outer training loop:  42%|####1     | 20948/50000 [12:39:35<14:55:00,  1.85s/it]
Outer training loop:  42%|####1     | 20949/50000 [12:39:37<14:38:54,  1.82s/it]
Outer training loop:  42%|####1     | 20950/50000 [12:39:39<15:09:08,  1.88s/it]
Outer training loop:  42%|####1     | 20951/50000 [12:39:41<15:18:05,  1.90s/it]
Outer training loop:  42%|####1     | 20952/50000 [12:39:42<15:06:12,  1.87s/it]
Outer training loop:  42%|####1     | 20953/50000 [12:39:44<14:50:35,  1.84s/it]
Outer training loop:  42%|####1     | 20954/50000 [12:39:46<15:01:41,  1.86s/it]
Outer training loop:  42%|####1     | 20955/50000 [12:39:48<14:55:09,  1.85s/it]
Outer training loop:  42%|####1     | 20956/50000 [12:39:50<14:45:32,  1.83s/it]
Outer training loop:  42%|####1     | 20957/50000 [12:39:52<14:58:30,  1.86s/it]
Outer training loop:  42%|####1     | 20958/50000 [12:39:53<15:02:23,  1.86s/it]
Outer training loop:  42%|####1     | 20959/50000 [12:39:55<14:55:24,  1.85s/it]
Outer training loop:  42%|####1     | 20960/50000 [12:39:57<15:18:37,  1.90s/it]
Outer training loop:  42%|####1     | 20961/50000 [12:39:59<14:55:26,  1.85s/it]
Outer training loop:  42%|####1     | 20962/50000 [12:40:01<14:28:19,  1.79s/it]
Outer training loop:  42%|####1     | 20963/50000 [12:40:02<14:21:50,  1.78s/it]
Outer training loop:  42%|####1     | 20964/50000 [12:40:04<14:59:31,  1.86s/it]
Outer training loop:  42%|####1     | 20965/50000 [12:40:06<15:12:17,  1.89s/it]
Outer training loop:  42%|####1     | 20966/50000 [12:40:08<15:25:52,  1.91s/it]
Outer training loop:  42%|####1     | 20967/50000 [12:40:10<15:38:27,  1.94s/it]
Outer training loop:  42%|####1     | 20968/50000 [12:40:12<15:19:08,  1.90s/it]
Outer training loop:  42%|####1     | 20969/50000 [12:40:14<15:14:57,  1.89s/it]
Outer training loop:  42%|####1     | 20970/50000 [12:40:16<15:34:01,  1.93s/it]
Outer training loop:  42%|####1     | 20971/50000 [12:40:18<15:05:25,  1.87s/it]
Outer training loop:  42%|####1     | 20972/50000 [12:40:20<14:50:01,  1.84s/it]
Outer training loop:  42%|####1     | 20973/50000 [12:40:21<14:42:47,  1.82s/it]
Outer training loop:  42%|####1     | 20974/50000 [12:40:23<15:13:39,  1.89s/it]
Outer training loop:  42%|####1     | 20975/50000 [12:40:25<15:08:36,  1.88s/it]
Outer training loop:  42%|####1     | 20976/50000 [12:40:27<15:11:53,  1.89s/it]
Outer training loop:  42%|####1     | 20977/50000 [12:40:29<15:28:18,  1.92s/it]
Outer training loop:  42%|####1     | 20978/50000 [12:40:31<15:00:29,  1.86s/it]
Outer training loop:  42%|####1     | 20979/50000 [12:40:33<14:45:48,  1.83s/it]
Outer training loop:  42%|####1     | 20980/50000 [12:40:34<14:46:57,  1.83s/it]
Outer training loop:  42%|####1     | 20981/50000 [12:40:37<15:20:29,  1.90s/it]
Outer training loop:  42%|####1     | 20982/50000 [12:40:38<15:07:20,  1.88s/it]
Outer training loop:  42%|####1     | 20983/50000 [12:40:40<14:40:43,  1.82s/it]
Outer training loop:  42%|####1     | 20984/50000 [12:40:42<14:37:42,  1.81s/it]
Outer training loop:  42%|####1     | 20985/50000 [12:40:44<15:13:54,  1.89s/it]
Outer training loop:  42%|####1     | 20986/50000 [12:40:46<14:43:30,  1.83s/it]
Outer training loop:  42%|####1     | 20987/50000 [12:40:47<14:32:20,  1.80s/it]
Outer training loop:  42%|####1     | 20988/50000 [12:40:49<15:03:48,  1.87s/it]
Outer training loop:  42%|####1     | 20989/50000 [12:40:51<14:52:28,  1.85s/it]
Outer training loop:  42%|####1     | 20990/50000 [12:40:53<14:39:52,  1.82s/it]
Outer training loop:  42%|####1     | 20991/50000 [12:40:55<15:05:44,  1.87s/it]
Outer training loop:  42%|####1     | 20992/50000 [12:40:57<14:55:31,  1.85s/it]
Outer training loop:  42%|####1     | 20993/50000 [12:40:59<14:48:09,  1.84s/it]
Outer training loop:  42%|####1     | 20994/50000 [12:41:00<14:34:35,  1.81s/it]
Outer training loop:  42%|####1     | 20995/50000 [12:41:02<14:52:02,  1.85s/it]
Outer training loop:  42%|####1     | 20996/50000 [12:41:04<14:43:52,  1.83s/it]
Outer training loop:  42%|####1     | 20997/50000 [12:41:06<14:35:27,  1.81s/it]
Outer training loop:  42%|####1     | 20998/50000 [12:41:08<14:42:41,  1.83s/it]
Outer training loop:  42%|####1     | 20999/50000 [12:41:09<14:33:02,  1.81s/it]
Outer training loop:  42%|####2     | 21000/50000 [12:41:11<14:18:31,  1.78s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:45,  2.02s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:38,  1.96s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.90s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.86s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:30,  1.93s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:21,  1.86s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:20,  1.90s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:15,  1.85s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:05,  1.84s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:00,  1.80s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:03,  1.87s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:58,  1.82s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:54,  1.79s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:53,  1.79s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:40<01:53,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:42<01:52,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:46,  1.84s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:45,  1.88s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:42,  1.86s/it]

Evaluating policy:  35%|###4      | 29/83 [00:53<01:37,  1.80s/it]

Evaluating policy:  36%|###6      | 30/83 [00:55<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:57<01:40,  1.93s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:35,  1.87s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:32,  1.85s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:29,  1.82s/it]

Evaluating policy:  42%|####2     | 35/83 [01:04<01:24,  1.77s/it]

Evaluating policy:  43%|####3     | 36/83 [01:06<01:23,  1.78s/it]

Evaluating policy:  45%|####4     | 37/83 [01:08<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:10<01:22,  1.82s/it]

Evaluating policy:  47%|####6     | 39/83 [01:12<01:20,  1.82s/it]

Evaluating policy:  48%|####8     | 40/83 [01:13<01:16,  1.78s/it]

Evaluating policy:  49%|####9     | 41/83 [01:15<01:13,  1.75s/it]

Evaluating policy:  51%|#####     | 42/83 [01:17<01:12,  1.77s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:19<01:10,  1.75s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:21<01:12,  1.87s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:23<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:25<01:08,  1.86s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:26<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:28<01:06,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:30<01:04,  1.89s/it]

Evaluating policy:  60%|######    | 50/83 [01:32<01:01,  1.88s/it]

Evaluating policy:  61%|######1   | 51/83 [01:34<01:02,  1.96s/it]

Evaluating policy:  63%|######2   | 52/83 [01:36<00:59,  1.92s/it]

Evaluating policy:  64%|######3   | 53/83 [01:38<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:39<00:52,  1.80s/it]

Evaluating policy:  66%|######6   | 55/83 [01:41<00:50,  1.80s/it]

Evaluating policy:  67%|######7   | 56/83 [01:43<00:48,  1.81s/it]

Evaluating policy:  69%|######8   | 57/83 [01:45<00:47,  1.81s/it]

Evaluating policy:  70%|######9   | 58/83 [01:47<00:45,  1.81s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:49<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:51<00:43,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:53<00:42,  1.93s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:54<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:56<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:58<00:33,  1.79s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:00<00:31,  1.75s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:01<00:30,  1.77s/it]

Evaluating policy:  81%|########  | 67/83 [02:03<00:30,  1.88s/it]

Evaluating policy:  82%|########1 | 68/83 [02:05<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:07<00:25,  1.79s/it]

Evaluating policy:  84%|########4 | 70/83 [02:09<00:23,  1.80s/it]

Evaluating policy:  86%|########5 | 71/83 [02:10<00:21,  1.77s/it]

Evaluating policy:  87%|########6 | 72/83 [02:12<00:19,  1.75s/it]

Evaluating policy:  88%|########7 | 73/83 [02:14<00:17,  1.74s/it]

Evaluating policy:  89%|########9 | 74/83 [02:16<00:15,  1.77s/it]

Evaluating policy:  90%|######### | 75/83 [02:17<00:14,  1.76s/it]

Evaluating policy:  92%|#########1| 76/83 [02:20<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:22<00:11,  1.89s/it]

Evaluating policy:  94%|#########3| 78/83 [02:23<00:09,  1.84s/it]

Evaluating policy:  95%|#########5| 79/83 [02:25<00:07,  1.87s/it]

Evaluating policy:  96%|#########6| 80/83 [02:27<00:05,  1.83s/it]

Evaluating policy:  98%|#########7| 81/83 [02:29<00:03,  1.82s/it]

Evaluating policy:  99%|#########8| 82/83 [02:31<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:32<00:00,  1.76s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:19,  1.71s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:31,  1.86s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:25,  1.82s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:24,  1.83s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:18,  1.78s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:20,  1.82s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:13,  1.78s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:10,  1.77s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:12,  1.82s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:10,  1.81s/it]

Evaluating policy:  14%|#4        | 12/83 [00:21<02:07,  1.80s/it]

Evaluating policy:  16%|#5        | 13/83 [00:23<02:03,  1.76s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:05,  1.81s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:01,  1.78s/it]

Evaluating policy:  19%|#9        | 16/83 [00:28<01:59,  1.79s/it]

Evaluating policy:  20%|##        | 17/83 [00:30<01:54,  1.74s/it]

Evaluating policy:  22%|##1       | 18/83 [00:32<01:52,  1.72s/it]

Evaluating policy:  23%|##2       | 19/83 [00:34<01:55,  1.81s/it]

Evaluating policy:  24%|##4       | 20/83 [00:35<01:51,  1.78s/it]

Evaluating policy:  25%|##5       | 21/83 [00:37<01:47,  1.74s/it]

Evaluating policy:  27%|##6       | 22/83 [00:39<01:45,  1.72s/it]

Evaluating policy:  28%|##7       | 23/83 [00:41<01:47,  1.80s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:45,  1.78s/it]

Evaluating policy:  30%|###       | 25/83 [00:44<01:42,  1.76s/it]

Evaluating policy:  31%|###1      | 26/83 [00:46<01:39,  1.75s/it]

Evaluating policy:  33%|###2      | 27/83 [00:48<01:37,  1.74s/it]

Evaluating policy:  34%|###3      | 28/83 [00:50<01:38,  1.80s/it]

Evaluating policy:  35%|###4      | 29/83 [00:52<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:53<01:36,  1.82s/it]

Evaluating policy:  37%|###7      | 31/83 [00:55<01:33,  1.79s/it]

Evaluating policy:  39%|###8      | 32/83 [00:57<01:30,  1.78s/it]

Evaluating policy:  40%|###9      | 33/83 [00:59<01:28,  1.77s/it]

Evaluating policy:  41%|####      | 34/83 [01:01<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:02<01:27,  1.81s/it]

Evaluating policy:  43%|####3     | 36/83 [01:04<01:26,  1.84s/it]

Evaluating policy:  45%|####4     | 37/83 [01:06<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:08<01:19,  1.77s/it]

Evaluating policy:  47%|####6     | 39/83 [01:09<01:17,  1.75s/it]

Evaluating policy:  48%|####8     | 40/83 [01:12<01:20,  1.87s/it]

Evaluating policy:  49%|####9     | 41/83 [01:13<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:15<01:14,  1.82s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:17<01:10,  1.77s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:19<01:09,  1.78s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:20<01:07,  1.79s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:22<01:05,  1.76s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:24<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:26<01:02,  1.79s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:27<01:00,  1.77s/it]

Evaluating policy:  60%|######    | 50/83 [01:29<00:57,  1.75s/it]

Evaluating policy:  61%|######1   | 51/83 [01:31<00:55,  1.75s/it]

Evaluating policy:  63%|######2   | 52/83 [01:33<00:53,  1.72s/it]

Evaluating policy:  64%|######3   | 53/83 [01:34<00:51,  1.72s/it]

Evaluating policy:  65%|######5   | 54/83 [01:36<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:38<00:50,  1.80s/it]

Evaluating policy:  67%|######7   | 56/83 [01:40<00:48,  1.78s/it]

Evaluating policy:  69%|######8   | 57/83 [01:42<00:45,  1.76s/it]

Evaluating policy:  70%|######9   | 58/83 [01:43<00:44,  1.78s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:45<00:42,  1.78s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:47<00:40,  1.76s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:49<00:38,  1.73s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:51<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:52<00:35,  1.80s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:54<00:33,  1.76s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:56<00:31,  1.73s/it]

Evaluating policy:  80%|#######9  | 66/83 [01:57<00:29,  1.74s/it]

Evaluating policy:  81%|########  | 67/83 [01:59<00:27,  1.74s/it]

Evaluating policy:  82%|########1 | 68/83 [02:01<00:25,  1.72s/it]

Evaluating policy:  83%|########3 | 69/83 [02:03<00:24,  1.75s/it]

Evaluating policy:  84%|########4 | 70/83 [02:04<00:22,  1.73s/it]

Evaluating policy:  86%|########5 | 71/83 [02:07<00:22,  1.88s/it]

Evaluating policy:  87%|########6 | 72/83 [02:08<00:20,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:10<00:17,  1.79s/it]

Evaluating policy:  89%|########9 | 74/83 [02:12<00:15,  1.77s/it]

Evaluating policy:  90%|######### | 75/83 [02:14<00:14,  1.79s/it]

Evaluating policy:  92%|#########1| 76/83 [02:15<00:12,  1.76s/it]

Evaluating policy:  93%|#########2| 77/83 [02:17<00:10,  1.74s/it]

Evaluating policy:  94%|#########3| 78/83 [02:19<00:08,  1.77s/it]

Evaluating policy:  95%|#########5| 79/83 [02:21<00:07,  1.78s/it]

Evaluating policy:  96%|#########6| 80/83 [02:23<00:05,  1.94s/it]

Evaluating policy:  98%|#########7| 81/83 [02:25<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:26<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 83/83 [02:28<00:00,  1.88s/it]

Outer training loop:  42%|####2     | 21001/50000 [12:46:20<756:10:24, 93.87s/it]
Outer training loop:  42%|####2     | 21002/50000 [12:46:22<533:36:02, 66.24s/it]
Outer training loop:  42%|####2     | 21003/50000 [12:46:23<377:48:02, 46.90s/it]
Outer training loop:  42%|####2     | 21004/50000 [12:46:25<268:51:11, 33.38s/it]
Outer training loop:  42%|####2     | 21005/50000 [12:46:27<192:16:04, 23.87s/it]
Outer training loop:  42%|####2     | 21006/50000 [12:46:29<138:51:00, 17.24s/it]
Outer training loop:  42%|####2     | 21007/50000 [12:46:30<101:27:02, 12.60s/it]
Outer training loop:  42%|####2     | 21008/50000 [12:46:32<75:18:45,  9.35s/it] 
Outer training loop:  42%|####2     | 21009/50000 [12:46:35<58:38:08,  7.28s/it]
Outer training loop:  42%|####2     | 21010/50000 [12:46:36<45:13:46,  5.62s/it]
Outer training loop:  42%|####2     | 21011/50000 [12:46:38<35:44:23,  4.44s/it]
Outer training loop:  42%|####2     | 21012/50000 [12:46:40<30:02:18,  3.73s/it]
Outer training loop:  42%|####2     | 21013/50000 [12:46:42<25:17:52,  3.14s/it]
Outer training loop:  42%|####2     | 21014/50000 [12:46:44<22:14:14,  2.76s/it]
Outer training loop:  42%|####2     | 21015/50000 [12:46:46<20:31:12,  2.55s/it]
Outer training loop:  42%|####2     | 21016/50000 [12:46:48<18:53:15,  2.35s/it]
Outer training loop:  42%|####2     | 21017/50000 [12:46:50<17:53:57,  2.22s/it]
Outer training loop:  42%|####2     | 21018/50000 [12:46:52<17:17:34,  2.15s/it]
Outer training loop:  42%|####2     | 21019/50000 [12:46:53<16:21:18,  2.03s/it]
Outer training loop:  42%|####2     | 21020/50000 [12:46:55<16:06:18,  2.00s/it]
Outer training loop:  42%|####2     | 21021/50000 [12:46:57<16:13:00,  2.01s/it]
Outer training loop:  42%|####2     | 21022/50000 [12:46:59<15:44:52,  1.96s/it]
Outer training loop:  42%|####2     | 21023/50000 [12:47:01<15:23:58,  1.91s/it]
Outer training loop:  42%|####2     | 21024/50000 [12:47:03<15:41:33,  1.95s/it]
Outer training loop:  42%|####2     | 21025/50000 [12:47:05<15:46:44,  1.96s/it]
Outer training loop:  42%|####2     | 21026/50000 [12:47:07<15:30:52,  1.93s/it]
Outer training loop:  42%|####2     | 21027/50000 [12:47:09<15:26:00,  1.92s/it]
Outer training loop:  42%|####2     | 21028/50000 [12:47:11<15:43:16,  1.95s/it]
Outer training loop:  42%|####2     | 21029/50000 [12:47:13<15:15:37,  1.90s/it]
Outer training loop:  42%|####2     | 21030/50000 [12:47:14<15:12:27,  1.89s/it]
Outer training loop:  42%|####2     | 21031/50000 [12:47:17<15:38:48,  1.94s/it]
Outer training loop:  42%|####2     | 21032/50000 [12:47:18<15:29:34,  1.93s/it]
Outer training loop:  42%|####2     | 21033/50000 [12:47:20<15:35:25,  1.94s/it]
Outer training loop:  42%|####2     | 21034/50000 [12:47:22<15:33:08,  1.93s/it]
Outer training loop:  42%|####2     | 21035/50000 [12:47:24<15:07:18,  1.88s/it]
Outer training loop:  42%|####2     | 21036/50000 [12:47:26<14:58:53,  1.86s/it]
Outer training loop:  42%|####2     | 21037/50000 [12:47:28<15:21:48,  1.91s/it]
Outer training loop:  42%|####2     | 21038/50000 [12:47:30<15:05:32,  1.88s/it]
Outer training loop:  42%|####2     | 21039/50000 [12:47:32<14:59:35,  1.86s/it]
Outer training loop:  42%|####2     | 21040/50000 [12:47:34<15:34:50,  1.94s/it]
Outer training loop:  42%|####2     | 21041/50000 [12:47:36<15:32:51,  1.93s/it]
Outer training loop:  42%|####2     | 21042/50000 [12:47:37<15:15:44,  1.90s/it]
Outer training loop:  42%|####2     | 21043/50000 [12:47:39<14:45:33,  1.83s/it]
Outer training loop:  42%|####2     | 21044/50000 [12:47:41<15:13:11,  1.89s/it]
Outer training loop:  42%|####2     | 21045/50000 [12:47:43<15:16:13,  1.90s/it]
Outer training loop:  42%|####2     | 21046/50000 [12:47:45<15:30:42,  1.93s/it]
Outer training loop:  42%|####2     | 21047/50000 [12:47:47<15:58:42,  1.99s/it]
Outer training loop:  42%|####2     | 21048/50000 [12:47:49<15:50:25,  1.97s/it]
Outer training loop:  42%|####2     | 21049/50000 [12:47:51<15:23:27,  1.91s/it]
Outer training loop:  42%|####2     | 21050/50000 [12:47:53<15:27:22,  1.92s/it]
Outer training loop:  42%|####2     | 21051/50000 [12:47:55<15:58:42,  1.99s/it]
Outer training loop:  42%|####2     | 21052/50000 [12:47:57<15:43:23,  1.96s/it]
Outer training loop:  42%|####2     | 21053/50000 [12:47:59<15:40:34,  1.95s/it]
Outer training loop:  42%|####2     | 21054/50000 [12:48:01<15:23:54,  1.92s/it]
Outer training loop:  42%|####2     | 21055/50000 [12:48:02<15:25:31,  1.92s/it]
Outer training loop:  42%|####2     | 21056/50000 [12:48:04<15:37:05,  1.94s/it]
Outer training loop:  42%|####2     | 21057/50000 [12:48:06<15:37:01,  1.94s/it]
Outer training loop:  42%|####2     | 21058/50000 [12:48:08<15:49:38,  1.97s/it]
Outer training loop:  42%|####2     | 21059/50000 [12:48:10<15:33:20,  1.93s/it]
Outer training loop:  42%|####2     | 21060/50000 [12:48:12<15:04:24,  1.88s/it]
Outer training loop:  42%|####2     | 21061/50000 [12:48:14<15:35:55,  1.94s/it]
Outer training loop:  42%|####2     | 21062/50000 [12:48:16<15:27:57,  1.92s/it]
Outer training loop:  42%|####2     | 21063/50000 [12:48:18<15:11:22,  1.89s/it]
Outer training loop:  42%|####2     | 21064/50000 [12:48:20<15:14:31,  1.90s/it]
Outer training loop:  42%|####2     | 21065/50000 [12:48:22<15:46:04,  1.96s/it]
Outer training loop:  42%|####2     | 21066/50000 [12:48:24<15:14:00,  1.90s/it]
Outer training loop:  42%|####2     | 21067/50000 [12:48:25<15:15:48,  1.90s/it]
Outer training loop:  42%|####2     | 21068/50000 [12:48:28<15:40:41,  1.95s/it]
Outer training loop:  42%|####2     | 21069/50000 [12:48:29<15:08:50,  1.88s/it]
Outer training loop:  42%|####2     | 21070/50000 [12:48:31<14:48:31,  1.84s/it]
Outer training loop:  42%|####2     | 21071/50000 [12:48:33<14:49:39,  1.85s/it]
Outer training loop:  42%|####2     | 21072/50000 [12:48:35<15:33:57,  1.94s/it]
Outer training loop:  42%|####2     | 21073/50000 [12:48:37<15:09:13,  1.89s/it]
Outer training loop:  42%|####2     | 21074/50000 [12:48:38<14:32:26,  1.81s/it]
Outer training loop:  42%|####2     | 21075/50000 [12:48:41<15:24:19,  1.92s/it]
Outer training loop:  42%|####2     | 21076/50000 [12:48:42<15:15:11,  1.90s/it]
Outer training loop:  42%|####2     | 21077/50000 [12:48:44<15:07:37,  1.88s/it]
Outer training loop:  42%|####2     | 21078/50000 [12:48:46<15:19:57,  1.91s/it]
Outer training loop:  42%|####2     | 21079/50000 [12:48:48<15:12:34,  1.89s/it]
Outer training loop:  42%|####2     | 21080/50000 [12:48:50<15:23:10,  1.92s/it]
Outer training loop:  42%|####2     | 21081/50000 [12:48:52<15:28:52,  1.93s/it]
Outer training loop:  42%|####2     | 21082/50000 [12:48:54<15:21:01,  1.91s/it]
Outer training loop:  42%|####2     | 21083/50000 [12:48:56<14:59:56,  1.87s/it]
Outer training loop:  42%|####2     | 21084/50000 [12:48:58<15:14:06,  1.90s/it]
Outer training loop:  42%|####2     | 21085/50000 [12:49:00<15:07:20,  1.88s/it]
Outer training loop:  42%|####2     | 21086/50000 [12:49:01<15:05:25,  1.88s/it]
Outer training loop:  42%|####2     | 21087/50000 [12:49:03<14:45:15,  1.84s/it]
Outer training loop:  42%|####2     | 21088/50000 [12:49:05<15:13:27,  1.90s/it]
Outer training loop:  42%|####2     | 21089/50000 [12:49:07<15:05:56,  1.88s/it]
Outer training loop:  42%|####2     | 21090/50000 [12:49:09<14:49:35,  1.85s/it]
Outer training loop:  42%|####2     | 21091/50000 [12:49:11<15:11:05,  1.89s/it]
Outer training loop:  42%|####2     | 21092/50000 [12:49:13<14:53:04,  1.85s/it]
Outer training loop:  42%|####2     | 21093/50000 [12:49:14<14:59:53,  1.87s/it]
Outer training loop:  42%|####2     | 21094/50000 [12:49:16<14:56:25,  1.86s/it]
Outer training loop:  42%|####2     | 21095/50000 [12:49:18<15:07:21,  1.88s/it]
Outer training loop:  42%|####2     | 21096/50000 [12:49:20<15:50:26,  1.97s/it]
Outer training loop:  42%|####2     | 21097/50000 [12:49:22<15:40:04,  1.95s/it]
Outer training loop:  42%|####2     | 21098/50000 [12:49:24<15:53:31,  1.98s/it]
Outer training loop:  42%|####2     | 21099/50000 [12:49:26<15:23:47,  1.92s/it]
Outer training loop:  42%|####2     | 21100/50000 [12:49:28<15:14:13,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.98s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  42%|####2     | 21101/50000 [12:49:35<28:37:46,  3.57s/it]
Outer training loop:  42%|####2     | 21102/50000 [12:49:38<25:06:45,  3.13s/it]
Outer training loop:  42%|####2     | 21103/50000 [12:49:39<21:56:01,  2.73s/it]
Outer training loop:  42%|####2     | 21104/50000 [12:49:41<19:40:14,  2.45s/it]
Outer training loop:  42%|####2     | 21105/50000 [12:49:43<18:34:28,  2.31s/it]
Outer training loop:  42%|####2     | 21106/50000 [12:49:45<17:52:50,  2.23s/it]
Outer training loop:  42%|####2     | 21107/50000 [12:49:47<16:49:02,  2.10s/it]
Outer training loop:  42%|####2     | 21108/50000 [12:49:49<16:05:31,  2.01s/it]
Outer training loop:  42%|####2     | 21109/50000 [12:49:51<16:06:36,  2.01s/it]
Outer training loop:  42%|####2     | 21110/50000 [12:49:53<15:43:20,  1.96s/it]
Outer training loop:  42%|####2     | 21111/50000 [12:49:54<15:19:25,  1.91s/it]
Outer training loop:  42%|####2     | 21112/50000 [12:49:56<15:06:19,  1.88s/it]
Outer training loop:  42%|####2     | 21113/50000 [12:49:58<15:58:30,  1.99s/it]
Outer training loop:  42%|####2     | 21114/50000 [12:50:00<15:40:10,  1.95s/it]
Outer training loop:  42%|####2     | 21115/50000 [12:50:02<15:30:10,  1.93s/it]
Outer training loop:  42%|####2     | 21116/50000 [12:50:04<15:24:10,  1.92s/it]
Outer training loop:  42%|####2     | 21117/50000 [12:50:06<15:36:17,  1.94s/it]
Outer training loop:  42%|####2     | 21118/50000 [12:50:08<15:16:59,  1.90s/it]
Outer training loop:  42%|####2     | 21119/50000 [12:50:10<15:15:58,  1.90s/it]
Outer training loop:  42%|####2     | 21120/50000 [12:50:12<15:29:07,  1.93s/it]
Outer training loop:  42%|####2     | 21121/50000 [12:50:14<15:24:05,  1.92s/it]
Outer training loop:  42%|####2     | 21122/50000 [12:50:16<15:08:14,  1.89s/it]
Outer training loop:  42%|####2     | 21123/50000 [12:50:17<15:21:10,  1.91s/it]
Outer training loop:  42%|####2     | 21124/50000 [12:50:19<15:20:44,  1.91s/it]
Outer training loop:  42%|####2     | 21125/50000 [12:50:21<14:58:57,  1.87s/it]
Outer training loop:  42%|####2     | 21126/50000 [12:50:23<15:27:47,  1.93s/it]
Outer training loop:  42%|####2     | 21127/50000 [12:50:25<15:05:47,  1.88s/it]
Outer training loop:  42%|####2     | 21128/50000 [12:50:27<14:50:20,  1.85s/it]
Outer training loop:  42%|####2     | 21129/50000 [12:50:29<14:54:59,  1.86s/it]
Outer training loop:  42%|####2     | 21130/50000 [12:50:31<15:02:27,  1.88s/it]
Outer training loop:  42%|####2     | 21131/50000 [12:50:32<14:59:46,  1.87s/it]
Outer training loop:  42%|####2     | 21132/50000 [12:50:34<14:59:41,  1.87s/it]
Outer training loop:  42%|####2     | 21133/50000 [12:50:36<15:29:06,  1.93s/it]
Outer training loop:  42%|####2     | 21134/50000 [12:50:38<15:20:00,  1.91s/it]
Outer training loop:  42%|####2     | 21135/50000 [12:50:40<15:17:27,  1.91s/it]
Outer training loop:  42%|####2     | 21136/50000 [12:50:42<15:42:12,  1.96s/it]
Outer training loop:  42%|####2     | 21137/50000 [12:50:44<15:19:41,  1.91s/it]
Outer training loop:  42%|####2     | 21138/50000 [12:50:46<14:55:31,  1.86s/it]
Outer training loop:  42%|####2     | 21139/50000 [12:50:48<15:23:08,  1.92s/it]
Outer training loop:  42%|####2     | 21140/50000 [12:50:50<15:44:42,  1.96s/it]
Outer training loop:  42%|####2     | 21141/50000 [12:50:52<15:12:40,  1.90s/it]
Outer training loop:  42%|####2     | 21142/50000 [12:50:53<14:49:31,  1.85s/it]
Outer training loop:  42%|####2     | 21143/50000 [12:50:55<15:27:36,  1.93s/it]
Outer training loop:  42%|####2     | 21144/50000 [12:50:57<15:06:38,  1.89s/it]
Outer training loop:  42%|####2     | 21145/50000 [12:50:59<15:01:17,  1.87s/it]
Outer training loop:  42%|####2     | 21146/50000 [12:51:01<15:10:16,  1.89s/it]
Outer training loop:  42%|####2     | 21147/50000 [12:51:03<15:04:20,  1.88s/it]
Outer training loop:  42%|####2     | 21148/50000 [12:51:05<15:12:37,  1.90s/it]
Outer training loop:  42%|####2     | 21149/50000 [12:51:07<15:16:04,  1.91s/it]
Outer training loop:  42%|####2     | 21150/50000 [12:51:09<15:11:01,  1.89s/it]
Outer training loop:  42%|####2     | 21151/50000 [12:51:10<15:06:28,  1.89s/it]
Outer training loop:  42%|####2     | 21152/50000 [12:51:13<15:45:38,  1.97s/it]
Outer training loop:  42%|####2     | 21153/50000 [12:51:14<15:22:21,  1.92s/it]
Outer training loop:  42%|####2     | 21154/50000 [12:51:16<15:01:11,  1.87s/it]
Outer training loop:  42%|####2     | 21155/50000 [12:51:18<15:01:55,  1.88s/it]
Outer training loop:  42%|####2     | 21156/50000 [12:51:20<15:25:18,  1.92s/it]
Outer training loop:  42%|####2     | 21157/50000 [12:51:22<15:07:07,  1.89s/it]
Outer training loop:  42%|####2     | 21158/50000 [12:51:24<14:55:49,  1.86s/it]
Outer training loop:  42%|####2     | 21159/50000 [12:51:26<15:37:16,  1.95s/it]
Outer training loop:  42%|####2     | 21160/50000 [12:51:28<15:19:22,  1.91s/it]
Outer training loop:  42%|####2     | 21161/50000 [12:51:30<15:03:57,  1.88s/it]
Outer training loop:  42%|####2     | 21162/50000 [12:51:32<15:38:18,  1.95s/it]
Outer training loop:  42%|####2     | 21163/50000 [12:51:33<15:19:54,  1.91s/it]
Outer training loop:  42%|####2     | 21164/50000 [12:51:35<15:10:59,  1.90s/it]
Outer training loop:  42%|####2     | 21165/50000 [12:51:37<14:43:10,  1.84s/it]
Outer training loop:  42%|####2     | 21166/50000 [12:51:39<15:09:05,  1.89s/it]
Outer training loop:  42%|####2     | 21167/50000 [12:51:41<15:09:50,  1.89s/it]
Outer training loop:  42%|####2     | 21168/50000 [12:51:43<14:58:15,  1.87s/it]
Outer training loop:  42%|####2     | 21169/50000 [12:51:45<15:28:08,  1.93s/it]
Outer training loop:  42%|####2     | 21170/50000 [12:51:47<15:01:20,  1.88s/it]
Outer training loop:  42%|####2     | 21171/50000 [12:51:48<15:05:26,  1.88s/it]
Outer training loop:  42%|####2     | 21172/50000 [12:51:51<15:41:24,  1.96s/it]
Outer training loop:  42%|####2     | 21173/50000 [12:51:52<15:20:17,  1.92s/it]
Outer training loop:  42%|####2     | 21174/50000 [12:51:54<15:11:18,  1.90s/it]
Outer training loop:  42%|####2     | 21175/50000 [12:51:56<15:38:27,  1.95s/it]
Outer training loop:  42%|####2     | 21176/50000 [12:51:58<15:23:20,  1.92s/it]
Outer training loop:  42%|####2     | 21177/50000 [12:52:00<15:09:59,  1.89s/it]
Outer training loop:  42%|####2     | 21178/50000 [12:52:02<14:43:59,  1.84s/it]
Outer training loop:  42%|####2     | 21179/50000 [12:52:04<15:29:33,  1.94s/it]
Outer training loop:  42%|####2     | 21180/50000 [12:52:06<15:15:46,  1.91s/it]
Outer training loop:  42%|####2     | 21181/50000 [12:52:08<14:55:46,  1.86s/it]
Outer training loop:  42%|####2     | 21182/50000 [12:52:09<14:57:35,  1.87s/it]
Outer training loop:  42%|####2     | 21183/50000 [12:52:11<15:10:14,  1.90s/it]
Outer training loop:  42%|####2     | 21184/50000 [12:52:13<14:49:31,  1.85s/it]
Outer training loop:  42%|####2     | 21185/50000 [12:52:15<14:48:41,  1.85s/it]
Outer training loop:  42%|####2     | 21186/50000 [12:52:17<15:06:11,  1.89s/it]
Outer training loop:  42%|####2     | 21187/50000 [12:52:19<15:03:39,  1.88s/it]
Outer training loop:  42%|####2     | 21188/50000 [12:52:21<14:58:37,  1.87s/it]
Outer training loop:  42%|####2     | 21189/50000 [12:52:22<14:45:28,  1.84s/it]
Outer training loop:  42%|####2     | 21190/50000 [12:52:24<15:10:00,  1.90s/it]
Outer training loop:  42%|####2     | 21191/50000 [12:52:26<15:18:20,  1.91s/it]
Outer training loop:  42%|####2     | 21192/50000 [12:52:28<15:11:16,  1.90s/it]
Outer training loop:  42%|####2     | 21193/50000 [12:52:30<15:47:11,  1.97s/it]
Outer training loop:  42%|####2     | 21194/50000 [12:52:32<15:20:24,  1.92s/it]
Outer training loop:  42%|####2     | 21195/50000 [12:52:34<15:16:04,  1.91s/it]
Outer training loop:  42%|####2     | 21196/50000 [12:52:36<15:27:34,  1.93s/it]
Outer training loop:  42%|####2     | 21197/50000 [12:52:38<15:17:23,  1.91s/it]
Outer training loop:  42%|####2     | 21198/50000 [12:52:40<15:21:54,  1.92s/it]
Outer training loop:  42%|####2     | 21199/50000 [12:52:42<15:31:55,  1.94s/it]
Outer training loop:  42%|####2     | 21200/50000 [12:52:44<15:12:55,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.95s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.99s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.93s/it]

Outer training loop:  42%|####2     | 21201/50000 [12:52:51<28:54:01,  3.61s/it]
Outer training loop:  42%|####2     | 21202/50000 [12:52:53<24:49:04,  3.10s/it]
Outer training loop:  42%|####2     | 21203/50000 [12:52:55<22:42:00,  2.84s/it]
Outer training loop:  42%|####2     | 21204/50000 [12:52:57<20:34:00,  2.57s/it]
Outer training loop:  42%|####2     | 21205/50000 [12:52:59<19:04:35,  2.38s/it]
Outer training loop:  42%|####2     | 21206/50000 [12:53:01<18:31:09,  2.32s/it]
Outer training loop:  42%|####2     | 21207/50000 [12:53:03<17:23:50,  2.18s/it]
Outer training loop:  42%|####2     | 21208/50000 [12:53:05<16:37:55,  2.08s/it]
Outer training loop:  42%|####2     | 21209/50000 [12:53:07<16:20:21,  2.04s/it]
Outer training loop:  42%|####2     | 21210/50000 [12:53:09<16:05:42,  2.01s/it]
Outer training loop:  42%|####2     | 21211/50000 [12:53:11<16:04:45,  2.01s/it]
Outer training loop:  42%|####2     | 21212/50000 [12:53:13<16:42:08,  2.09s/it]
Outer training loop:  42%|####2     | 21213/50000 [12:53:15<16:21:36,  2.05s/it]
Outer training loop:  42%|####2     | 21214/50000 [12:53:17<15:52:23,  1.99s/it]
Outer training loop:  42%|####2     | 21215/50000 [12:53:19<15:45:14,  1.97s/it]
Outer training loop:  42%|####2     | 21216/50000 [12:53:21<16:07:05,  2.02s/it]
Outer training loop:  42%|####2     | 21217/50000 [12:53:23<15:42:20,  1.96s/it]
Outer training loop:  42%|####2     | 21218/50000 [12:53:25<15:24:32,  1.93s/it]
Outer training loop:  42%|####2     | 21219/50000 [12:53:27<15:38:03,  1.96s/it]
Outer training loop:  42%|####2     | 21220/50000 [12:53:29<15:34:58,  1.95s/it]
Outer training loop:  42%|####2     | 21221/50000 [12:53:31<15:22:44,  1.92s/it]
Outer training loop:  42%|####2     | 21222/50000 [12:53:33<15:17:13,  1.91s/it]
Outer training loop:  42%|####2     | 21223/50000 [12:53:35<15:38:45,  1.96s/it]
Outer training loop:  42%|####2     | 21224/50000 [12:53:36<15:08:03,  1.89s/it]
Outer training loop:  42%|####2     | 21225/50000 [12:53:38<15:01:48,  1.88s/it]
Outer training loop:  42%|####2     | 21226/50000 [12:53:40<15:25:48,  1.93s/it]
Outer training loop:  42%|####2     | 21227/50000 [12:53:42<15:23:58,  1.93s/it]
Outer training loop:  42%|####2     | 21228/50000 [12:53:44<15:17:56,  1.91s/it]
Outer training loop:  42%|####2     | 21229/50000 [12:53:46<15:01:59,  1.88s/it]
Outer training loop:  42%|####2     | 21230/50000 [12:53:48<15:21:09,  1.92s/it]
Outer training loop:  42%|####2     | 21231/50000 [12:53:50<15:07:22,  1.89s/it]
Outer training loop:  42%|####2     | 21232/50000 [12:53:52<14:51:03,  1.86s/it]
Outer training loop:  42%|####2     | 21233/50000 [12:53:54<15:19:43,  1.92s/it]
Outer training loop:  42%|####2     | 21234/50000 [12:53:55<15:01:24,  1.88s/it]
Outer training loop:  42%|####2     | 21235/50000 [12:53:57<15:17:43,  1.91s/it]
Outer training loop:  42%|####2     | 21236/50000 [12:53:59<15:33:04,  1.95s/it]
Outer training loop:  42%|####2     | 21237/50000 [12:54:01<15:25:01,  1.93s/it]
Outer training loop:  42%|####2     | 21238/50000 [12:54:03<15:18:26,  1.92s/it]
Outer training loop:  42%|####2     | 21239/50000 [12:54:05<15:31:51,  1.94s/it]
Outer training loop:  42%|####2     | 21240/50000 [12:54:07<15:38:40,  1.96s/it]
Outer training loop:  42%|####2     | 21241/50000 [12:54:09<15:15:06,  1.91s/it]
Outer training loop:  42%|####2     | 21242/50000 [12:54:11<14:51:43,  1.86s/it]
Outer training loop:  42%|####2     | 21243/50000 [12:54:13<15:27:42,  1.94s/it]
Outer training loop:  42%|####2     | 21244/50000 [12:54:15<15:09:13,  1.90s/it]
Outer training loop:  42%|####2     | 21245/50000 [12:54:17<15:09:07,  1.90s/it]
Outer training loop:  42%|####2     | 21246/50000 [12:54:18<15:03:27,  1.89s/it]
Outer training loop:  42%|####2     | 21247/50000 [12:54:20<15:37:30,  1.96s/it]
Outer training loop:  42%|####2     | 21248/50000 [12:54:22<15:20:47,  1.92s/it]
Outer training loop:  42%|####2     | 21249/50000 [12:54:24<14:56:57,  1.87s/it]
Outer training loop:  42%|####2     | 21250/50000 [12:54:26<15:39:17,  1.96s/it]
Outer training loop:  43%|####2     | 21251/50000 [12:54:28<15:20:39,  1.92s/it]
Outer training loop:  43%|####2     | 21252/50000 [12:54:30<15:21:32,  1.92s/it]
Outer training loop:  43%|####2     | 21253/50000 [12:54:32<15:30:52,  1.94s/it]
Outer training loop:  43%|####2     | 21254/50000 [12:54:34<15:17:01,  1.91s/it]
Outer training loop:  43%|####2     | 21255/50000 [12:54:36<15:09:16,  1.90s/it]
Outer training loop:  43%|####2     | 21256/50000 [12:54:38<15:22:50,  1.93s/it]
Outer training loop:  43%|####2     | 21257/50000 [12:54:40<15:16:25,  1.91s/it]
Outer training loop:  43%|####2     | 21258/50000 [12:54:42<15:25:06,  1.93s/it]
Outer training loop:  43%|####2     | 21259/50000 [12:54:43<15:18:33,  1.92s/it]
Outer training loop:  43%|####2     | 21260/50000 [12:54:45<15:33:31,  1.95s/it]
Outer training loop:  43%|####2     | 21261/50000 [12:54:47<15:29:30,  1.94s/it]
Outer training loop:  43%|####2     | 21262/50000 [12:54:49<15:07:33,  1.89s/it]
Outer training loop:  43%|####2     | 21263/50000 [12:54:51<15:28:08,  1.94s/it]
Outer training loop:  43%|####2     | 21264/50000 [12:54:53<15:17:13,  1.92s/it]
Outer training loop:  43%|####2     | 21265/50000 [12:54:55<15:06:20,  1.89s/it]
Outer training loop:  43%|####2     | 21266/50000 [12:54:57<15:05:39,  1.89s/it]
Outer training loop:  43%|####2     | 21267/50000 [12:54:59<15:19:22,  1.92s/it]
Outer training loop:  43%|####2     | 21268/50000 [12:55:01<14:59:48,  1.88s/it]
Outer training loop:  43%|####2     | 21269/50000 [12:55:02<14:57:06,  1.87s/it]
Outer training loop:  43%|####2     | 21270/50000 [12:55:04<15:10:24,  1.90s/it]
Outer training loop:  43%|####2     | 21271/50000 [12:55:06<15:12:14,  1.91s/it]
Outer training loop:  43%|####2     | 21272/50000 [12:55:08<15:19:45,  1.92s/it]
Outer training loop:  43%|####2     | 21273/50000 [12:55:10<15:20:34,  1.92s/it]
Outer training loop:  43%|####2     | 21274/50000 [12:55:12<15:26:29,  1.94s/it]
Outer training loop:  43%|####2     | 21275/50000 [12:55:14<15:29:48,  1.94s/it]
Outer training loop:  43%|####2     | 21276/50000 [12:55:16<15:07:09,  1.89s/it]
Outer training loop:  43%|####2     | 21277/50000 [12:55:18<15:15:47,  1.91s/it]
Outer training loop:  43%|####2     | 21278/50000 [12:55:20<15:07:47,  1.90s/it]
Outer training loop:  43%|####2     | 21279/50000 [12:55:21<14:47:41,  1.85s/it]
Outer training loop:  43%|####2     | 21280/50000 [12:55:23<15:03:44,  1.89s/it]
Outer training loop:  43%|####2     | 21281/50000 [12:55:25<15:05:32,  1.89s/it]
Outer training loop:  43%|####2     | 21282/50000 [12:55:27<14:55:31,  1.87s/it]
Outer training loop:  43%|####2     | 21283/50000 [12:55:29<15:01:49,  1.88s/it]
Outer training loop:  43%|####2     | 21284/50000 [12:55:31<15:24:18,  1.93s/it]
Outer training loop:  43%|####2     | 21285/50000 [12:55:33<14:56:56,  1.87s/it]
Outer training loop:  43%|####2     | 21286/50000 [12:55:35<15:02:58,  1.89s/it]
Outer training loop:  43%|####2     | 21287/50000 [12:55:37<15:16:33,  1.92s/it]
Outer training loop:  43%|####2     | 21288/50000 [12:55:38<14:46:37,  1.85s/it]
Outer training loop:  43%|####2     | 21289/50000 [12:55:40<14:35:13,  1.83s/it]
Outer training loop:  43%|####2     | 21290/50000 [12:55:42<15:07:17,  1.90s/it]
Outer training loop:  43%|####2     | 21291/50000 [12:55:44<15:26:27,  1.94s/it]
Outer training loop:  43%|####2     | 21292/50000 [12:55:46<15:10:27,  1.90s/it]
Outer training loop:  43%|####2     | 21293/50000 [12:55:48<14:54:53,  1.87s/it]
Outer training loop:  43%|####2     | 21294/50000 [12:55:50<15:26:29,  1.94s/it]
Outer training loop:  43%|####2     | 21295/50000 [12:55:52<15:10:31,  1.90s/it]
Outer training loop:  43%|####2     | 21296/50000 [12:55:54<15:14:19,  1.91s/it]
Outer training loop:  43%|####2     | 21297/50000 [12:55:56<15:36:33,  1.96s/it]
Outer training loop:  43%|####2     | 21298/50000 [12:55:58<15:22:44,  1.93s/it]
Outer training loop:  43%|####2     | 21299/50000 [12:56:00<15:08:01,  1.90s/it]
Outer training loop:  43%|####2     | 21300/50000 [12:56:01<14:50:52,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.88s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  43%|####2     | 21301/50000 [12:56:09<28:49:27,  3.62s/it]
Outer training loop:  43%|####2     | 21302/50000 [12:56:11<24:42:13,  3.10s/it]
Outer training loop:  43%|####2     | 21303/50000 [12:56:13<21:08:21,  2.65s/it]
Outer training loop:  43%|####2     | 21304/50000 [12:56:14<19:02:24,  2.39s/it]
Outer training loop:  43%|####2     | 21305/50000 [12:56:16<18:00:11,  2.26s/it]
Outer training loop:  43%|####2     | 21306/50000 [12:56:18<16:39:49,  2.09s/it]
Outer training loop:  43%|####2     | 21307/50000 [12:56:20<15:58:15,  2.00s/it]
Outer training loop:  43%|####2     | 21308/50000 [12:56:22<16:01:55,  2.01s/it]
Outer training loop:  43%|####2     | 21309/50000 [12:56:24<15:26:14,  1.94s/it]
Outer training loop:  43%|####2     | 21310/50000 [12:56:25<15:16:26,  1.92s/it]
Outer training loop:  43%|####2     | 21311/50000 [12:56:28<15:48:56,  1.98s/it]
Outer training loop:  43%|####2     | 21312/50000 [12:56:29<15:30:14,  1.95s/it]
Outer training loop:  43%|####2     | 21313/50000 [12:56:31<15:11:35,  1.91s/it]
Outer training loop:  43%|####2     | 21314/50000 [12:56:33<15:04:01,  1.89s/it]
Outer training loop:  43%|####2     | 21315/50000 [12:56:35<15:38:35,  1.96s/it]
Outer training loop:  43%|####2     | 21316/50000 [12:56:37<15:19:36,  1.92s/it]
Outer training loop:  43%|####2     | 21317/50000 [12:56:39<15:10:08,  1.90s/it]
Outer training loop:  43%|####2     | 21318/50000 [12:56:41<15:09:27,  1.90s/it]
Outer training loop:  43%|####2     | 21319/50000 [12:56:43<15:02:33,  1.89s/it]
Outer training loop:  43%|####2     | 21320/50000 [12:56:45<15:04:03,  1.89s/it]
Outer training loop:  43%|####2     | 21321/50000 [12:56:46<15:09:47,  1.90s/it]
Outer training loop:  43%|####2     | 21322/50000 [12:56:48<14:45:51,  1.85s/it]
Outer training loop:  43%|####2     | 21323/50000 [12:56:50<14:37:24,  1.84s/it]
Outer training loop:  43%|####2     | 21324/50000 [12:56:52<14:16:41,  1.79s/it]
Outer training loop:  43%|####2     | 21325/50000 [12:56:54<14:43:14,  1.85s/it]
Outer training loop:  43%|####2     | 21326/50000 [12:56:56<14:46:25,  1.85s/it]
Outer training loop:  43%|####2     | 21327/50000 [12:56:57<14:50:14,  1.86s/it]
Outer training loop:  43%|####2     | 21328/50000 [12:56:59<15:07:18,  1.90s/it]
Outer training loop:  43%|####2     | 21329/50000 [12:57:01<14:56:09,  1.88s/it]
Outer training loop:  43%|####2     | 21330/50000 [12:57:03<14:42:39,  1.85s/it]
Outer training loop:  43%|####2     | 21331/50000 [12:57:05<15:18:09,  1.92s/it]
Outer training loop:  43%|####2     | 21332/50000 [12:57:07<14:55:51,  1.87s/it]
Outer training loop:  43%|####2     | 21333/50000 [12:57:09<14:44:25,  1.85s/it]
Outer training loop:  43%|####2     | 21334/50000 [12:57:11<14:57:17,  1.88s/it]
Outer training loop:  43%|####2     | 21335/50000 [12:57:13<15:10:58,  1.91s/it]
Outer training loop:  43%|####2     | 21336/50000 [12:57:14<14:51:44,  1.87s/it]
Outer training loop:  43%|####2     | 21337/50000 [12:57:17<15:32:06,  1.95s/it]
Outer training loop:  43%|####2     | 21338/50000 [12:57:18<15:33:10,  1.95s/it]
Outer training loop:  43%|####2     | 21339/50000 [12:57:20<15:11:12,  1.91s/it]
Outer training loop:  43%|####2     | 21340/50000 [12:57:22<15:07:13,  1.90s/it]
Outer training loop:  43%|####2     | 21341/50000 [12:57:24<15:29:26,  1.95s/it]
Outer training loop:  43%|####2     | 21342/50000 [12:57:26<15:31:05,  1.95s/it]
Outer training loop:  43%|####2     | 21343/50000 [12:57:28<15:20:53,  1.93s/it]
Outer training loop:  43%|####2     | 21344/50000 [12:57:30<14:46:44,  1.86s/it]
Outer training loop:  43%|####2     | 21345/50000 [12:57:32<15:06:11,  1.90s/it]
Outer training loop:  43%|####2     | 21346/50000 [12:57:34<14:48:20,  1.86s/it]
Outer training loop:  43%|####2     | 21347/50000 [12:57:35<14:43:08,  1.85s/it]
Outer training loop:  43%|####2     | 21348/50000 [12:57:37<14:55:26,  1.88s/it]
Outer training loop:  43%|####2     | 21349/50000 [12:57:39<14:45:46,  1.85s/it]
Outer training loop:  43%|####2     | 21350/50000 [12:57:41<14:51:27,  1.87s/it]
Outer training loop:  43%|####2     | 21351/50000 [12:57:43<15:15:52,  1.92s/it]
Outer training loop:  43%|####2     | 21352/50000 [12:57:45<15:10:20,  1.91s/it]
Outer training loop:  43%|####2     | 21353/50000 [12:57:47<15:00:03,  1.89s/it]
Outer training loop:  43%|####2     | 21354/50000 [12:57:49<14:46:42,  1.86s/it]
Outer training loop:  43%|####2     | 21355/50000 [12:57:51<15:38:10,  1.97s/it]
Outer training loop:  43%|####2     | 21356/50000 [12:57:53<15:10:16,  1.91s/it]
Outer training loop:  43%|####2     | 21357/50000 [12:57:54<15:02:28,  1.89s/it]
Outer training loop:  43%|####2     | 21358/50000 [12:57:56<15:27:26,  1.94s/it]
Outer training loop:  43%|####2     | 21359/50000 [12:57:58<15:18:57,  1.93s/it]
Outer training loop:  43%|####2     | 21360/50000 [12:58:00<14:50:29,  1.87s/it]
Outer training loop:  43%|####2     | 21361/50000 [12:58:02<15:17:04,  1.92s/it]
Outer training loop:  43%|####2     | 21362/50000 [12:58:04<15:03:37,  1.89s/it]
Outer training loop:  43%|####2     | 21363/50000 [12:58:06<15:01:02,  1.89s/it]
Outer training loop:  43%|####2     | 21364/50000 [12:58:08<14:40:32,  1.84s/it]
Outer training loop:  43%|####2     | 21365/50000 [12:58:10<15:10:18,  1.91s/it]
Outer training loop:  43%|####2     | 21366/50000 [12:58:11<15:04:54,  1.90s/it]
Outer training loop:  43%|####2     | 21367/50000 [12:58:13<14:51:43,  1.87s/it]
Outer training loop:  43%|####2     | 21368/50000 [12:58:15<15:08:45,  1.90s/it]
Outer training loop:  43%|####2     | 21369/50000 [12:58:17<15:07:16,  1.90s/it]
Outer training loop:  43%|####2     | 21370/50000 [12:58:19<15:01:16,  1.89s/it]
Outer training loop:  43%|####2     | 21371/50000 [12:58:21<14:58:59,  1.88s/it]
Outer training loop:  43%|####2     | 21372/50000 [12:58:23<15:03:38,  1.89s/it]
Outer training loop:  43%|####2     | 21373/50000 [12:58:25<14:58:20,  1.88s/it]
Outer training loop:  43%|####2     | 21374/50000 [12:58:26<14:42:59,  1.85s/it]
Outer training loop:  43%|####2     | 21375/50000 [12:58:28<15:06:40,  1.90s/it]
Outer training loop:  43%|####2     | 21376/50000 [12:58:30<14:41:44,  1.85s/it]
Outer training loop:  43%|####2     | 21377/50000 [12:58:32<14:32:53,  1.83s/it]
Outer training loop:  43%|####2     | 21378/50000 [12:58:34<15:08:33,  1.90s/it]
Outer training loop:  43%|####2     | 21379/50000 [12:58:36<15:12:44,  1.91s/it]
Outer training loop:  43%|####2     | 21380/50000 [12:58:38<14:59:41,  1.89s/it]
Outer training loop:  43%|####2     | 21381/50000 [12:58:40<14:52:33,  1.87s/it]
Outer training loop:  43%|####2     | 21382/50000 [12:58:42<15:19:07,  1.93s/it]
Outer training loop:  43%|####2     | 21383/50000 [12:58:44<15:20:59,  1.93s/it]
Outer training loop:  43%|####2     | 21384/50000 [12:58:45<15:06:04,  1.90s/it]
Outer training loop:  43%|####2     | 21385/50000 [12:58:48<15:27:45,  1.95s/it]
Outer training loop:  43%|####2     | 21386/50000 [12:58:49<15:11:48,  1.91s/it]
Outer training loop:  43%|####2     | 21387/50000 [12:58:51<15:24:20,  1.94s/it]
Outer training loop:  43%|####2     | 21388/50000 [12:58:53<15:35:21,  1.96s/it]
Outer training loop:  43%|####2     | 21389/50000 [12:58:55<15:10:57,  1.91s/it]
Outer training loop:  43%|####2     | 21390/50000 [12:58:57<15:00:33,  1.89s/it]
Outer training loop:  43%|####2     | 21391/50000 [12:58:59<14:57:11,  1.88s/it]
Outer training loop:  43%|####2     | 21392/50000 [12:59:01<15:13:43,  1.92s/it]
Outer training loop:  43%|####2     | 21393/50000 [12:59:03<15:01:38,  1.89s/it]
Outer training loop:  43%|####2     | 21394/50000 [12:59:05<14:57:07,  1.88s/it]
Outer training loop:  43%|####2     | 21395/50000 [12:59:07<15:16:38,  1.92s/it]
Outer training loop:  43%|####2     | 21396/50000 [12:59:08<14:47:16,  1.86s/it]
Outer training loop:  43%|####2     | 21397/50000 [12:59:10<14:42:08,  1.85s/it]
Outer training loop:  43%|####2     | 21398/50000 [12:59:12<15:18:38,  1.93s/it]
Outer training loop:  43%|####2     | 21399/50000 [12:59:14<15:06:14,  1.90s/it]
Outer training loop:  43%|####2     | 21400/50000 [12:59:16<15:04:27,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.02s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.99s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  43%|####2     | 21401/50000 [12:59:23<27:55:09,  3.51s/it]
Outer training loop:  43%|####2     | 21402/50000 [12:59:25<24:25:27,  3.07s/it]
Outer training loop:  43%|####2     | 21403/50000 [12:59:27<21:30:52,  2.71s/it]
Outer training loop:  43%|####2     | 21404/50000 [12:59:29<19:20:12,  2.43s/it]
Outer training loop:  43%|####2     | 21405/50000 [12:59:31<17:54:36,  2.25s/it]
Outer training loop:  43%|####2     | 21406/50000 [12:59:33<17:16:11,  2.17s/it]
Outer training loop:  43%|####2     | 21407/50000 [12:59:35<16:30:39,  2.08s/it]
Outer training loop:  43%|####2     | 21408/50000 [12:59:37<16:08:55,  2.03s/it]
Outer training loop:  43%|####2     | 21409/50000 [12:59:38<15:58:50,  2.01s/it]
Outer training loop:  43%|####2     | 21410/50000 [12:59:40<15:44:31,  1.98s/it]
Outer training loop:  43%|####2     | 21411/50000 [12:59:42<15:20:56,  1.93s/it]
Outer training loop:  43%|####2     | 21412/50000 [12:59:44<15:14:53,  1.92s/it]
Outer training loop:  43%|####2     | 21413/50000 [12:59:46<15:23:37,  1.94s/it]
Outer training loop:  43%|####2     | 21414/50000 [12:59:48<15:06:42,  1.90s/it]
Outer training loop:  43%|####2     | 21415/50000 [12:59:50<15:09:04,  1.91s/it]
Outer training loop:  43%|####2     | 21416/50000 [12:59:52<15:19:54,  1.93s/it]
Outer training loop:  43%|####2     | 21417/50000 [12:59:54<15:06:11,  1.90s/it]
Outer training loop:  43%|####2     | 21418/50000 [12:59:56<15:06:49,  1.90s/it]
Outer training loop:  43%|####2     | 21419/50000 [12:59:58<15:31:38,  1.96s/it]
Outer training loop:  43%|####2     | 21420/50000 [12:59:59<15:12:02,  1.91s/it]
Outer training loop:  43%|####2     | 21421/50000 [13:00:01<14:56:27,  1.88s/it]
Outer training loop:  43%|####2     | 21422/50000 [13:00:03<14:43:58,  1.86s/it]
Outer training loop:  43%|####2     | 21423/50000 [13:00:05<15:04:03,  1.90s/it]
Outer training loop:  43%|####2     | 21424/50000 [13:00:07<14:44:49,  1.86s/it]
Outer training loop:  43%|####2     | 21425/50000 [13:00:08<14:07:41,  1.78s/it]
Outer training loop:  43%|####2     | 21426/50000 [13:00:10<14:32:06,  1.83s/it]
Outer training loop:  43%|####2     | 21427/50000 [13:00:12<14:20:16,  1.81s/it]
Outer training loop:  43%|####2     | 21428/50000 [13:00:14<14:26:54,  1.82s/it]
Outer training loop:  43%|####2     | 21429/50000 [13:00:16<14:16:16,  1.80s/it]
Outer training loop:  43%|####2     | 21430/50000 [13:00:18<15:14:31,  1.92s/it]
Outer training loop:  43%|####2     | 21431/50000 [13:00:20<15:04:04,  1.90s/it]
Outer training loop:  43%|####2     | 21432/50000 [13:00:22<14:59:57,  1.89s/it]
Outer training loop:  43%|####2     | 21433/50000 [13:00:24<15:12:17,  1.92s/it]
Outer training loop:  43%|####2     | 21434/50000 [13:00:26<15:27:58,  1.95s/it]
Outer training loop:  43%|####2     | 21435/50000 [13:00:28<15:26:42,  1.95s/it]
Outer training loop:  43%|####2     | 21436/50000 [13:00:29<14:56:57,  1.88s/it]
Outer training loop:  43%|####2     | 21437/50000 [13:00:31<15:01:59,  1.89s/it]
Outer training loop:  43%|####2     | 21438/50000 [13:00:33<14:57:55,  1.89s/it]
Outer training loop:  43%|####2     | 21439/50000 [13:00:35<15:06:31,  1.90s/it]
Outer training loop:  43%|####2     | 21440/50000 [13:00:37<15:25:34,  1.94s/it]
Outer training loop:  43%|####2     | 21441/50000 [13:00:39<15:03:50,  1.90s/it]
Outer training loop:  43%|####2     | 21442/50000 [13:00:41<14:45:47,  1.86s/it]
Outer training loop:  43%|####2     | 21443/50000 [13:00:43<15:12:09,  1.92s/it]
Outer training loop:  43%|####2     | 21444/50000 [13:00:45<14:58:57,  1.89s/it]
Outer training loop:  43%|####2     | 21445/50000 [13:00:46<14:46:55,  1.86s/it]
Outer training loop:  43%|####2     | 21446/50000 [13:00:48<14:56:30,  1.88s/it]
Outer training loop:  43%|####2     | 21447/50000 [13:00:50<14:47:56,  1.87s/it]
Outer training loop:  43%|####2     | 21448/50000 [13:00:52<14:26:15,  1.82s/it]
Outer training loop:  43%|####2     | 21449/50000 [13:00:54<14:36:42,  1.84s/it]
Outer training loop:  43%|####2     | 21450/50000 [13:00:56<15:09:54,  1.91s/it]
Outer training loop:  43%|####2     | 21451/50000 [13:00:58<14:48:27,  1.87s/it]
Outer training loop:  43%|####2     | 21452/50000 [13:00:59<14:26:02,  1.82s/it]
Outer training loop:  43%|####2     | 21453/50000 [13:01:01<15:01:23,  1.89s/it]
Outer training loop:  43%|####2     | 21454/50000 [13:01:03<14:52:30,  1.88s/it]
Outer training loop:  43%|####2     | 21455/50000 [13:01:05<14:46:56,  1.86s/it]
Outer training loop:  43%|####2     | 21456/50000 [13:01:07<15:08:38,  1.91s/it]
Outer training loop:  43%|####2     | 21457/50000 [13:01:09<14:41:04,  1.85s/it]
Outer training loop:  43%|####2     | 21458/50000 [13:01:11<14:43:42,  1.86s/it]
Outer training loop:  43%|####2     | 21459/50000 [13:01:12<14:22:48,  1.81s/it]
Outer training loop:  43%|####2     | 21460/50000 [13:01:14<14:31:33,  1.83s/it]
Outer training loop:  43%|####2     | 21461/50000 [13:01:16<14:29:54,  1.83s/it]
Outer training loop:  43%|####2     | 21462/50000 [13:01:18<14:23:30,  1.82s/it]
Outer training loop:  43%|####2     | 21463/50000 [13:01:20<14:55:31,  1.88s/it]
Outer training loop:  43%|####2     | 21464/50000 [13:01:22<14:42:48,  1.86s/it]
Outer training loop:  43%|####2     | 21465/50000 [13:01:23<14:19:07,  1.81s/it]
Outer training loop:  43%|####2     | 21466/50000 [13:01:25<14:35:08,  1.84s/it]
Outer training loop:  43%|####2     | 21467/50000 [13:01:27<15:06:23,  1.91s/it]
Outer training loop:  43%|####2     | 21468/50000 [13:01:29<15:09:15,  1.91s/it]
Outer training loop:  43%|####2     | 21469/50000 [13:01:31<15:02:52,  1.90s/it]
Outer training loop:  43%|####2     | 21470/50000 [13:01:33<15:10:40,  1.92s/it]
Outer training loop:  43%|####2     | 21471/50000 [13:01:35<14:52:17,  1.88s/it]
Outer training loop:  43%|####2     | 21472/50000 [13:01:37<14:38:10,  1.85s/it]
Outer training loop:  43%|####2     | 21473/50000 [13:01:39<14:58:34,  1.89s/it]
Outer training loop:  43%|####2     | 21474/50000 [13:01:40<14:58:49,  1.89s/it]
Outer training loop:  43%|####2     | 21475/50000 [13:01:42<15:00:25,  1.89s/it]
Outer training loop:  43%|####2     | 21476/50000 [13:01:44<15:20:15,  1.94s/it]
Outer training loop:  43%|####2     | 21477/50000 [13:01:46<15:00:38,  1.89s/it]
Outer training loop:  43%|####2     | 21478/50000 [13:01:48<14:52:41,  1.88s/it]
Outer training loop:  43%|####2     | 21479/50000 [13:01:50<14:34:01,  1.84s/it]
Outer training loop:  43%|####2     | 21480/50000 [13:01:52<14:59:02,  1.89s/it]
Outer training loop:  43%|####2     | 21481/50000 [13:01:54<14:36:47,  1.84s/it]
Outer training loop:  43%|####2     | 21482/50000 [13:01:55<14:38:44,  1.85s/it]
Outer training loop:  43%|####2     | 21483/50000 [13:01:57<15:13:16,  1.92s/it]
Outer training loop:  43%|####2     | 21484/50000 [13:02:00<15:35:15,  1.97s/it]
Outer training loop:  43%|####2     | 21485/50000 [13:02:02<15:33:08,  1.96s/it]
Outer training loop:  43%|####2     | 21486/50000 [13:02:03<15:15:05,  1.93s/it]
Outer training loop:  43%|####2     | 21487/50000 [13:02:05<15:23:40,  1.94s/it]
Outer training loop:  43%|####2     | 21488/50000 [13:02:07<15:32:16,  1.96s/it]
Outer training loop:  43%|####2     | 21489/50000 [13:02:09<15:23:01,  1.94s/it]
Outer training loop:  43%|####2     | 21490/50000 [13:02:11<15:25:39,  1.95s/it]
Outer training loop:  43%|####2     | 21491/50000 [13:02:13<15:17:56,  1.93s/it]
Outer training loop:  43%|####2     | 21492/50000 [13:02:15<14:52:05,  1.88s/it]
Outer training loop:  43%|####2     | 21493/50000 [13:02:17<14:39:23,  1.85s/it]
Outer training loop:  43%|####2     | 21494/50000 [13:02:19<14:55:02,  1.88s/it]
Outer training loop:  43%|####2     | 21495/50000 [13:02:20<14:52:03,  1.88s/it]
Outer training loop:  43%|####2     | 21496/50000 [13:02:22<14:57:18,  1.89s/it]
Outer training loop:  43%|####2     | 21497/50000 [13:02:25<15:35:38,  1.97s/it]
Outer training loop:  43%|####2     | 21498/50000 [13:02:26<14:59:51,  1.89s/it]
Outer training loop:  43%|####2     | 21499/50000 [13:02:28<15:03:55,  1.90s/it]
Outer training loop:  43%|####3     | 21500/50000 [13:02:30<14:38:10,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:38,  1.94s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:33,  1.90s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:33,  1.92s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:32,  1.93s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.90s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.93s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:18,  1.90s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:16,  1.89s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:18,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.91s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:00,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:01,  1.97s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:51,  1.93s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:54,  2.01s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:51,  1.99s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:47,  1.95s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:43,  1.96s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:45,  2.03s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:41,  1.99s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:37,  1.94s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:28,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:28,  1.93s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:25,  1.90s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:23,  1.90s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:20,  1.87s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:13,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:11,  1.88s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:09,  1.87s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:01,  1.81s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:59,  1.92s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:46,  1.85s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:47,  1.96s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:44,  1.93s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:42,  1.92s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.91s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:32,  1.93s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:30,  1.89s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.87s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.84s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:19,  1.79s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:17,  1.79s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:16,  2.00s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.98s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.92s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.94s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.91s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.82s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:21,  1.72s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:34,  1.90s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.89s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:29,  1.91s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:20,  1.87s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.93s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:18,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.88s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:14,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:13,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:11,  1.97s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.92s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:57,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:55,  1.84s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.86s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:47,  1.83s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:45,  1.82s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:50,  1.93s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.88s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:42,  1.86s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:36,  1.82s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:31,  1.83s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:29,  1.83s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:27,  1.85s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:27,  1.94s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:14,  1.81s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:15,  1.95s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:13,  1.93s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:10,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:10,  1.95s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:06,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:04,  1.90s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:03,  1.98s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.91s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:57,  1.91s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:47,  1.82s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:49,  1.98s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:46,  1.94s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.90s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.81s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:31,  1.96s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.91s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.92s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.92s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.87s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.87s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  2.00s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.94s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.86s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.83s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.85s/it]

Outer training loop:  43%|####3     | 21501/50000 [13:07:51<772:54:02, 97.63s/it]
Outer training loop:  43%|####3     | 21502/50000 [13:07:53<545:20:13, 68.89s/it]
Outer training loop:  43%|####3     | 21503/50000 [13:07:55<387:26:05, 48.94s/it]
Outer training loop:  43%|####3     | 21504/50000 [13:07:57<275:36:54, 34.82s/it]
Outer training loop:  43%|####3     | 21505/50000 [13:07:59<197:25:16, 24.94s/it]
Outer training loop:  43%|####3     | 21506/50000 [13:08:01<142:14:29, 17.97s/it]
Outer training loop:  43%|####3     | 21507/50000 [13:08:03<104:29:45, 13.20s/it]
Outer training loop:  43%|####3     | 21508/50000 [13:08:05<77:38:16,  9.81s/it] 
Outer training loop:  43%|####3     | 21509/50000 [13:08:06<58:20:48,  7.37s/it]
Outer training loop:  43%|####3     | 21510/50000 [13:08:08<45:30:58,  5.75s/it]
Outer training loop:  43%|####3     | 21511/50000 [13:08:10<36:24:56,  4.60s/it]
Outer training loop:  43%|####3     | 21512/50000 [13:08:12<29:38:47,  3.75s/it]
Outer training loop:  43%|####3     | 21513/50000 [13:08:14<25:31:57,  3.23s/it]
Outer training loop:  43%|####3     | 21514/50000 [13:08:16<21:57:54,  2.78s/it]
Outer training loop:  43%|####3     | 21515/50000 [13:08:18<20:13:52,  2.56s/it]
Outer training loop:  43%|####3     | 21516/50000 [13:08:20<18:42:24,  2.36s/it]
Outer training loop:  43%|####3     | 21517/50000 [13:08:22<17:39:11,  2.23s/it]
Outer training loop:  43%|####3     | 21518/50000 [13:08:24<16:56:30,  2.14s/it]
Outer training loop:  43%|####3     | 21519/50000 [13:08:25<16:07:29,  2.04s/it]
Outer training loop:  43%|####3     | 21520/50000 [13:08:28<16:32:08,  2.09s/it]
Outer training loop:  43%|####3     | 21521/50000 [13:08:30<16:10:49,  2.05s/it]
Outer training loop:  43%|####3     | 21522/50000 [13:08:31<15:47:34,  2.00s/it]
Outer training loop:  43%|####3     | 21523/50000 [13:08:33<15:27:55,  1.96s/it]
Outer training loop:  43%|####3     | 21524/50000 [13:08:35<15:52:12,  2.01s/it]
Outer training loop:  43%|####3     | 21525/50000 [13:08:37<15:20:45,  1.94s/it]
Outer training loop:  43%|####3     | 21526/50000 [13:08:39<15:13:23,  1.92s/it]
Outer training loop:  43%|####3     | 21527/50000 [13:08:41<15:19:56,  1.94s/it]
Outer training loop:  43%|####3     | 21528/50000 [13:08:43<15:00:48,  1.90s/it]
Outer training loop:  43%|####3     | 21529/50000 [13:08:45<14:40:33,  1.86s/it]
Outer training loop:  43%|####3     | 21530/50000 [13:08:47<14:52:39,  1.88s/it]
Outer training loop:  43%|####3     | 21531/50000 [13:08:48<14:37:02,  1.85s/it]
Outer training loop:  43%|####3     | 21532/50000 [13:08:50<14:46:28,  1.87s/it]
Outer training loop:  43%|####3     | 21533/50000 [13:08:52<14:56:24,  1.89s/it]
Outer training loop:  43%|####3     | 21534/50000 [13:08:54<14:44:29,  1.86s/it]
Outer training loop:  43%|####3     | 21535/50000 [13:08:56<14:30:03,  1.83s/it]
Outer training loop:  43%|####3     | 21536/50000 [13:08:58<14:25:46,  1.83s/it]
Outer training loop:  43%|####3     | 21537/50000 [13:08:59<14:43:14,  1.86s/it]
Outer training loop:  43%|####3     | 21538/50000 [13:09:01<14:30:39,  1.84s/it]
Outer training loop:  43%|####3     | 21539/50000 [13:09:03<14:20:25,  1.81s/it]
Outer training loop:  43%|####3     | 21540/50000 [13:09:05<14:58:51,  1.89s/it]
Outer training loop:  43%|####3     | 21541/50000 [13:09:07<14:35:42,  1.85s/it]
Outer training loop:  43%|####3     | 21542/50000 [13:09:09<14:20:26,  1.81s/it]
Outer training loop:  43%|####3     | 21543/50000 [13:09:10<14:34:19,  1.84s/it]
Outer training loop:  43%|####3     | 21544/50000 [13:09:12<14:59:47,  1.90s/it]
Outer training loop:  43%|####3     | 21545/50000 [13:09:14<14:37:21,  1.85s/it]
Outer training loop:  43%|####3     | 21546/50000 [13:09:16<14:33:55,  1.84s/it]
Outer training loop:  43%|####3     | 21547/50000 [13:09:18<14:57:04,  1.89s/it]
Outer training loop:  43%|####3     | 21548/50000 [13:09:20<15:04:23,  1.91s/it]
Outer training loop:  43%|####3     | 21549/50000 [13:09:22<14:36:40,  1.85s/it]
Outer training loop:  43%|####3     | 21550/50000 [13:09:24<14:37:32,  1.85s/it]
Outer training loop:  43%|####3     | 21551/50000 [13:09:26<14:59:04,  1.90s/it]
Outer training loop:  43%|####3     | 21552/50000 [13:09:27<14:33:07,  1.84s/it]
Outer training loop:  43%|####3     | 21553/50000 [13:09:29<14:39:05,  1.85s/it]
Outer training loop:  43%|####3     | 21554/50000 [13:09:31<14:55:38,  1.89s/it]
Outer training loop:  43%|####3     | 21555/50000 [13:09:33<14:50:22,  1.88s/it]
Outer training loop:  43%|####3     | 21556/50000 [13:09:35<14:42:09,  1.86s/it]
Outer training loop:  43%|####3     | 21557/50000 [13:09:37<14:44:54,  1.87s/it]
Outer training loop:  43%|####3     | 21558/50000 [13:09:39<14:46:11,  1.87s/it]
Outer training loop:  43%|####3     | 21559/50000 [13:09:40<14:46:24,  1.87s/it]
Outer training loop:  43%|####3     | 21560/50000 [13:09:42<15:01:28,  1.90s/it]
Outer training loop:  43%|####3     | 21561/50000 [13:09:44<15:10:14,  1.92s/it]
Outer training loop:  43%|####3     | 21562/50000 [13:09:46<14:48:25,  1.87s/it]
Outer training loop:  43%|####3     | 21563/50000 [13:09:48<14:44:32,  1.87s/it]
Outer training loop:  43%|####3     | 21564/50000 [13:09:50<15:14:16,  1.93s/it]
Outer training loop:  43%|####3     | 21565/50000 [13:09:52<15:36:46,  1.98s/it]
Outer training loop:  43%|####3     | 21566/50000 [13:09:54<15:12:02,  1.92s/it]
Outer training loop:  43%|####3     | 21567/50000 [13:09:56<15:20:22,  1.94s/it]
Outer training loop:  43%|####3     | 21568/50000 [13:09:58<15:12:22,  1.93s/it]
Outer training loop:  43%|####3     | 21569/50000 [13:10:00<14:59:21,  1.90s/it]
Outer training loop:  43%|####3     | 21570/50000 [13:10:02<14:52:16,  1.88s/it]
Outer training loop:  43%|####3     | 21571/50000 [13:10:04<15:17:26,  1.94s/it]
Outer training loop:  43%|####3     | 21572/50000 [13:10:05<15:02:04,  1.90s/it]
Outer training loop:  43%|####3     | 21573/50000 [13:10:07<14:55:50,  1.89s/it]
Outer training loop:  43%|####3     | 21574/50000 [13:10:09<15:38:00,  1.98s/it]
Outer training loop:  43%|####3     | 21575/50000 [13:10:11<15:19:03,  1.94s/it]
Outer training loop:  43%|####3     | 21576/50000 [13:10:13<15:09:41,  1.92s/it]
Outer training loop:  43%|####3     | 21577/50000 [13:10:15<15:12:17,  1.93s/it]
Outer training loop:  43%|####3     | 21578/50000 [13:10:17<15:11:26,  1.92s/it]
Outer training loop:  43%|####3     | 21579/50000 [13:10:19<14:57:13,  1.89s/it]
Outer training loop:  43%|####3     | 21580/50000 [13:10:21<14:27:14,  1.83s/it]
Outer training loop:  43%|####3     | 21581/50000 [13:10:23<14:54:16,  1.89s/it]
Outer training loop:  43%|####3     | 21582/50000 [13:10:24<14:39:27,  1.86s/it]
Outer training loop:  43%|####3     | 21583/50000 [13:10:26<14:35:09,  1.85s/it]
Outer training loop:  43%|####3     | 21584/50000 [13:10:28<14:46:53,  1.87s/it]
Outer training loop:  43%|####3     | 21585/50000 [13:10:30<14:45:48,  1.87s/it]
Outer training loop:  43%|####3     | 21586/50000 [13:10:32<14:36:11,  1.85s/it]
Outer training loop:  43%|####3     | 21587/50000 [13:10:34<14:36:54,  1.85s/it]
Outer training loop:  43%|####3     | 21588/50000 [13:10:36<15:12:57,  1.93s/it]
Outer training loop:  43%|####3     | 21589/50000 [13:10:38<15:04:58,  1.91s/it]
Outer training loop:  43%|####3     | 21590/50000 [13:10:39<14:43:07,  1.87s/it]
Outer training loop:  43%|####3     | 21591/50000 [13:10:41<15:01:41,  1.90s/it]
Outer training loop:  43%|####3     | 21592/50000 [13:10:43<15:04:48,  1.91s/it]
Outer training loop:  43%|####3     | 21593/50000 [13:10:45<14:57:07,  1.89s/it]
Outer training loop:  43%|####3     | 21594/50000 [13:10:47<15:25:00,  1.95s/it]
Outer training loop:  43%|####3     | 21595/50000 [13:10:49<15:22:54,  1.95s/it]
Outer training loop:  43%|####3     | 21596/50000 [13:10:51<15:08:50,  1.92s/it]
Outer training loop:  43%|####3     | 21597/50000 [13:10:53<14:53:32,  1.89s/it]
Outer training loop:  43%|####3     | 21598/50000 [13:10:55<15:21:24,  1.95s/it]
Outer training loop:  43%|####3     | 21599/50000 [13:10:57<14:56:38,  1.89s/it]
Outer training loop:  43%|####3     | 21600/50000 [13:10:59<15:21:08,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.99s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  43%|####3     | 21601/50000 [13:11:06<28:19:26,  3.59s/it]
Outer training loop:  43%|####3     | 21602/50000 [13:11:08<24:09:03,  3.06s/it]
Outer training loop:  43%|####3     | 21603/50000 [13:11:10<21:38:21,  2.74s/it]
Outer training loop:  43%|####3     | 21604/50000 [13:11:12<19:25:08,  2.46s/it]
Outer training loop:  43%|####3     | 21605/50000 [13:11:14<17:53:30,  2.27s/it]
Outer training loop:  43%|####3     | 21606/50000 [13:11:16<17:11:53,  2.18s/it]
Outer training loop:  43%|####3     | 21607/50000 [13:11:17<16:25:13,  2.08s/it]
Outer training loop:  43%|####3     | 21608/50000 [13:11:19<15:49:15,  2.01s/it]
Outer training loop:  43%|####3     | 21609/50000 [13:11:21<16:02:18,  2.03s/it]
Outer training loop:  43%|####3     | 21610/50000 [13:11:23<15:34:42,  1.98s/it]
Outer training loop:  43%|####3     | 21611/50000 [13:11:25<15:16:26,  1.94s/it]
Outer training loop:  43%|####3     | 21612/50000 [13:11:27<15:44:29,  2.00s/it]
Outer training loop:  43%|####3     | 21613/50000 [13:11:29<15:29:19,  1.96s/it]
Outer training loop:  43%|####3     | 21614/50000 [13:11:31<14:55:38,  1.89s/it]
Outer training loop:  43%|####3     | 21615/50000 [13:11:33<14:35:10,  1.85s/it]
Outer training loop:  43%|####3     | 21616/50000 [13:11:35<15:20:23,  1.95s/it]
Outer training loop:  43%|####3     | 21617/50000 [13:11:37<15:00:08,  1.90s/it]
Outer training loop:  43%|####3     | 21618/50000 [13:11:38<14:45:50,  1.87s/it]
Outer training loop:  43%|####3     | 21619/50000 [13:11:40<15:03:19,  1.91s/it]
Outer training loop:  43%|####3     | 21620/50000 [13:11:42<15:00:28,  1.90s/it]
Outer training loop:  43%|####3     | 21621/50000 [13:11:44<14:36:15,  1.85s/it]
Outer training loop:  43%|####3     | 21622/50000 [13:11:46<14:25:02,  1.83s/it]
Outer training loop:  43%|####3     | 21623/50000 [13:11:48<14:45:50,  1.87s/it]
Outer training loop:  43%|####3     | 21624/50000 [13:11:50<14:54:14,  1.89s/it]
Outer training loop:  43%|####3     | 21625/50000 [13:11:52<14:49:46,  1.88s/it]
Outer training loop:  43%|####3     | 21626/50000 [13:11:53<14:52:14,  1.89s/it]
Outer training loop:  43%|####3     | 21627/50000 [13:11:55<14:41:20,  1.86s/it]
Outer training loop:  43%|####3     | 21628/50000 [13:11:57<14:32:15,  1.84s/it]
Outer training loop:  43%|####3     | 21629/50000 [13:11:59<14:58:18,  1.90s/it]
Outer training loop:  43%|####3     | 21630/50000 [13:12:01<14:31:58,  1.84s/it]
Outer training loop:  43%|####3     | 21631/50000 [13:12:03<14:32:12,  1.84s/it]
Outer training loop:  43%|####3     | 21632/50000 [13:12:05<14:42:46,  1.87s/it]
Outer training loop:  43%|####3     | 21633/50000 [13:12:06<14:48:09,  1.88s/it]
Outer training loop:  43%|####3     | 21634/50000 [13:12:08<14:33:33,  1.85s/it]
Outer training loop:  43%|####3     | 21635/50000 [13:12:10<14:29:38,  1.84s/it]
Outer training loop:  43%|####3     | 21636/50000 [13:12:12<15:15:32,  1.94s/it]
Outer training loop:  43%|####3     | 21637/50000 [13:12:14<15:08:57,  1.92s/it]
Outer training loop:  43%|####3     | 21638/50000 [13:12:16<15:04:49,  1.91s/it]
Outer training loop:  43%|####3     | 21639/50000 [13:12:18<15:22:27,  1.95s/it]
Outer training loop:  43%|####3     | 21640/50000 [13:12:20<14:57:55,  1.90s/it]
Outer training loop:  43%|####3     | 21641/50000 [13:12:22<14:53:08,  1.89s/it]
Outer training loop:  43%|####3     | 21642/50000 [13:12:23<14:35:26,  1.85s/it]
Outer training loop:  43%|####3     | 21643/50000 [13:12:25<15:02:26,  1.91s/it]
Outer training loop:  43%|####3     | 21644/50000 [13:12:27<14:40:55,  1.86s/it]
Outer training loop:  43%|####3     | 21645/50000 [13:12:29<14:41:29,  1.87s/it]
Outer training loop:  43%|####3     | 21646/50000 [13:12:31<14:55:03,  1.89s/it]
Outer training loop:  43%|####3     | 21647/50000 [13:12:33<14:23:18,  1.83s/it]
Outer training loop:  43%|####3     | 21648/50000 [13:12:35<14:27:35,  1.84s/it]
Outer training loop:  43%|####3     | 21649/50000 [13:12:36<14:23:20,  1.83s/it]
Outer training loop:  43%|####3     | 21650/50000 [13:12:38<14:44:43,  1.87s/it]
Outer training loop:  43%|####3     | 21651/50000 [13:12:40<14:40:03,  1.86s/it]
Outer training loop:  43%|####3     | 21652/50000 [13:12:42<14:26:53,  1.83s/it]
Outer training loop:  43%|####3     | 21653/50000 [13:12:44<15:06:28,  1.92s/it]
Outer training loop:  43%|####3     | 21654/50000 [13:12:46<14:47:18,  1.88s/it]
Outer training loop:  43%|####3     | 21655/50000 [13:12:48<14:26:11,  1.83s/it]
Outer training loop:  43%|####3     | 21656/50000 [13:12:50<14:37:03,  1.86s/it]
Outer training loop:  43%|####3     | 21657/50000 [13:12:52<15:11:03,  1.93s/it]
Outer training loop:  43%|####3     | 21658/50000 [13:12:53<14:55:39,  1.90s/it]
Outer training loop:  43%|####3     | 21659/50000 [13:12:55<14:36:39,  1.86s/it]
Outer training loop:  43%|####3     | 21660/50000 [13:12:57<14:51:10,  1.89s/it]
Outer training loop:  43%|####3     | 21661/50000 [13:12:59<14:49:04,  1.88s/it]
Outer training loop:  43%|####3     | 21662/50000 [13:13:01<15:18:18,  1.94s/it]
Outer training loop:  43%|####3     | 21663/50000 [13:13:03<15:22:47,  1.95s/it]
Outer training loop:  43%|####3     | 21664/50000 [13:13:05<15:26:52,  1.96s/it]
Outer training loop:  43%|####3     | 21665/50000 [13:13:07<14:51:40,  1.89s/it]
Outer training loop:  43%|####3     | 21666/50000 [13:13:09<15:17:29,  1.94s/it]
Outer training loop:  43%|####3     | 21667/50000 [13:13:11<15:21:09,  1.95s/it]
Outer training loop:  43%|####3     | 21668/50000 [13:13:13<15:58:37,  2.03s/it]
Outer training loop:  43%|####3     | 21669/50000 [13:13:15<16:04:57,  2.04s/it]
Outer training loop:  43%|####3     | 21670/50000 [13:13:17<16:10:09,  2.05s/it]
Outer training loop:  43%|####3     | 21671/50000 [13:13:19<15:44:16,  2.00s/it]
Outer training loop:  43%|####3     | 21672/50000 [13:13:21<15:44:32,  2.00s/it]
Outer training loop:  43%|####3     | 21673/50000 [13:13:23<15:46:59,  2.01s/it]
Outer training loop:  43%|####3     | 21674/50000 [13:13:25<15:35:56,  1.98s/it]
Outer training loop:  43%|####3     | 21675/50000 [13:13:27<15:22:09,  1.95s/it]
Outer training loop:  43%|####3     | 21676/50000 [13:13:29<15:08:15,  1.92s/it]
Outer training loop:  43%|####3     | 21677/50000 [13:13:31<15:24:14,  1.96s/it]
Outer training loop:  43%|####3     | 21678/50000 [13:13:33<15:03:48,  1.91s/it]
Outer training loop:  43%|####3     | 21679/50000 [13:13:35<15:03:01,  1.91s/it]
Outer training loop:  43%|####3     | 21680/50000 [13:13:37<15:14:31,  1.94s/it]
Outer training loop:  43%|####3     | 21681/50000 [13:13:38<14:49:43,  1.89s/it]
Outer training loop:  43%|####3     | 21682/50000 [13:13:40<14:58:06,  1.90s/it]
Outer training loop:  43%|####3     | 21683/50000 [13:13:42<15:38:58,  1.99s/it]
Outer training loop:  43%|####3     | 21684/50000 [13:13:44<15:17:37,  1.94s/it]
Outer training loop:  43%|####3     | 21685/50000 [13:13:46<14:53:34,  1.89s/it]
Outer training loop:  43%|####3     | 21686/50000 [13:13:48<14:47:32,  1.88s/it]
Outer training loop:  43%|####3     | 21687/50000 [13:13:50<15:22:03,  1.95s/it]
Outer training loop:  43%|####3     | 21688/50000 [13:13:52<14:47:55,  1.88s/it]
Outer training loop:  43%|####3     | 21689/50000 [13:13:53<14:24:01,  1.83s/it]
Outer training loop:  43%|####3     | 21690/50000 [13:13:56<15:06:05,  1.92s/it]
Outer training loop:  43%|####3     | 21691/50000 [13:13:57<15:02:34,  1.91s/it]
Outer training loop:  43%|####3     | 21692/50000 [13:13:59<15:00:56,  1.91s/it]
Outer training loop:  43%|####3     | 21693/50000 [13:14:01<15:26:28,  1.96s/it]
Outer training loop:  43%|####3     | 21694/50000 [13:14:03<15:16:54,  1.94s/it]
Outer training loop:  43%|####3     | 21695/50000 [13:14:05<15:01:46,  1.91s/it]
Outer training loop:  43%|####3     | 21696/50000 [13:14:07<14:58:40,  1.91s/it]
Outer training loop:  43%|####3     | 21697/50000 [13:14:09<15:07:17,  1.92s/it]
Outer training loop:  43%|####3     | 21698/50000 [13:14:11<14:59:15,  1.91s/it]
Outer training loop:  43%|####3     | 21699/50000 [13:14:13<14:53:33,  1.89s/it]
Outer training loop:  43%|####3     | 21700/50000 [13:14:15<14:58:47,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  43%|####3     | 21701/50000 [13:14:22<27:53:29,  3.55s/it]
Outer training loop:  43%|####3     | 21702/50000 [13:14:24<23:32:05,  2.99s/it]
Outer training loop:  43%|####3     | 21703/50000 [13:14:26<20:44:04,  2.64s/it]
Outer training loop:  43%|####3     | 21704/50000 [13:14:28<19:12:05,  2.44s/it]
Outer training loop:  43%|####3     | 21705/50000 [13:14:29<17:31:12,  2.23s/it]
Outer training loop:  43%|####3     | 21706/50000 [13:14:31<16:13:08,  2.06s/it]
Outer training loop:  43%|####3     | 21707/50000 [13:14:33<15:39:10,  1.99s/it]
Outer training loop:  43%|####3     | 21708/50000 [13:14:35<16:18:10,  2.07s/it]
Outer training loop:  43%|####3     | 21709/50000 [13:14:37<16:08:27,  2.05s/it]
Outer training loop:  43%|####3     | 21710/50000 [13:14:39<15:15:57,  1.94s/it]
Outer training loop:  43%|####3     | 21711/50000 [13:14:41<15:31:32,  1.98s/it]
Outer training loop:  43%|####3     | 21712/50000 [13:14:43<15:24:36,  1.96s/it]
Outer training loop:  43%|####3     | 21713/50000 [13:14:45<15:14:08,  1.94s/it]
Outer training loop:  43%|####3     | 21714/50000 [13:14:47<15:02:34,  1.91s/it]
Outer training loop:  43%|####3     | 21715/50000 [13:14:48<15:07:31,  1.93s/it]
Outer training loop:  43%|####3     | 21716/50000 [13:14:50<15:02:48,  1.92s/it]
Outer training loop:  43%|####3     | 21717/50000 [13:14:52<15:00:15,  1.91s/it]
Outer training loop:  43%|####3     | 21718/50000 [13:14:54<15:18:09,  1.95s/it]
Outer training loop:  43%|####3     | 21719/50000 [13:14:56<14:54:07,  1.90s/it]
Outer training loop:  43%|####3     | 21720/50000 [13:14:58<15:03:23,  1.92s/it]
Outer training loop:  43%|####3     | 21721/50000 [13:15:00<15:41:54,  2.00s/it]
Outer training loop:  43%|####3     | 21722/50000 [13:15:02<15:28:34,  1.97s/it]
Outer training loop:  43%|####3     | 21723/50000 [13:15:04<15:16:22,  1.94s/it]
Outer training loop:  43%|####3     | 21724/50000 [13:15:06<14:58:19,  1.91s/it]
Outer training loop:  43%|####3     | 21725/50000 [13:15:08<15:19:53,  1.95s/it]
Outer training loop:  43%|####3     | 21726/50000 [13:15:10<15:20:17,  1.95s/it]
Outer training loop:  43%|####3     | 21727/50000 [13:15:12<14:49:03,  1.89s/it]
Outer training loop:  43%|####3     | 21728/50000 [13:15:14<15:11:22,  1.93s/it]
Outer training loop:  43%|####3     | 21729/50000 [13:15:16<15:17:07,  1.95s/it]
Outer training loop:  43%|####3     | 21730/50000 [13:15:17<14:45:05,  1.88s/it]
Outer training loop:  43%|####3     | 21731/50000 [13:15:19<14:45:20,  1.88s/it]
Outer training loop:  43%|####3     | 21732/50000 [13:15:21<15:05:20,  1.92s/it]
Outer training loop:  43%|####3     | 21733/50000 [13:15:23<14:50:47,  1.89s/it]
Outer training loop:  43%|####3     | 21734/50000 [13:15:25<15:01:01,  1.91s/it]
Outer training loop:  43%|####3     | 21735/50000 [13:15:27<15:22:53,  1.96s/it]
Outer training loop:  43%|####3     | 21736/50000 [13:15:29<15:04:43,  1.92s/it]
Outer training loop:  43%|####3     | 21737/50000 [13:15:31<14:49:22,  1.89s/it]
Outer training loop:  43%|####3     | 21738/50000 [13:15:33<15:09:34,  1.93s/it]
Outer training loop:  43%|####3     | 21739/50000 [13:15:35<15:10:20,  1.93s/it]
Outer training loop:  43%|####3     | 21740/50000 [13:15:37<15:00:51,  1.91s/it]
Outer training loop:  43%|####3     | 21741/50000 [13:15:38<14:39:23,  1.87s/it]
Outer training loop:  43%|####3     | 21742/50000 [13:15:40<15:16:48,  1.95s/it]
Outer training loop:  43%|####3     | 21743/50000 [13:15:42<14:43:03,  1.88s/it]
Outer training loop:  43%|####3     | 21744/50000 [13:15:44<14:40:25,  1.87s/it]
Outer training loop:  43%|####3     | 21745/50000 [13:15:46<14:59:08,  1.91s/it]
Outer training loop:  43%|####3     | 21746/50000 [13:15:48<14:39:58,  1.87s/it]
Outer training loop:  43%|####3     | 21747/50000 [13:15:50<14:50:48,  1.89s/it]
Outer training loop:  43%|####3     | 21748/50000 [13:15:52<15:11:46,  1.94s/it]
Outer training loop:  43%|####3     | 21749/50000 [13:15:54<14:49:23,  1.89s/it]
Outer training loop:  44%|####3     | 21750/50000 [13:15:55<14:44:58,  1.88s/it]
Outer training loop:  44%|####3     | 21751/50000 [13:15:57<14:54:17,  1.90s/it]
Outer training loop:  44%|####3     | 21752/50000 [13:15:59<15:09:35,  1.93s/it]
Outer training loop:  44%|####3     | 21753/50000 [13:16:01<14:40:10,  1.87s/it]
Outer training loop:  44%|####3     | 21754/50000 [13:16:03<14:37:00,  1.86s/it]
Outer training loop:  44%|####3     | 21755/50000 [13:16:05<15:03:06,  1.92s/it]
Outer training loop:  44%|####3     | 21756/50000 [13:16:07<14:33:52,  1.86s/it]
Outer training loop:  44%|####3     | 21757/50000 [13:16:08<14:10:37,  1.81s/it]
Outer training loop:  44%|####3     | 21758/50000 [13:16:10<14:35:11,  1.86s/it]
Outer training loop:  44%|####3     | 21759/50000 [13:16:13<15:33:13,  1.98s/it]
Outer training loop:  44%|####3     | 21760/50000 [13:16:14<15:00:08,  1.91s/it]
Outer training loop:  44%|####3     | 21761/50000 [13:16:16<14:35:56,  1.86s/it]
Outer training loop:  44%|####3     | 21762/50000 [13:16:18<14:20:46,  1.83s/it]
Outer training loop:  44%|####3     | 21763/50000 [13:16:20<15:09:08,  1.93s/it]
Outer training loop:  44%|####3     | 21764/50000 [13:16:22<14:48:01,  1.89s/it]
Outer training loop:  44%|####3     | 21765/50000 [13:16:24<14:40:46,  1.87s/it]
Outer training loop:  44%|####3     | 21766/50000 [13:16:26<15:11:10,  1.94s/it]
Outer training loop:  44%|####3     | 21767/50000 [13:16:27<14:39:45,  1.87s/it]
Outer training loop:  44%|####3     | 21768/50000 [13:16:29<14:52:04,  1.90s/it]
Outer training loop:  44%|####3     | 21769/50000 [13:16:31<15:06:19,  1.93s/it]
Outer training loop:  44%|####3     | 21770/50000 [13:16:33<14:52:02,  1.90s/it]
Outer training loop:  44%|####3     | 21771/50000 [13:16:35<14:54:21,  1.90s/it]
Outer training loop:  44%|####3     | 21772/50000 [13:16:37<15:09:40,  1.93s/it]
Outer training loop:  44%|####3     | 21773/50000 [13:16:39<15:07:29,  1.93s/it]
Outer training loop:  44%|####3     | 21774/50000 [13:16:41<14:44:51,  1.88s/it]
Outer training loop:  44%|####3     | 21775/50000 [13:16:43<14:32:33,  1.85s/it]
Outer training loop:  44%|####3     | 21776/50000 [13:16:45<14:53:24,  1.90s/it]
Outer training loop:  44%|####3     | 21777/50000 [13:16:46<14:40:44,  1.87s/it]
Outer training loop:  44%|####3     | 21778/50000 [13:16:48<14:12:18,  1.81s/it]
Outer training loop:  44%|####3     | 21779/50000 [13:16:50<14:40:14,  1.87s/it]
Outer training loop:  44%|####3     | 21780/50000 [13:16:52<14:12:50,  1.81s/it]
Outer training loop:  44%|####3     | 21781/50000 [13:16:54<14:36:32,  1.86s/it]
Outer training loop:  44%|####3     | 21782/50000 [13:16:56<14:22:46,  1.83s/it]
Outer training loop:  44%|####3     | 21783/50000 [13:16:58<14:45:46,  1.88s/it]
Outer training loop:  44%|####3     | 21784/50000 [13:16:59<14:33:00,  1.86s/it]
Outer training loop:  44%|####3     | 21785/50000 [13:17:01<14:18:39,  1.83s/it]
Outer training loop:  44%|####3     | 21786/50000 [13:17:03<14:24:54,  1.84s/it]
Outer training loop:  44%|####3     | 21787/50000 [13:17:05<14:10:52,  1.81s/it]
Outer training loop:  44%|####3     | 21788/50000 [13:17:06<13:52:00,  1.77s/it]
Outer training loop:  44%|####3     | 21789/50000 [13:17:08<14:20:27,  1.83s/it]
Outer training loop:  44%|####3     | 21790/50000 [13:17:10<14:07:31,  1.80s/it]
Outer training loop:  44%|####3     | 21791/50000 [13:17:12<13:51:34,  1.77s/it]
Outer training loop:  44%|####3     | 21792/50000 [13:17:14<13:49:42,  1.76s/it]
Outer training loop:  44%|####3     | 21793/50000 [13:17:16<14:26:18,  1.84s/it]
Outer training loop:  44%|####3     | 21794/50000 [13:17:17<14:11:07,  1.81s/it]
Outer training loop:  44%|####3     | 21795/50000 [13:17:19<14:05:53,  1.80s/it]
Outer training loop:  44%|####3     | 21796/50000 [13:17:21<14:31:13,  1.85s/it]
Outer training loop:  44%|####3     | 21797/50000 [13:17:23<14:15:05,  1.82s/it]
Outer training loop:  44%|####3     | 21798/50000 [13:17:25<14:04:21,  1.80s/it]
Outer training loop:  44%|####3     | 21799/50000 [13:17:26<14:06:51,  1.80s/it]
Outer training loop:  44%|####3     | 21800/50000 [13:17:28<14:27:38,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  44%|####3     | 21801/50000 [13:17:35<26:56:04,  3.44s/it]
Outer training loop:  44%|####3     | 21802/50000 [13:17:37<22:55:14,  2.93s/it]
Outer training loop:  44%|####3     | 21803/50000 [13:17:39<20:15:12,  2.59s/it]
Outer training loop:  44%|####3     | 21804/50000 [13:17:41<18:34:52,  2.37s/it]
Outer training loop:  44%|####3     | 21805/50000 [13:17:42<16:51:40,  2.15s/it]
Outer training loop:  44%|####3     | 21806/50000 [13:17:44<16:16:43,  2.08s/it]
Outer training loop:  44%|####3     | 21807/50000 [13:17:46<16:02:32,  2.05s/it]
Outer training loop:  44%|####3     | 21808/50000 [13:17:48<15:34:23,  1.99s/it]
Outer training loop:  44%|####3     | 21809/50000 [13:17:50<15:04:53,  1.93s/it]
Outer training loop:  44%|####3     | 21810/50000 [13:17:52<15:34:44,  1.99s/it]
Outer training loop:  44%|####3     | 21811/50000 [13:17:54<14:53:31,  1.90s/it]
Outer training loop:  44%|####3     | 21812/50000 [13:17:56<14:31:14,  1.85s/it]
Outer training loop:  44%|####3     | 21813/50000 [13:17:57<14:14:59,  1.82s/it]
Outer training loop:  44%|####3     | 21814/50000 [13:17:59<14:47:40,  1.89s/it]
Outer training loop:  44%|####3     | 21815/50000 [13:18:01<14:26:59,  1.85s/it]
Outer training loop:  44%|####3     | 21816/50000 [13:18:03<14:04:31,  1.80s/it]
Outer training loop:  44%|####3     | 21817/50000 [13:18:05<14:33:50,  1.86s/it]
Outer training loop:  44%|####3     | 21818/50000 [13:18:07<14:16:38,  1.82s/it]
Outer training loop:  44%|####3     | 21819/50000 [13:18:08<14:11:22,  1.81s/it]
Outer training loop:  44%|####3     | 21820/50000 [13:18:10<14:11:05,  1.81s/it]
Outer training loop:  44%|####3     | 21821/50000 [13:18:12<14:31:49,  1.86s/it]
Outer training loop:  44%|####3     | 21822/50000 [13:18:14<14:18:16,  1.83s/it]
Outer training loop:  44%|####3     | 21823/50000 [13:18:16<14:13:39,  1.82s/it]
Outer training loop:  44%|####3     | 21824/50000 [13:18:17<14:14:05,  1.82s/it]
Outer training loop:  44%|####3     | 21825/50000 [13:18:19<14:14:19,  1.82s/it]
Outer training loop:  44%|####3     | 21826/50000 [13:18:21<14:20:02,  1.83s/it]
Outer training loop:  44%|####3     | 21827/50000 [13:18:23<14:35:40,  1.86s/it]
Outer training loop:  44%|####3     | 21828/50000 [13:18:25<14:21:47,  1.84s/it]
Outer training loop:  44%|####3     | 21829/50000 [13:18:27<14:17:53,  1.83s/it]
Outer training loop:  44%|####3     | 21830/50000 [13:18:28<14:16:23,  1.82s/it]
Outer training loop:  44%|####3     | 21831/50000 [13:18:30<14:39:20,  1.87s/it]
Outer training loop:  44%|####3     | 21832/50000 [13:18:32<14:39:44,  1.87s/it]
Outer training loop:  44%|####3     | 21833/50000 [13:18:34<14:33:13,  1.86s/it]
Outer training loop:  44%|####3     | 21834/50000 [13:18:36<14:41:25,  1.88s/it]
Outer training loop:  44%|####3     | 21835/50000 [13:18:38<14:34:26,  1.86s/it]
Outer training loop:  44%|####3     | 21836/50000 [13:18:40<14:00:53,  1.79s/it]
Outer training loop:  44%|####3     | 21837/50000 [13:18:41<14:13:29,  1.82s/it]
Outer training loop:  44%|####3     | 21838/50000 [13:18:43<14:11:43,  1.81s/it]
Outer training loop:  44%|####3     | 21839/50000 [13:18:45<14:09:35,  1.81s/it]
Outer training loop:  44%|####3     | 21840/50000 [13:18:47<14:35:52,  1.87s/it]
Outer training loop:  44%|####3     | 21841/50000 [13:18:49<14:33:44,  1.86s/it]
Outer training loop:  44%|####3     | 21842/50000 [13:18:51<14:12:26,  1.82s/it]
Outer training loop:  44%|####3     | 21843/50000 [13:18:53<14:25:19,  1.84s/it]
Outer training loop:  44%|####3     | 21844/50000 [13:18:54<14:10:09,  1.81s/it]
Outer training loop:  44%|####3     | 21845/50000 [13:18:56<14:06:45,  1.80s/it]
Outer training loop:  44%|####3     | 21846/50000 [13:18:58<14:38:55,  1.87s/it]
Outer training loop:  44%|####3     | 21847/50000 [13:19:00<14:27:26,  1.85s/it]
Outer training loop:  44%|####3     | 21848/50000 [13:19:02<14:22:36,  1.84s/it]
Outer training loop:  44%|####3     | 21849/50000 [13:19:04<14:21:24,  1.84s/it]
Outer training loop:  44%|####3     | 21850/50000 [13:19:05<14:32:49,  1.86s/it]
Outer training loop:  44%|####3     | 21851/50000 [13:19:07<14:16:57,  1.83s/it]
Outer training loop:  44%|####3     | 21852/50000 [13:19:09<14:27:23,  1.85s/it]
Outer training loop:  44%|####3     | 21853/50000 [13:19:11<14:53:27,  1.90s/it]
Outer training loop:  44%|####3     | 21854/50000 [13:19:13<14:39:51,  1.88s/it]
Outer training loop:  44%|####3     | 21855/50000 [13:19:15<14:16:50,  1.83s/it]
Outer training loop:  44%|####3     | 21856/50000 [13:19:17<14:59:43,  1.92s/it]
Outer training loop:  44%|####3     | 21857/50000 [13:19:19<14:42:06,  1.88s/it]
Outer training loop:  44%|####3     | 21858/50000 [13:19:20<14:36:00,  1.87s/it]
Outer training loop:  44%|####3     | 21859/50000 [13:19:22<14:09:44,  1.81s/it]
Outer training loop:  44%|####3     | 21860/50000 [13:19:24<14:23:19,  1.84s/it]
Outer training loop:  44%|####3     | 21861/50000 [13:19:26<14:25:35,  1.85s/it]
Outer training loop:  44%|####3     | 21862/50000 [13:19:28<14:29:24,  1.85s/it]
Outer training loop:  44%|####3     | 21863/50000 [13:19:30<14:32:25,  1.86s/it]
Outer training loop:  44%|####3     | 21864/50000 [13:19:31<14:39:23,  1.88s/it]
Outer training loop:  44%|####3     | 21865/50000 [13:19:33<14:36:38,  1.87s/it]
Outer training loop:  44%|####3     | 21866/50000 [13:19:35<14:31:02,  1.86s/it]
Outer training loop:  44%|####3     | 21867/50000 [13:19:37<14:57:41,  1.91s/it]
Outer training loop:  44%|####3     | 21868/50000 [13:19:39<14:30:10,  1.86s/it]
Outer training loop:  44%|####3     | 21869/50000 [13:19:41<14:25:49,  1.85s/it]
Outer training loop:  44%|####3     | 21870/50000 [13:19:43<14:41:33,  1.88s/it]
Outer training loop:  44%|####3     | 21871/50000 [13:19:45<14:33:46,  1.86s/it]
Outer training loop:  44%|####3     | 21872/50000 [13:19:46<14:25:18,  1.85s/it]
Outer training loop:  44%|####3     | 21873/50000 [13:19:48<14:43:11,  1.88s/it]
Outer training loop:  44%|####3     | 21874/50000 [13:19:50<14:29:53,  1.86s/it]
Outer training loop:  44%|####3     | 21875/50000 [13:19:52<14:29:45,  1.86s/it]
Outer training loop:  44%|####3     | 21876/50000 [13:19:54<14:19:07,  1.83s/it]
Outer training loop:  44%|####3     | 21877/50000 [13:19:56<14:46:31,  1.89s/it]
Outer training loop:  44%|####3     | 21878/50000 [13:19:58<14:42:54,  1.88s/it]
Outer training loop:  44%|####3     | 21879/50000 [13:19:59<14:35:28,  1.87s/it]
Outer training loop:  44%|####3     | 21880/50000 [13:20:02<14:59:11,  1.92s/it]
Outer training loop:  44%|####3     | 21881/50000 [13:20:03<14:25:57,  1.85s/it]
Outer training loop:  44%|####3     | 21882/50000 [13:20:05<14:41:11,  1.88s/it]
Outer training loop:  44%|####3     | 21883/50000 [13:20:07<14:59:05,  1.92s/it]
Outer training loop:  44%|####3     | 21884/50000 [13:20:09<14:39:03,  1.88s/it]
Outer training loop:  44%|####3     | 21885/50000 [13:20:11<14:29:57,  1.86s/it]
Outer training loop:  44%|####3     | 21886/50000 [13:20:13<14:32:21,  1.86s/it]
Outer training loop:  44%|####3     | 21887/50000 [13:20:15<14:45:04,  1.89s/it]
Outer training loop:  44%|####3     | 21888/50000 [13:20:16<14:32:39,  1.86s/it]
Outer training loop:  44%|####3     | 21889/50000 [13:20:18<14:14:40,  1.82s/it]
Outer training loop:  44%|####3     | 21890/50000 [13:20:20<14:31:32,  1.86s/it]
Outer training loop:  44%|####3     | 21891/50000 [13:20:22<14:02:20,  1.80s/it]
Outer training loop:  44%|####3     | 21892/50000 [13:20:24<14:07:29,  1.81s/it]
Outer training loop:  44%|####3     | 21893/50000 [13:20:26<14:35:44,  1.87s/it]
Outer training loop:  44%|####3     | 21894/50000 [13:20:27<14:35:44,  1.87s/it]
Outer training loop:  44%|####3     | 21895/50000 [13:20:29<14:21:24,  1.84s/it]
Outer training loop:  44%|####3     | 21896/50000 [13:20:31<14:00:43,  1.79s/it]
Outer training loop:  44%|####3     | 21897/50000 [13:20:33<14:34:09,  1.87s/it]
Outer training loop:  44%|####3     | 21898/50000 [13:20:35<14:30:02,  1.86s/it]
Outer training loop:  44%|####3     | 21899/50000 [13:20:37<14:14:11,  1.82s/it]
Outer training loop:  44%|####3     | 21900/50000 [13:20:39<14:41:44,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  44%|####3     | 21901/50000 [13:20:46<27:10:19,  3.48s/it]
Outer training loop:  44%|####3     | 21902/50000 [13:20:47<22:57:56,  2.94s/it]
Outer training loop:  44%|####3     | 21903/50000 [13:20:50<21:38:35,  2.77s/it]
Outer training loop:  44%|####3     | 21904/50000 [13:20:52<19:18:36,  2.47s/it]
Outer training loop:  44%|####3     | 21905/50000 [13:20:53<17:36:48,  2.26s/it]
Outer training loop:  44%|####3     | 21906/50000 [13:20:55<16:27:30,  2.11s/it]
Outer training loop:  44%|####3     | 21907/50000 [13:20:57<16:10:17,  2.07s/it]
Outer training loop:  44%|####3     | 21908/50000 [13:20:59<15:40:00,  2.01s/it]
Outer training loop:  44%|####3     | 21909/50000 [13:21:01<15:04:56,  1.93s/it]
Outer training loop:  44%|####3     | 21910/50000 [13:21:03<15:09:41,  1.94s/it]
Outer training loop:  44%|####3     | 21911/50000 [13:21:05<15:06:13,  1.94s/it]
Outer training loop:  44%|####3     | 21912/50000 [13:21:07<15:14:58,  1.95s/it]
Outer training loop:  44%|####3     | 21913/50000 [13:21:09<15:11:37,  1.95s/it]
Outer training loop:  44%|####3     | 21914/50000 [13:21:11<15:31:00,  1.99s/it]
Outer training loop:  44%|####3     | 21915/50000 [13:21:12<15:04:39,  1.93s/it]
Outer training loop:  44%|####3     | 21916/50000 [13:21:14<14:52:32,  1.91s/it]
Outer training loop:  44%|####3     | 21917/50000 [13:21:16<15:03:35,  1.93s/it]
Outer training loop:  44%|####3     | 21918/50000 [13:21:18<14:46:59,  1.90s/it]
Outer training loop:  44%|####3     | 21919/50000 [13:21:20<14:35:35,  1.87s/it]
Outer training loop:  44%|####3     | 21920/50000 [13:21:22<14:59:57,  1.92s/it]
Outer training loop:  44%|####3     | 21921/50000 [13:21:24<14:44:22,  1.89s/it]
Outer training loop:  44%|####3     | 21922/50000 [13:21:26<14:39:04,  1.88s/it]
Outer training loop:  44%|####3     | 21923/50000 [13:21:27<14:18:51,  1.84s/it]
Outer training loop:  44%|####3     | 21924/50000 [13:21:29<14:45:01,  1.89s/it]
Outer training loop:  44%|####3     | 21925/50000 [13:21:31<14:37:31,  1.88s/it]
Outer training loop:  44%|####3     | 21926/50000 [13:21:33<14:38:37,  1.88s/it]
Outer training loop:  44%|####3     | 21927/50000 [13:21:35<15:14:01,  1.95s/it]
Outer training loop:  44%|####3     | 21928/50000 [13:21:37<14:51:29,  1.91s/it]
Outer training loop:  44%|####3     | 21929/50000 [13:21:39<14:45:24,  1.89s/it]
Outer training loop:  44%|####3     | 21930/50000 [13:21:41<14:23:14,  1.85s/it]
Outer training loop:  44%|####3     | 21931/50000 [13:21:42<14:33:56,  1.87s/it]
Outer training loop:  44%|####3     | 21932/50000 [13:21:44<14:27:30,  1.85s/it]
Outer training loop:  44%|####3     | 21933/50000 [13:21:46<14:22:28,  1.84s/it]
Outer training loop:  44%|####3     | 21934/50000 [13:21:48<14:24:12,  1.85s/it]
Outer training loop:  44%|####3     | 21935/50000 [13:21:50<14:33:29,  1.87s/it]
Outer training loop:  44%|####3     | 21936/50000 [13:21:52<14:17:01,  1.83s/it]
Outer training loop:  44%|####3     | 21937/50000 [13:21:53<14:09:14,  1.82s/it]
Outer training loop:  44%|####3     | 21938/50000 [13:21:56<14:56:48,  1.92s/it]
Outer training loop:  44%|####3     | 21939/50000 [13:21:57<14:37:18,  1.88s/it]
Outer training loop:  44%|####3     | 21940/50000 [13:21:59<14:23:22,  1.85s/it]
Outer training loop:  44%|####3     | 21941/50000 [13:22:01<14:09:09,  1.82s/it]
Outer training loop:  44%|####3     | 21942/50000 [13:22:03<14:32:26,  1.87s/it]
Outer training loop:  44%|####3     | 21943/50000 [13:22:05<14:23:47,  1.85s/it]
Outer training loop:  44%|####3     | 21944/50000 [13:22:07<14:28:26,  1.86s/it]
Outer training loop:  44%|####3     | 21945/50000 [13:22:09<14:46:44,  1.90s/it]
Outer training loop:  44%|####3     | 21946/50000 [13:22:10<14:21:41,  1.84s/it]
Outer training loop:  44%|####3     | 21947/50000 [13:22:12<14:09:52,  1.82s/it]
Outer training loop:  44%|####3     | 21948/50000 [13:22:14<14:35:53,  1.87s/it]
Outer training loop:  44%|####3     | 21949/50000 [13:22:16<14:18:12,  1.84s/it]
Outer training loop:  44%|####3     | 21950/50000 [13:22:18<14:14:46,  1.83s/it]
Outer training loop:  44%|####3     | 21951/50000 [13:22:19<14:10:06,  1.82s/it]
Outer training loop:  44%|####3     | 21952/50000 [13:22:21<14:39:23,  1.88s/it]
Outer training loop:  44%|####3     | 21953/50000 [13:22:23<14:56:03,  1.92s/it]
Outer training loop:  44%|####3     | 21954/50000 [13:22:25<14:46:58,  1.90s/it]
Outer training loop:  44%|####3     | 21955/50000 [13:22:27<14:38:24,  1.88s/it]
Outer training loop:  44%|####3     | 21956/50000 [13:22:29<14:54:49,  1.91s/it]
Outer training loop:  44%|####3     | 21957/50000 [13:22:31<14:56:08,  1.92s/it]
Outer training loop:  44%|####3     | 21958/50000 [13:22:33<14:49:50,  1.90s/it]
Outer training loop:  44%|####3     | 21959/50000 [13:22:35<15:01:06,  1.93s/it]
Outer training loop:  44%|####3     | 21960/50000 [13:22:37<14:47:03,  1.90s/it]
Outer training loop:  44%|####3     | 21961/50000 [13:22:38<14:22:37,  1.85s/it]
Outer training loop:  44%|####3     | 21962/50000 [13:22:40<14:53:28,  1.91s/it]
Outer training loop:  44%|####3     | 21963/50000 [13:22:42<14:53:06,  1.91s/it]
Outer training loop:  44%|####3     | 21964/50000 [13:22:44<14:53:30,  1.91s/it]
Outer training loop:  44%|####3     | 21965/50000 [13:22:46<14:54:21,  1.91s/it]
Outer training loop:  44%|####3     | 21966/50000 [13:22:48<14:58:14,  1.92s/it]
Outer training loop:  44%|####3     | 21967/50000 [13:22:50<14:39:37,  1.88s/it]
Outer training loop:  44%|####3     | 21968/50000 [13:22:52<14:24:04,  1.85s/it]
Outer training loop:  44%|####3     | 21969/50000 [13:22:54<14:42:04,  1.89s/it]
Outer training loop:  44%|####3     | 21970/50000 [13:22:56<14:43:58,  1.89s/it]
Outer training loop:  44%|####3     | 21971/50000 [13:22:58<14:48:09,  1.90s/it]
Outer training loop:  44%|####3     | 21972/50000 [13:22:59<14:35:51,  1.87s/it]
Outer training loop:  44%|####3     | 21973/50000 [13:23:01<14:53:00,  1.91s/it]
Outer training loop:  44%|####3     | 21974/50000 [13:23:03<14:46:26,  1.90s/it]
Outer training loop:  44%|####3     | 21975/50000 [13:23:05<14:43:30,  1.89s/it]
Outer training loop:  44%|####3     | 21976/50000 [13:23:07<14:58:11,  1.92s/it]
Outer training loop:  44%|####3     | 21977/50000 [13:23:09<14:27:31,  1.86s/it]
Outer training loop:  44%|####3     | 21978/50000 [13:23:11<14:15:42,  1.83s/it]
Outer training loop:  44%|####3     | 21979/50000 [13:23:12<14:03:13,  1.81s/it]
Outer training loop:  44%|####3     | 21980/50000 [13:23:14<14:30:32,  1.86s/it]
Outer training loop:  44%|####3     | 21981/50000 [13:23:16<14:14:36,  1.83s/it]
Outer training loop:  44%|####3     | 21982/50000 [13:23:18<14:09:59,  1.82s/it]
Outer training loop:  44%|####3     | 21983/50000 [13:23:20<14:36:40,  1.88s/it]
Outer training loop:  44%|####3     | 21984/50000 [13:23:22<14:13:59,  1.83s/it]
Outer training loop:  44%|####3     | 21985/50000 [13:23:23<14:06:40,  1.81s/it]
Outer training loop:  44%|####3     | 21986/50000 [13:23:25<14:01:21,  1.80s/it]
Outer training loop:  44%|####3     | 21987/50000 [13:23:27<14:45:07,  1.90s/it]
Outer training loop:  44%|####3     | 21988/50000 [13:23:29<14:30:36,  1.86s/it]
Outer training loop:  44%|####3     | 21989/50000 [13:23:31<14:20:55,  1.84s/it]
Outer training loop:  44%|####3     | 21990/50000 [13:23:33<14:23:58,  1.85s/it]
Outer training loop:  44%|####3     | 21991/50000 [13:23:34<14:12:11,  1.83s/it]
Outer training loop:  44%|####3     | 21992/50000 [13:23:36<14:04:19,  1.81s/it]
Outer training loop:  44%|####3     | 21993/50000 [13:23:38<14:17:51,  1.84s/it]
Outer training loop:  44%|####3     | 21994/50000 [13:23:40<14:10:57,  1.82s/it]
Outer training loop:  44%|####3     | 21995/50000 [13:23:42<13:57:52,  1.80s/it]
Outer training loop:  44%|####3     | 21996/50000 [13:23:43<13:56:41,  1.79s/it]
Outer training loop:  44%|####3     | 21997/50000 [13:23:45<14:26:55,  1.86s/it]
Outer training loop:  44%|####3     | 21998/50000 [13:23:47<14:09:13,  1.82s/it]
Outer training loop:  44%|####3     | 21999/50000 [13:23:49<14:31:59,  1.87s/it]
Outer training loop:  44%|####4     | 22000/50000 [13:23:51<14:56:30,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:26,  1.79s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:29,  1.87s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:25,  1.84s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.86s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.93s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:20,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:15,  1.91s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:09,  1.85s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:03,  1.84s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:00,  1.82s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:57,  1.81s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:01,  1.90s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:53,  1.83s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:50,  1.81s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:49,  1.83s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:51,  1.92s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:49,  1.96s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:45,  1.95s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:40,  1.94s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:34,  1.92s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:20,  1.80s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:20,  1.82s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:17,  1.80s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.89s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:15,  1.88s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:11,  1.83s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:06,  1.81s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:27<01:04,  1.78s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:06,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:04,  1.91s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<01:01,  1.85s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:42<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:52,  1.96s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:49,  1.92s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:46,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:42,  1.86s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.83s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:38,  1.93s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.88s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:32,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:30,  1.80s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:28,  1.81s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:08<00:25,  1.79s/it]

Evaluating policy:  84%|########4 | 70/83 [02:10<00:23,  1.81s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:21,  1.80s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:21,  1.94s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.88s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.87s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.89s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.86s/it]

Evaluating policy:  93%|#########2| 77/83 [02:23<00:11,  1.86s/it]

Evaluating policy:  94%|#########3| 78/83 [02:25<00:09,  1.89s/it]

Evaluating policy:  95%|#########5| 79/83 [02:27<00:07,  1.93s/it]

Evaluating policy:  96%|#########6| 80/83 [02:29<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:04,  2.01s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.88s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.75s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:20,  1.74s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:20,  1.76s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:22,  1.80s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.79s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:18,  1.80s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.83s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:29,  2.00s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:25,  1.96s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:19,  1.96s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.88s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:11,  1.96s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:10,  1.98s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:05,  1.94s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.90s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:01,  1.96s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:57,  1.92s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:51,  1.96s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.93s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:42,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:41,  1.98s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:36,  1.93s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.88s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.86s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.89s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:27,  1.94s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:23,  1.90s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:22,  1.91s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:18,  1.88s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:16,  1.97s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:13,  1.93s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:08,  1.90s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:03,  1.86s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:02,  1.97s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:59,  1.92s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:52,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.86s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:49,  1.98s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:46,  1.92s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:43,  1.90s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:32,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:32,  1.90s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:32,  2.02s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:29,  1.96s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:27,  1.94s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.86s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.83s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.82s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.98s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.93s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.87s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.82s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.82s/it]

Outer training loop:  44%|####4     | 22001/50000 [13:29:12<757:47:33, 97.43s/it]
Outer training loop:  44%|####4     | 22002/50000 [13:29:13<534:40:35, 68.75s/it]
Outer training loop:  44%|####4     | 22003/50000 [13:29:15<378:32:30, 48.67s/it]
Outer training loop:  44%|####4     | 22004/50000 [13:29:18<270:28:50, 34.78s/it]
Outer training loop:  44%|####4     | 22005/50000 [13:29:19<193:30:38, 24.88s/it]
Outer training loop:  44%|####4     | 22006/50000 [13:29:21<139:36:55, 17.95s/it]
Outer training loop:  44%|####4     | 22007/50000 [13:29:23<102:31:17, 13.18s/it]
Outer training loop:  44%|####4     | 22008/50000 [13:29:25<76:13:21,  9.80s/it] 
Outer training loop:  44%|####4     | 22009/50000 [13:29:27<57:38:10,  7.41s/it]
Outer training loop:  44%|####4     | 22010/50000 [13:29:29<44:56:12,  5.78s/it]
Outer training loop:  44%|####4     | 22011/50000 [13:29:31<35:56:15,  4.62s/it]
Outer training loop:  44%|####4     | 22012/50000 [13:29:33<29:23:27,  3.78s/it]
Outer training loop:  44%|####4     | 22013/50000 [13:29:34<24:54:28,  3.20s/it]
Outer training loop:  44%|####4     | 22014/50000 [13:29:36<22:09:53,  2.85s/it]
Outer training loop:  44%|####4     | 22015/50000 [13:29:38<20:08:33,  2.59s/it]
Outer training loop:  44%|####4     | 22016/50000 [13:29:40<18:28:01,  2.38s/it]
Outer training loop:  44%|####4     | 22017/50000 [13:29:42<17:53:26,  2.30s/it]
Outer training loop:  44%|####4     | 22018/50000 [13:29:44<16:47:55,  2.16s/it]
Outer training loop:  44%|####4     | 22019/50000 [13:29:46<16:00:23,  2.06s/it]
Outer training loop:  44%|####4     | 22020/50000 [13:29:48<15:51:22,  2.04s/it]
Outer training loop:  44%|####4     | 22021/50000 [13:29:50<15:36:31,  2.01s/it]
Outer training loop:  44%|####4     | 22022/50000 [13:29:52<15:19:41,  1.97s/it]
Outer training loop:  44%|####4     | 22023/50000 [13:29:54<15:21:53,  1.98s/it]
Outer training loop:  44%|####4     | 22024/50000 [13:29:56<15:03:52,  1.94s/it]
Outer training loop:  44%|####4     | 22025/50000 [13:29:58<14:46:09,  1.90s/it]
Outer training loop:  44%|####4     | 22026/50000 [13:29:59<14:41:53,  1.89s/it]
Outer training loop:  44%|####4     | 22027/50000 [13:30:01<14:47:23,  1.90s/it]
Outer training loop:  44%|####4     | 22028/50000 [13:30:03<14:45:49,  1.90s/it]
Outer training loop:  44%|####4     | 22029/50000 [13:30:05<14:40:55,  1.89s/it]
Outer training loop:  44%|####4     | 22030/50000 [13:30:07<14:59:03,  1.93s/it]
Outer training loop:  44%|####4     | 22031/50000 [13:30:09<14:43:41,  1.90s/it]
Outer training loop:  44%|####4     | 22032/50000 [13:30:11<14:46:09,  1.90s/it]
Outer training loop:  44%|####4     | 22033/50000 [13:30:13<15:05:27,  1.94s/it]
Outer training loop:  44%|####4     | 22034/50000 [13:30:15<15:11:38,  1.96s/it]
Outer training loop:  44%|####4     | 22035/50000 [13:30:17<14:57:34,  1.93s/it]
Outer training loop:  44%|####4     | 22036/50000 [13:30:19<14:48:57,  1.91s/it]
Outer training loop:  44%|####4     | 22037/50000 [13:30:21<15:15:36,  1.96s/it]
Outer training loop:  44%|####4     | 22038/50000 [13:30:23<14:53:43,  1.92s/it]
Outer training loop:  44%|####4     | 22039/50000 [13:30:25<14:57:51,  1.93s/it]
Outer training loop:  44%|####4     | 22040/50000 [13:30:27<15:09:22,  1.95s/it]
Outer training loop:  44%|####4     | 22041/50000 [13:30:28<15:03:30,  1.94s/it]
Outer training loop:  44%|####4     | 22042/50000 [13:30:30<14:34:36,  1.88s/it]
Outer training loop:  44%|####4     | 22043/50000 [13:30:32<15:08:33,  1.95s/it]
Outer training loop:  44%|####4     | 22044/50000 [13:30:34<15:04:09,  1.94s/it]
Outer training loop:  44%|####4     | 22045/50000 [13:30:36<15:00:40,  1.93s/it]
Outer training loop:  44%|####4     | 22046/50000 [13:30:38<14:48:28,  1.91s/it]
Outer training loop:  44%|####4     | 22047/50000 [13:30:40<15:18:05,  1.97s/it]
Outer training loop:  44%|####4     | 22048/50000 [13:30:42<15:10:56,  1.96s/it]
Outer training loop:  44%|####4     | 22049/50000 [13:30:44<14:55:04,  1.92s/it]
Outer training loop:  44%|####4     | 22050/50000 [13:30:46<15:08:34,  1.95s/it]
Outer training loop:  44%|####4     | 22051/50000 [13:30:48<14:45:49,  1.90s/it]
Outer training loop:  44%|####4     | 22052/50000 [13:30:50<14:38:16,  1.89s/it]
Outer training loop:  44%|####4     | 22053/50000 [13:30:51<14:22:04,  1.85s/it]
Outer training loop:  44%|####4     | 22054/50000 [13:30:53<14:51:53,  1.91s/it]
Outer training loop:  44%|####4     | 22055/50000 [13:30:55<14:48:04,  1.91s/it]
Outer training loop:  44%|####4     | 22056/50000 [13:30:57<14:30:10,  1.87s/it]
Outer training loop:  44%|####4     | 22057/50000 [13:30:59<14:38:20,  1.89s/it]
Outer training loop:  44%|####4     | 22058/50000 [13:31:01<14:46:57,  1.90s/it]
Outer training loop:  44%|####4     | 22059/50000 [13:31:03<14:32:06,  1.87s/it]
Outer training loop:  44%|####4     | 22060/50000 [13:31:05<14:38:09,  1.89s/it]
Outer training loop:  44%|####4     | 22061/50000 [13:31:07<14:56:30,  1.93s/it]
Outer training loop:  44%|####4     | 22062/50000 [13:31:08<14:31:40,  1.87s/it]
Outer training loop:  44%|####4     | 22063/50000 [13:31:10<14:38:35,  1.89s/it]
Outer training loop:  44%|####4     | 22064/50000 [13:31:12<14:56:36,  1.93s/it]
Outer training loop:  44%|####4     | 22065/50000 [13:31:14<14:29:16,  1.87s/it]
Outer training loop:  44%|####4     | 22066/50000 [13:31:16<14:09:53,  1.83s/it]
Outer training loop:  44%|####4     | 22067/50000 [13:31:17<13:54:54,  1.79s/it]
Outer training loop:  44%|####4     | 22068/50000 [13:31:20<14:31:13,  1.87s/it]
Outer training loop:  44%|####4     | 22069/50000 [13:31:21<14:36:04,  1.88s/it]
Outer training loop:  44%|####4     | 22070/50000 [13:31:23<14:13:39,  1.83s/it]
Outer training loop:  44%|####4     | 22071/50000 [13:31:25<15:09:09,  1.95s/it]
Outer training loop:  44%|####4     | 22072/50000 [13:31:27<14:51:00,  1.91s/it]
Outer training loop:  44%|####4     | 22073/50000 [13:31:29<14:30:16,  1.87s/it]
Outer training loop:  44%|####4     | 22074/50000 [13:31:31<14:12:55,  1.83s/it]
Outer training loop:  44%|####4     | 22075/50000 [13:31:33<14:47:40,  1.91s/it]
Outer training loop:  44%|####4     | 22076/50000 [13:31:35<15:01:36,  1.94s/it]
Outer training loop:  44%|####4     | 22077/50000 [13:31:37<14:37:49,  1.89s/it]
Outer training loop:  44%|####4     | 22078/50000 [13:31:39<15:10:13,  1.96s/it]
Outer training loop:  44%|####4     | 22079/50000 [13:31:41<15:20:21,  1.98s/it]
Outer training loop:  44%|####4     | 22080/50000 [13:31:42<14:49:14,  1.91s/it]
Outer training loop:  44%|####4     | 22081/50000 [13:31:44<15:00:08,  1.93s/it]
Outer training loop:  44%|####4     | 22082/50000 [13:31:46<14:38:32,  1.89s/it]
Outer training loop:  44%|####4     | 22083/50000 [13:31:48<14:28:09,  1.87s/it]
Outer training loop:  44%|####4     | 22084/50000 [13:31:50<15:06:11,  1.95s/it]
Outer training loop:  44%|####4     | 22085/50000 [13:31:52<15:07:58,  1.95s/it]
Outer training loop:  44%|####4     | 22086/50000 [13:31:54<14:48:57,  1.91s/it]
Outer training loop:  44%|####4     | 22087/50000 [13:31:56<14:48:02,  1.91s/it]
Outer training loop:  44%|####4     | 22088/50000 [13:31:58<14:52:43,  1.92s/it]
Outer training loop:  44%|####4     | 22089/50000 [13:32:00<14:23:02,  1.86s/it]
Outer training loop:  44%|####4     | 22090/50000 [13:32:01<14:22:40,  1.85s/it]
Outer training loop:  44%|####4     | 22091/50000 [13:32:03<14:20:48,  1.85s/it]
Outer training loop:  44%|####4     | 22092/50000 [13:32:05<14:59:21,  1.93s/it]
Outer training loop:  44%|####4     | 22093/50000 [13:32:07<14:45:19,  1.90s/it]
Outer training loop:  44%|####4     | 22094/50000 [13:32:09<14:14:47,  1.84s/it]
Outer training loop:  44%|####4     | 22095/50000 [13:32:11<14:39:56,  1.89s/it]
Outer training loop:  44%|####4     | 22096/50000 [13:32:13<14:41:06,  1.89s/it]
Outer training loop:  44%|####4     | 22097/50000 [13:32:15<14:31:02,  1.87s/it]
Outer training loop:  44%|####4     | 22098/50000 [13:32:17<14:42:24,  1.90s/it]
Outer training loop:  44%|####4     | 22099/50000 [13:32:18<14:38:15,  1.89s/it]
Outer training loop:  44%|####4     | 22100/50000 [13:32:20<14:38:21,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  44%|####4     | 22101/50000 [13:32:28<27:17:55,  3.52s/it]
Outer training loop:  44%|####4     | 22102/50000 [13:32:30<23:45:07,  3.07s/it]
Outer training loop:  44%|####4     | 22103/50000 [13:32:31<20:37:36,  2.66s/it]
Outer training loop:  44%|####4     | 22104/50000 [13:32:33<18:49:52,  2.43s/it]
Outer training loop:  44%|####4     | 22105/50000 [13:32:35<17:20:35,  2.24s/it]
Outer training loop:  44%|####4     | 22106/50000 [13:32:37<16:40:42,  2.15s/it]
Outer training loop:  44%|####4     | 22107/50000 [13:32:39<15:43:09,  2.03s/it]
Outer training loop:  44%|####4     | 22108/50000 [13:32:41<15:28:40,  2.00s/it]
Outer training loop:  44%|####4     | 22109/50000 [13:32:42<14:58:17,  1.93s/it]
Outer training loop:  44%|####4     | 22110/50000 [13:32:44<15:06:08,  1.95s/it]
Outer training loop:  44%|####4     | 22111/50000 [13:32:46<14:44:01,  1.90s/it]
Outer training loop:  44%|####4     | 22112/50000 [13:32:48<14:17:51,  1.85s/it]
Outer training loop:  44%|####4     | 22113/50000 [13:32:50<14:50:26,  1.92s/it]
Outer training loop:  44%|####4     | 22114/50000 [13:32:52<14:36:37,  1.89s/it]
Outer training loop:  44%|####4     | 22115/50000 [13:32:54<14:26:17,  1.86s/it]
Outer training loop:  44%|####4     | 22116/50000 [13:32:55<14:16:55,  1.84s/it]
Outer training loop:  44%|####4     | 22117/50000 [13:32:57<14:35:07,  1.88s/it]
Outer training loop:  44%|####4     | 22118/50000 [13:32:59<14:23:25,  1.86s/it]
Outer training loop:  44%|####4     | 22119/50000 [13:33:01<14:15:27,  1.84s/it]
Outer training loop:  44%|####4     | 22120/50000 [13:33:03<14:36:52,  1.89s/it]
Outer training loop:  44%|####4     | 22121/50000 [13:33:05<14:20:56,  1.85s/it]
Outer training loop:  44%|####4     | 22122/50000 [13:33:07<14:27:48,  1.87s/it]
Outer training loop:  44%|####4     | 22123/50000 [13:33:09<14:38:37,  1.89s/it]
Outer training loop:  44%|####4     | 22124/50000 [13:33:11<14:35:35,  1.88s/it]
Outer training loop:  44%|####4     | 22125/50000 [13:33:12<14:32:55,  1.88s/it]
Outer training loop:  44%|####4     | 22126/50000 [13:33:14<15:00:31,  1.94s/it]
Outer training loop:  44%|####4     | 22127/50000 [13:33:16<14:48:49,  1.91s/it]
Outer training loop:  44%|####4     | 22128/50000 [13:33:18<14:31:34,  1.88s/it]
Outer training loop:  44%|####4     | 22129/50000 [13:33:20<14:24:06,  1.86s/it]
Outer training loop:  44%|####4     | 22130/50000 [13:33:22<14:28:31,  1.87s/it]
Outer training loop:  44%|####4     | 22131/50000 [13:33:24<14:42:13,  1.90s/it]
Outer training loop:  44%|####4     | 22132/50000 [13:33:26<14:40:48,  1.90s/it]
Outer training loop:  44%|####4     | 22133/50000 [13:33:28<14:54:52,  1.93s/it]
Outer training loop:  44%|####4     | 22134/50000 [13:33:30<14:42:43,  1.90s/it]
Outer training loop:  44%|####4     | 22135/50000 [13:33:31<14:47:04,  1.91s/it]
Outer training loop:  44%|####4     | 22136/50000 [13:33:33<14:46:39,  1.91s/it]
Outer training loop:  44%|####4     | 22137/50000 [13:33:35<14:49:37,  1.92s/it]
Outer training loop:  44%|####4     | 22138/50000 [13:33:37<14:32:09,  1.88s/it]
Outer training loop:  44%|####4     | 22139/50000 [13:33:39<14:44:27,  1.90s/it]
Outer training loop:  44%|####4     | 22140/50000 [13:33:41<14:55:33,  1.93s/it]
Outer training loop:  44%|####4     | 22141/50000 [13:33:43<14:35:13,  1.88s/it]
Outer training loop:  44%|####4     | 22142/50000 [13:33:45<14:14:04,  1.84s/it]
Outer training loop:  44%|####4     | 22143/50000 [13:33:46<14:28:14,  1.87s/it]
Outer training loop:  44%|####4     | 22144/50000 [13:33:48<14:18:46,  1.85s/it]
Outer training loop:  44%|####4     | 22145/50000 [13:33:50<14:26:46,  1.87s/it]
Outer training loop:  44%|####4     | 22146/50000 [13:33:52<14:32:03,  1.88s/it]
Outer training loop:  44%|####4     | 22147/50000 [13:33:54<14:11:16,  1.83s/it]
Outer training loop:  44%|####4     | 22148/50000 [13:33:56<14:08:07,  1.83s/it]
Outer training loop:  44%|####4     | 22149/50000 [13:33:58<14:13:25,  1.84s/it]
Outer training loop:  44%|####4     | 22150/50000 [13:34:00<14:36:19,  1.89s/it]
Outer training loop:  44%|####4     | 22151/50000 [13:34:01<14:21:03,  1.86s/it]
Outer training loop:  44%|####4     | 22152/50000 [13:34:03<14:12:00,  1.84s/it]
Outer training loop:  44%|####4     | 22153/50000 [13:34:05<14:44:28,  1.91s/it]
Outer training loop:  44%|####4     | 22154/50000 [13:34:07<14:43:20,  1.90s/it]
Outer training loop:  44%|####4     | 22155/50000 [13:34:09<14:25:53,  1.87s/it]
Outer training loop:  44%|####4     | 22156/50000 [13:34:11<14:51:20,  1.92s/it]
Outer training loop:  44%|####4     | 22157/50000 [13:34:13<14:42:57,  1.90s/it]
Outer training loop:  44%|####4     | 22158/50000 [13:34:14<14:21:43,  1.86s/it]
Outer training loop:  44%|####4     | 22159/50000 [13:34:16<14:09:17,  1.83s/it]
Outer training loop:  44%|####4     | 22160/50000 [13:34:18<14:27:58,  1.87s/it]
Outer training loop:  44%|####4     | 22161/50000 [13:34:20<14:26:16,  1.87s/it]
Outer training loop:  44%|####4     | 22162/50000 [13:34:22<14:19:12,  1.85s/it]
Outer training loop:  44%|####4     | 22163/50000 [13:34:24<14:07:58,  1.83s/it]
Outer training loop:  44%|####4     | 22164/50000 [13:34:26<15:00:33,  1.94s/it]
Outer training loop:  44%|####4     | 22165/50000 [13:34:28<14:49:41,  1.92s/it]
Outer training loop:  44%|####4     | 22166/50000 [13:34:30<14:52:46,  1.92s/it]
Outer training loop:  44%|####4     | 22167/50000 [13:34:32<14:49:14,  1.92s/it]
Outer training loop:  44%|####4     | 22168/50000 [13:34:33<14:31:04,  1.88s/it]
Outer training loop:  44%|####4     | 22169/50000 [13:34:35<14:37:00,  1.89s/it]
Outer training loop:  44%|####4     | 22170/50000 [13:34:37<14:30:11,  1.88s/it]
Outer training loop:  44%|####4     | 22171/50000 [13:34:39<14:30:47,  1.88s/it]
Outer training loop:  44%|####4     | 22172/50000 [13:34:41<14:41:32,  1.90s/it]
Outer training loop:  44%|####4     | 22173/50000 [13:34:43<14:34:50,  1.89s/it]
Outer training loop:  44%|####4     | 22174/50000 [13:34:45<14:31:52,  1.88s/it]
Outer training loop:  44%|####4     | 22175/50000 [13:34:46<14:14:21,  1.84s/it]
Outer training loop:  44%|####4     | 22176/50000 [13:34:48<14:03:39,  1.82s/it]
Outer training loop:  44%|####4     | 22177/50000 [13:34:50<14:09:50,  1.83s/it]
Outer training loop:  44%|####4     | 22178/50000 [13:34:52<14:14:08,  1.84s/it]
Outer training loop:  44%|####4     | 22179/50000 [13:34:54<13:57:37,  1.81s/it]
Outer training loop:  44%|####4     | 22180/50000 [13:34:55<13:49:37,  1.79s/it]
Outer training loop:  44%|####4     | 22181/50000 [13:34:58<14:35:37,  1.89s/it]
Outer training loop:  44%|####4     | 22182/50000 [13:34:59<14:37:22,  1.89s/it]
Outer training loop:  44%|####4     | 22183/50000 [13:35:01<14:34:17,  1.89s/it]
Outer training loop:  44%|####4     | 22184/50000 [13:35:03<14:17:35,  1.85s/it]
Outer training loop:  44%|####4     | 22185/50000 [13:35:05<14:22:49,  1.86s/it]
Outer training loop:  44%|####4     | 22186/50000 [13:35:07<14:22:07,  1.86s/it]
Outer training loop:  44%|####4     | 22187/50000 [13:35:09<14:22:36,  1.86s/it]
Outer training loop:  44%|####4     | 22188/50000 [13:35:11<14:24:30,  1.87s/it]
Outer training loop:  44%|####4     | 22189/50000 [13:35:13<14:42:23,  1.90s/it]
Outer training loop:  44%|####4     | 22190/50000 [13:35:14<14:19:37,  1.85s/it]
Outer training loop:  44%|####4     | 22191/50000 [13:35:16<14:20:07,  1.86s/it]
Outer training loop:  44%|####4     | 22192/50000 [13:35:18<14:36:26,  1.89s/it]
Outer training loop:  44%|####4     | 22193/50000 [13:35:20<14:56:04,  1.93s/it]
Outer training loop:  44%|####4     | 22194/50000 [13:35:22<14:48:40,  1.92s/it]
Outer training loop:  44%|####4     | 22195/50000 [13:35:24<14:53:11,  1.93s/it]
Outer training loop:  44%|####4     | 22196/50000 [13:35:26<14:42:56,  1.91s/it]
Outer training loop:  44%|####4     | 22197/50000 [13:35:28<14:51:35,  1.92s/it]
Outer training loop:  44%|####4     | 22198/50000 [13:35:30<14:45:11,  1.91s/it]
Outer training loop:  44%|####4     | 22199/50000 [13:35:32<14:39:38,  1.90s/it]
Outer training loop:  44%|####4     | 22200/50000 [13:35:33<14:27:25,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  44%|####4     | 22201/50000 [13:35:41<27:06:47,  3.51s/it]
Outer training loop:  44%|####4     | 22202/50000 [13:35:43<23:12:16,  3.01s/it]
Outer training loop:  44%|####4     | 22203/50000 [13:35:44<20:43:44,  2.68s/it]
Outer training loop:  44%|####4     | 22204/50000 [13:35:46<18:35:11,  2.41s/it]
Outer training loop:  44%|####4     | 22205/50000 [13:35:48<17:06:32,  2.22s/it]
Outer training loop:  44%|####4     | 22206/50000 [13:35:50<16:39:12,  2.16s/it]
Outer training loop:  44%|####4     | 22207/50000 [13:35:52<15:35:26,  2.02s/it]
Outer training loop:  44%|####4     | 22208/50000 [13:35:54<15:12:57,  1.97s/it]
Outer training loop:  44%|####4     | 22209/50000 [13:35:55<14:44:01,  1.91s/it]
Outer training loop:  44%|####4     | 22210/50000 [13:35:57<15:22:34,  1.99s/it]
Outer training loop:  44%|####4     | 22211/50000 [13:35:59<14:53:12,  1.93s/it]
Outer training loop:  44%|####4     | 22212/50000 [13:36:01<14:43:17,  1.91s/it]
Outer training loop:  44%|####4     | 22213/50000 [13:36:03<15:02:12,  1.95s/it]
Outer training loop:  44%|####4     | 22214/50000 [13:36:05<14:44:11,  1.91s/it]
Outer training loop:  44%|####4     | 22215/50000 [13:36:07<14:16:13,  1.85s/it]
Outer training loop:  44%|####4     | 22216/50000 [13:36:09<14:32:20,  1.88s/it]
Outer training loop:  44%|####4     | 22217/50000 [13:36:11<14:26:30,  1.87s/it]
Outer training loop:  44%|####4     | 22218/50000 [13:36:12<14:18:56,  1.86s/it]
Outer training loop:  44%|####4     | 22219/50000 [13:36:14<14:03:20,  1.82s/it]
Outer training loop:  44%|####4     | 22220/50000 [13:36:16<14:28:37,  1.88s/it]
Outer training loop:  44%|####4     | 22221/50000 [13:36:18<14:01:12,  1.82s/it]
Outer training loop:  44%|####4     | 22222/50000 [13:36:20<13:57:04,  1.81s/it]
Outer training loop:  44%|####4     | 22223/50000 [13:36:22<14:51:25,  1.93s/it]
Outer training loop:  44%|####4     | 22224/50000 [13:36:24<14:46:39,  1.92s/it]
Outer training loop:  44%|####4     | 22225/50000 [13:36:26<14:52:48,  1.93s/it]
Outer training loop:  44%|####4     | 22226/50000 [13:36:28<14:59:59,  1.94s/it]
Outer training loop:  44%|####4     | 22227/50000 [13:36:29<14:56:06,  1.94s/it]
Outer training loop:  44%|####4     | 22228/50000 [13:36:32<15:14:45,  1.98s/it]
Outer training loop:  44%|####4     | 22229/50000 [13:36:34<15:10:50,  1.97s/it]
Outer training loop:  44%|####4     | 22230/50000 [13:36:35<14:54:22,  1.93s/it]
Outer training loop:  44%|####4     | 22231/50000 [13:36:37<14:33:06,  1.89s/it]
Outer training loop:  44%|####4     | 22232/50000 [13:36:39<14:32:27,  1.89s/it]
Outer training loop:  44%|####4     | 22233/50000 [13:36:41<15:05:26,  1.96s/it]
Outer training loop:  44%|####4     | 22234/50000 [13:36:43<15:06:06,  1.96s/it]
Outer training loop:  44%|####4     | 22235/50000 [13:36:45<14:41:50,  1.91s/it]
Outer training loop:  44%|####4     | 22236/50000 [13:36:47<14:43:54,  1.91s/it]
Outer training loop:  44%|####4     | 22237/50000 [13:36:49<15:12:04,  1.97s/it]
Outer training loop:  44%|####4     | 22238/50000 [13:36:51<14:48:50,  1.92s/it]
Outer training loop:  44%|####4     | 22239/50000 [13:36:53<14:34:09,  1.89s/it]
Outer training loop:  44%|####4     | 22240/50000 [13:36:55<14:58:43,  1.94s/it]
Outer training loop:  44%|####4     | 22241/50000 [13:36:56<14:36:31,  1.89s/it]
Outer training loop:  44%|####4     | 22242/50000 [13:36:58<14:43:35,  1.91s/it]
Outer training loop:  44%|####4     | 22243/50000 [13:37:00<15:02:15,  1.95s/it]
Outer training loop:  44%|####4     | 22244/50000 [13:37:02<14:52:53,  1.93s/it]
Outer training loop:  44%|####4     | 22245/50000 [13:37:04<15:08:10,  1.96s/it]
Outer training loop:  44%|####4     | 22246/50000 [13:37:06<15:17:06,  1.98s/it]
Outer training loop:  44%|####4     | 22247/50000 [13:37:08<14:54:22,  1.93s/it]
Outer training loop:  44%|####4     | 22248/50000 [13:37:10<14:56:22,  1.94s/it]
Outer training loop:  44%|####4     | 22249/50000 [13:37:12<14:47:03,  1.92s/it]
Outer training loop:  44%|####4     | 22250/50000 [13:37:14<14:50:07,  1.92s/it]
Outer training loop:  45%|####4     | 22251/50000 [13:37:16<14:47:38,  1.92s/it]
Outer training loop:  45%|####4     | 22252/50000 [13:37:18<14:38:04,  1.90s/it]
Outer training loop:  45%|####4     | 22253/50000 [13:37:20<15:09:43,  1.97s/it]
Outer training loop:  45%|####4     | 22254/50000 [13:37:22<14:56:06,  1.94s/it]
Outer training loop:  45%|####4     | 22255/50000 [13:37:23<14:32:47,  1.89s/it]
Outer training loop:  45%|####4     | 22256/50000 [13:37:25<14:55:52,  1.94s/it]
Outer training loop:  45%|####4     | 22257/50000 [13:37:27<14:53:26,  1.93s/it]
Outer training loop:  45%|####4     | 22258/50000 [13:37:29<14:35:31,  1.89s/it]
Outer training loop:  45%|####4     | 22259/50000 [13:37:31<14:23:50,  1.87s/it]
Outer training loop:  45%|####4     | 22260/50000 [13:37:33<14:51:13,  1.93s/it]
Outer training loop:  45%|####4     | 22261/50000 [13:37:35<14:55:56,  1.94s/it]
Outer training loop:  45%|####4     | 22262/50000 [13:37:37<14:57:28,  1.94s/it]
Outer training loop:  45%|####4     | 22263/50000 [13:37:39<15:17:06,  1.98s/it]
Outer training loop:  45%|####4     | 22264/50000 [13:37:41<14:38:40,  1.90s/it]
Outer training loop:  45%|####4     | 22265/50000 [13:37:43<14:57:02,  1.94s/it]
Outer training loop:  45%|####4     | 22266/50000 [13:37:45<14:49:54,  1.93s/it]
Outer training loop:  45%|####4     | 22267/50000 [13:37:47<15:18:04,  1.99s/it]
Outer training loop:  45%|####4     | 22268/50000 [13:37:49<15:00:55,  1.95s/it]
Outer training loop:  45%|####4     | 22269/50000 [13:37:50<14:34:30,  1.89s/it]
Outer training loop:  45%|####4     | 22270/50000 [13:37:53<15:02:22,  1.95s/it]
Outer training loop:  45%|####4     | 22271/50000 [13:37:54<14:57:49,  1.94s/it]
Outer training loop:  45%|####4     | 22272/50000 [13:37:56<14:27:21,  1.88s/it]
Outer training loop:  45%|####4     | 22273/50000 [13:37:58<14:42:16,  1.91s/it]
Outer training loop:  45%|####4     | 22274/50000 [13:38:00<14:33:19,  1.89s/it]
Outer training loop:  45%|####4     | 22275/50000 [13:38:02<14:21:34,  1.86s/it]
Outer training loop:  45%|####4     | 22276/50000 [13:38:03<13:53:34,  1.80s/it]
Outer training loop:  45%|####4     | 22277/50000 [13:38:06<14:38:05,  1.90s/it]
Outer training loop:  45%|####4     | 22278/50000 [13:38:08<15:13:18,  1.98s/it]
Outer training loop:  45%|####4     | 22279/50000 [13:38:10<15:11:47,  1.97s/it]
Outer training loop:  45%|####4     | 22280/50000 [13:38:12<15:19:34,  1.99s/it]
Outer training loop:  45%|####4     | 22281/50000 [13:38:14<15:16:00,  1.98s/it]
Outer training loop:  45%|####4     | 22282/50000 [13:38:16<15:01:57,  1.95s/it]
Outer training loop:  45%|####4     | 22283/50000 [13:38:18<15:14:33,  1.98s/it]
Outer training loop:  45%|####4     | 22284/50000 [13:38:20<15:09:45,  1.97s/it]
Outer training loop:  45%|####4     | 22285/50000 [13:38:21<14:52:12,  1.93s/it]
Outer training loop:  45%|####4     | 22286/50000 [13:38:23<14:35:02,  1.89s/it]
Outer training loop:  45%|####4     | 22287/50000 [13:38:25<15:09:46,  1.97s/it]
Outer training loop:  45%|####4     | 22288/50000 [13:38:27<14:39:49,  1.90s/it]
Outer training loop:  45%|####4     | 22289/50000 [13:38:29<14:43:42,  1.91s/it]
Outer training loop:  45%|####4     | 22290/50000 [13:38:31<14:52:33,  1.93s/it]
Outer training loop:  45%|####4     | 22291/50000 [13:38:33<14:40:47,  1.91s/it]
Outer training loop:  45%|####4     | 22292/50000 [13:38:35<14:51:18,  1.93s/it]
Outer training loop:  45%|####4     | 22293/50000 [13:38:37<14:29:29,  1.88s/it]
Outer training loop:  45%|####4     | 22294/50000 [13:38:39<14:42:50,  1.91s/it]
Outer training loop:  45%|####4     | 22295/50000 [13:38:40<14:27:31,  1.88s/it]
Outer training loop:  45%|####4     | 22296/50000 [13:38:42<14:13:28,  1.85s/it]
Outer training loop:  45%|####4     | 22297/50000 [13:38:44<14:28:15,  1.88s/it]
Outer training loop:  45%|####4     | 22298/50000 [13:38:46<14:27:34,  1.88s/it]
Outer training loop:  45%|####4     | 22299/50000 [13:38:48<14:19:01,  1.86s/it]
Outer training loop:  45%|####4     | 22300/50000 [13:38:50<14:21:33,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  45%|####4     | 22301/50000 [13:38:57<27:28:58,  3.57s/it]
Outer training loop:  45%|####4     | 22302/50000 [13:38:59<23:53:48,  3.11s/it]
Outer training loop:  45%|####4     | 22303/50000 [13:39:01<21:06:51,  2.74s/it]
Outer training loop:  45%|####4     | 22304/50000 [13:39:03<19:19:43,  2.51s/it]
Outer training loop:  45%|####4     | 22305/50000 [13:39:05<17:53:00,  2.32s/it]
Outer training loop:  45%|####4     | 22306/50000 [13:39:07<16:37:16,  2.16s/it]
Outer training loop:  45%|####4     | 22307/50000 [13:39:09<15:51:33,  2.06s/it]
Outer training loop:  45%|####4     | 22308/50000 [13:39:11<15:46:38,  2.05s/it]
Outer training loop:  45%|####4     | 22309/50000 [13:39:13<15:30:14,  2.02s/it]
Outer training loop:  45%|####4     | 22310/50000 [13:39:14<15:03:19,  1.96s/it]
Outer training loop:  45%|####4     | 22311/50000 [13:39:16<15:04:05,  1.96s/it]
Outer training loop:  45%|####4     | 22312/50000 [13:39:18<14:32:08,  1.89s/it]
Outer training loop:  45%|####4     | 22313/50000 [13:39:20<14:25:29,  1.88s/it]
Outer training loop:  45%|####4     | 22314/50000 [13:39:22<14:48:35,  1.93s/it]
Outer training loop:  45%|####4     | 22315/50000 [13:39:24<14:30:20,  1.89s/it]
Outer training loop:  45%|####4     | 22316/50000 [13:39:26<14:14:27,  1.85s/it]
Outer training loop:  45%|####4     | 22317/50000 [13:39:27<14:16:08,  1.86s/it]
Outer training loop:  45%|####4     | 22318/50000 [13:39:29<14:36:26,  1.90s/it]
Outer training loop:  45%|####4     | 22319/50000 [13:39:31<14:32:14,  1.89s/it]
Outer training loop:  45%|####4     | 22320/50000 [13:39:33<14:25:30,  1.88s/it]
Outer training loop:  45%|####4     | 22321/50000 [13:39:35<15:03:52,  1.96s/it]
Outer training loop:  45%|####4     | 22322/50000 [13:39:37<14:43:41,  1.92s/it]
Outer training loop:  45%|####4     | 22323/50000 [13:39:39<14:11:47,  1.85s/it]
Outer training loop:  45%|####4     | 22324/50000 [13:39:41<14:12:33,  1.85s/it]
Outer training loop:  45%|####4     | 22325/50000 [13:39:43<15:21:55,  2.00s/it]
Outer training loop:  45%|####4     | 22326/50000 [13:39:45<14:53:17,  1.94s/it]
Outer training loop:  45%|####4     | 22327/50000 [13:39:47<14:29:48,  1.89s/it]
Outer training loop:  45%|####4     | 22328/50000 [13:39:49<14:40:03,  1.91s/it]
Outer training loop:  45%|####4     | 22329/50000 [13:39:50<14:36:43,  1.90s/it]
Outer training loop:  45%|####4     | 22330/50000 [13:39:52<14:04:39,  1.83s/it]
Outer training loop:  45%|####4     | 22331/50000 [13:39:54<14:09:13,  1.84s/it]
Outer training loop:  45%|####4     | 22332/50000 [13:39:56<14:28:06,  1.88s/it]
Outer training loop:  45%|####4     | 22333/50000 [13:39:58<14:25:30,  1.88s/it]
Outer training loop:  45%|####4     | 22334/50000 [13:40:00<14:00:51,  1.82s/it]
Outer training loop:  45%|####4     | 22335/50000 [13:40:01<14:03:28,  1.83s/it]
Outer training loop:  45%|####4     | 22336/50000 [13:40:03<13:56:48,  1.81s/it]
Outer training loop:  45%|####4     | 22337/50000 [13:40:05<14:01:25,  1.83s/it]
Outer training loop:  45%|####4     | 22338/50000 [13:40:07<14:41:38,  1.91s/it]
Outer training loop:  45%|####4     | 22339/50000 [13:40:09<14:39:45,  1.91s/it]
Outer training loop:  45%|####4     | 22340/50000 [13:40:11<14:37:17,  1.90s/it]
Outer training loop:  45%|####4     | 22341/50000 [13:40:13<14:24:37,  1.88s/it]
Outer training loop:  45%|####4     | 22342/50000 [13:40:15<14:27:20,  1.88s/it]
Outer training loop:  45%|####4     | 22343/50000 [13:40:16<14:14:18,  1.85s/it]
Outer training loop:  45%|####4     | 22344/50000 [13:40:18<14:25:00,  1.88s/it]
Outer training loop:  45%|####4     | 22345/50000 [13:40:20<14:41:58,  1.91s/it]
Outer training loop:  45%|####4     | 22346/50000 [13:40:22<14:30:54,  1.89s/it]
Outer training loop:  45%|####4     | 22347/50000 [13:40:24<14:10:54,  1.85s/it]
Outer training loop:  45%|####4     | 22348/50000 [13:40:26<14:31:39,  1.89s/it]
Outer training loop:  45%|####4     | 22349/50000 [13:40:28<14:25:38,  1.88s/it]
Outer training loop:  45%|####4     | 22350/50000 [13:40:30<14:08:16,  1.84s/it]
Outer training loop:  45%|####4     | 22351/50000 [13:40:31<14:06:07,  1.84s/it]
Outer training loop:  45%|####4     | 22352/50000 [13:40:33<14:24:04,  1.88s/it]
Outer training loop:  45%|####4     | 22353/50000 [13:40:35<14:13:44,  1.85s/it]
Outer training loop:  45%|####4     | 22354/50000 [13:40:37<13:54:42,  1.81s/it]
Outer training loop:  45%|####4     | 22355/50000 [13:40:39<14:32:12,  1.89s/it]
Outer training loop:  45%|####4     | 22356/50000 [13:40:41<14:33:23,  1.90s/it]
Outer training loop:  45%|####4     | 22357/50000 [13:40:43<14:26:17,  1.88s/it]
Outer training loop:  45%|####4     | 22358/50000 [13:40:45<14:35:01,  1.90s/it]
Outer training loop:  45%|####4     | 22359/50000 [13:40:46<14:22:28,  1.87s/it]
Outer training loop:  45%|####4     | 22360/50000 [13:40:48<14:09:48,  1.84s/it]
Outer training loop:  45%|####4     | 22361/50000 [13:40:50<14:27:46,  1.88s/it]
Outer training loop:  45%|####4     | 22362/50000 [13:40:52<15:00:22,  1.95s/it]
Outer training loop:  45%|####4     | 22363/50000 [13:40:54<14:45:01,  1.92s/it]
Outer training loop:  45%|####4     | 22364/50000 [13:40:56<14:37:51,  1.91s/it]
Outer training loop:  45%|####4     | 22365/50000 [13:40:58<15:04:19,  1.96s/it]
Outer training loop:  45%|####4     | 22366/50000 [13:41:00<15:02:59,  1.96s/it]
Outer training loop:  45%|####4     | 22367/50000 [13:41:02<14:58:51,  1.95s/it]
Outer training loop:  45%|####4     | 22368/50000 [13:41:04<15:17:03,  1.99s/it]
Outer training loop:  45%|####4     | 22369/50000 [13:41:06<14:58:58,  1.95s/it]
Outer training loop:  45%|####4     | 22370/50000 [13:41:08<14:36:42,  1.90s/it]
Outer training loop:  45%|####4     | 22371/50000 [13:41:10<14:36:09,  1.90s/it]
Outer training loop:  45%|####4     | 22372/50000 [13:41:12<14:46:47,  1.93s/it]
Outer training loop:  45%|####4     | 22373/50000 [13:41:13<14:29:06,  1.89s/it]
Outer training loop:  45%|####4     | 22374/50000 [13:41:15<14:11:23,  1.85s/it]
Outer training loop:  45%|####4     | 22375/50000 [13:41:17<14:40:53,  1.91s/it]
Outer training loop:  45%|####4     | 22376/50000 [13:41:19<14:32:12,  1.89s/it]
Outer training loop:  45%|####4     | 22377/50000 [13:41:21<14:21:18,  1.87s/it]
Outer training loop:  45%|####4     | 22378/50000 [13:41:23<14:19:53,  1.87s/it]
Outer training loop:  45%|####4     | 22379/50000 [13:41:25<15:04:58,  1.97s/it]
Outer training loop:  45%|####4     | 22380/50000 [13:41:27<14:40:23,  1.91s/it]
Outer training loop:  45%|####4     | 22381/50000 [13:41:29<14:42:49,  1.92s/it]
Outer training loop:  45%|####4     | 22382/50000 [13:41:31<14:36:16,  1.90s/it]
Outer training loop:  45%|####4     | 22383/50000 [13:41:32<14:27:56,  1.89s/it]
Outer training loop:  45%|####4     | 22384/50000 [13:41:34<14:15:09,  1.86s/it]
Outer training loop:  45%|####4     | 22385/50000 [13:41:36<14:29:26,  1.89s/it]
Outer training loop:  45%|####4     | 22386/50000 [13:41:38<14:15:25,  1.86s/it]
Outer training loop:  45%|####4     | 22387/50000 [13:41:40<14:06:15,  1.84s/it]
Outer training loop:  45%|####4     | 22388/50000 [13:41:41<13:49:08,  1.80s/it]
Outer training loop:  45%|####4     | 22389/50000 [13:41:43<14:22:28,  1.87s/it]
Outer training loop:  45%|####4     | 22390/50000 [13:41:45<14:41:15,  1.92s/it]
Outer training loop:  45%|####4     | 22391/50000 [13:41:47<14:28:54,  1.89s/it]
Outer training loop:  45%|####4     | 22392/50000 [13:41:49<14:48:35,  1.93s/it]
Outer training loop:  45%|####4     | 22393/50000 [13:41:51<14:42:41,  1.92s/it]
Outer training loop:  45%|####4     | 22394/50000 [13:41:53<14:24:20,  1.88s/it]
Outer training loop:  45%|####4     | 22395/50000 [13:41:55<14:53:18,  1.94s/it]
Outer training loop:  45%|####4     | 22396/50000 [13:41:57<14:27:38,  1.89s/it]
Outer training loop:  45%|####4     | 22397/50000 [13:41:59<14:12:09,  1.85s/it]
Outer training loop:  45%|####4     | 22398/50000 [13:42:00<14:04:15,  1.84s/it]
Outer training loop:  45%|####4     | 22399/50000 [13:42:02<14:23:02,  1.88s/it]
Outer training loop:  45%|####4     | 22400/50000 [13:42:04<14:01:31,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.95s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  45%|####4     | 22401/50000 [13:42:12<27:00:06,  3.52s/it]
Outer training loop:  45%|####4     | 22402/50000 [13:42:14<23:24:58,  3.05s/it]
Outer training loop:  45%|####4     | 22403/50000 [13:42:15<20:21:33,  2.66s/it]
Outer training loop:  45%|####4     | 22404/50000 [13:42:17<18:18:07,  2.39s/it]
Outer training loop:  45%|####4     | 22405/50000 [13:42:19<17:00:14,  2.22s/it]
Outer training loop:  45%|####4     | 22406/50000 [13:42:21<16:33:24,  2.16s/it]
Outer training loop:  45%|####4     | 22407/50000 [13:42:23<15:32:20,  2.03s/it]
Outer training loop:  45%|####4     | 22408/50000 [13:42:24<15:07:55,  1.97s/it]
Outer training loop:  45%|####4     | 22409/50000 [13:42:26<15:18:26,  2.00s/it]
Outer training loop:  45%|####4     | 22410/50000 [13:42:28<14:53:21,  1.94s/it]
Outer training loop:  45%|####4     | 22411/50000 [13:42:30<14:34:46,  1.90s/it]
Outer training loop:  45%|####4     | 22412/50000 [13:42:32<14:18:47,  1.87s/it]
Outer training loop:  45%|####4     | 22413/50000 [13:42:34<14:42:57,  1.92s/it]
Outer training loop:  45%|####4     | 22414/50000 [13:42:36<14:27:37,  1.89s/it]
Outer training loop:  45%|####4     | 22415/50000 [13:42:38<14:14:29,  1.86s/it]
Outer training loop:  45%|####4     | 22416/50000 [13:42:40<14:35:21,  1.90s/it]
Outer training loop:  45%|####4     | 22417/50000 [13:42:41<14:23:51,  1.88s/it]
Outer training loop:  45%|####4     | 22418/50000 [13:42:43<14:33:16,  1.90s/it]
Outer training loop:  45%|####4     | 22419/50000 [13:42:45<14:59:11,  1.96s/it]
Outer training loop:  45%|####4     | 22420/50000 [13:42:47<14:32:21,  1.90s/it]
Outer training loop:  45%|####4     | 22421/50000 [13:42:49<14:24:27,  1.88s/it]
Outer training loop:  45%|####4     | 22422/50000 [13:42:51<15:19:24,  2.00s/it]
Outer training loop:  45%|####4     | 22423/50000 [13:42:53<14:32:28,  1.90s/it]
Outer training loop:  45%|####4     | 22424/50000 [13:42:55<14:25:52,  1.88s/it]
Outer training loop:  45%|####4     | 22425/50000 [13:42:57<14:00:54,  1.83s/it]
Outer training loop:  45%|####4     | 22426/50000 [13:42:59<14:26:37,  1.89s/it]
Outer training loop:  45%|####4     | 22427/50000 [13:43:00<14:10:19,  1.85s/it]
Outer training loop:  45%|####4     | 22428/50000 [13:43:02<14:20:10,  1.87s/it]
Outer training loop:  45%|####4     | 22429/50000 [13:43:04<14:36:12,  1.91s/it]
Outer training loop:  45%|####4     | 22430/50000 [13:43:06<14:29:12,  1.89s/it]
Outer training loop:  45%|####4     | 22431/50000 [13:43:08<14:11:18,  1.85s/it]
Outer training loop:  45%|####4     | 22432/50000 [13:43:10<14:10:14,  1.85s/it]
Outer training loop:  45%|####4     | 22433/50000 [13:43:12<14:28:46,  1.89s/it]
Outer training loop:  45%|####4     | 22434/50000 [13:43:14<14:29:13,  1.89s/it]
Outer training loop:  45%|####4     | 22435/50000 [13:43:15<14:11:48,  1.85s/it]
Outer training loop:  45%|####4     | 22436/50000 [13:43:17<14:34:39,  1.90s/it]
Outer training loop:  45%|####4     | 22437/50000 [13:43:19<14:59:03,  1.96s/it]
Outer training loop:  45%|####4     | 22438/50000 [13:43:21<14:36:10,  1.91s/it]
Outer training loop:  45%|####4     | 22439/50000 [13:43:23<14:31:12,  1.90s/it]
Outer training loop:  45%|####4     | 22440/50000 [13:43:25<14:56:34,  1.95s/it]
Outer training loop:  45%|####4     | 22441/50000 [13:43:27<14:31:27,  1.90s/it]
Outer training loop:  45%|####4     | 22442/50000 [13:43:29<14:17:59,  1.87s/it]
Outer training loop:  45%|####4     | 22443/50000 [13:43:31<14:31:26,  1.90s/it]
Outer training loop:  45%|####4     | 22444/50000 [13:43:32<14:06:07,  1.84s/it]
Outer training loop:  45%|####4     | 22445/50000 [13:43:34<14:08:47,  1.85s/it]
Outer training loop:  45%|####4     | 22446/50000 [13:43:36<14:22:28,  1.88s/it]
Outer training loop:  45%|####4     | 22447/50000 [13:43:38<14:13:43,  1.86s/it]
Outer training loop:  45%|####4     | 22448/50000 [13:43:40<14:14:54,  1.86s/it]
Outer training loop:  45%|####4     | 22449/50000 [13:43:42<13:53:44,  1.82s/it]
Outer training loop:  45%|####4     | 22450/50000 [13:43:44<14:30:12,  1.90s/it]
Outer training loop:  45%|####4     | 22451/50000 [13:43:46<14:33:14,  1.90s/it]
Outer training loop:  45%|####4     | 22452/50000 [13:43:47<14:18:55,  1.87s/it]
Outer training loop:  45%|####4     | 22453/50000 [13:43:50<14:59:14,  1.96s/it]
Outer training loop:  45%|####4     | 22454/50000 [13:43:51<14:33:10,  1.90s/it]
Outer training loop:  45%|####4     | 22455/50000 [13:43:53<14:04:02,  1.84s/it]
Outer training loop:  45%|####4     | 22456/50000 [13:43:55<14:01:38,  1.83s/it]
Outer training loop:  45%|####4     | 22457/50000 [13:43:57<14:24:38,  1.88s/it]
Outer training loop:  45%|####4     | 22458/50000 [13:43:59<14:15:35,  1.86s/it]
Outer training loop:  45%|####4     | 22459/50000 [13:44:00<13:57:18,  1.82s/it]
Outer training loop:  45%|####4     | 22460/50000 [13:44:02<14:22:37,  1.88s/it]
Outer training loop:  45%|####4     | 22461/50000 [13:44:04<14:20:20,  1.87s/it]
Outer training loop:  45%|####4     | 22462/50000 [13:44:06<14:25:56,  1.89s/it]
Outer training loop:  45%|####4     | 22463/50000 [13:44:08<14:02:50,  1.84s/it]
Outer training loop:  45%|####4     | 22464/50000 [13:44:10<14:31:59,  1.90s/it]
Outer training loop:  45%|####4     | 22465/50000 [13:44:12<14:14:44,  1.86s/it]
Outer training loop:  45%|####4     | 22466/50000 [13:44:14<14:11:18,  1.86s/it]
Outer training loop:  45%|####4     | 22467/50000 [13:44:16<14:25:43,  1.89s/it]
Outer training loop:  45%|####4     | 22468/50000 [13:44:17<14:16:32,  1.87s/it]
Outer training loop:  45%|####4     | 22469/50000 [13:44:19<14:21:11,  1.88s/it]
Outer training loop:  45%|####4     | 22470/50000 [13:44:21<14:02:05,  1.84s/it]
Outer training loop:  45%|####4     | 22471/50000 [13:44:23<14:25:45,  1.89s/it]
Outer training loop:  45%|####4     | 22472/50000 [13:44:25<14:53:11,  1.95s/it]
Outer training loop:  45%|####4     | 22473/50000 [13:44:27<14:21:57,  1.88s/it]
Outer training loop:  45%|####4     | 22474/50000 [13:44:29<14:18:23,  1.87s/it]
Outer training loop:  45%|####4     | 22475/50000 [13:44:31<14:23:23,  1.88s/it]
Outer training loop:  45%|####4     | 22476/50000 [13:44:32<13:52:55,  1.82s/it]
Outer training loop:  45%|####4     | 22477/50000 [13:44:34<14:10:09,  1.85s/it]
Outer training loop:  45%|####4     | 22478/50000 [13:44:36<14:45:48,  1.93s/it]
Outer training loop:  45%|####4     | 22479/50000 [13:44:38<14:24:40,  1.89s/it]
Outer training loop:  45%|####4     | 22480/50000 [13:44:40<14:20:23,  1.88s/it]
Outer training loop:  45%|####4     | 22481/50000 [13:44:42<14:44:39,  1.93s/it]
Outer training loop:  45%|####4     | 22482/50000 [13:44:44<14:22:18,  1.88s/it]
Outer training loop:  45%|####4     | 22483/50000 [13:44:46<14:14:04,  1.86s/it]
Outer training loop:  45%|####4     | 22484/50000 [13:44:47<14:17:03,  1.87s/it]
Outer training loop:  45%|####4     | 22485/50000 [13:44:49<14:26:03,  1.89s/it]
Outer training loop:  45%|####4     | 22486/50000 [13:44:51<14:20:33,  1.88s/it]
Outer training loop:  45%|####4     | 22487/50000 [13:44:53<14:03:22,  1.84s/it]
Outer training loop:  45%|####4     | 22488/50000 [13:44:55<14:36:47,  1.91s/it]
Outer training loop:  45%|####4     | 22489/50000 [13:44:57<14:27:10,  1.89s/it]
Outer training loop:  45%|####4     | 22490/50000 [13:44:59<14:23:09,  1.88s/it]
Outer training loop:  45%|####4     | 22491/50000 [13:45:00<13:53:27,  1.82s/it]
Outer training loop:  45%|####4     | 22492/50000 [13:45:02<14:18:39,  1.87s/it]
Outer training loop:  45%|####4     | 22493/50000 [13:45:04<14:27:27,  1.89s/it]
Outer training loop:  45%|####4     | 22494/50000 [13:45:06<14:03:47,  1.84s/it]
Outer training loop:  45%|####4     | 22495/50000 [13:45:08<14:25:53,  1.89s/it]
Outer training loop:  45%|####4     | 22496/50000 [13:45:10<14:31:24,  1.90s/it]
Outer training loop:  45%|####4     | 22497/50000 [13:45:12<14:03:51,  1.84s/it]
Outer training loop:  45%|####4     | 22498/50000 [13:45:14<14:41:35,  1.92s/it]
Outer training loop:  45%|####4     | 22499/50000 [13:45:16<14:29:11,  1.90s/it]
Outer training loop:  45%|####5     | 22500/50000 [13:45:17<14:18:56,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.96s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:48,  2.06s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:41,  1.99s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:34,  1.93s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.87s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:29,  1.92s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:20,  1.85s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.90s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:20,  1.89s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.86s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.88s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:06,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:02,  1.83s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.90s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:01,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:59,  1.92s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.90s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:47,  1.85s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.93s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.93s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.85s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.85s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:40,  1.97s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:36,  1.93s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:25,  1.83s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:23,  1.82s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:26,  1.91s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:15,  1.81s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:15,  1.84s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:10,  1.80s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:12,  1.91s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:08,  1.85s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:05,  1.82s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:29<01:02,  1.80s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.79s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:56,  1.77s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:59,  1.93s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:57,  1.91s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:54,  1.87s/it]

Evaluating policy:  66%|######6   | 55/83 [01:42<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:49,  1.84s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:45,  1.83s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:43,  1.80s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:44,  1.95s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:42,  1.93s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.90s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:57<00:36,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:34,  1.80s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:02<00:30,  1.79s/it]

Evaluating policy:  81%|########  | 67/83 [02:04<00:28,  1.76s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:29,  1.99s/it]

Evaluating policy:  83%|########3 | 69/83 [02:08<00:26,  1.92s/it]

Evaluating policy:  84%|########4 | 70/83 [02:10<00:24,  1.90s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:22,  1.87s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:20,  1.85s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.86s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.88s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:21<00:13,  1.87s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:12,  2.01s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.94s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:29<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:31<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:17,  1.68s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:19,  1.72s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:33,  1.92s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.90s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:24,  1.86s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:20,  1.83s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.92s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:20,  1.87s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:16,  1.84s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:14,  1.84s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:18,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:12,  1.87s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:09,  1.85s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:03,  1.87s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:01,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:57,  1.83s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:51,  1.83s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:50,  1.84s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:47,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:45,  1.84s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:41,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:39,  1.81s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:37,  1.80s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:57<01:36,  1.85s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:31,  1.83s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:29,  1.82s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:34,  2.02s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:29,  1.95s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:12<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:14<01:18,  1.82s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:20,  1.95s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:16,  1.92s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:14,  1.90s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:10,  1.90s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:07,  1.88s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:09,  1.97s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:05,  1.93s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<01:01,  1.88s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:01,  1.91s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.81s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:55,  1.97s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:52,  1.95s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:49,  1.91s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.90s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.88s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:42,  1.85s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:40,  1.84s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.81s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:39,  1.96s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:36,  1.93s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.88s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:21,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:21,  1.95s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:19,  1.91s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.86s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.86s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.81s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.83s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.82s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.96s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.93s/it]

Outer training loop:  45%|####5     | 22501/50000 [13:50:37<741:12:30, 97.03s/it]
Outer training loop:  45%|####5     | 22502/50000 [13:50:38<523:04:30, 68.48s/it]
Outer training loop:  45%|####5     | 22503/50000 [13:50:41<370:56:28, 48.56s/it]
Outer training loop:  45%|####5     | 22504/50000 [13:50:42<263:41:19, 34.52s/it]
Outer training loop:  45%|####5     | 22505/50000 [13:50:44<188:39:49, 24.70s/it]
Outer training loop:  45%|####5     | 22506/50000 [13:50:46<136:13:14, 17.84s/it]
Outer training loop:  45%|####5     | 22507/50000 [13:50:48<99:28:40, 13.03s/it] 
Outer training loop:  45%|####5     | 22508/50000 [13:50:49<73:42:11,  9.65s/it]
Outer training loop:  45%|####5     | 22509/50000 [13:50:52<57:30:16,  7.53s/it]
Outer training loop:  45%|####5     | 22510/50000 [13:50:54<44:30:12,  5.83s/it]
Outer training loop:  45%|####5     | 22511/50000 [13:50:56<35:21:33,  4.63s/it]
Outer training loop:  45%|####5     | 22512/50000 [13:50:58<29:34:22,  3.87s/it]
Outer training loop:  45%|####5     | 22513/50000 [13:51:00<24:55:58,  3.27s/it]
Outer training loop:  45%|####5     | 22514/50000 [13:51:02<21:43:47,  2.85s/it]
Outer training loop:  45%|####5     | 22515/50000 [13:51:03<19:21:03,  2.53s/it]
Outer training loop:  45%|####5     | 22516/50000 [13:51:05<18:09:17,  2.38s/it]
Outer training loop:  45%|####5     | 22517/50000 [13:51:07<16:51:11,  2.21s/it]
Outer training loop:  45%|####5     | 22518/50000 [13:51:09<16:01:14,  2.10s/it]
Outer training loop:  45%|####5     | 22519/50000 [13:51:11<15:48:03,  2.07s/it]
Outer training loop:  45%|####5     | 22520/50000 [13:51:13<15:19:13,  2.01s/it]
Outer training loop:  45%|####5     | 22521/50000 [13:51:15<14:55:27,  1.96s/it]
Outer training loop:  45%|####5     | 22522/50000 [13:51:17<14:44:49,  1.93s/it]
Outer training loop:  45%|####5     | 22523/50000 [13:51:18<14:40:28,  1.92s/it]
Outer training loop:  45%|####5     | 22524/50000 [13:51:20<14:35:22,  1.91s/it]
Outer training loop:  45%|####5     | 22525/50000 [13:51:22<14:27:52,  1.90s/it]
Outer training loop:  45%|####5     | 22526/50000 [13:51:24<15:05:21,  1.98s/it]
Outer training loop:  45%|####5     | 22527/50000 [13:51:26<14:32:23,  1.91s/it]
Outer training loop:  45%|####5     | 22528/50000 [13:51:28<14:21:10,  1.88s/it]
Outer training loop:  45%|####5     | 22529/50000 [13:51:30<14:42:46,  1.93s/it]
Outer training loop:  45%|####5     | 22530/50000 [13:51:32<14:32:19,  1.91s/it]
Outer training loop:  45%|####5     | 22531/50000 [13:51:34<14:19:42,  1.88s/it]
Outer training loop:  45%|####5     | 22532/50000 [13:51:36<14:15:48,  1.87s/it]
Outer training loop:  45%|####5     | 22533/50000 [13:51:37<14:28:42,  1.90s/it]
Outer training loop:  45%|####5     | 22534/50000 [13:51:39<14:16:53,  1.87s/it]
Outer training loop:  45%|####5     | 22535/50000 [13:51:41<13:51:45,  1.82s/it]
Outer training loop:  45%|####5     | 22536/50000 [13:51:43<14:06:57,  1.85s/it]
Outer training loop:  45%|####5     | 22537/50000 [13:51:45<14:00:16,  1.84s/it]
Outer training loop:  45%|####5     | 22538/50000 [13:51:46<13:51:34,  1.82s/it]
Outer training loop:  45%|####5     | 22539/50000 [13:51:48<14:15:25,  1.87s/it]
Outer training loop:  45%|####5     | 22540/50000 [13:51:50<14:16:42,  1.87s/it]
Outer training loop:  45%|####5     | 22541/50000 [13:51:52<14:11:03,  1.86s/it]
Outer training loop:  45%|####5     | 22542/50000 [13:51:54<14:24:44,  1.89s/it]
Outer training loop:  45%|####5     | 22543/50000 [13:51:56<14:22:38,  1.89s/it]
Outer training loop:  45%|####5     | 22544/50000 [13:51:58<14:15:18,  1.87s/it]
Outer training loop:  45%|####5     | 22545/50000 [13:52:00<14:09:11,  1.86s/it]
Outer training loop:  45%|####5     | 22546/50000 [13:52:02<14:23:17,  1.89s/it]
Outer training loop:  45%|####5     | 22547/50000 [13:52:04<14:28:24,  1.90s/it]
Outer training loop:  45%|####5     | 22548/50000 [13:52:05<14:20:45,  1.88s/it]
Outer training loop:  45%|####5     | 22549/50000 [13:52:07<14:22:32,  1.89s/it]
Outer training loop:  45%|####5     | 22550/50000 [13:52:09<14:01:01,  1.84s/it]
Outer training loop:  45%|####5     | 22551/50000 [13:52:11<14:16:11,  1.87s/it]
Outer training loop:  45%|####5     | 22552/50000 [13:52:13<14:29:53,  1.90s/it]
Outer training loop:  45%|####5     | 22553/50000 [13:52:15<14:51:48,  1.95s/it]
Outer training loop:  45%|####5     | 22554/50000 [13:52:17<14:41:04,  1.93s/it]
Outer training loop:  45%|####5     | 22555/50000 [13:52:19<14:55:38,  1.96s/it]
Outer training loop:  45%|####5     | 22556/50000 [13:52:21<14:39:03,  1.92s/it]
Outer training loop:  45%|####5     | 22557/50000 [13:52:23<14:29:47,  1.90s/it]
Outer training loop:  45%|####5     | 22558/50000 [13:52:25<14:38:45,  1.92s/it]
Outer training loop:  45%|####5     | 22559/50000 [13:52:27<14:50:13,  1.95s/it]
Outer training loop:  45%|####5     | 22560/50000 [13:52:28<14:38:20,  1.92s/it]
Outer training loop:  45%|####5     | 22561/50000 [13:52:30<14:39:26,  1.92s/it]
Outer training loop:  45%|####5     | 22562/50000 [13:52:32<14:47:05,  1.94s/it]
Outer training loop:  45%|####5     | 22563/50000 [13:52:34<14:44:05,  1.93s/it]
Outer training loop:  45%|####5     | 22564/50000 [13:52:36<14:26:09,  1.89s/it]
Outer training loop:  45%|####5     | 22565/50000 [13:52:38<14:19:41,  1.88s/it]
Outer training loop:  45%|####5     | 22566/50000 [13:52:40<14:30:56,  1.90s/it]
Outer training loop:  45%|####5     | 22567/50000 [13:52:42<14:17:45,  1.88s/it]
Outer training loop:  45%|####5     | 22568/50000 [13:52:44<14:22:07,  1.89s/it]
Outer training loop:  45%|####5     | 22569/50000 [13:52:46<14:44:45,  1.94s/it]
Outer training loop:  45%|####5     | 22570/50000 [13:52:47<14:13:12,  1.87s/it]
Outer training loop:  45%|####5     | 22571/50000 [13:52:49<14:27:25,  1.90s/it]
Outer training loop:  45%|####5     | 22572/50000 [13:52:51<14:17:57,  1.88s/it]
Outer training loop:  45%|####5     | 22573/50000 [13:52:53<14:32:07,  1.91s/it]
Outer training loop:  45%|####5     | 22574/50000 [13:52:55<14:36:36,  1.92s/it]
Outer training loop:  45%|####5     | 22575/50000 [13:52:57<14:27:46,  1.90s/it]
Outer training loop:  45%|####5     | 22576/50000 [13:52:59<14:48:35,  1.94s/it]
Outer training loop:  45%|####5     | 22577/50000 [13:53:01<14:31:24,  1.91s/it]
Outer training loop:  45%|####5     | 22578/50000 [13:53:03<14:07:04,  1.85s/it]
Outer training loop:  45%|####5     | 22579/50000 [13:53:04<14:10:54,  1.86s/it]
Outer training loop:  45%|####5     | 22580/50000 [13:53:06<14:41:53,  1.93s/it]
Outer training loop:  45%|####5     | 22581/50000 [13:53:08<14:24:57,  1.89s/it]
Outer training loop:  45%|####5     | 22582/50000 [13:53:10<14:12:09,  1.86s/it]
Outer training loop:  45%|####5     | 22583/50000 [13:53:12<14:26:51,  1.90s/it]
Outer training loop:  45%|####5     | 22584/50000 [13:53:14<14:16:09,  1.87s/it]
Outer training loop:  45%|####5     | 22585/50000 [13:53:16<14:09:09,  1.86s/it]
Outer training loop:  45%|####5     | 22586/50000 [13:53:17<13:58:04,  1.83s/it]
Outer training loop:  45%|####5     | 22587/50000 [13:53:20<14:43:59,  1.93s/it]
Outer training loop:  45%|####5     | 22588/50000 [13:53:21<14:19:15,  1.88s/it]
Outer training loop:  45%|####5     | 22589/50000 [13:53:23<14:02:44,  1.84s/it]
Outer training loop:  45%|####5     | 22590/50000 [13:53:25<14:15:56,  1.87s/it]
Outer training loop:  45%|####5     | 22591/50000 [13:53:27<14:03:17,  1.85s/it]
Outer training loop:  45%|####5     | 22592/50000 [13:53:29<13:55:22,  1.83s/it]
Outer training loop:  45%|####5     | 22593/50000 [13:53:30<13:51:33,  1.82s/it]
Outer training loop:  45%|####5     | 22594/50000 [13:53:33<14:26:00,  1.90s/it]
Outer training loop:  45%|####5     | 22595/50000 [13:53:34<14:17:09,  1.88s/it]
Outer training loop:  45%|####5     | 22596/50000 [13:53:36<14:30:44,  1.91s/it]
Outer training loop:  45%|####5     | 22597/50000 [13:53:38<14:39:12,  1.93s/it]
Outer training loop:  45%|####5     | 22598/50000 [13:53:40<14:03:58,  1.85s/it]
Outer training loop:  45%|####5     | 22599/50000 [13:53:42<13:56:54,  1.83s/it]
Outer training loop:  45%|####5     | 22600/50000 [13:53:44<13:57:53,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  45%|####5     | 22601/50000 [13:53:52<27:48:38,  3.65s/it]
Outer training loop:  45%|####5     | 22602/50000 [13:53:53<23:43:56,  3.12s/it]
Outer training loop:  45%|####5     | 22603/50000 [13:53:55<20:58:01,  2.76s/it]
Outer training loop:  45%|####5     | 22604/50000 [13:53:57<18:37:09,  2.45s/it]
Outer training loop:  45%|####5     | 22605/50000 [13:53:59<17:43:43,  2.33s/it]
Outer training loop:  45%|####5     | 22606/50000 [13:54:01<16:37:04,  2.18s/it]
Outer training loop:  45%|####5     | 22607/50000 [13:54:03<15:34:17,  2.05s/it]
Outer training loop:  45%|####5     | 22608/50000 [13:54:05<15:33:59,  2.05s/it]
Outer training loop:  45%|####5     | 22609/50000 [13:54:07<15:04:08,  1.98s/it]
Outer training loop:  45%|####5     | 22610/50000 [13:54:08<14:37:22,  1.92s/it]
Outer training loop:  45%|####5     | 22611/50000 [13:54:10<14:51:30,  1.95s/it]
Outer training loop:  45%|####5     | 22612/50000 [13:54:12<14:42:53,  1.93s/it]
Outer training loop:  45%|####5     | 22613/50000 [13:54:14<14:33:16,  1.91s/it]
Outer training loop:  45%|####5     | 22614/50000 [13:54:16<14:21:22,  1.89s/it]
Outer training loop:  45%|####5     | 22615/50000 [13:54:18<14:35:39,  1.92s/it]
Outer training loop:  45%|####5     | 22616/50000 [13:54:20<14:23:06,  1.89s/it]
Outer training loop:  45%|####5     | 22617/50000 [13:54:22<14:14:26,  1.87s/it]
Outer training loop:  45%|####5     | 22618/50000 [13:54:24<14:31:53,  1.91s/it]
Outer training loop:  45%|####5     | 22619/50000 [13:54:25<14:18:44,  1.88s/it]
Outer training loop:  45%|####5     | 22620/50000 [13:54:27<14:16:30,  1.88s/it]
Outer training loop:  45%|####5     | 22621/50000 [13:54:29<14:20:04,  1.88s/it]
Outer training loop:  45%|####5     | 22622/50000 [13:54:31<13:48:03,  1.81s/it]
Outer training loop:  45%|####5     | 22623/50000 [13:54:33<13:41:42,  1.80s/it]
Outer training loop:  45%|####5     | 22624/50000 [13:54:35<14:23:26,  1.89s/it]
Outer training loop:  45%|####5     | 22625/50000 [13:54:37<14:15:58,  1.88s/it]
Outer training loop:  45%|####5     | 22626/50000 [13:54:38<14:06:52,  1.86s/it]
Outer training loop:  45%|####5     | 22627/50000 [13:54:40<13:54:45,  1.83s/it]
Outer training loop:  45%|####5     | 22628/50000 [13:54:42<14:09:30,  1.86s/it]
Outer training loop:  45%|####5     | 22629/50000 [13:54:44<14:16:33,  1.88s/it]
Outer training loop:  45%|####5     | 22630/50000 [13:54:46<14:16:05,  1.88s/it]
Outer training loop:  45%|####5     | 22631/50000 [13:54:48<14:30:07,  1.91s/it]
Outer training loop:  45%|####5     | 22632/50000 [13:54:50<14:19:32,  1.88s/it]
Outer training loop:  45%|####5     | 22633/50000 [13:54:51<14:07:21,  1.86s/it]
Outer training loop:  45%|####5     | 22634/50000 [13:54:54<14:35:43,  1.92s/it]
Outer training loop:  45%|####5     | 22635/50000 [13:54:55<14:19:18,  1.88s/it]
Outer training loop:  45%|####5     | 22636/50000 [13:54:57<14:05:53,  1.85s/it]
Outer training loop:  45%|####5     | 22637/50000 [13:54:59<14:07:51,  1.86s/it]
Outer training loop:  45%|####5     | 22638/50000 [13:55:01<14:17:49,  1.88s/it]
Outer training loop:  45%|####5     | 22639/50000 [13:55:03<14:02:58,  1.85s/it]
Outer training loop:  45%|####5     | 22640/50000 [13:55:05<14:08:46,  1.86s/it]
Outer training loop:  45%|####5     | 22641/50000 [13:55:07<14:52:38,  1.96s/it]
Outer training loop:  45%|####5     | 22642/50000 [13:55:09<14:38:36,  1.93s/it]
Outer training loop:  45%|####5     | 22643/50000 [13:55:10<14:26:01,  1.90s/it]
Outer training loop:  45%|####5     | 22644/50000 [13:55:12<14:40:46,  1.93s/it]
Outer training loop:  45%|####5     | 22645/50000 [13:55:14<14:14:28,  1.87s/it]
Outer training loop:  45%|####5     | 22646/50000 [13:55:16<13:58:37,  1.84s/it]
Outer training loop:  45%|####5     | 22647/50000 [13:55:18<13:46:08,  1.81s/it]
Outer training loop:  45%|####5     | 22648/50000 [13:55:20<14:18:57,  1.88s/it]
Outer training loop:  45%|####5     | 22649/50000 [13:55:21<14:01:37,  1.85s/it]
Outer training loop:  45%|####5     | 22650/50000 [13:55:23<14:14:20,  1.87s/it]
Outer training loop:  45%|####5     | 22651/50000 [13:55:25<14:30:09,  1.91s/it]
Outer training loop:  45%|####5     | 22652/50000 [13:55:27<14:00:45,  1.84s/it]
Outer training loop:  45%|####5     | 22653/50000 [13:55:29<13:53:17,  1.83s/it]
Outer training loop:  45%|####5     | 22654/50000 [13:55:31<14:01:32,  1.85s/it]
Outer training loop:  45%|####5     | 22655/50000 [13:55:32<13:40:29,  1.80s/it]
Outer training loop:  45%|####5     | 22656/50000 [13:55:34<13:56:54,  1.84s/it]
Outer training loop:  45%|####5     | 22657/50000 [13:55:36<14:11:41,  1.87s/it]
Outer training loop:  45%|####5     | 22658/50000 [13:55:38<14:25:56,  1.90s/it]
Outer training loop:  45%|####5     | 22659/50000 [13:55:40<14:23:25,  1.89s/it]
Outer training loop:  45%|####5     | 22660/50000 [13:55:42<14:14:35,  1.88s/it]
Outer training loop:  45%|####5     | 22661/50000 [13:55:44<14:34:02,  1.92s/it]
Outer training loop:  45%|####5     | 22662/50000 [13:55:46<14:19:06,  1.89s/it]
Outer training loop:  45%|####5     | 22663/50000 [13:55:48<14:23:20,  1.89s/it]
Outer training loop:  45%|####5     | 22664/50000 [13:55:50<14:08:01,  1.86s/it]
Outer training loop:  45%|####5     | 22665/50000 [13:55:52<14:18:32,  1.88s/it]
Outer training loop:  45%|####5     | 22666/50000 [13:55:53<13:57:54,  1.84s/it]
Outer training loop:  45%|####5     | 22667/50000 [13:55:55<13:49:10,  1.82s/it]
Outer training loop:  45%|####5     | 22668/50000 [13:55:57<14:02:57,  1.85s/it]
Outer training loop:  45%|####5     | 22669/50000 [13:55:59<14:02:21,  1.85s/it]
Outer training loop:  45%|####5     | 22670/50000 [13:56:01<13:48:31,  1.82s/it]
Outer training loop:  45%|####5     | 22671/50000 [13:56:02<13:42:32,  1.81s/it]
Outer training loop:  45%|####5     | 22672/50000 [13:56:04<14:25:23,  1.90s/it]
Outer training loop:  45%|####5     | 22673/50000 [13:56:06<14:09:09,  1.86s/it]
Outer training loop:  45%|####5     | 22674/50000 [13:56:08<13:54:14,  1.83s/it]
Outer training loop:  45%|####5     | 22675/50000 [13:56:10<14:21:04,  1.89s/it]
Outer training loop:  45%|####5     | 22676/50000 [13:56:12<14:13:22,  1.87s/it]
Outer training loop:  45%|####5     | 22677/50000 [13:56:14<14:03:39,  1.85s/it]
Outer training loop:  45%|####5     | 22678/50000 [13:56:16<14:22:07,  1.89s/it]
Outer training loop:  45%|####5     | 22679/50000 [13:56:17<14:15:02,  1.88s/it]
Outer training loop:  45%|####5     | 22680/50000 [13:56:19<14:10:05,  1.87s/it]
Outer training loop:  45%|####5     | 22681/50000 [13:56:21<14:24:45,  1.90s/it]
Outer training loop:  45%|####5     | 22682/50000 [13:56:23<14:39:03,  1.93s/it]
Outer training loop:  45%|####5     | 22683/50000 [13:56:25<14:21:01,  1.89s/it]
Outer training loop:  45%|####5     | 22684/50000 [13:56:27<14:06:27,  1.86s/it]
Outer training loop:  45%|####5     | 22685/50000 [13:56:29<14:34:41,  1.92s/it]
Outer training loop:  45%|####5     | 22686/50000 [13:56:31<14:23:43,  1.90s/it]
Outer training loop:  45%|####5     | 22687/50000 [13:56:33<14:07:52,  1.86s/it]
Outer training loop:  45%|####5     | 22688/50000 [13:56:34<13:53:54,  1.83s/it]
Outer training loop:  45%|####5     | 22689/50000 [13:56:36<14:27:20,  1.91s/it]
Outer training loop:  45%|####5     | 22690/50000 [13:56:38<14:34:45,  1.92s/it]
Outer training loop:  45%|####5     | 22691/50000 [13:56:40<14:19:55,  1.89s/it]
Outer training loop:  45%|####5     | 22692/50000 [13:56:42<14:41:06,  1.94s/it]
Outer training loop:  45%|####5     | 22693/50000 [13:56:44<14:18:42,  1.89s/it]
Outer training loop:  45%|####5     | 22694/50000 [13:56:46<14:09:01,  1.87s/it]
Outer training loop:  45%|####5     | 22695/50000 [13:56:48<14:42:41,  1.94s/it]
Outer training loop:  45%|####5     | 22696/50000 [13:56:50<14:43:03,  1.94s/it]
Outer training loop:  45%|####5     | 22697/50000 [13:56:52<14:15:42,  1.88s/it]
Outer training loop:  45%|####5     | 22698/50000 [13:56:54<14:30:03,  1.91s/it]
Outer training loop:  45%|####5     | 22699/50000 [13:56:55<14:12:15,  1.87s/it]
Outer training loop:  45%|####5     | 22700/50000 [13:56:57<14:46:48,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.96s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  45%|####5     | 22701/50000 [13:57:05<27:11:18,  3.59s/it]
Outer training loop:  45%|####5     | 22702/50000 [13:57:07<23:39:13,  3.12s/it]
Outer training loop:  45%|####5     | 22703/50000 [13:57:09<20:50:44,  2.75s/it]
Outer training loop:  45%|####5     | 22704/50000 [13:57:11<18:56:27,  2.50s/it]
Outer training loop:  45%|####5     | 22705/50000 [13:57:13<17:46:55,  2.35s/it]
Outer training loop:  45%|####5     | 22706/50000 [13:57:15<16:53:22,  2.23s/it]
Outer training loop:  45%|####5     | 22707/50000 [13:57:17<16:04:48,  2.12s/it]
Outer training loop:  45%|####5     | 22708/50000 [13:57:18<15:15:12,  2.01s/it]
Outer training loop:  45%|####5     | 22709/50000 [13:57:20<14:44:08,  1.94s/it]
Outer training loop:  45%|####5     | 22710/50000 [13:57:22<14:55:55,  1.97s/it]
Outer training loop:  45%|####5     | 22711/50000 [13:57:24<14:38:01,  1.93s/it]
Outer training loop:  45%|####5     | 22712/50000 [13:57:26<14:16:57,  1.88s/it]
Outer training loop:  45%|####5     | 22713/50000 [13:57:28<14:35:55,  1.93s/it]
Outer training loop:  45%|####5     | 22714/50000 [13:57:30<14:16:00,  1.88s/it]
Outer training loop:  45%|####5     | 22715/50000 [13:57:31<14:04:17,  1.86s/it]
Outer training loop:  45%|####5     | 22716/50000 [13:57:33<14:32:17,  1.92s/it]
Outer training loop:  45%|####5     | 22717/50000 [13:57:35<14:34:59,  1.92s/it]
Outer training loop:  45%|####5     | 22718/50000 [13:57:37<14:24:29,  1.90s/it]
Outer training loop:  45%|####5     | 22719/50000 [13:57:39<14:37:14,  1.93s/it]
Outer training loop:  45%|####5     | 22720/50000 [13:57:41<14:19:53,  1.89s/it]
Outer training loop:  45%|####5     | 22721/50000 [13:57:43<14:19:05,  1.89s/it]
Outer training loop:  45%|####5     | 22722/50000 [13:57:45<14:07:38,  1.86s/it]
Outer training loop:  45%|####5     | 22723/50000 [13:57:47<14:30:10,  1.91s/it]
Outer training loop:  45%|####5     | 22724/50000 [13:57:48<13:49:17,  1.82s/it]
Outer training loop:  45%|####5     | 22725/50000 [13:57:50<13:59:56,  1.85s/it]
Outer training loop:  45%|####5     | 22726/50000 [13:57:52<14:23:10,  1.90s/it]
Outer training loop:  45%|####5     | 22727/50000 [13:57:54<14:15:10,  1.88s/it]
Outer training loop:  45%|####5     | 22728/50000 [13:57:56<14:02:59,  1.85s/it]
Outer training loop:  45%|####5     | 22729/50000 [13:57:58<13:56:47,  1.84s/it]
Outer training loop:  45%|####5     | 22730/50000 [13:58:00<14:15:17,  1.88s/it]
Outer training loop:  45%|####5     | 22731/50000 [13:58:02<14:19:11,  1.89s/it]
Outer training loop:  45%|####5     | 22732/50000 [13:58:03<14:16:17,  1.88s/it]
Outer training loop:  45%|####5     | 22733/50000 [13:58:05<14:36:54,  1.93s/it]
Outer training loop:  45%|####5     | 22734/50000 [13:58:07<14:10:15,  1.87s/it]
Outer training loop:  45%|####5     | 22735/50000 [13:58:09<13:56:38,  1.84s/it]
Outer training loop:  45%|####5     | 22736/50000 [13:58:11<13:55:48,  1.84s/it]
Outer training loop:  45%|####5     | 22737/50000 [13:58:13<14:28:16,  1.91s/it]
Outer training loop:  45%|####5     | 22738/50000 [13:58:15<14:10:58,  1.87s/it]
Outer training loop:  45%|####5     | 22739/50000 [13:58:16<13:57:03,  1.84s/it]
Outer training loop:  45%|####5     | 22740/50000 [13:58:18<14:08:49,  1.87s/it]
Outer training loop:  45%|####5     | 22741/50000 [13:58:20<14:20:53,  1.89s/it]
Outer training loop:  45%|####5     | 22742/50000 [13:58:22<14:01:17,  1.85s/it]
Outer training loop:  45%|####5     | 22743/50000 [13:58:24<14:13:16,  1.88s/it]
Outer training loop:  45%|####5     | 22744/50000 [13:58:26<14:06:39,  1.86s/it]
Outer training loop:  45%|####5     | 22745/50000 [13:58:28<14:15:20,  1.88s/it]
Outer training loop:  45%|####5     | 22746/50000 [13:58:30<14:03:59,  1.86s/it]
Outer training loop:  45%|####5     | 22747/50000 [13:58:32<14:48:28,  1.96s/it]
Outer training loop:  45%|####5     | 22748/50000 [13:58:34<14:20:29,  1.89s/it]
Outer training loop:  45%|####5     | 22749/50000 [13:58:35<14:13:08,  1.88s/it]
Outer training loop:  46%|####5     | 22750/50000 [13:58:37<14:18:24,  1.89s/it]
Outer training loop:  46%|####5     | 22751/50000 [13:58:39<14:02:03,  1.85s/it]
Outer training loop:  46%|####5     | 22752/50000 [13:58:41<13:58:54,  1.85s/it]
Outer training loop:  46%|####5     | 22753/50000 [13:58:43<13:53:31,  1.84s/it]
Outer training loop:  46%|####5     | 22754/50000 [13:58:45<14:25:18,  1.91s/it]
Outer training loop:  46%|####5     | 22755/50000 [13:58:47<14:12:06,  1.88s/it]
Outer training loop:  46%|####5     | 22756/50000 [13:58:48<14:07:37,  1.87s/it]
Outer training loop:  46%|####5     | 22757/50000 [13:58:50<14:37:23,  1.93s/it]
Outer training loop:  46%|####5     | 22758/50000 [13:58:52<14:24:24,  1.90s/it]
Outer training loop:  46%|####5     | 22759/50000 [13:58:54<14:02:45,  1.86s/it]
Outer training loop:  46%|####5     | 22760/50000 [13:58:56<13:55:03,  1.84s/it]
Outer training loop:  46%|####5     | 22761/50000 [13:58:58<14:22:22,  1.90s/it]
Outer training loop:  46%|####5     | 22762/50000 [13:59:00<13:54:45,  1.84s/it]
Outer training loop:  46%|####5     | 22763/50000 [13:59:01<13:48:11,  1.82s/it]
Outer training loop:  46%|####5     | 22764/50000 [13:59:03<14:21:21,  1.90s/it]
Outer training loop:  46%|####5     | 22765/50000 [13:59:05<14:13:53,  1.88s/it]
Outer training loop:  46%|####5     | 22766/50000 [13:59:07<13:54:51,  1.84s/it]
Outer training loop:  46%|####5     | 22767/50000 [13:59:09<13:43:54,  1.82s/it]
Outer training loop:  46%|####5     | 22768/50000 [13:59:11<14:23:37,  1.90s/it]
Outer training loop:  46%|####5     | 22769/50000 [13:59:13<14:14:55,  1.88s/it]
Outer training loop:  46%|####5     | 22770/50000 [13:59:15<14:18:44,  1.89s/it]
Outer training loop:  46%|####5     | 22771/50000 [13:59:17<14:58:58,  1.98s/it]
Outer training loop:  46%|####5     | 22772/50000 [13:59:19<14:31:59,  1.92s/it]
Outer training loop:  46%|####5     | 22773/50000 [13:59:20<14:20:07,  1.90s/it]
Outer training loop:  46%|####5     | 22774/50000 [13:59:22<14:10:15,  1.87s/it]
Outer training loop:  46%|####5     | 22775/50000 [13:59:24<14:42:20,  1.94s/it]
Outer training loop:  46%|####5     | 22776/50000 [13:59:26<14:11:34,  1.88s/it]
Outer training loop:  46%|####5     | 22777/50000 [13:59:28<14:12:50,  1.88s/it]
Outer training loop:  46%|####5     | 22778/50000 [13:59:30<14:21:23,  1.90s/it]
Outer training loop:  46%|####5     | 22779/50000 [13:59:32<14:05:17,  1.86s/it]
Outer training loop:  46%|####5     | 22780/50000 [13:59:34<13:57:07,  1.85s/it]
Outer training loop:  46%|####5     | 22781/50000 [13:59:36<14:26:49,  1.91s/it]
Outer training loop:  46%|####5     | 22782/50000 [13:59:37<14:22:27,  1.90s/it]
Outer training loop:  46%|####5     | 22783/50000 [13:59:39<13:48:59,  1.83s/it]
Outer training loop:  46%|####5     | 22784/50000 [13:59:41<13:49:03,  1.83s/it]
Outer training loop:  46%|####5     | 22785/50000 [13:59:43<14:05:25,  1.86s/it]
Outer training loop:  46%|####5     | 22786/50000 [13:59:45<14:01:27,  1.86s/it]
Outer training loop:  46%|####5     | 22787/50000 [13:59:47<13:52:59,  1.84s/it]
Outer training loop:  46%|####5     | 22788/50000 [13:59:48<14:09:44,  1.87s/it]
Outer training loop:  46%|####5     | 22789/50000 [13:59:50<14:18:01,  1.89s/it]
Outer training loop:  46%|####5     | 22790/50000 [13:59:52<14:17:01,  1.89s/it]
Outer training loop:  46%|####5     | 22791/50000 [13:59:54<14:22:43,  1.90s/it]
Outer training loop:  46%|####5     | 22792/50000 [13:59:56<14:15:00,  1.89s/it]
Outer training loop:  46%|####5     | 22793/50000 [13:59:58<14:08:02,  1.87s/it]
Outer training loop:  46%|####5     | 22794/50000 [14:00:00<14:01:20,  1.86s/it]
Outer training loop:  46%|####5     | 22795/50000 [14:00:02<14:23:22,  1.90s/it]
Outer training loop:  46%|####5     | 22796/50000 [14:00:04<14:02:04,  1.86s/it]
Outer training loop:  46%|####5     | 22797/50000 [14:00:06<14:24:52,  1.91s/it]
Outer training loop:  46%|####5     | 22798/50000 [14:00:08<14:33:49,  1.93s/it]
Outer training loop:  46%|####5     | 22799/50000 [14:00:09<14:28:01,  1.91s/it]
Outer training loop:  46%|####5     | 22800/50000 [14:00:11<14:22:12,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  46%|####5     | 22801/50000 [14:00:19<26:51:23,  3.55s/it]
Outer training loop:  46%|####5     | 22802/50000 [14:00:21<23:21:44,  3.09s/it]
Outer training loop:  46%|####5     | 22803/50000 [14:00:23<20:32:24,  2.72s/it]
Outer training loop:  46%|####5     | 22804/50000 [14:00:24<18:26:23,  2.44s/it]
Outer training loop:  46%|####5     | 22805/50000 [14:00:26<17:29:42,  2.32s/it]
Outer training loop:  46%|####5     | 22806/50000 [14:00:28<16:23:55,  2.17s/it]
Outer training loop:  46%|####5     | 22807/50000 [14:00:30<15:28:53,  2.05s/it]
Outer training loop:  46%|####5     | 22808/50000 [14:00:32<15:16:57,  2.02s/it]
Outer training loop:  46%|####5     | 22809/50000 [14:00:34<14:54:15,  1.97s/it]
Outer training loop:  46%|####5     | 22810/50000 [14:00:36<14:34:37,  1.93s/it]
Outer training loop:  46%|####5     | 22811/50000 [14:00:38<14:44:43,  1.95s/it]
Outer training loop:  46%|####5     | 22812/50000 [14:00:39<14:32:58,  1.93s/it]
Outer training loop:  46%|####5     | 22813/50000 [14:00:41<14:22:21,  1.90s/it]
Outer training loop:  46%|####5     | 22814/50000 [14:00:43<14:10:36,  1.88s/it]
Outer training loop:  46%|####5     | 22815/50000 [14:00:45<14:32:52,  1.93s/it]
Outer training loop:  46%|####5     | 22816/50000 [14:00:47<14:29:20,  1.92s/it]
Outer training loop:  46%|####5     | 22817/50000 [14:00:49<14:19:14,  1.90s/it]
Outer training loop:  46%|####5     | 22818/50000 [14:00:51<14:30:41,  1.92s/it]
Outer training loop:  46%|####5     | 22819/50000 [14:00:53<14:09:21,  1.87s/it]
Outer training loop:  46%|####5     | 22820/50000 [14:00:54<13:50:26,  1.83s/it]
Outer training loop:  46%|####5     | 22821/50000 [14:00:56<14:16:13,  1.89s/it]
Outer training loop:  46%|####5     | 22822/50000 [14:00:58<13:52:13,  1.84s/it]
Outer training loop:  46%|####5     | 22823/50000 [14:01:00<13:34:46,  1.80s/it]
Outer training loop:  46%|####5     | 22824/50000 [14:01:02<13:41:19,  1.81s/it]
Outer training loop:  46%|####5     | 22825/50000 [14:01:04<14:26:45,  1.91s/it]
Outer training loop:  46%|####5     | 22826/50000 [14:01:06<14:12:49,  1.88s/it]
Outer training loop:  46%|####5     | 22827/50000 [14:01:07<13:53:22,  1.84s/it]
Outer training loop:  46%|####5     | 22828/50000 [14:01:09<14:11:59,  1.88s/it]
Outer training loop:  46%|####5     | 22829/50000 [14:01:11<14:10:44,  1.88s/it]
Outer training loop:  46%|####5     | 22830/50000 [14:01:13<14:00:51,  1.86s/it]
Outer training loop:  46%|####5     | 22831/50000 [14:01:15<13:56:00,  1.85s/it]
Outer training loop:  46%|####5     | 22832/50000 [14:01:17<14:12:48,  1.88s/it]
Outer training loop:  46%|####5     | 22833/50000 [14:01:19<13:45:01,  1.82s/it]
Outer training loop:  46%|####5     | 22834/50000 [14:01:20<13:51:42,  1.84s/it]
Outer training loop:  46%|####5     | 22835/50000 [14:01:22<14:06:29,  1.87s/it]
Outer training loop:  46%|####5     | 22836/50000 [14:01:24<13:47:15,  1.83s/it]
Outer training loop:  46%|####5     | 22837/50000 [14:01:26<13:46:02,  1.82s/it]
Outer training loop:  46%|####5     | 22838/50000 [14:01:28<14:25:22,  1.91s/it]
Outer training loop:  46%|####5     | 22839/50000 [14:01:30<13:55:27,  1.85s/it]
Outer training loop:  46%|####5     | 22840/50000 [14:01:31<13:49:42,  1.83s/it]
Outer training loop:  46%|####5     | 22841/50000 [14:01:33<13:50:55,  1.84s/it]
Outer training loop:  46%|####5     | 22842/50000 [14:01:35<14:24:24,  1.91s/it]
Outer training loop:  46%|####5     | 22843/50000 [14:01:37<14:05:32,  1.87s/it]
Outer training loop:  46%|####5     | 22844/50000 [14:01:39<14:29:49,  1.92s/it]
Outer training loop:  46%|####5     | 22845/50000 [14:01:41<14:38:35,  1.94s/it]
Outer training loop:  46%|####5     | 22846/50000 [14:01:43<14:26:34,  1.91s/it]
Outer training loop:  46%|####5     | 22847/50000 [14:01:45<14:21:38,  1.90s/it]
Outer training loop:  46%|####5     | 22848/50000 [14:01:47<14:26:28,  1.91s/it]
Outer training loop:  46%|####5     | 22849/50000 [14:01:49<13:58:45,  1.85s/it]
Outer training loop:  46%|####5     | 22850/50000 [14:01:50<14:00:26,  1.86s/it]
Outer training loop:  46%|####5     | 22851/50000 [14:01:53<14:32:33,  1.93s/it]
Outer training loop:  46%|####5     | 22852/50000 [14:01:54<14:18:48,  1.90s/it]
Outer training loop:  46%|####5     | 22853/50000 [14:01:56<14:17:06,  1.89s/it]
Outer training loop:  46%|####5     | 22854/50000 [14:01:58<14:38:47,  1.94s/it]
Outer training loop:  46%|####5     | 22855/50000 [14:02:00<14:17:11,  1.89s/it]
Outer training loop:  46%|####5     | 22856/50000 [14:02:02<14:11:23,  1.88s/it]
Outer training loop:  46%|####5     | 22857/50000 [14:02:04<14:21:23,  1.90s/it]
Outer training loop:  46%|####5     | 22858/50000 [14:02:06<14:41:12,  1.95s/it]
Outer training loop:  46%|####5     | 22859/50000 [14:02:08<14:31:24,  1.93s/it]
Outer training loop:  46%|####5     | 22860/50000 [14:02:10<14:23:54,  1.91s/it]
Outer training loop:  46%|####5     | 22861/50000 [14:02:12<14:44:21,  1.96s/it]
Outer training loop:  46%|####5     | 22862/50000 [14:02:14<14:37:24,  1.94s/it]
Outer training loop:  46%|####5     | 22863/50000 [14:02:15<14:16:27,  1.89s/it]
Outer training loop:  46%|####5     | 22864/50000 [14:02:18<14:40:59,  1.95s/it]
Outer training loop:  46%|####5     | 22865/50000 [14:02:19<14:15:59,  1.89s/it]
Outer training loop:  46%|####5     | 22866/50000 [14:02:21<14:09:21,  1.88s/it]
Outer training loop:  46%|####5     | 22867/50000 [14:02:23<14:30:25,  1.92s/it]
Outer training loop:  46%|####5     | 22868/50000 [14:02:25<14:17:04,  1.90s/it]
Outer training loop:  46%|####5     | 22869/50000 [14:02:27<14:03:36,  1.87s/it]
Outer training loop:  46%|####5     | 22870/50000 [14:02:29<14:31:04,  1.93s/it]
Outer training loop:  46%|####5     | 22871/50000 [14:02:31<14:25:29,  1.91s/it]
Outer training loop:  46%|####5     | 22872/50000 [14:02:33<14:11:05,  1.88s/it]
Outer training loop:  46%|####5     | 22873/50000 [14:02:34<14:03:27,  1.87s/it]
Outer training loop:  46%|####5     | 22874/50000 [14:02:36<14:25:15,  1.91s/it]
Outer training loop:  46%|####5     | 22875/50000 [14:02:38<14:07:57,  1.88s/it]
Outer training loop:  46%|####5     | 22876/50000 [14:02:40<14:10:14,  1.88s/it]
Outer training loop:  46%|####5     | 22877/50000 [14:02:42<14:18:11,  1.90s/it]
Outer training loop:  46%|####5     | 22878/50000 [14:02:44<14:12:05,  1.89s/it]
Outer training loop:  46%|####5     | 22879/50000 [14:02:46<14:18:07,  1.90s/it]
Outer training loop:  46%|####5     | 22880/50000 [14:02:47<13:42:48,  1.82s/it]
Outer training loop:  46%|####5     | 22881/50000 [14:02:49<14:04:39,  1.87s/it]
Outer training loop:  46%|####5     | 22882/50000 [14:02:51<13:54:54,  1.85s/it]
Outer training loop:  46%|####5     | 22883/50000 [14:02:53<13:40:58,  1.82s/it]
Outer training loop:  46%|####5     | 22884/50000 [14:02:55<14:17:49,  1.90s/it]
Outer training loop:  46%|####5     | 22885/50000 [14:02:57<14:03:03,  1.87s/it]
Outer training loop:  46%|####5     | 22886/50000 [14:02:59<14:03:02,  1.87s/it]
Outer training loop:  46%|####5     | 22887/50000 [14:03:01<14:14:09,  1.89s/it]
Outer training loop:  46%|####5     | 22888/50000 [14:03:02<13:58:32,  1.86s/it]
Outer training loop:  46%|####5     | 22889/50000 [14:03:04<14:06:41,  1.87s/it]
Outer training loop:  46%|####5     | 22890/50000 [14:03:06<13:52:58,  1.84s/it]
Outer training loop:  46%|####5     | 22891/50000 [14:03:08<14:08:10,  1.88s/it]
Outer training loop:  46%|####5     | 22892/50000 [14:03:10<14:08:20,  1.88s/it]
Outer training loop:  46%|####5     | 22893/50000 [14:03:12<14:05:00,  1.87s/it]
Outer training loop:  46%|####5     | 22894/50000 [14:03:14<14:53:50,  1.98s/it]
Outer training loop:  46%|####5     | 22895/50000 [14:03:16<14:32:54,  1.93s/it]
Outer training loop:  46%|####5     | 22896/50000 [14:03:18<14:19:25,  1.90s/it]
Outer training loop:  46%|####5     | 22897/50000 [14:03:20<14:51:03,  1.97s/it]
Outer training loop:  46%|####5     | 22898/50000 [14:03:22<14:05:24,  1.87s/it]
Outer training loop:  46%|####5     | 22899/50000 [14:03:23<14:01:25,  1.86s/it]
Outer training loop:  46%|####5     | 22900/50000 [14:03:25<13:51:13,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  46%|####5     | 22901/50000 [14:03:32<26:07:22,  3.47s/it]
Outer training loop:  46%|####5     | 22902/50000 [14:03:34<22:49:18,  3.03s/it]
Outer training loop:  46%|####5     | 22903/50000 [14:03:36<20:05:00,  2.67s/it]
Outer training loop:  46%|####5     | 22904/50000 [14:03:38<18:34:04,  2.47s/it]
Outer training loop:  46%|####5     | 22905/50000 [14:03:40<17:40:02,  2.35s/it]
Outer training loop:  46%|####5     | 22906/50000 [14:03:42<16:34:38,  2.20s/it]
Outer training loop:  46%|####5     | 22907/50000 [14:03:44<15:38:33,  2.08s/it]
Outer training loop:  46%|####5     | 22908/50000 [14:03:46<15:27:21,  2.05s/it]
Outer training loop:  46%|####5     | 22909/50000 [14:03:48<14:47:20,  1.97s/it]
Outer training loop:  46%|####5     | 22910/50000 [14:03:50<14:34:36,  1.94s/it]
Outer training loop:  46%|####5     | 22911/50000 [14:03:51<14:05:40,  1.87s/it]
Outer training loop:  46%|####5     | 22912/50000 [14:03:53<14:42:38,  1.96s/it]
Outer training loop:  46%|####5     | 22913/50000 [14:03:55<14:20:20,  1.91s/it]
Outer training loop:  46%|####5     | 22914/50000 [14:03:57<14:06:10,  1.87s/it]
Outer training loop:  46%|####5     | 22915/50000 [14:03:59<14:18:09,  1.90s/it]
Outer training loop:  46%|####5     | 22916/50000 [14:04:01<13:57:25,  1.86s/it]
Outer training loop:  46%|####5     | 22917/50000 [14:04:03<13:57:15,  1.85s/it]
Outer training loop:  46%|####5     | 22918/50000 [14:04:05<14:16:53,  1.90s/it]
Outer training loop:  46%|####5     | 22919/50000 [14:04:06<14:09:28,  1.88s/it]
Outer training loop:  46%|####5     | 22920/50000 [14:04:08<13:52:59,  1.85s/it]
Outer training loop:  46%|####5     | 22921/50000 [14:04:10<13:58:11,  1.86s/it]
Outer training loop:  46%|####5     | 22922/50000 [14:04:12<14:24:41,  1.92s/it]
Outer training loop:  46%|####5     | 22923/50000 [14:04:14<14:04:28,  1.87s/it]
Outer training loop:  46%|####5     | 22924/50000 [14:04:16<14:03:11,  1.87s/it]
Outer training loop:  46%|####5     | 22925/50000 [14:04:18<14:24:09,  1.92s/it]
Outer training loop:  46%|####5     | 22926/50000 [14:04:20<14:29:36,  1.93s/it]
Outer training loop:  46%|####5     | 22927/50000 [14:04:22<14:11:09,  1.89s/it]
Outer training loop:  46%|####5     | 22928/50000 [14:04:24<14:34:11,  1.94s/it]
Outer training loop:  46%|####5     | 22929/50000 [14:04:25<14:25:49,  1.92s/it]
Outer training loop:  46%|####5     | 22930/50000 [14:04:27<14:31:53,  1.93s/it]
Outer training loop:  46%|####5     | 22931/50000 [14:04:29<14:16:23,  1.90s/it]
Outer training loop:  46%|####5     | 22932/50000 [14:04:31<14:37:06,  1.94s/it]
Outer training loop:  46%|####5     | 22933/50000 [14:04:33<14:35:25,  1.94s/it]
Outer training loop:  46%|####5     | 22934/50000 [14:04:35<14:44:28,  1.96s/it]
Outer training loop:  46%|####5     | 22935/50000 [14:04:37<14:24:05,  1.92s/it]
Outer training loop:  46%|####5     | 22936/50000 [14:04:39<14:36:07,  1.94s/it]
Outer training loop:  46%|####5     | 22937/50000 [14:04:41<14:29:59,  1.93s/it]
Outer training loop:  46%|####5     | 22938/50000 [14:04:43<14:11:18,  1.89s/it]
Outer training loop:  46%|####5     | 22939/50000 [14:04:45<14:30:26,  1.93s/it]
Outer training loop:  46%|####5     | 22940/50000 [14:04:47<14:19:35,  1.91s/it]
Outer training loop:  46%|####5     | 22941/50000 [14:04:49<14:19:07,  1.91s/it]
Outer training loop:  46%|####5     | 22942/50000 [14:04:51<14:46:43,  1.97s/it]
Outer training loop:  46%|####5     | 22943/50000 [14:04:53<14:32:20,  1.93s/it]
Outer training loop:  46%|####5     | 22944/50000 [14:04:55<14:40:10,  1.95s/it]
Outer training loop:  46%|####5     | 22945/50000 [14:04:56<14:35:19,  1.94s/it]
Outer training loop:  46%|####5     | 22946/50000 [14:04:58<14:36:02,  1.94s/it]
Outer training loop:  46%|####5     | 22947/50000 [14:05:00<14:33:40,  1.94s/it]
Outer training loop:  46%|####5     | 22948/50000 [14:05:02<14:46:15,  1.97s/it]
Outer training loop:  46%|####5     | 22949/50000 [14:05:04<14:28:55,  1.93s/it]
Outer training loop:  46%|####5     | 22950/50000 [14:05:06<14:21:02,  1.91s/it]
Outer training loop:  46%|####5     | 22951/50000 [14:05:08<14:04:21,  1.87s/it]
Outer training loop:  46%|####5     | 22952/50000 [14:05:10<14:44:25,  1.96s/it]
Outer training loop:  46%|####5     | 22953/50000 [14:05:12<14:33:05,  1.94s/it]
Outer training loop:  46%|####5     | 22954/50000 [14:05:14<14:23:13,  1.92s/it]
Outer training loop:  46%|####5     | 22955/50000 [14:05:16<14:40:50,  1.95s/it]
Outer training loop:  46%|####5     | 22956/50000 [14:05:18<14:14:54,  1.90s/it]
Outer training loop:  46%|####5     | 22957/50000 [14:05:19<14:03:56,  1.87s/it]
Outer training loop:  46%|####5     | 22958/50000 [14:05:21<14:24:01,  1.92s/it]
Outer training loop:  46%|####5     | 22959/50000 [14:05:23<14:05:22,  1.88s/it]
Outer training loop:  46%|####5     | 22960/50000 [14:05:25<13:59:33,  1.86s/it]
Outer training loop:  46%|####5     | 22961/50000 [14:05:27<14:24:12,  1.92s/it]
Outer training loop:  46%|####5     | 22962/50000 [14:05:29<14:13:06,  1.89s/it]
Outer training loop:  46%|####5     | 22963/50000 [14:05:31<14:00:27,  1.87s/it]
Outer training loop:  46%|####5     | 22964/50000 [14:05:33<13:56:51,  1.86s/it]
Outer training loop:  46%|####5     | 22965/50000 [14:05:35<14:32:01,  1.94s/it]
Outer training loop:  46%|####5     | 22966/50000 [14:05:36<14:11:39,  1.89s/it]
Outer training loop:  46%|####5     | 22967/50000 [14:05:38<13:59:00,  1.86s/it]
Outer training loop:  46%|####5     | 22968/50000 [14:05:40<14:17:18,  1.90s/it]
Outer training loop:  46%|####5     | 22969/50000 [14:05:42<13:55:11,  1.85s/it]
Outer training loop:  46%|####5     | 22970/50000 [14:05:44<14:05:10,  1.88s/it]
Outer training loop:  46%|####5     | 22971/50000 [14:05:46<14:10:18,  1.89s/it]
Outer training loop:  46%|####5     | 22972/50000 [14:05:48<14:09:16,  1.89s/it]
Outer training loop:  46%|####5     | 22973/50000 [14:05:50<14:10:46,  1.89s/it]
Outer training loop:  46%|####5     | 22974/50000 [14:05:51<14:11:23,  1.89s/it]
Outer training loop:  46%|####5     | 22975/50000 [14:05:53<14:12:21,  1.89s/it]
Outer training loop:  46%|####5     | 22976/50000 [14:05:55<14:13:16,  1.89s/it]
Outer training loop:  46%|####5     | 22977/50000 [14:05:57<14:06:39,  1.88s/it]
Outer training loop:  46%|####5     | 22978/50000 [14:05:59<14:27:21,  1.93s/it]
Outer training loop:  46%|####5     | 22979/50000 [14:06:01<14:20:54,  1.91s/it]
Outer training loop:  46%|####5     | 22980/50000 [14:06:03<14:15:29,  1.90s/it]
Outer training loop:  46%|####5     | 22981/50000 [14:06:05<14:06:10,  1.88s/it]
Outer training loop:  46%|####5     | 22982/50000 [14:06:07<14:22:49,  1.92s/it]
Outer training loop:  46%|####5     | 22983/50000 [14:06:09<14:18:01,  1.91s/it]
Outer training loop:  46%|####5     | 22984/50000 [14:06:10<14:14:12,  1.90s/it]
Outer training loop:  46%|####5     | 22985/50000 [14:06:12<14:21:22,  1.91s/it]
Outer training loop:  46%|####5     | 22986/50000 [14:06:14<14:16:45,  1.90s/it]
Outer training loop:  46%|####5     | 22987/50000 [14:06:16<14:03:15,  1.87s/it]
Outer training loop:  46%|####5     | 22988/50000 [14:06:18<13:28:14,  1.80s/it]
Outer training loop:  46%|####5     | 22989/50000 [14:06:20<13:37:20,  1.82s/it]
Outer training loop:  46%|####5     | 22990/50000 [14:06:21<13:40:57,  1.82s/it]
Outer training loop:  46%|####5     | 22991/50000 [14:06:23<14:11:28,  1.89s/it]
Outer training loop:  46%|####5     | 22992/50000 [14:06:25<14:28:22,  1.93s/it]
Outer training loop:  46%|####5     | 22993/50000 [14:06:27<14:11:01,  1.89s/it]
Outer training loop:  46%|####5     | 22994/50000 [14:06:29<13:50:17,  1.84s/it]
Outer training loop:  46%|####5     | 22995/50000 [14:06:31<14:17:36,  1.91s/it]
Outer training loop:  46%|####5     | 22996/50000 [14:06:33<13:59:15,  1.86s/it]
Outer training loop:  46%|####5     | 22997/50000 [14:06:35<14:45:50,  1.97s/it]
Outer training loop:  46%|####5     | 22998/50000 [14:06:37<14:18:13,  1.91s/it]
Outer training loop:  46%|####5     | 22999/50000 [14:06:39<14:27:24,  1.93s/it]
Outer training loop:  46%|####6     | 23000/50000 [14:06:41<14:18:36,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.97s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:40,  1.96s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:33,  1.89s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:29,  1.87s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:25,  1.84s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.94s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:18,  1.85s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.94s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:18,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.89s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:15,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:16,  1.95s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:02,  1.83s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.92s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:00,  1.85s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:54,  1.85s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:56,  1.91s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:45,  1.85s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:49,  1.96s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.85s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:38,  1.93s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:26,  1.81s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:26,  1.84s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:26,  1.88s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:26,  1.93s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.90s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:16,  1.86s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:14,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:11,  1.84s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:13,  1.93s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<01:00,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<01:01,  1.99s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:57,  1.92s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:46,  1.85s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:43,  1.82s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:45,  1.96s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:43,  1.96s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:40,  1.92s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:30,  1.81s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:28,  1.80s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:29,  1.98s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.92s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.91s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.87s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.85s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:14,  1.87s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.88s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:12,  2.06s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.99s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.99s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.95s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.92s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.86s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.77s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.78s/it]

Evaluating policy:   5%|4         | 4/83 [00:08<02:41,  2.05s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:32,  1.95s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:25,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:27,  1.97s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:18,  1.90s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:16,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:14,  1.92s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.88s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:06,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:10,  1.95s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.93s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:57,  1.86s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.92s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:55,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:55,  1.99s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:50,  1.94s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:42,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:41,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:41,  1.94s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.94s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:27,  1.83s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:32,  1.98s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:29,  1.95s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:25,  1.91s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:20,  1.95s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:16,  1.92s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:08,  1.85s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:06,  1.97s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:03,  1.91s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.84s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.88s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.85s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:53,  1.99s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:50,  1.94s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:48,  1.94s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:43,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:38,  2.01s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:35,  1.97s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:33,  1.95s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:30,  1.90s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.92s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:27,  1.93s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.88s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:19<00:20,  2.02s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:17,  1.97s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:15,  1.95s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:13,  1.96s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:11,  1.96s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.89s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.90s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.91s/it]

Outer training loop:  46%|####6     | 23001/50000 [14:12:03<734:42:09, 97.96s/it]
Outer training loop:  46%|####6     | 23002/50000 [14:12:05<520:00:20, 69.34s/it]
Outer training loop:  46%|####6     | 23003/50000 [14:12:07<368:00:43, 49.07s/it]
Outer training loop:  46%|####6     | 23004/50000 [14:12:09<261:48:48, 34.91s/it]
Outer training loop:  46%|####6     | 23005/50000 [14:12:11<187:55:33, 25.06s/it]
Outer training loop:  46%|####6     | 23006/50000 [14:12:13<135:42:27, 18.10s/it]
Outer training loop:  46%|####6     | 23007/50000 [14:12:15<99:12:03, 13.23s/it] 
Outer training loop:  46%|####6     | 23008/50000 [14:12:17<74:03:37,  9.88s/it]
Outer training loop:  46%|####6     | 23009/50000 [14:12:19<56:16:18,  7.51s/it]
Outer training loop:  46%|####6     | 23010/50000 [14:12:21<43:46:12,  5.84s/it]
Outer training loop:  46%|####6     | 23011/50000 [14:12:23<35:23:33,  4.72s/it]
Outer training loop:  46%|####6     | 23012/50000 [14:12:25<28:53:24,  3.85s/it]
Outer training loop:  46%|####6     | 23013/50000 [14:12:26<24:19:11,  3.24s/it]
Outer training loop:  46%|####6     | 23014/50000 [14:12:28<21:23:16,  2.85s/it]
Outer training loop:  46%|####6     | 23015/50000 [14:12:31<19:47:42,  2.64s/it]
Outer training loop:  46%|####6     | 23016/50000 [14:12:32<18:04:47,  2.41s/it]
Outer training loop:  46%|####6     | 23017/50000 [14:12:34<16:56:58,  2.26s/it]
Outer training loop:  46%|####6     | 23018/50000 [14:12:36<16:28:20,  2.20s/it]
Outer training loop:  46%|####6     | 23019/50000 [14:12:38<15:47:40,  2.11s/it]
Outer training loop:  46%|####6     | 23020/50000 [14:12:40<15:27:13,  2.06s/it]
Outer training loop:  46%|####6     | 23021/50000 [14:12:42<14:58:35,  2.00s/it]
Outer training loop:  46%|####6     | 23022/50000 [14:12:44<15:37:12,  2.08s/it]
Outer training loop:  46%|####6     | 23023/50000 [14:12:46<15:16:37,  2.04s/it]
Outer training loop:  46%|####6     | 23024/50000 [14:12:48<14:31:42,  1.94s/it]
Outer training loop:  46%|####6     | 23025/50000 [14:12:50<14:55:24,  1.99s/it]
Outer training loop:  46%|####6     | 23026/50000 [14:12:52<14:49:40,  1.98s/it]
Outer training loop:  46%|####6     | 23027/50000 [14:12:54<14:30:54,  1.94s/it]
Outer training loop:  46%|####6     | 23028/50000 [14:12:56<14:16:35,  1.91s/it]
Outer training loop:  46%|####6     | 23029/50000 [14:12:58<14:40:38,  1.96s/it]
Outer training loop:  46%|####6     | 23030/50000 [14:13:00<14:30:37,  1.94s/it]
Outer training loop:  46%|####6     | 23031/50000 [14:13:02<14:20:36,  1.91s/it]
Outer training loop:  46%|####6     | 23032/50000 [14:13:04<14:42:20,  1.96s/it]
Outer training loop:  46%|####6     | 23033/50000 [14:13:06<14:25:35,  1.93s/it]
Outer training loop:  46%|####6     | 23034/50000 [14:13:07<14:05:21,  1.88s/it]
Outer training loop:  46%|####6     | 23035/50000 [14:13:09<14:15:45,  1.90s/it]
Outer training loop:  46%|####6     | 23036/50000 [14:13:11<14:13:39,  1.90s/it]
Outer training loop:  46%|####6     | 23037/50000 [14:13:13<14:00:49,  1.87s/it]
Outer training loop:  46%|####6     | 23038/50000 [14:13:15<13:34:44,  1.81s/it]
Outer training loop:  46%|####6     | 23039/50000 [14:13:17<14:14:27,  1.90s/it]
Outer training loop:  46%|####6     | 23040/50000 [14:13:19<14:06:53,  1.88s/it]
Outer training loop:  46%|####6     | 23041/50000 [14:13:20<13:59:10,  1.87s/it]
Outer training loop:  46%|####6     | 23042/50000 [14:13:22<14:15:16,  1.90s/it]
Outer training loop:  46%|####6     | 23043/50000 [14:13:24<14:11:00,  1.89s/it]
Outer training loop:  46%|####6     | 23044/50000 [14:13:26<14:11:12,  1.89s/it]
Outer training loop:  46%|####6     | 23045/50000 [14:13:28<14:44:01,  1.97s/it]
Outer training loop:  46%|####6     | 23046/50000 [14:13:30<14:32:19,  1.94s/it]
Outer training loop:  46%|####6     | 23047/50000 [14:13:32<14:08:12,  1.89s/it]
Outer training loop:  46%|####6     | 23048/50000 [14:13:34<14:20:44,  1.92s/it]
Outer training loop:  46%|####6     | 23049/50000 [14:13:36<14:09:45,  1.89s/it]
Outer training loop:  46%|####6     | 23050/50000 [14:13:38<14:08:31,  1.89s/it]
Outer training loop:  46%|####6     | 23051/50000 [14:13:40<14:28:29,  1.93s/it]
Outer training loop:  46%|####6     | 23052/50000 [14:13:42<14:35:04,  1.95s/it]
Outer training loop:  46%|####6     | 23053/50000 [14:13:43<14:15:22,  1.90s/it]
Outer training loop:  46%|####6     | 23054/50000 [14:13:45<14:05:56,  1.88s/it]
Outer training loop:  46%|####6     | 23055/50000 [14:13:47<14:38:14,  1.96s/it]
Outer training loop:  46%|####6     | 23056/50000 [14:13:49<14:21:07,  1.92s/it]
Outer training loop:  46%|####6     | 23057/50000 [14:13:51<13:56:48,  1.86s/it]
Outer training loop:  46%|####6     | 23058/50000 [14:13:53<14:12:53,  1.90s/it]
Outer training loop:  46%|####6     | 23059/50000 [14:13:55<14:08:53,  1.89s/it]
Outer training loop:  46%|####6     | 23060/50000 [14:13:57<13:58:05,  1.87s/it]
Outer training loop:  46%|####6     | 23061/50000 [14:13:58<13:47:40,  1.84s/it]
Outer training loop:  46%|####6     | 23062/50000 [14:14:00<14:08:25,  1.89s/it]
Outer training loop:  46%|####6     | 23063/50000 [14:14:02<14:06:43,  1.89s/it]
Outer training loop:  46%|####6     | 23064/50000 [14:14:04<14:04:31,  1.88s/it]
Outer training loop:  46%|####6     | 23065/50000 [14:14:06<14:26:53,  1.93s/it]
Outer training loop:  46%|####6     | 23066/50000 [14:14:08<14:07:40,  1.89s/it]
Outer training loop:  46%|####6     | 23067/50000 [14:14:10<13:54:44,  1.86s/it]
Outer training loop:  46%|####6     | 23068/50000 [14:14:12<14:07:12,  1.89s/it]
Outer training loop:  46%|####6     | 23069/50000 [14:14:14<13:53:28,  1.86s/it]
Outer training loop:  46%|####6     | 23070/50000 [14:14:15<13:45:50,  1.84s/it]
Outer training loop:  46%|####6     | 23071/50000 [14:14:17<13:29:06,  1.80s/it]
Outer training loop:  46%|####6     | 23072/50000 [14:14:19<14:29:30,  1.94s/it]
Outer training loop:  46%|####6     | 23073/50000 [14:14:21<14:10:12,  1.89s/it]
Outer training loop:  46%|####6     | 23074/50000 [14:14:23<14:02:38,  1.88s/it]
Outer training loop:  46%|####6     | 23075/50000 [14:14:25<14:24:27,  1.93s/it]
Outer training loop:  46%|####6     | 23076/50000 [14:14:27<14:12:12,  1.90s/it]
Outer training loop:  46%|####6     | 23077/50000 [14:14:29<14:13:10,  1.90s/it]
Outer training loop:  46%|####6     | 23078/50000 [14:14:31<14:17:02,  1.91s/it]
Outer training loop:  46%|####6     | 23079/50000 [14:14:33<14:19:57,  1.92s/it]
Outer training loop:  46%|####6     | 23080/50000 [14:14:34<14:08:16,  1.89s/it]
Outer training loop:  46%|####6     | 23081/50000 [14:14:36<14:20:07,  1.92s/it]
Outer training loop:  46%|####6     | 23082/50000 [14:14:39<14:48:26,  1.98s/it]
Outer training loop:  46%|####6     | 23083/50000 [14:14:40<14:22:43,  1.92s/it]
Outer training loop:  46%|####6     | 23084/50000 [14:14:42<14:01:58,  1.88s/it]
Outer training loop:  46%|####6     | 23085/50000 [14:14:44<14:12:53,  1.90s/it]
Outer training loop:  46%|####6     | 23086/50000 [14:14:46<14:19:10,  1.92s/it]
Outer training loop:  46%|####6     | 23087/50000 [14:14:48<14:11:27,  1.90s/it]
Outer training loop:  46%|####6     | 23088/50000 [14:14:50<13:54:24,  1.86s/it]
Outer training loop:  46%|####6     | 23089/50000 [14:14:52<14:06:29,  1.89s/it]
Outer training loop:  46%|####6     | 23090/50000 [14:14:53<13:58:21,  1.87s/it]
Outer training loop:  46%|####6     | 23091/50000 [14:14:55<14:12:53,  1.90s/it]
Outer training loop:  46%|####6     | 23092/50000 [14:14:57<14:08:29,  1.89s/it]
Outer training loop:  46%|####6     | 23093/50000 [14:14:59<14:15:39,  1.91s/it]
Outer training loop:  46%|####6     | 23094/50000 [14:15:01<14:25:14,  1.93s/it]
Outer training loop:  46%|####6     | 23095/50000 [14:15:03<14:11:41,  1.90s/it]
Outer training loop:  46%|####6     | 23096/50000 [14:15:05<14:42:11,  1.97s/it]
Outer training loop:  46%|####6     | 23097/50000 [14:15:07<14:04:31,  1.88s/it]
Outer training loop:  46%|####6     | 23098/50000 [14:15:09<13:54:00,  1.86s/it]
Outer training loop:  46%|####6     | 23099/50000 [14:15:11<14:02:54,  1.88s/it]
Outer training loop:  46%|####6     | 23100/50000 [14:15:13<14:11:25,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  46%|####6     | 23101/50000 [14:15:20<26:46:37,  3.58s/it]
Outer training loop:  46%|####6     | 23102/50000 [14:15:22<22:45:07,  3.05s/it]
Outer training loop:  46%|####6     | 23103/50000 [14:15:24<20:01:44,  2.68s/it]
Outer training loop:  46%|####6     | 23104/50000 [14:15:26<18:46:11,  2.51s/it]
Outer training loop:  46%|####6     | 23105/50000 [14:15:28<17:04:59,  2.29s/it]
Outer training loop:  46%|####6     | 23106/50000 [14:15:30<16:32:47,  2.21s/it]
Outer training loop:  46%|####6     | 23107/50000 [14:15:31<15:14:42,  2.04s/it]
Outer training loop:  46%|####6     | 23108/50000 [14:15:33<15:08:24,  2.03s/it]
Outer training loop:  46%|####6     | 23109/50000 [14:15:35<14:41:25,  1.97s/it]
Outer training loop:  46%|####6     | 23110/50000 [14:15:37<14:15:45,  1.91s/it]
Outer training loop:  46%|####6     | 23111/50000 [14:15:39<14:18:11,  1.91s/it]
Outer training loop:  46%|####6     | 23112/50000 [14:15:40<13:53:50,  1.86s/it]
Outer training loop:  46%|####6     | 23113/50000 [14:15:42<13:41:11,  1.83s/it]
Outer training loop:  46%|####6     | 23114/50000 [14:15:44<13:46:23,  1.84s/it]
Outer training loop:  46%|####6     | 23115/50000 [14:15:46<13:42:50,  1.84s/it]
Outer training loop:  46%|####6     | 23116/50000 [14:15:48<13:38:14,  1.83s/it]
Outer training loop:  46%|####6     | 23117/50000 [14:15:49<13:30:31,  1.81s/it]
Outer training loop:  46%|####6     | 23118/50000 [14:15:52<14:01:33,  1.88s/it]
Outer training loop:  46%|####6     | 23119/50000 [14:15:54<14:15:57,  1.91s/it]
Outer training loop:  46%|####6     | 23120/50000 [14:15:55<14:08:44,  1.89s/it]
Outer training loop:  46%|####6     | 23121/50000 [14:15:57<14:20:05,  1.92s/it]
Outer training loop:  46%|####6     | 23122/50000 [14:15:59<14:42:43,  1.97s/it]
Outer training loop:  46%|####6     | 23123/50000 [14:16:01<14:15:06,  1.91s/it]
Outer training loop:  46%|####6     | 23124/50000 [14:16:03<14:11:08,  1.90s/it]
Outer training loop:  46%|####6     | 23125/50000 [14:16:05<14:33:43,  1.95s/it]
Outer training loop:  46%|####6     | 23126/50000 [14:16:07<14:21:36,  1.92s/it]
Outer training loop:  46%|####6     | 23127/50000 [14:16:09<14:14:07,  1.91s/it]
Outer training loop:  46%|####6     | 23128/50000 [14:16:11<14:24:48,  1.93s/it]
Outer training loop:  46%|####6     | 23129/50000 [14:16:13<14:19:14,  1.92s/it]
Outer training loop:  46%|####6     | 23130/50000 [14:16:15<14:14:42,  1.91s/it]
Outer training loop:  46%|####6     | 23131/50000 [14:16:16<13:51:00,  1.86s/it]
Outer training loop:  46%|####6     | 23132/50000 [14:16:18<14:21:46,  1.92s/it]
Outer training loop:  46%|####6     | 23133/50000 [14:16:20<14:30:09,  1.94s/it]
Outer training loop:  46%|####6     | 23134/50000 [14:16:22<14:23:55,  1.93s/it]
Outer training loop:  46%|####6     | 23135/50000 [14:16:24<14:38:42,  1.96s/it]
Outer training loop:  46%|####6     | 23136/50000 [14:16:26<14:11:48,  1.90s/it]
Outer training loop:  46%|####6     | 23137/50000 [14:16:28<14:14:39,  1.91s/it]
Outer training loop:  46%|####6     | 23138/50000 [14:16:30<14:25:15,  1.93s/it]
Outer training loop:  46%|####6     | 23139/50000 [14:16:32<14:12:28,  1.90s/it]
Outer training loop:  46%|####6     | 23140/50000 [14:16:34<13:55:45,  1.87s/it]
Outer training loop:  46%|####6     | 23141/50000 [14:16:36<13:51:30,  1.86s/it]
Outer training loop:  46%|####6     | 23142/50000 [14:16:37<13:51:47,  1.86s/it]
Outer training loop:  46%|####6     | 23143/50000 [14:16:39<13:55:18,  1.87s/it]
Outer training loop:  46%|####6     | 23144/50000 [14:16:41<13:56:03,  1.87s/it]
Outer training loop:  46%|####6     | 23145/50000 [14:16:43<14:14:16,  1.91s/it]
Outer training loop:  46%|####6     | 23146/50000 [14:16:45<13:59:04,  1.87s/it]
Outer training loop:  46%|####6     | 23147/50000 [14:16:47<13:52:10,  1.86s/it]
Outer training loop:  46%|####6     | 23148/50000 [14:16:49<13:52:26,  1.86s/it]
Outer training loop:  46%|####6     | 23149/50000 [14:16:51<14:18:58,  1.92s/it]
Outer training loop:  46%|####6     | 23150/50000 [14:16:52<13:55:37,  1.87s/it]
Outer training loop:  46%|####6     | 23151/50000 [14:16:54<13:49:43,  1.85s/it]
Outer training loop:  46%|####6     | 23152/50000 [14:16:56<14:21:30,  1.93s/it]
Outer training loop:  46%|####6     | 23153/50000 [14:16:58<13:57:34,  1.87s/it]
Outer training loop:  46%|####6     | 23154/50000 [14:17:00<13:35:54,  1.82s/it]
Outer training loop:  46%|####6     | 23155/50000 [14:17:02<13:43:16,  1.84s/it]
Outer training loop:  46%|####6     | 23156/50000 [14:17:03<13:23:11,  1.80s/it]
Outer training loop:  46%|####6     | 23157/50000 [14:17:05<13:35:36,  1.82s/it]
Outer training loop:  46%|####6     | 23158/50000 [14:17:07<14:03:30,  1.89s/it]
Outer training loop:  46%|####6     | 23159/50000 [14:17:09<13:46:05,  1.85s/it]
Outer training loop:  46%|####6     | 23160/50000 [14:17:11<13:49:00,  1.85s/it]
Outer training loop:  46%|####6     | 23161/50000 [14:17:13<13:46:55,  1.85s/it]
Outer training loop:  46%|####6     | 23162/50000 [14:17:15<13:54:31,  1.87s/it]
Outer training loop:  46%|####6     | 23163/50000 [14:17:16<13:44:22,  1.84s/it]
Outer training loop:  46%|####6     | 23164/50000 [14:17:18<13:15:09,  1.78s/it]
Outer training loop:  46%|####6     | 23165/50000 [14:17:20<13:53:38,  1.86s/it]
Outer training loop:  46%|####6     | 23166/50000 [14:17:22<13:39:04,  1.83s/it]
Outer training loop:  46%|####6     | 23167/50000 [14:17:24<13:26:41,  1.80s/it]
Outer training loop:  46%|####6     | 23168/50000 [14:17:26<13:50:27,  1.86s/it]
Outer training loop:  46%|####6     | 23169/50000 [14:17:28<14:21:52,  1.93s/it]
Outer training loop:  46%|####6     | 23170/50000 [14:17:29<13:57:06,  1.87s/it]
Outer training loop:  46%|####6     | 23171/50000 [14:17:31<14:00:27,  1.88s/it]
Outer training loop:  46%|####6     | 23172/50000 [14:17:33<14:04:21,  1.89s/it]
Outer training loop:  46%|####6     | 23173/50000 [14:17:35<13:57:52,  1.87s/it]
Outer training loop:  46%|####6     | 23174/50000 [14:17:37<13:37:10,  1.83s/it]
Outer training loop:  46%|####6     | 23175/50000 [14:17:39<13:30:41,  1.81s/it]
Outer training loop:  46%|####6     | 23176/50000 [14:17:41<13:54:37,  1.87s/it]
Outer training loop:  46%|####6     | 23177/50000 [14:17:42<13:53:21,  1.86s/it]
Outer training loop:  46%|####6     | 23178/50000 [14:17:44<13:38:48,  1.83s/it]
Outer training loop:  46%|####6     | 23179/50000 [14:17:46<13:57:23,  1.87s/it]
Outer training loop:  46%|####6     | 23180/50000 [14:17:48<13:37:33,  1.83s/it]
Outer training loop:  46%|####6     | 23181/50000 [14:17:50<13:32:57,  1.82s/it]
Outer training loop:  46%|####6     | 23182/50000 [14:17:52<13:50:27,  1.86s/it]
Outer training loop:  46%|####6     | 23183/50000 [14:17:53<13:37:08,  1.83s/it]
Outer training loop:  46%|####6     | 23184/50000 [14:17:55<13:18:48,  1.79s/it]
Outer training loop:  46%|####6     | 23185/50000 [14:17:57<13:08:23,  1.76s/it]
Outer training loop:  46%|####6     | 23186/50000 [14:17:59<13:29:56,  1.81s/it]
Outer training loop:  46%|####6     | 23187/50000 [14:18:00<13:25:37,  1.80s/it]
Outer training loop:  46%|####6     | 23188/50000 [14:18:02<13:23:58,  1.80s/it]
Outer training loop:  46%|####6     | 23189/50000 [14:18:04<13:48:27,  1.85s/it]
Outer training loop:  46%|####6     | 23190/50000 [14:18:06<13:45:38,  1.85s/it]
Outer training loop:  46%|####6     | 23191/50000 [14:18:08<13:42:26,  1.84s/it]
Outer training loop:  46%|####6     | 23192/50000 [14:18:10<13:56:44,  1.87s/it]
Outer training loop:  46%|####6     | 23193/50000 [14:18:12<13:40:34,  1.84s/it]
Outer training loop:  46%|####6     | 23194/50000 [14:18:14<13:48:05,  1.85s/it]
Outer training loop:  46%|####6     | 23195/50000 [14:18:15<13:39:19,  1.83s/it]
Outer training loop:  46%|####6     | 23196/50000 [14:18:17<13:46:50,  1.85s/it]
Outer training loop:  46%|####6     | 23197/50000 [14:18:19<13:51:02,  1.86s/it]
Outer training loop:  46%|####6     | 23198/50000 [14:18:21<13:40:03,  1.84s/it]
Outer training loop:  46%|####6     | 23199/50000 [14:18:23<13:45:00,  1.85s/it]
Outer training loop:  46%|####6     | 23200/50000 [14:18:25<13:46:08,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.91s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  46%|####6     | 23201/50000 [14:18:32<25:35:34,  3.44s/it]
Outer training loop:  46%|####6     | 23202/50000 [14:18:34<22:29:41,  3.02s/it]
Outer training loop:  46%|####6     | 23203/50000 [14:18:36<19:50:17,  2.67s/it]
Outer training loop:  46%|####6     | 23204/50000 [14:18:37<17:48:09,  2.39s/it]
Outer training loop:  46%|####6     | 23205/50000 [14:18:39<16:17:53,  2.19s/it]
Outer training loop:  46%|####6     | 23206/50000 [14:18:41<15:45:20,  2.12s/it]
Outer training loop:  46%|####6     | 23207/50000 [14:18:43<15:05:05,  2.03s/it]
Outer training loop:  46%|####6     | 23208/50000 [14:18:44<14:13:16,  1.91s/it]
Outer training loop:  46%|####6     | 23209/50000 [14:18:46<14:20:14,  1.93s/it]
Outer training loop:  46%|####6     | 23210/50000 [14:18:48<13:53:17,  1.87s/it]
Outer training loop:  46%|####6     | 23211/50000 [14:18:50<13:36:18,  1.83s/it]
Outer training loop:  46%|####6     | 23212/50000 [14:18:52<13:21:20,  1.79s/it]
Outer training loop:  46%|####6     | 23213/50000 [14:18:54<13:41:05,  1.84s/it]
Outer training loop:  46%|####6     | 23214/50000 [14:18:55<13:33:30,  1.82s/it]
Outer training loop:  46%|####6     | 23215/50000 [14:18:57<13:16:09,  1.78s/it]
Outer training loop:  46%|####6     | 23216/50000 [14:18:59<14:02:43,  1.89s/it]
Outer training loop:  46%|####6     | 23217/50000 [14:19:01<13:37:19,  1.83s/it]
Outer training loop:  46%|####6     | 23218/50000 [14:19:03<13:22:43,  1.80s/it]
Outer training loop:  46%|####6     | 23219/50000 [14:19:04<13:18:27,  1.79s/it]
Outer training loop:  46%|####6     | 23220/50000 [14:19:06<13:49:23,  1.86s/it]
Outer training loop:  46%|####6     | 23221/50000 [14:19:08<13:19:40,  1.79s/it]
Outer training loop:  46%|####6     | 23222/50000 [14:19:10<13:28:23,  1.81s/it]
Outer training loop:  46%|####6     | 23223/50000 [14:19:12<13:48:30,  1.86s/it]
Outer training loop:  46%|####6     | 23224/50000 [14:19:14<13:43:57,  1.85s/it]
Outer training loop:  46%|####6     | 23225/50000 [14:19:15<13:35:30,  1.83s/it]
Outer training loop:  46%|####6     | 23226/50000 [14:19:17<13:46:33,  1.85s/it]
Outer training loop:  46%|####6     | 23227/50000 [14:19:19<13:34:13,  1.82s/it]
Outer training loop:  46%|####6     | 23228/50000 [14:19:21<13:29:31,  1.81s/it]
Outer training loop:  46%|####6     | 23229/50000 [14:19:23<13:54:14,  1.87s/it]
Outer training loop:  46%|####6     | 23230/50000 [14:19:25<13:48:55,  1.86s/it]
Outer training loop:  46%|####6     | 23231/50000 [14:19:27<13:37:02,  1.83s/it]
Outer training loop:  46%|####6     | 23232/50000 [14:19:28<13:33:46,  1.82s/it]
Outer training loop:  46%|####6     | 23233/50000 [14:19:30<14:00:10,  1.88s/it]
Outer training loop:  46%|####6     | 23234/50000 [14:19:32<13:34:55,  1.83s/it]
Outer training loop:  46%|####6     | 23235/50000 [14:19:34<13:26:08,  1.81s/it]
Outer training loop:  46%|####6     | 23236/50000 [14:19:36<13:58:29,  1.88s/it]
Outer training loop:  46%|####6     | 23237/50000 [14:19:38<13:44:55,  1.85s/it]
Outer training loop:  46%|####6     | 23238/50000 [14:19:39<13:28:07,  1.81s/it]
Outer training loop:  46%|####6     | 23239/50000 [14:19:41<13:44:53,  1.85s/it]
Outer training loop:  46%|####6     | 23240/50000 [14:19:43<13:27:32,  1.81s/it]
Outer training loop:  46%|####6     | 23241/50000 [14:19:45<13:26:29,  1.81s/it]
Outer training loop:  46%|####6     | 23242/50000 [14:19:47<13:49:22,  1.86s/it]
Outer training loop:  46%|####6     | 23243/50000 [14:19:48<13:23:58,  1.80s/it]
Outer training loop:  46%|####6     | 23244/50000 [14:19:50<13:17:40,  1.79s/it]
Outer training loop:  46%|####6     | 23245/50000 [14:19:52<13:34:27,  1.83s/it]
Outer training loop:  46%|####6     | 23246/50000 [14:19:54<13:32:02,  1.82s/it]
Outer training loop:  46%|####6     | 23247/50000 [14:19:56<13:45:23,  1.85s/it]
Outer training loop:  46%|####6     | 23248/50000 [14:19:58<14:16:34,  1.92s/it]
Outer training loop:  46%|####6     | 23249/50000 [14:20:00<14:01:14,  1.89s/it]
Outer training loop:  46%|####6     | 23250/50000 [14:20:02<13:48:06,  1.86s/it]
Outer training loop:  47%|####6     | 23251/50000 [14:20:03<13:56:23,  1.88s/it]
Outer training loop:  47%|####6     | 23252/50000 [14:20:05<14:03:30,  1.89s/it]
Outer training loop:  47%|####6     | 23253/50000 [14:20:07<13:42:27,  1.84s/it]
Outer training loop:  47%|####6     | 23254/50000 [14:20:09<13:35:20,  1.83s/it]
Outer training loop:  47%|####6     | 23255/50000 [14:20:11<14:04:59,  1.90s/it]
Outer training loop:  47%|####6     | 23256/50000 [14:20:13<13:43:56,  1.85s/it]
Outer training loop:  47%|####6     | 23257/50000 [14:20:15<13:39:31,  1.84s/it]
Outer training loop:  47%|####6     | 23258/50000 [14:20:16<13:29:31,  1.82s/it]
Outer training loop:  47%|####6     | 23259/50000 [14:20:18<13:50:42,  1.86s/it]
Outer training loop:  47%|####6     | 23260/50000 [14:20:20<13:56:50,  1.88s/it]
Outer training loop:  47%|####6     | 23261/50000 [14:20:22<13:38:24,  1.84s/it]
Outer training loop:  47%|####6     | 23262/50000 [14:20:24<13:43:59,  1.85s/it]
Outer training loop:  47%|####6     | 23263/50000 [14:20:26<13:39:15,  1.84s/it]
Outer training loop:  47%|####6     | 23264/50000 [14:20:27<13:40:35,  1.84s/it]
Outer training loop:  47%|####6     | 23265/50000 [14:20:29<13:21:12,  1.80s/it]
Outer training loop:  47%|####6     | 23266/50000 [14:20:31<14:08:09,  1.90s/it]
Outer training loop:  47%|####6     | 23267/50000 [14:20:33<14:11:48,  1.91s/it]
Outer training loop:  47%|####6     | 23268/50000 [14:20:35<14:13:23,  1.92s/it]
Outer training loop:  47%|####6     | 23269/50000 [14:20:37<14:34:22,  1.96s/it]
Outer training loop:  47%|####6     | 23270/50000 [14:20:39<14:19:44,  1.93s/it]
Outer training loop:  47%|####6     | 23271/50000 [14:20:41<14:17:31,  1.92s/it]
Outer training loop:  47%|####6     | 23272/50000 [14:20:43<14:27:15,  1.95s/it]
Outer training loop:  47%|####6     | 23273/50000 [14:20:45<14:12:01,  1.91s/it]
Outer training loop:  47%|####6     | 23274/50000 [14:20:47<13:56:13,  1.88s/it]
Outer training loop:  47%|####6     | 23275/50000 [14:20:49<14:03:08,  1.89s/it]
Outer training loop:  47%|####6     | 23276/50000 [14:20:51<14:14:50,  1.92s/it]
Outer training loop:  47%|####6     | 23277/50000 [14:20:52<13:55:36,  1.88s/it]
Outer training loop:  47%|####6     | 23278/50000 [14:20:54<13:43:47,  1.85s/it]
Outer training loop:  47%|####6     | 23279/50000 [14:20:56<14:07:23,  1.90s/it]
Outer training loop:  47%|####6     | 23280/50000 [14:20:58<14:01:03,  1.89s/it]
Outer training loop:  47%|####6     | 23281/50000 [14:21:00<13:48:27,  1.86s/it]
Outer training loop:  47%|####6     | 23282/50000 [14:21:02<14:17:06,  1.92s/it]
Outer training loop:  47%|####6     | 23283/50000 [14:21:04<14:12:15,  1.91s/it]
Outer training loop:  47%|####6     | 23284/50000 [14:21:06<14:11:28,  1.91s/it]
Outer training loop:  47%|####6     | 23285/50000 [14:21:07<14:01:40,  1.89s/it]
Outer training loop:  47%|####6     | 23286/50000 [14:21:09<14:13:09,  1.92s/it]
Outer training loop:  47%|####6     | 23287/50000 [14:21:11<14:07:23,  1.90s/it]
Outer training loop:  47%|####6     | 23288/50000 [14:21:13<14:15:32,  1.92s/it]
Outer training loop:  47%|####6     | 23289/50000 [14:21:15<14:28:09,  1.95s/it]
Outer training loop:  47%|####6     | 23290/50000 [14:21:17<14:12:46,  1.92s/it]
Outer training loop:  47%|####6     | 23291/50000 [14:21:19<13:47:13,  1.86s/it]
Outer training loop:  47%|####6     | 23292/50000 [14:21:21<13:33:40,  1.83s/it]
Outer training loop:  47%|####6     | 23293/50000 [14:21:23<14:05:57,  1.90s/it]
Outer training loop:  47%|####6     | 23294/50000 [14:21:24<13:42:07,  1.85s/it]
Outer training loop:  47%|####6     | 23295/50000 [14:21:26<13:37:39,  1.84s/it]
Outer training loop:  47%|####6     | 23296/50000 [14:21:28<14:04:16,  1.90s/it]
Outer training loop:  47%|####6     | 23297/50000 [14:21:30<13:51:59,  1.87s/it]
Outer training loop:  47%|####6     | 23298/50000 [14:21:32<13:48:03,  1.86s/it]
Outer training loop:  47%|####6     | 23299/50000 [14:21:34<13:39:33,  1.84s/it]
Outer training loop:  47%|####6     | 23300/50000 [14:21:36<14:30:57,  1.96s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  47%|####6     | 23301/50000 [14:21:43<26:22:53,  3.56s/it]
Outer training loop:  47%|####6     | 23302/50000 [14:21:45<22:17:34,  3.01s/it]
Outer training loop:  47%|####6     | 23303/50000 [14:21:47<19:42:33,  2.66s/it]
Outer training loop:  47%|####6     | 23304/50000 [14:21:49<18:22:49,  2.48s/it]
Outer training loop:  47%|####6     | 23305/50000 [14:21:51<17:12:55,  2.32s/it]
Outer training loop:  47%|####6     | 23306/50000 [14:21:53<15:58:53,  2.16s/it]
Outer training loop:  47%|####6     | 23307/50000 [14:21:55<15:36:56,  2.11s/it]
Outer training loop:  47%|####6     | 23308/50000 [14:21:56<14:47:03,  1.99s/it]
Outer training loop:  47%|####6     | 23309/50000 [14:21:58<14:28:11,  1.95s/it]
Outer training loop:  47%|####6     | 23310/50000 [14:22:00<13:58:33,  1.89s/it]
Outer training loop:  47%|####6     | 23311/50000 [14:22:02<14:25:13,  1.95s/it]
Outer training loop:  47%|####6     | 23312/50000 [14:22:04<14:14:02,  1.92s/it]
Outer training loop:  47%|####6     | 23313/50000 [14:22:06<14:15:32,  1.92s/it]
Outer training loop:  47%|####6     | 23314/50000 [14:22:08<14:17:25,  1.93s/it]
Outer training loop:  47%|####6     | 23315/50000 [14:22:10<14:05:28,  1.90s/it]
Outer training loop:  47%|####6     | 23316/50000 [14:22:11<14:09:01,  1.91s/it]
Outer training loop:  47%|####6     | 23317/50000 [14:22:13<14:24:42,  1.94s/it]
Outer training loop:  47%|####6     | 23318/50000 [14:22:15<13:59:50,  1.89s/it]
Outer training loop:  47%|####6     | 23319/50000 [14:22:17<13:43:11,  1.85s/it]
Outer training loop:  47%|####6     | 23320/50000 [14:22:19<13:30:56,  1.82s/it]
Outer training loop:  47%|####6     | 23321/50000 [14:22:21<13:45:02,  1.86s/it]
Outer training loop:  47%|####6     | 23322/50000 [14:22:23<13:41:44,  1.85s/it]
Outer training loop:  47%|####6     | 23323/50000 [14:22:24<13:23:19,  1.81s/it]
Outer training loop:  47%|####6     | 23324/50000 [14:22:26<13:55:55,  1.88s/it]
Outer training loop:  47%|####6     | 23325/50000 [14:22:28<14:05:07,  1.90s/it]
Outer training loop:  47%|####6     | 23326/50000 [14:22:30<13:46:31,  1.86s/it]
Outer training loop:  47%|####6     | 23327/50000 [14:22:32<13:18:16,  1.80s/it]
Outer training loop:  47%|####6     | 23328/50000 [14:22:34<13:55:31,  1.88s/it]
Outer training loop:  47%|####6     | 23329/50000 [14:22:36<13:40:49,  1.85s/it]
Outer training loop:  47%|####6     | 23330/50000 [14:22:37<13:40:39,  1.85s/it]
Outer training loop:  47%|####6     | 23331/50000 [14:22:39<13:47:26,  1.86s/it]
Outer training loop:  47%|####6     | 23332/50000 [14:22:41<14:00:22,  1.89s/it]
Outer training loop:  47%|####6     | 23333/50000 [14:22:43<14:00:56,  1.89s/it]
Outer training loop:  47%|####6     | 23334/50000 [14:22:45<14:07:41,  1.91s/it]
Outer training loop:  47%|####6     | 23335/50000 [14:22:47<14:06:08,  1.90s/it]
Outer training loop:  47%|####6     | 23336/50000 [14:22:49<14:22:55,  1.94s/it]
Outer training loop:  47%|####6     | 23337/50000 [14:22:51<14:32:05,  1.96s/it]
Outer training loop:  47%|####6     | 23338/50000 [14:22:53<14:26:28,  1.95s/it]
Outer training loop:  47%|####6     | 23339/50000 [14:22:55<13:57:43,  1.89s/it]
Outer training loop:  47%|####6     | 23340/50000 [14:22:56<13:41:53,  1.85s/it]
Outer training loop:  47%|####6     | 23341/50000 [14:22:59<14:14:52,  1.92s/it]
Outer training loop:  47%|####6     | 23342/50000 [14:23:00<14:07:33,  1.91s/it]
Outer training loop:  47%|####6     | 23343/50000 [14:23:02<14:05:29,  1.90s/it]
Outer training loop:  47%|####6     | 23344/50000 [14:23:04<14:41:36,  1.98s/it]
Outer training loop:  47%|####6     | 23345/50000 [14:23:06<14:24:44,  1.95s/it]
Outer training loop:  47%|####6     | 23346/50000 [14:23:08<14:01:00,  1.89s/it]
Outer training loop:  47%|####6     | 23347/50000 [14:23:10<13:55:12,  1.88s/it]
Outer training loop:  47%|####6     | 23348/50000 [14:23:12<14:09:17,  1.91s/it]
Outer training loop:  47%|####6     | 23349/50000 [14:23:14<14:09:25,  1.91s/it]
Outer training loop:  47%|####6     | 23350/50000 [14:23:16<14:06:03,  1.90s/it]
Outer training loop:  47%|####6     | 23351/50000 [14:23:18<13:55:02,  1.88s/it]
Outer training loop:  47%|####6     | 23352/50000 [14:23:19<13:36:20,  1.84s/it]
Outer training loop:  47%|####6     | 23353/50000 [14:23:21<13:26:23,  1.82s/it]
Outer training loop:  47%|####6     | 23354/50000 [14:23:23<13:46:52,  1.86s/it]
Outer training loop:  47%|####6     | 23355/50000 [14:23:25<13:55:55,  1.88s/it]
Outer training loop:  47%|####6     | 23356/50000 [14:23:27<13:37:11,  1.84s/it]
Outer training loop:  47%|####6     | 23357/50000 [14:23:29<14:01:39,  1.90s/it]
Outer training loop:  47%|####6     | 23358/50000 [14:23:31<13:56:49,  1.88s/it]
Outer training loop:  47%|####6     | 23359/50000 [14:23:32<13:53:53,  1.88s/it]
Outer training loop:  47%|####6     | 23360/50000 [14:23:34<14:18:34,  1.93s/it]
Outer training loop:  47%|####6     | 23361/50000 [14:23:36<14:24:00,  1.95s/it]
Outer training loop:  47%|####6     | 23362/50000 [14:23:38<14:15:07,  1.93s/it]
Outer training loop:  47%|####6     | 23363/50000 [14:23:40<14:26:14,  1.95s/it]
Outer training loop:  47%|####6     | 23364/50000 [14:23:42<14:48:52,  2.00s/it]
Outer training loop:  47%|####6     | 23365/50000 [14:23:44<14:15:29,  1.93s/it]
Outer training loop:  47%|####6     | 23366/50000 [14:23:46<13:48:59,  1.87s/it]
Outer training loop:  47%|####6     | 23367/50000 [14:23:48<14:15:33,  1.93s/it]
Outer training loop:  47%|####6     | 23368/50000 [14:23:50<14:10:21,  1.92s/it]
Outer training loop:  47%|####6     | 23369/50000 [14:23:52<13:57:10,  1.89s/it]
Outer training loop:  47%|####6     | 23370/50000 [14:23:53<13:33:39,  1.83s/it]
Outer training loop:  47%|####6     | 23371/50000 [14:23:56<14:09:02,  1.91s/it]
Outer training loop:  47%|####6     | 23372/50000 [14:23:57<13:52:04,  1.87s/it]
Outer training loop:  47%|####6     | 23373/50000 [14:23:59<13:55:02,  1.88s/it]
Outer training loop:  47%|####6     | 23374/50000 [14:24:01<14:29:23,  1.96s/it]
Outer training loop:  47%|####6     | 23375/50000 [14:24:03<14:04:07,  1.90s/it]
Outer training loop:  47%|####6     | 23376/50000 [14:24:05<14:03:34,  1.90s/it]
Outer training loop:  47%|####6     | 23377/50000 [14:24:07<14:36:23,  1.98s/it]
Outer training loop:  47%|####6     | 23378/50000 [14:24:09<14:07:26,  1.91s/it]
Outer training loop:  47%|####6     | 23379/50000 [14:24:11<13:42:46,  1.85s/it]
Outer training loop:  47%|####6     | 23380/50000 [14:24:13<13:53:31,  1.88s/it]
Outer training loop:  47%|####6     | 23381/50000 [14:24:15<14:00:16,  1.89s/it]
Outer training loop:  47%|####6     | 23382/50000 [14:24:16<13:58:52,  1.89s/it]
Outer training loop:  47%|####6     | 23383/50000 [14:24:18<13:56:31,  1.89s/it]
Outer training loop:  47%|####6     | 23384/50000 [14:24:20<14:06:33,  1.91s/it]
Outer training loop:  47%|####6     | 23385/50000 [14:24:22<13:37:16,  1.84s/it]
Outer training loop:  47%|####6     | 23386/50000 [14:24:24<13:29:58,  1.83s/it]
Outer training loop:  47%|####6     | 23387/50000 [14:24:26<14:07:00,  1.91s/it]
Outer training loop:  47%|####6     | 23388/50000 [14:24:28<14:01:56,  1.90s/it]
Outer training loop:  47%|####6     | 23389/50000 [14:24:29<13:37:36,  1.84s/it]
Outer training loop:  47%|####6     | 23390/50000 [14:24:31<13:40:59,  1.85s/it]
Outer training loop:  47%|####6     | 23391/50000 [14:24:33<13:50:44,  1.87s/it]
Outer training loop:  47%|####6     | 23392/50000 [14:24:35<13:51:19,  1.87s/it]
Outer training loop:  47%|####6     | 23393/50000 [14:24:37<13:35:27,  1.84s/it]
Outer training loop:  47%|####6     | 23394/50000 [14:24:39<13:54:29,  1.88s/it]
Outer training loop:  47%|####6     | 23395/50000 [14:24:41<13:47:31,  1.87s/it]
Outer training loop:  47%|####6     | 23396/50000 [14:24:43<14:07:49,  1.91s/it]
Outer training loop:  47%|####6     | 23397/50000 [14:24:44<13:41:30,  1.85s/it]
Outer training loop:  47%|####6     | 23398/50000 [14:24:46<14:00:47,  1.90s/it]
Outer training loop:  47%|####6     | 23399/50000 [14:24:48<13:43:14,  1.86s/it]
Outer training loop:  47%|####6     | 23400/50000 [14:24:50<13:36:52,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.69s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  47%|####6     | 23401/50000 [14:24:57<25:50:39,  3.50s/it]
Outer training loop:  47%|####6     | 23402/50000 [14:24:59<22:25:11,  3.03s/it]
Outer training loop:  47%|####6     | 23403/50000 [14:25:01<19:36:07,  2.65s/it]
Outer training loop:  47%|####6     | 23404/50000 [14:25:03<17:40:52,  2.39s/it]
Outer training loop:  47%|####6     | 23405/50000 [14:25:05<17:24:34,  2.36s/it]
Outer training loop:  47%|####6     | 23406/50000 [14:25:07<16:16:56,  2.20s/it]
Outer training loop:  47%|####6     | 23407/50000 [14:25:09<15:18:36,  2.07s/it]
Outer training loop:  47%|####6     | 23408/50000 [14:25:11<14:50:41,  2.01s/it]
Outer training loop:  47%|####6     | 23409/50000 [14:25:13<14:59:14,  2.03s/it]
Outer training loop:  47%|####6     | 23410/50000 [14:25:15<14:57:52,  2.03s/it]
Outer training loop:  47%|####6     | 23411/50000 [14:25:16<14:17:14,  1.93s/it]
Outer training loop:  47%|####6     | 23412/50000 [14:25:18<14:28:22,  1.96s/it]
Outer training loop:  47%|####6     | 23413/50000 [14:25:20<14:16:09,  1.93s/it]
Outer training loop:  47%|####6     | 23414/50000 [14:25:22<14:18:03,  1.94s/it]
Outer training loop:  47%|####6     | 23415/50000 [14:25:24<14:18:04,  1.94s/it]
Outer training loop:  47%|####6     | 23416/50000 [14:25:26<14:09:56,  1.92s/it]
Outer training loop:  47%|####6     | 23417/50000 [14:25:28<14:01:09,  1.90s/it]
Outer training loop:  47%|####6     | 23418/50000 [14:25:30<14:06:43,  1.91s/it]
Outer training loop:  47%|####6     | 23419/50000 [14:25:32<13:46:18,  1.87s/it]
Outer training loop:  47%|####6     | 23420/50000 [14:25:34<14:01:23,  1.90s/it]
Outer training loop:  47%|####6     | 23421/50000 [14:25:35<13:56:58,  1.89s/it]
Outer training loop:  47%|####6     | 23422/50000 [14:25:37<14:17:44,  1.94s/it]
Outer training loop:  47%|####6     | 23423/50000 [14:25:39<14:13:30,  1.93s/it]
Outer training loop:  47%|####6     | 23424/50000 [14:25:41<14:06:44,  1.91s/it]
Outer training loop:  47%|####6     | 23425/50000 [14:25:43<14:26:26,  1.96s/it]
Outer training loop:  47%|####6     | 23426/50000 [14:25:45<14:06:57,  1.91s/it]
Outer training loop:  47%|####6     | 23427/50000 [14:25:47<13:56:28,  1.89s/it]
Outer training loop:  47%|####6     | 23428/50000 [14:25:49<14:17:53,  1.94s/it]
Outer training loop:  47%|####6     | 23429/50000 [14:25:51<14:18:22,  1.94s/it]
Outer training loop:  47%|####6     | 23430/50000 [14:25:53<13:55:55,  1.89s/it]
Outer training loop:  47%|####6     | 23431/50000 [14:25:55<14:02:15,  1.90s/it]
Outer training loop:  47%|####6     | 23432/50000 [14:25:57<14:27:41,  1.96s/it]
Outer training loop:  47%|####6     | 23433/50000 [14:25:59<14:11:57,  1.92s/it]
Outer training loop:  47%|####6     | 23434/50000 [14:26:00<13:41:10,  1.85s/it]
Outer training loop:  47%|####6     | 23435/50000 [14:26:02<14:03:05,  1.90s/it]
Outer training loop:  47%|####6     | 23436/50000 [14:26:04<13:56:58,  1.89s/it]
Outer training loop:  47%|####6     | 23437/50000 [14:26:06<13:52:55,  1.88s/it]
Outer training loop:  47%|####6     | 23438/50000 [14:26:08<13:43:54,  1.86s/it]
Outer training loop:  47%|####6     | 23439/50000 [14:26:10<14:03:56,  1.91s/it]
Outer training loop:  47%|####6     | 23440/50000 [14:26:12<14:02:56,  1.90s/it]
Outer training loop:  47%|####6     | 23441/50000 [14:26:14<13:52:44,  1.88s/it]
Outer training loop:  47%|####6     | 23442/50000 [14:26:16<14:11:52,  1.92s/it]
Outer training loop:  47%|####6     | 23443/50000 [14:26:17<13:58:48,  1.90s/it]
Outer training loop:  47%|####6     | 23444/50000 [14:26:19<14:03:38,  1.91s/it]
Outer training loop:  47%|####6     | 23445/50000 [14:26:21<14:21:14,  1.95s/it]
Outer training loop:  47%|####6     | 23446/50000 [14:26:23<14:01:32,  1.90s/it]
Outer training loop:  47%|####6     | 23447/50000 [14:26:25<13:58:38,  1.90s/it]
Outer training loop:  47%|####6     | 23448/50000 [14:26:27<14:09:03,  1.92s/it]
Outer training loop:  47%|####6     | 23449/50000 [14:26:29<13:53:31,  1.88s/it]
Outer training loop:  47%|####6     | 23450/50000 [14:26:31<13:34:26,  1.84s/it]
Outer training loop:  47%|####6     | 23451/50000 [14:26:32<13:28:34,  1.83s/it]
Outer training loop:  47%|####6     | 23452/50000 [14:26:34<13:55:13,  1.89s/it]
Outer training loop:  47%|####6     | 23453/50000 [14:26:36<13:51:20,  1.88s/it]
Outer training loop:  47%|####6     | 23454/50000 [14:26:38<13:43:44,  1.86s/it]
Outer training loop:  47%|####6     | 23455/50000 [14:26:40<14:11:14,  1.92s/it]
Outer training loop:  47%|####6     | 23456/50000 [14:26:42<13:46:05,  1.87s/it]
Outer training loop:  47%|####6     | 23457/50000 [14:26:44<13:45:58,  1.87s/it]
Outer training loop:  47%|####6     | 23458/50000 [14:26:46<13:36:19,  1.85s/it]
Outer training loop:  47%|####6     | 23459/50000 [14:26:48<14:02:40,  1.90s/it]
Outer training loop:  47%|####6     | 23460/50000 [14:26:50<14:21:57,  1.95s/it]
Outer training loop:  47%|####6     | 23461/50000 [14:26:51<13:59:05,  1.90s/it]
Outer training loop:  47%|####6     | 23462/50000 [14:26:53<13:39:48,  1.85s/it]
Outer training loop:  47%|####6     | 23463/50000 [14:26:55<13:56:15,  1.89s/it]
Outer training loop:  47%|####6     | 23464/50000 [14:26:57<13:30:51,  1.83s/it]
Outer training loop:  47%|####6     | 23465/50000 [14:26:59<13:34:49,  1.84s/it]
Outer training loop:  47%|####6     | 23466/50000 [14:27:00<13:23:23,  1.82s/it]
Outer training loop:  47%|####6     | 23467/50000 [14:27:03<13:55:28,  1.89s/it]
Outer training loop:  47%|####6     | 23468/50000 [14:27:04<13:46:51,  1.87s/it]
Outer training loop:  47%|####6     | 23469/50000 [14:27:06<13:49:40,  1.88s/it]
Outer training loop:  47%|####6     | 23470/50000 [14:27:08<14:01:10,  1.90s/it]
Outer training loop:  47%|####6     | 23471/50000 [14:27:10<13:44:30,  1.86s/it]
Outer training loop:  47%|####6     | 23472/50000 [14:27:12<13:29:30,  1.83s/it]
Outer training loop:  47%|####6     | 23473/50000 [14:27:14<13:58:42,  1.90s/it]
Outer training loop:  47%|####6     | 23474/50000 [14:27:16<13:37:34,  1.85s/it]
Outer training loop:  47%|####6     | 23475/50000 [14:27:17<13:32:27,  1.84s/it]
Outer training loop:  47%|####6     | 23476/50000 [14:27:19<13:54:26,  1.89s/it]
Outer training loop:  47%|####6     | 23477/50000 [14:27:21<13:58:27,  1.90s/it]
Outer training loop:  47%|####6     | 23478/50000 [14:27:23<13:44:41,  1.87s/it]
Outer training loop:  47%|####6     | 23479/50000 [14:27:25<13:41:43,  1.86s/it]
Outer training loop:  47%|####6     | 23480/50000 [14:27:27<13:58:09,  1.90s/it]
Outer training loop:  47%|####6     | 23481/50000 [14:27:29<13:56:47,  1.89s/it]
Outer training loop:  47%|####6     | 23482/50000 [14:27:31<13:50:50,  1.88s/it]
Outer training loop:  47%|####6     | 23483/50000 [14:27:32<13:48:11,  1.87s/it]
Outer training loop:  47%|####6     | 23484/50000 [14:27:34<14:07:05,  1.92s/it]
Outer training loop:  47%|####6     | 23485/50000 [14:27:36<13:49:12,  1.88s/it]
Outer training loop:  47%|####6     | 23486/50000 [14:27:38<13:31:39,  1.84s/it]
Outer training loop:  47%|####6     | 23487/50000 [14:27:40<14:09:53,  1.92s/it]
Outer training loop:  47%|####6     | 23488/50000 [14:27:42<13:57:43,  1.90s/it]
Outer training loop:  47%|####6     | 23489/50000 [14:27:44<13:45:24,  1.87s/it]
Outer training loop:  47%|####6     | 23490/50000 [14:27:46<13:35:09,  1.84s/it]
Outer training loop:  47%|####6     | 23491/50000 [14:27:48<13:59:56,  1.90s/it]
Outer training loop:  47%|####6     | 23492/50000 [14:27:49<13:46:39,  1.87s/it]
Outer training loop:  47%|####6     | 23493/50000 [14:27:51<13:45:50,  1.87s/it]
Outer training loop:  47%|####6     | 23494/50000 [14:27:53<13:54:42,  1.89s/it]
Outer training loop:  47%|####6     | 23495/50000 [14:27:55<13:49:02,  1.88s/it]
Outer training loop:  47%|####6     | 23496/50000 [14:27:57<13:31:35,  1.84s/it]
Outer training loop:  47%|####6     | 23497/50000 [14:27:59<13:57:27,  1.90s/it]
Outer training loop:  47%|####6     | 23498/50000 [14:28:01<13:47:51,  1.87s/it]
Outer training loop:  47%|####6     | 23499/50000 [14:28:02<13:43:39,  1.86s/it]
Outer training loop:  47%|####6     | 23500/50000 [14:28:04<13:41:54,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:27,  1.80s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:24,  1.81s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:26,  1.86s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.86s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:25,  1.94s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.95s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.90s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:22,  1.98s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:19,  1.97s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:15,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.88s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:06,  1.85s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.90s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:05,  1.99s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.88s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:53,  1.96s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.92s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:36,  1.83s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:41,  1.95s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.90s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:31,  1.99s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:25,  1.94s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:21,  1.90s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:20,  2.01s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:16,  1.95s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:09,  1.87s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:07,  1.87s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.86s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:03,  1.93s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:59,  1.93s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.89s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.88s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:52,  1.93s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:50,  1.96s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:51,  2.06s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:47,  1.99s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:44,  1.95s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:42,  1.92s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:40,  1.91s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:37,  1.89s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:35,  1.98s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:06<00:32,  1.93s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:30,  1.90s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.88s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:25,  1.84s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.89s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.87s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:18,  2.00s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:15,  1.99s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:13,  1.93s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.89s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.91s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<03:04,  2.25s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:48,  2.08s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:39,  2.00s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:39,  2.02s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.94s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.90s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:25,  1.94s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:19,  1.89s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:14,  1.85s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:13,  1.86s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:15,  1.91s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.88s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:08,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:04,  1.83s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.93s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:01,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:01,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:59,  1.93s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:52,  1.97s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.88s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:43,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.85s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:34,  1.82s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:38,  1.92s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.86s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:15,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:13,  1.84s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:16,  1.97s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:14,  1.97s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:10,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:07,  1.87s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.84s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:02,  1.96s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.92s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.81s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:50,  1.80s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.84s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:47,  1.83s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:49,  1.98s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:46,  1.92s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:43,  1.90s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.82s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.80s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:34,  1.80s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:32,  1.78s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:32,  1.91s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:30,  1.92s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:28,  1.89s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:21,  1.82s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:19,  1.80s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.96s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.91s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:08,  1.80s/it]

Evaluating policy:  95%|#########5| 79/83 [02:27<00:07,  1.83s/it]

Evaluating policy:  96%|#########6| 80/83 [02:29<00:05,  1.85s/it]

Evaluating policy:  98%|#########7| 81/83 [02:31<00:03,  1.81s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.85s/it]

Outer training loop:  47%|####7     | 23501/50000 [14:33:25<718:43:46, 97.64s/it]
Outer training loop:  47%|####7     | 23502/50000 [14:33:27<507:15:01, 68.91s/it]
Outer training loop:  47%|####7     | 23503/50000 [14:33:30<360:13:06, 48.94s/it]
Outer training loop:  47%|####7     | 23504/50000 [14:33:32<256:18:04, 34.82s/it]
Outer training loop:  47%|####7     | 23505/50000 [14:33:33<183:17:49, 24.91s/it]
Outer training loop:  47%|####7     | 23506/50000 [14:33:35<132:52:07, 18.05s/it]
Outer training loop:  47%|####7     | 23507/50000 [14:33:37<97:06:26, 13.20s/it] 
Outer training loop:  47%|####7     | 23508/50000 [14:33:39<71:44:33,  9.75s/it]
Outer training loop:  47%|####7     | 23509/50000 [14:33:41<54:36:53,  7.42s/it]
Outer training loop:  47%|####7     | 23510/50000 [14:33:43<42:31:48,  5.78s/it]
Outer training loop:  47%|####7     | 23511/50000 [14:33:45<33:42:56,  4.58s/it]
Outer training loop:  47%|####7     | 23512/50000 [14:33:47<27:34:55,  3.75s/it]
Outer training loop:  47%|####7     | 23513/50000 [14:33:49<23:46:39,  3.23s/it]
Outer training loop:  47%|####7     | 23514/50000 [14:33:51<21:00:09,  2.85s/it]
Outer training loop:  47%|####7     | 23515/50000 [14:33:52<18:43:03,  2.54s/it]
Outer training loop:  47%|####7     | 23516/50000 [14:33:54<17:39:00,  2.40s/it]
Outer training loop:  47%|####7     | 23517/50000 [14:33:56<16:28:59,  2.24s/it]
Outer training loop:  47%|####7     | 23518/50000 [14:33:58<15:46:39,  2.14s/it]
Outer training loop:  47%|####7     | 23519/50000 [14:34:00<15:02:34,  2.05s/it]
Outer training loop:  47%|####7     | 23520/50000 [14:34:02<15:04:20,  2.05s/it]
Outer training loop:  47%|####7     | 23521/50000 [14:34:04<14:28:35,  1.97s/it]
Outer training loop:  47%|####7     | 23522/50000 [14:34:06<14:06:49,  1.92s/it]
Outer training loop:  47%|####7     | 23523/50000 [14:34:08<14:22:16,  1.95s/it]
Outer training loop:  47%|####7     | 23524/50000 [14:34:10<14:14:37,  1.94s/it]
Outer training loop:  47%|####7     | 23525/50000 [14:34:11<14:11:42,  1.93s/it]
Outer training loop:  47%|####7     | 23526/50000 [14:34:14<14:44:33,  2.00s/it]
Outer training loop:  47%|####7     | 23527/50000 [14:34:15<14:19:05,  1.95s/it]
Outer training loop:  47%|####7     | 23528/50000 [14:34:17<13:42:08,  1.86s/it]
Outer training loop:  47%|####7     | 23529/50000 [14:34:19<13:27:05,  1.83s/it]
Outer training loop:  47%|####7     | 23530/50000 [14:34:21<13:45:31,  1.87s/it]
Outer training loop:  47%|####7     | 23531/50000 [14:34:23<13:30:01,  1.84s/it]
Outer training loop:  47%|####7     | 23532/50000 [14:34:25<13:37:51,  1.85s/it]
Outer training loop:  47%|####7     | 23533/50000 [14:34:26<13:54:25,  1.89s/it]
Outer training loop:  47%|####7     | 23534/50000 [14:34:28<13:47:10,  1.88s/it]
Outer training loop:  47%|####7     | 23535/50000 [14:34:30<13:38:08,  1.85s/it]
Outer training loop:  47%|####7     | 23536/50000 [14:34:32<13:57:52,  1.90s/it]
Outer training loop:  47%|####7     | 23537/50000 [14:34:34<13:58:45,  1.90s/it]
Outer training loop:  47%|####7     | 23538/50000 [14:34:36<13:51:53,  1.89s/it]
Outer training loop:  47%|####7     | 23539/50000 [14:34:38<14:05:25,  1.92s/it]
Outer training loop:  47%|####7     | 23540/50000 [14:34:40<13:44:31,  1.87s/it]
Outer training loop:  47%|####7     | 23541/50000 [14:34:42<14:00:59,  1.91s/it]
Outer training loop:  47%|####7     | 23542/50000 [14:34:43<13:52:59,  1.89s/it]
Outer training loop:  47%|####7     | 23543/50000 [14:34:46<14:17:06,  1.94s/it]
Outer training loop:  47%|####7     | 23544/50000 [14:34:47<13:53:21,  1.89s/it]
Outer training loop:  47%|####7     | 23545/50000 [14:34:49<13:37:38,  1.85s/it]
Outer training loop:  47%|####7     | 23546/50000 [14:34:51<13:56:45,  1.90s/it]
Outer training loop:  47%|####7     | 23547/50000 [14:34:53<13:35:41,  1.85s/it]
Outer training loop:  47%|####7     | 23548/50000 [14:34:55<13:20:36,  1.82s/it]
Outer training loop:  47%|####7     | 23549/50000 [14:34:56<13:30:12,  1.84s/it]
Outer training loop:  47%|####7     | 23550/50000 [14:34:58<13:22:54,  1.82s/it]
Outer training loop:  47%|####7     | 23551/50000 [14:35:00<13:27:02,  1.83s/it]
Outer training loop:  47%|####7     | 23552/50000 [14:35:02<13:11:34,  1.80s/it]
Outer training loop:  47%|####7     | 23553/50000 [14:35:04<13:45:55,  1.87s/it]
Outer training loop:  47%|####7     | 23554/50000 [14:35:06<13:49:16,  1.88s/it]
Outer training loop:  47%|####7     | 23555/50000 [14:35:08<13:39:34,  1.86s/it]
Outer training loop:  47%|####7     | 23556/50000 [14:35:10<14:00:43,  1.91s/it]
Outer training loop:  47%|####7     | 23557/50000 [14:35:11<13:46:41,  1.88s/it]
Outer training loop:  47%|####7     | 23558/50000 [14:35:13<13:51:50,  1.89s/it]
Outer training loop:  47%|####7     | 23559/50000 [14:35:15<13:38:56,  1.86s/it]
Outer training loop:  47%|####7     | 23560/50000 [14:35:17<13:54:34,  1.89s/it]
Outer training loop:  47%|####7     | 23561/50000 [14:35:19<13:48:36,  1.88s/it]
Outer training loop:  47%|####7     | 23562/50000 [14:35:21<13:57:01,  1.90s/it]
Outer training loop:  47%|####7     | 23563/50000 [14:35:23<14:12:36,  1.94s/it]
Outer training loop:  47%|####7     | 23564/50000 [14:35:25<13:59:35,  1.91s/it]
Outer training loop:  47%|####7     | 23565/50000 [14:35:27<14:07:29,  1.92s/it]
Outer training loop:  47%|####7     | 23566/50000 [14:35:29<14:05:13,  1.92s/it]
Outer training loop:  47%|####7     | 23567/50000 [14:35:31<14:26:14,  1.97s/it]
Outer training loop:  47%|####7     | 23568/50000 [14:35:33<14:11:19,  1.93s/it]
Outer training loop:  47%|####7     | 23569/50000 [14:35:34<13:57:32,  1.90s/it]
Outer training loop:  47%|####7     | 23570/50000 [14:35:36<14:26:13,  1.97s/it]
Outer training loop:  47%|####7     | 23571/50000 [14:35:38<14:11:11,  1.93s/it]
Outer training loop:  47%|####7     | 23572/50000 [14:35:40<14:00:09,  1.91s/it]
Outer training loop:  47%|####7     | 23573/50000 [14:35:42<13:43:40,  1.87s/it]
Outer training loop:  47%|####7     | 23574/50000 [14:35:44<14:05:39,  1.92s/it]
Outer training loop:  47%|####7     | 23575/50000 [14:35:46<13:48:38,  1.88s/it]
Outer training loop:  47%|####7     | 23576/50000 [14:35:48<13:27:30,  1.83s/it]
Outer training loop:  47%|####7     | 23577/50000 [14:35:49<13:46:58,  1.88s/it]
Outer training loop:  47%|####7     | 23578/50000 [14:35:51<13:23:49,  1.83s/it]
Outer training loop:  47%|####7     | 23579/50000 [14:35:53<13:32:24,  1.84s/it]
Outer training loop:  47%|####7     | 23580/50000 [14:35:55<14:19:59,  1.95s/it]
Outer training loop:  47%|####7     | 23581/50000 [14:35:57<14:16:25,  1.95s/it]
Outer training loop:  47%|####7     | 23582/50000 [14:35:59<14:21:34,  1.96s/it]
Outer training loop:  47%|####7     | 23583/50000 [14:36:01<14:24:00,  1.96s/it]
Outer training loop:  47%|####7     | 23584/50000 [14:36:03<14:02:58,  1.91s/it]
Outer training loop:  47%|####7     | 23585/50000 [14:36:05<14:04:09,  1.92s/it]
Outer training loop:  47%|####7     | 23586/50000 [14:36:07<13:46:02,  1.88s/it]
Outer training loop:  47%|####7     | 23587/50000 [14:36:09<13:46:37,  1.88s/it]
Outer training loop:  47%|####7     | 23588/50000 [14:36:10<13:45:54,  1.88s/it]
Outer training loop:  47%|####7     | 23589/50000 [14:36:12<13:48:26,  1.88s/it]
Outer training loop:  47%|####7     | 23590/50000 [14:36:14<14:11:40,  1.93s/it]
Outer training loop:  47%|####7     | 23591/50000 [14:36:16<14:12:38,  1.94s/it]
Outer training loop:  47%|####7     | 23592/50000 [14:36:18<13:43:43,  1.87s/it]
Outer training loop:  47%|####7     | 23593/50000 [14:36:20<13:40:17,  1.86s/it]
Outer training loop:  47%|####7     | 23594/50000 [14:36:22<13:53:26,  1.89s/it]
Outer training loop:  47%|####7     | 23595/50000 [14:36:24<13:36:29,  1.86s/it]
Outer training loop:  47%|####7     | 23596/50000 [14:36:25<13:38:04,  1.86s/it]
Outer training loop:  47%|####7     | 23597/50000 [14:36:28<14:02:45,  1.92s/it]
Outer training loop:  47%|####7     | 23598/50000 [14:36:29<13:44:31,  1.87s/it]
Outer training loop:  47%|####7     | 23599/50000 [14:36:31<13:31:33,  1.84s/it]
Outer training loop:  47%|####7     | 23600/50000 [14:36:33<13:53:38,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  47%|####7     | 23601/50000 [14:36:41<26:20:24,  3.59s/it]
Outer training loop:  47%|####7     | 23602/50000 [14:36:43<22:34:11,  3.08s/it]
Outer training loop:  47%|####7     | 23603/50000 [14:36:44<19:45:54,  2.70s/it]
Outer training loop:  47%|####7     | 23604/50000 [14:36:46<18:14:24,  2.49s/it]
Outer training loop:  47%|####7     | 23605/50000 [14:36:48<16:37:05,  2.27s/it]
Outer training loop:  47%|####7     | 23606/50000 [14:36:50<16:02:42,  2.19s/it]
Outer training loop:  47%|####7     | 23607/50000 [14:36:52<15:44:43,  2.15s/it]
Outer training loop:  47%|####7     | 23608/50000 [14:36:54<14:51:11,  2.03s/it]
Outer training loop:  47%|####7     | 23609/50000 [14:36:56<14:13:05,  1.94s/it]
Outer training loop:  47%|####7     | 23610/50000 [14:36:58<14:09:11,  1.93s/it]
Outer training loop:  47%|####7     | 23611/50000 [14:36:59<13:50:17,  1.89s/it]
Outer training loop:  47%|####7     | 23612/50000 [14:37:01<13:37:28,  1.86s/it]
Outer training loop:  47%|####7     | 23613/50000 [14:37:03<13:31:46,  1.85s/it]
Outer training loop:  47%|####7     | 23614/50000 [14:37:05<14:11:57,  1.94s/it]
Outer training loop:  47%|####7     | 23615/50000 [14:37:07<14:05:37,  1.92s/it]
Outer training loop:  47%|####7     | 23616/50000 [14:37:09<13:50:13,  1.89s/it]
Outer training loop:  47%|####7     | 23617/50000 [14:37:11<14:08:57,  1.93s/it]
Outer training loop:  47%|####7     | 23618/50000 [14:37:13<14:07:52,  1.93s/it]
Outer training loop:  47%|####7     | 23619/50000 [14:37:15<13:53:57,  1.90s/it]
Outer training loop:  47%|####7     | 23620/50000 [14:37:17<14:10:40,  1.93s/it]
Outer training loop:  47%|####7     | 23621/50000 [14:37:18<13:48:25,  1.88s/it]
Outer training loop:  47%|####7     | 23622/50000 [14:37:20<13:50:40,  1.89s/it]
Outer training loop:  47%|####7     | 23623/50000 [14:37:22<13:32:13,  1.85s/it]
Outer training loop:  47%|####7     | 23624/50000 [14:37:24<13:37:20,  1.86s/it]
Outer training loop:  47%|####7     | 23625/50000 [14:37:26<13:35:47,  1.86s/it]
Outer training loop:  47%|####7     | 23626/50000 [14:37:28<13:49:15,  1.89s/it]
Outer training loop:  47%|####7     | 23627/50000 [14:37:30<14:03:31,  1.92s/it]
Outer training loop:  47%|####7     | 23628/50000 [14:37:32<14:01:00,  1.91s/it]
Outer training loop:  47%|####7     | 23629/50000 [14:37:34<14:09:13,  1.93s/it]
Outer training loop:  47%|####7     | 23630/50000 [14:37:36<14:19:18,  1.96s/it]
Outer training loop:  47%|####7     | 23631/50000 [14:37:37<13:52:45,  1.89s/it]
Outer training loop:  47%|####7     | 23632/50000 [14:37:39<13:24:50,  1.83s/it]
Outer training loop:  47%|####7     | 23633/50000 [14:37:41<13:33:36,  1.85s/it]
Outer training loop:  47%|####7     | 23634/50000 [14:37:43<14:06:54,  1.93s/it]
Outer training loop:  47%|####7     | 23635/50000 [14:37:45<13:48:39,  1.89s/it]
Outer training loop:  47%|####7     | 23636/50000 [14:37:47<13:35:20,  1.86s/it]
Outer training loop:  47%|####7     | 23637/50000 [14:37:48<13:23:56,  1.83s/it]
Outer training loop:  47%|####7     | 23638/50000 [14:37:50<13:53:35,  1.90s/it]
Outer training loop:  47%|####7     | 23639/50000 [14:37:52<13:39:12,  1.86s/it]
Outer training loop:  47%|####7     | 23640/50000 [14:37:54<13:33:00,  1.85s/it]
Outer training loop:  47%|####7     | 23641/50000 [14:37:56<13:58:12,  1.91s/it]
Outer training loop:  47%|####7     | 23642/50000 [14:37:58<14:04:00,  1.92s/it]
Outer training loop:  47%|####7     | 23643/50000 [14:38:00<13:58:36,  1.91s/it]
Outer training loop:  47%|####7     | 23644/50000 [14:38:02<13:50:26,  1.89s/it]
Outer training loop:  47%|####7     | 23645/50000 [14:38:04<14:13:53,  1.94s/it]
Outer training loop:  47%|####7     | 23646/50000 [14:38:06<13:56:06,  1.90s/it]
Outer training loop:  47%|####7     | 23647/50000 [14:38:08<14:01:18,  1.92s/it]
Outer training loop:  47%|####7     | 23648/50000 [14:38:10<14:08:33,  1.93s/it]
Outer training loop:  47%|####7     | 23649/50000 [14:38:11<13:51:12,  1.89s/it]
Outer training loop:  47%|####7     | 23650/50000 [14:38:13<13:43:47,  1.88s/it]
Outer training loop:  47%|####7     | 23651/50000 [14:38:15<14:00:07,  1.91s/it]
Outer training loop:  47%|####7     | 23652/50000 [14:38:17<14:28:37,  1.98s/it]
Outer training loop:  47%|####7     | 23653/50000 [14:38:19<13:57:40,  1.91s/it]
Outer training loop:  47%|####7     | 23654/50000 [14:38:21<14:03:26,  1.92s/it]
Outer training loop:  47%|####7     | 23655/50000 [14:38:23<14:04:19,  1.92s/it]
Outer training loop:  47%|####7     | 23656/50000 [14:38:25<13:54:09,  1.90s/it]
Outer training loop:  47%|####7     | 23657/50000 [14:38:27<13:59:22,  1.91s/it]
Outer training loop:  47%|####7     | 23658/50000 [14:38:29<13:46:08,  1.88s/it]
Outer training loop:  47%|####7     | 23659/50000 [14:38:30<13:51:18,  1.89s/it]
Outer training loop:  47%|####7     | 23660/50000 [14:38:32<13:33:29,  1.85s/it]
Outer training loop:  47%|####7     | 23661/50000 [14:38:34<13:27:34,  1.84s/it]
Outer training loop:  47%|####7     | 23662/50000 [14:38:36<13:58:41,  1.91s/it]
Outer training loop:  47%|####7     | 23663/50000 [14:38:38<13:54:01,  1.90s/it]
Outer training loop:  47%|####7     | 23664/50000 [14:38:40<14:05:17,  1.93s/it]
Outer training loop:  47%|####7     | 23665/50000 [14:38:42<14:13:26,  1.94s/it]
Outer training loop:  47%|####7     | 23666/50000 [14:38:44<14:02:45,  1.92s/it]
Outer training loop:  47%|####7     | 23667/50000 [14:38:46<13:50:01,  1.89s/it]
Outer training loop:  47%|####7     | 23668/50000 [14:38:48<14:04:41,  1.92s/it]
Outer training loop:  47%|####7     | 23669/50000 [14:38:49<13:50:00,  1.89s/it]
Outer training loop:  47%|####7     | 23670/50000 [14:38:51<13:46:48,  1.88s/it]
Outer training loop:  47%|####7     | 23671/50000 [14:38:53<13:46:57,  1.88s/it]
Outer training loop:  47%|####7     | 23672/50000 [14:38:55<14:15:12,  1.95s/it]
Outer training loop:  47%|####7     | 23673/50000 [14:38:57<13:59:16,  1.91s/it]
Outer training loop:  47%|####7     | 23674/50000 [14:38:59<14:01:20,  1.92s/it]
Outer training loop:  47%|####7     | 23675/50000 [14:39:01<14:05:34,  1.93s/it]
Outer training loop:  47%|####7     | 23676/50000 [14:39:03<14:03:55,  1.92s/it]
Outer training loop:  47%|####7     | 23677/50000 [14:39:05<14:04:22,  1.92s/it]
Outer training loop:  47%|####7     | 23678/50000 [14:39:07<14:15:16,  1.95s/it]
Outer training loop:  47%|####7     | 23679/50000 [14:39:09<13:43:51,  1.88s/it]
Outer training loop:  47%|####7     | 23680/50000 [14:39:10<13:25:09,  1.84s/it]
Outer training loop:  47%|####7     | 23681/50000 [14:39:12<13:16:35,  1.82s/it]
Outer training loop:  47%|####7     | 23682/50000 [14:39:14<13:32:12,  1.85s/it]
Outer training loop:  47%|####7     | 23683/50000 [14:39:16<13:06:17,  1.79s/it]
Outer training loop:  47%|####7     | 23684/50000 [14:39:17<13:13:57,  1.81s/it]
Outer training loop:  47%|####7     | 23685/50000 [14:39:19<13:25:45,  1.84s/it]
Outer training loop:  47%|####7     | 23686/50000 [14:39:21<13:21:40,  1.83s/it]
Outer training loop:  47%|####7     | 23687/50000 [14:39:23<13:14:59,  1.81s/it]
Outer training loop:  47%|####7     | 23688/50000 [14:39:25<14:13:42,  1.95s/it]
Outer training loop:  47%|####7     | 23689/50000 [14:39:27<13:57:07,  1.91s/it]
Outer training loop:  47%|####7     | 23690/50000 [14:39:29<13:43:54,  1.88s/it]
Outer training loop:  47%|####7     | 23691/50000 [14:39:31<14:07:01,  1.93s/it]
Outer training loop:  47%|####7     | 23692/50000 [14:39:33<13:59:47,  1.92s/it]
Outer training loop:  47%|####7     | 23693/50000 [14:39:35<14:16:16,  1.95s/it]
Outer training loop:  47%|####7     | 23694/50000 [14:39:37<14:18:17,  1.96s/it]
Outer training loop:  47%|####7     | 23695/50000 [14:39:39<13:43:52,  1.88s/it]
Outer training loop:  47%|####7     | 23696/50000 [14:39:40<13:37:51,  1.87s/it]
Outer training loop:  47%|####7     | 23697/50000 [14:39:42<13:12:57,  1.81s/it]
Outer training loop:  47%|####7     | 23698/50000 [14:39:44<13:32:53,  1.85s/it]
Outer training loop:  47%|####7     | 23699/50000 [14:39:46<13:38:06,  1.87s/it]
Outer training loop:  47%|####7     | 23700/50000 [14:39:48<13:31:19,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  47%|####7     | 23701/50000 [14:39:55<26:10:00,  3.58s/it]
Outer training loop:  47%|####7     | 23702/50000 [14:39:57<22:45:05,  3.11s/it]
Outer training loop:  47%|####7     | 23703/50000 [14:39:59<20:02:01,  2.74s/it]
Outer training loop:  47%|####7     | 23704/50000 [14:40:01<17:44:44,  2.43s/it]
Outer training loop:  47%|####7     | 23705/50000 [14:40:03<16:52:36,  2.31s/it]
Outer training loop:  47%|####7     | 23706/50000 [14:40:05<15:58:16,  2.19s/it]
Outer training loop:  47%|####7     | 23707/50000 [14:40:07<15:29:54,  2.12s/it]
Outer training loop:  47%|####7     | 23708/50000 [14:40:09<15:09:29,  2.08s/it]
Outer training loop:  47%|####7     | 23709/50000 [14:40:11<15:02:34,  2.06s/it]
Outer training loop:  47%|####7     | 23710/50000 [14:40:13<14:58:44,  2.05s/it]
Outer training loop:  47%|####7     | 23711/50000 [14:40:15<14:41:58,  2.01s/it]
Outer training loop:  47%|####7     | 23712/50000 [14:40:17<14:49:29,  2.03s/it]
Outer training loop:  47%|####7     | 23713/50000 [14:40:19<14:12:44,  1.95s/it]
Outer training loop:  47%|####7     | 23714/50000 [14:40:21<14:13:26,  1.95s/it]
Outer training loop:  47%|####7     | 23715/50000 [14:40:23<14:38:24,  2.01s/it]
Outer training loop:  47%|####7     | 23716/50000 [14:40:25<14:27:14,  1.98s/it]
Outer training loop:  47%|####7     | 23717/50000 [14:40:26<14:02:22,  1.92s/it]
Outer training loop:  47%|####7     | 23718/50000 [14:40:28<13:49:19,  1.89s/it]
Outer training loop:  47%|####7     | 23719/50000 [14:40:30<13:59:18,  1.92s/it]
Outer training loop:  47%|####7     | 23720/50000 [14:40:32<14:06:12,  1.93s/it]
Outer training loop:  47%|####7     | 23721/50000 [14:40:34<13:54:51,  1.91s/it]
Outer training loop:  47%|####7     | 23722/50000 [14:40:36<14:32:19,  1.99s/it]
Outer training loop:  47%|####7     | 23723/50000 [14:40:38<14:01:54,  1.92s/it]
Outer training loop:  47%|####7     | 23724/50000 [14:40:40<14:08:14,  1.94s/it]
Outer training loop:  47%|####7     | 23725/50000 [14:40:42<14:35:43,  2.00s/it]
Outer training loop:  47%|####7     | 23726/50000 [14:40:44<14:15:55,  1.95s/it]
Outer training loop:  47%|####7     | 23727/50000 [14:40:46<14:07:38,  1.94s/it]
Outer training loop:  47%|####7     | 23728/50000 [14:40:48<13:47:03,  1.89s/it]
Outer training loop:  47%|####7     | 23729/50000 [14:40:50<14:13:17,  1.95s/it]
Outer training loop:  47%|####7     | 23730/50000 [14:40:52<14:16:18,  1.96s/it]
Outer training loop:  47%|####7     | 23731/50000 [14:40:54<14:12:08,  1.95s/it]
Outer training loop:  47%|####7     | 23732/50000 [14:40:56<14:32:58,  1.99s/it]
Outer training loop:  47%|####7     | 23733/50000 [14:40:58<14:26:08,  1.98s/it]
Outer training loop:  47%|####7     | 23734/50000 [14:40:59<13:48:45,  1.89s/it]
Outer training loop:  47%|####7     | 23735/50000 [14:41:01<14:15:45,  1.95s/it]
Outer training loop:  47%|####7     | 23736/50000 [14:41:03<13:49:07,  1.89s/it]
Outer training loop:  47%|####7     | 23737/50000 [14:41:05<14:09:11,  1.94s/it]
Outer training loop:  47%|####7     | 23738/50000 [14:41:07<14:00:23,  1.92s/it]
Outer training loop:  47%|####7     | 23739/50000 [14:41:09<14:27:09,  1.98s/it]
Outer training loop:  47%|####7     | 23740/50000 [14:41:11<14:32:42,  1.99s/it]
Outer training loop:  47%|####7     | 23741/50000 [14:41:13<14:12:16,  1.95s/it]
Outer training loop:  47%|####7     | 23742/50000 [14:41:15<14:24:47,  1.98s/it]
Outer training loop:  47%|####7     | 23743/50000 [14:41:17<14:20:53,  1.97s/it]
Outer training loop:  47%|####7     | 23744/50000 [14:41:19<14:30:31,  1.99s/it]
Outer training loop:  47%|####7     | 23745/50000 [14:41:21<14:29:06,  1.99s/it]
Outer training loop:  47%|####7     | 23746/50000 [14:41:23<14:22:18,  1.97s/it]
Outer training loop:  47%|####7     | 23747/50000 [14:41:25<14:40:46,  2.01s/it]
Outer training loop:  47%|####7     | 23748/50000 [14:41:27<14:31:41,  1.99s/it]
Outer training loop:  47%|####7     | 23749/50000 [14:41:29<14:46:45,  2.03s/it]
Outer training loop:  48%|####7     | 23750/50000 [14:41:31<14:15:18,  1.95s/it]
Outer training loop:  48%|####7     | 23751/50000 [14:41:33<14:18:07,  1.96s/it]
Outer training loop:  48%|####7     | 23752/50000 [14:41:35<14:58:13,  2.05s/it]
Outer training loop:  48%|####7     | 23753/50000 [14:41:37<14:36:26,  2.00s/it]
Outer training loop:  48%|####7     | 23754/50000 [14:41:39<14:23:29,  1.97s/it]
Outer training loop:  48%|####7     | 23755/50000 [14:41:41<14:19:00,  1.96s/it]
Outer training loop:  48%|####7     | 23756/50000 [14:41:43<14:34:46,  2.00s/it]
Outer training loop:  48%|####7     | 23757/50000 [14:41:45<14:18:54,  1.96s/it]
Outer training loop:  48%|####7     | 23758/50000 [14:41:47<13:49:09,  1.90s/it]
Outer training loop:  48%|####7     | 23759/50000 [14:41:49<14:02:45,  1.93s/it]
Outer training loop:  48%|####7     | 23760/50000 [14:41:50<13:42:06,  1.88s/it]
Outer training loop:  48%|####7     | 23761/50000 [14:41:52<13:40:16,  1.88s/it]
Outer training loop:  48%|####7     | 23762/50000 [14:41:54<13:22:09,  1.83s/it]
Outer training loop:  48%|####7     | 23763/50000 [14:41:56<13:58:30,  1.92s/it]
Outer training loop:  48%|####7     | 23764/50000 [14:41:58<13:48:22,  1.89s/it]
Outer training loop:  48%|####7     | 23765/50000 [14:42:00<13:43:48,  1.88s/it]
Outer training loop:  48%|####7     | 23766/50000 [14:42:02<13:59:38,  1.92s/it]
Outer training loop:  48%|####7     | 23767/50000 [14:42:04<13:43:21,  1.88s/it]
Outer training loop:  48%|####7     | 23768/50000 [14:42:05<13:40:03,  1.88s/it]
Outer training loop:  48%|####7     | 23769/50000 [14:42:08<14:18:51,  1.96s/it]
Outer training loop:  48%|####7     | 23770/50000 [14:42:10<14:10:30,  1.95s/it]
Outer training loop:  48%|####7     | 23771/50000 [14:42:11<13:57:54,  1.92s/it]
Outer training loop:  48%|####7     | 23772/50000 [14:42:13<14:04:19,  1.93s/it]
Outer training loop:  48%|####7     | 23773/50000 [14:42:15<13:53:37,  1.91s/it]
Outer training loop:  48%|####7     | 23774/50000 [14:42:17<13:47:47,  1.89s/it]
Outer training loop:  48%|####7     | 23775/50000 [14:42:19<13:46:05,  1.89s/it]
Outer training loop:  48%|####7     | 23776/50000 [14:42:21<14:18:06,  1.96s/it]
Outer training loop:  48%|####7     | 23777/50000 [14:42:23<13:58:14,  1.92s/it]
Outer training loop:  48%|####7     | 23778/50000 [14:42:25<13:49:21,  1.90s/it]
Outer training loop:  48%|####7     | 23779/50000 [14:42:27<14:07:18,  1.94s/it]
Outer training loop:  48%|####7     | 23780/50000 [14:42:29<13:51:57,  1.90s/it]
Outer training loop:  48%|####7     | 23781/50000 [14:42:30<13:42:35,  1.88s/it]
Outer training loop:  48%|####7     | 23782/50000 [14:42:33<14:16:47,  1.96s/it]
Outer training loop:  48%|####7     | 23783/50000 [14:42:34<13:58:35,  1.92s/it]
Outer training loop:  48%|####7     | 23784/50000 [14:42:36<14:06:26,  1.94s/it]
Outer training loop:  48%|####7     | 23785/50000 [14:42:39<14:37:45,  2.01s/it]
Outer training loop:  48%|####7     | 23786/50000 [14:42:40<14:15:07,  1.96s/it]
Outer training loop:  48%|####7     | 23787/50000 [14:42:42<14:13:51,  1.95s/it]
Outer training loop:  48%|####7     | 23788/50000 [14:42:44<13:49:37,  1.90s/it]
Outer training loop:  48%|####7     | 23789/50000 [14:42:46<14:05:42,  1.94s/it]
Outer training loop:  48%|####7     | 23790/50000 [14:42:48<13:42:03,  1.88s/it]
Outer training loop:  48%|####7     | 23791/50000 [14:42:50<13:48:10,  1.90s/it]
Outer training loop:  48%|####7     | 23792/50000 [14:42:52<14:15:21,  1.96s/it]
Outer training loop:  48%|####7     | 23793/50000 [14:42:54<14:18:59,  1.97s/it]
Outer training loop:  48%|####7     | 23794/50000 [14:42:56<14:11:07,  1.95s/it]
Outer training loop:  48%|####7     | 23795/50000 [14:42:58<14:23:23,  1.98s/it]
Outer training loop:  48%|####7     | 23796/50000 [14:43:00<14:01:58,  1.93s/it]
Outer training loop:  48%|####7     | 23797/50000 [14:43:02<13:51:46,  1.90s/it]
Outer training loop:  48%|####7     | 23798/50000 [14:43:03<13:46:49,  1.89s/it]
Outer training loop:  48%|####7     | 23799/50000 [14:43:05<14:01:16,  1.93s/it]
Outer training loop:  48%|####7     | 23800/50000 [14:43:07<13:39:42,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.94s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  48%|####7     | 23801/50000 [14:43:14<25:14:12,  3.47s/it]
Outer training loop:  48%|####7     | 23802/50000 [14:43:16<21:48:14,  3.00s/it]
Outer training loop:  48%|####7     | 23803/50000 [14:43:18<19:20:37,  2.66s/it]
Outer training loop:  48%|####7     | 23804/50000 [14:43:20<17:56:41,  2.47s/it]
Outer training loop:  48%|####7     | 23805/50000 [14:43:22<16:42:06,  2.30s/it]
Outer training loop:  48%|####7     | 23806/50000 [14:43:24<15:58:58,  2.20s/it]
Outer training loop:  48%|####7     | 23807/50000 [14:43:26<15:18:31,  2.10s/it]
Outer training loop:  48%|####7     | 23808/50000 [14:43:28<15:01:23,  2.06s/it]
Outer training loop:  48%|####7     | 23809/50000 [14:43:30<15:04:10,  2.07s/it]
Outer training loop:  48%|####7     | 23810/50000 [14:43:32<14:09:46,  1.95s/it]
Outer training loop:  48%|####7     | 23811/50000 [14:43:33<13:45:26,  1.89s/it]
Outer training loop:  48%|####7     | 23812/50000 [14:43:35<13:29:34,  1.85s/it]
Outer training loop:  48%|####7     | 23813/50000 [14:43:37<13:56:55,  1.92s/it]
Outer training loop:  48%|####7     | 23814/50000 [14:43:39<13:53:04,  1.91s/it]
Outer training loop:  48%|####7     | 23815/50000 [14:43:41<13:40:38,  1.88s/it]
Outer training loop:  48%|####7     | 23816/50000 [14:43:43<13:51:37,  1.91s/it]
Outer training loop:  48%|####7     | 23817/50000 [14:43:45<13:33:03,  1.86s/it]
Outer training loop:  48%|####7     | 23818/50000 [14:43:46<13:18:22,  1.83s/it]
Outer training loop:  48%|####7     | 23819/50000 [14:43:48<13:40:22,  1.88s/it]
Outer training loop:  48%|####7     | 23820/50000 [14:43:50<13:43:30,  1.89s/it]
Outer training loop:  48%|####7     | 23821/50000 [14:43:52<13:29:51,  1.86s/it]
Outer training loop:  48%|####7     | 23822/50000 [14:43:54<13:27:34,  1.85s/it]
Outer training loop:  48%|####7     | 23823/50000 [14:43:56<13:53:02,  1.91s/it]
Outer training loop:  48%|####7     | 23824/50000 [14:43:58<13:36:14,  1.87s/it]
Outer training loop:  48%|####7     | 23825/50000 [14:43:59<13:10:20,  1.81s/it]
Outer training loop:  48%|####7     | 23826/50000 [14:44:01<13:26:55,  1.85s/it]
Outer training loop:  48%|####7     | 23827/50000 [14:44:03<13:17:26,  1.83s/it]
Outer training loop:  48%|####7     | 23828/50000 [14:44:05<13:23:30,  1.84s/it]
Outer training loop:  48%|####7     | 23829/50000 [14:44:07<12:59:28,  1.79s/it]
Outer training loop:  48%|####7     | 23830/50000 [14:44:09<13:17:21,  1.83s/it]
Outer training loop:  48%|####7     | 23831/50000 [14:44:10<13:20:56,  1.84s/it]
Outer training loop:  48%|####7     | 23832/50000 [14:44:12<13:47:39,  1.90s/it]
Outer training loop:  48%|####7     | 23833/50000 [14:44:14<13:48:28,  1.90s/it]
Outer training loop:  48%|####7     | 23834/50000 [14:44:16<13:27:45,  1.85s/it]
Outer training loop:  48%|####7     | 23835/50000 [14:44:18<13:21:10,  1.84s/it]
Outer training loop:  48%|####7     | 23836/50000 [14:44:20<13:29:56,  1.86s/it]
Outer training loop:  48%|####7     | 23837/50000 [14:44:22<13:50:10,  1.90s/it]
Outer training loop:  48%|####7     | 23838/50000 [14:44:24<13:54:49,  1.91s/it]
Outer training loop:  48%|####7     | 23839/50000 [14:44:26<13:51:59,  1.91s/it]
Outer training loop:  48%|####7     | 23840/50000 [14:44:28<14:17:33,  1.97s/it]
Outer training loop:  48%|####7     | 23841/50000 [14:44:30<13:48:59,  1.90s/it]
Outer training loop:  48%|####7     | 23842/50000 [14:44:31<13:46:17,  1.90s/it]
Outer training loop:  48%|####7     | 23843/50000 [14:44:33<14:03:56,  1.94s/it]
Outer training loop:  48%|####7     | 23844/50000 [14:44:35<13:46:45,  1.90s/it]
Outer training loop:  48%|####7     | 23845/50000 [14:44:37<13:43:06,  1.89s/it]
Outer training loop:  48%|####7     | 23846/50000 [14:44:39<13:24:52,  1.85s/it]
Outer training loop:  48%|####7     | 23847/50000 [14:44:41<13:52:23,  1.91s/it]
Outer training loop:  48%|####7     | 23848/50000 [14:44:43<13:52:07,  1.91s/it]
Outer training loop:  48%|####7     | 23849/50000 [14:44:45<13:35:16,  1.87s/it]
Outer training loop:  48%|####7     | 23850/50000 [14:44:47<13:51:48,  1.91s/it]
Outer training loop:  48%|####7     | 23851/50000 [14:44:48<13:44:00,  1.89s/it]
Outer training loop:  48%|####7     | 23852/50000 [14:44:50<13:54:15,  1.91s/it]
Outer training loop:  48%|####7     | 23853/50000 [14:44:52<14:07:47,  1.95s/it]
Outer training loop:  48%|####7     | 23854/50000 [14:44:54<14:03:05,  1.93s/it]
Outer training loop:  48%|####7     | 23855/50000 [14:44:56<14:05:16,  1.94s/it]
Outer training loop:  48%|####7     | 23856/50000 [14:44:58<14:24:48,  1.98s/it]
Outer training loop:  48%|####7     | 23857/50000 [14:45:00<14:16:01,  1.96s/it]
Outer training loop:  48%|####7     | 23858/50000 [14:45:02<14:06:20,  1.94s/it]
Outer training loop:  48%|####7     | 23859/50000 [14:45:04<14:17:30,  1.97s/it]
Outer training loop:  48%|####7     | 23860/50000 [14:45:06<13:45:49,  1.90s/it]
Outer training loop:  48%|####7     | 23861/50000 [14:45:08<13:37:31,  1.88s/it]
Outer training loop:  48%|####7     | 23862/50000 [14:45:10<13:20:01,  1.84s/it]
Outer training loop:  48%|####7     | 23863/50000 [14:45:12<13:46:45,  1.90s/it]
Outer training loop:  48%|####7     | 23864/50000 [14:45:13<13:33:15,  1.87s/it]
Outer training loop:  48%|####7     | 23865/50000 [14:45:15<13:15:38,  1.83s/it]
Outer training loop:  48%|####7     | 23866/50000 [14:45:17<13:53:49,  1.91s/it]
Outer training loop:  48%|####7     | 23867/50000 [14:45:19<13:57:16,  1.92s/it]
Outer training loop:  48%|####7     | 23868/50000 [14:45:21<13:43:41,  1.89s/it]
Outer training loop:  48%|####7     | 23869/50000 [14:45:23<13:51:05,  1.91s/it]
Outer training loop:  48%|####7     | 23870/50000 [14:45:25<14:13:53,  1.96s/it]
Outer training loop:  48%|####7     | 23871/50000 [14:45:27<13:56:06,  1.92s/it]
Outer training loop:  48%|####7     | 23872/50000 [14:45:29<13:37:40,  1.88s/it]
Outer training loop:  48%|####7     | 23873/50000 [14:45:31<14:05:31,  1.94s/it]
Outer training loop:  48%|####7     | 23874/50000 [14:45:32<13:43:05,  1.89s/it]
Outer training loop:  48%|####7     | 23875/50000 [14:45:34<13:20:22,  1.84s/it]
Outer training loop:  48%|####7     | 23876/50000 [14:45:36<13:34:37,  1.87s/it]
Outer training loop:  48%|####7     | 23877/50000 [14:45:38<13:31:40,  1.86s/it]
Outer training loop:  48%|####7     | 23878/50000 [14:45:40<13:18:41,  1.83s/it]
Outer training loop:  48%|####7     | 23879/50000 [14:45:41<12:58:59,  1.79s/it]
Outer training loop:  48%|####7     | 23880/50000 [14:45:43<13:33:31,  1.87s/it]
Outer training loop:  48%|####7     | 23881/50000 [14:45:45<13:23:12,  1.85s/it]
Outer training loop:  48%|####7     | 23882/50000 [14:45:47<13:38:39,  1.88s/it]
Outer training loop:  48%|####7     | 23883/50000 [14:45:49<13:48:04,  1.90s/it]
Outer training loop:  48%|####7     | 23884/50000 [14:45:51<13:29:23,  1.86s/it]
Outer training loop:  48%|####7     | 23885/50000 [14:45:53<13:20:57,  1.84s/it]
Outer training loop:  48%|####7     | 23886/50000 [14:45:55<13:11:41,  1.82s/it]
Outer training loop:  48%|####7     | 23887/50000 [14:45:57<13:50:10,  1.91s/it]
Outer training loop:  48%|####7     | 23888/50000 [14:45:59<14:00:55,  1.93s/it]
Outer training loop:  48%|####7     | 23889/50000 [14:46:01<14:04:28,  1.94s/it]
Outer training loop:  48%|####7     | 23890/50000 [14:46:03<14:17:53,  1.97s/it]
Outer training loop:  48%|####7     | 23891/50000 [14:46:05<14:10:56,  1.96s/it]
Outer training loop:  48%|####7     | 23892/50000 [14:46:06<14:05:56,  1.94s/it]
Outer training loop:  48%|####7     | 23893/50000 [14:46:08<14:04:42,  1.94s/it]
Outer training loop:  48%|####7     | 23894/50000 [14:46:10<13:58:11,  1.93s/it]
Outer training loop:  48%|####7     | 23895/50000 [14:46:12<13:56:18,  1.92s/it]
Outer training loop:  48%|####7     | 23896/50000 [14:46:14<13:52:40,  1.91s/it]
Outer training loop:  48%|####7     | 23897/50000 [14:46:16<14:07:43,  1.95s/it]
Outer training loop:  48%|####7     | 23898/50000 [14:46:18<13:53:46,  1.92s/it]
Outer training loop:  48%|####7     | 23899/50000 [14:46:20<13:45:16,  1.90s/it]
Outer training loop:  48%|####7     | 23900/50000 [14:46:22<13:58:56,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.86s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  48%|####7     | 23901/50000 [14:46:29<26:09:02,  3.61s/it]
Outer training loop:  48%|####7     | 23902/50000 [14:46:31<22:31:21,  3.11s/it]
Outer training loop:  48%|####7     | 23903/50000 [14:46:33<19:44:21,  2.72s/it]
Outer training loop:  48%|####7     | 23904/50000 [14:46:35<18:21:41,  2.53s/it]
Outer training loop:  48%|####7     | 23905/50000 [14:46:37<16:47:03,  2.32s/it]
Outer training loop:  48%|####7     | 23906/50000 [14:46:39<15:45:57,  2.18s/it]
Outer training loop:  48%|####7     | 23907/50000 [14:46:41<15:11:04,  2.09s/it]
Outer training loop:  48%|####7     | 23908/50000 [14:46:43<15:22:47,  2.12s/it]
Outer training loop:  48%|####7     | 23909/50000 [14:46:45<15:08:09,  2.09s/it]
Outer training loop:  48%|####7     | 23910/50000 [14:46:47<14:33:11,  2.01s/it]
Outer training loop:  48%|####7     | 23911/50000 [14:46:49<14:44:35,  2.03s/it]
Outer training loop:  48%|####7     | 23912/50000 [14:46:51<14:41:49,  2.03s/it]
Outer training loop:  48%|####7     | 23913/50000 [14:46:53<14:31:40,  2.00s/it]
Outer training loop:  48%|####7     | 23914/50000 [14:46:55<14:06:28,  1.95s/it]
Outer training loop:  48%|####7     | 23915/50000 [14:46:57<14:13:26,  1.96s/it]
Outer training loop:  48%|####7     | 23916/50000 [14:46:58<13:54:31,  1.92s/it]
Outer training loop:  48%|####7     | 23917/50000 [14:47:00<13:38:19,  1.88s/it]
Outer training loop:  48%|####7     | 23918/50000 [14:47:02<13:48:05,  1.90s/it]
Outer training loop:  48%|####7     | 23919/50000 [14:47:04<13:39:59,  1.89s/it]
Outer training loop:  48%|####7     | 23920/50000 [14:47:06<13:40:17,  1.89s/it]
Outer training loop:  48%|####7     | 23921/50000 [14:47:08<13:29:04,  1.86s/it]
Outer training loop:  48%|####7     | 23922/50000 [14:47:10<13:46:47,  1.90s/it]
Outer training loop:  48%|####7     | 23923/50000 [14:47:12<13:48:29,  1.91s/it]
Outer training loop:  48%|####7     | 23924/50000 [14:47:13<13:32:44,  1.87s/it]
Outer training loop:  48%|####7     | 23925/50000 [14:47:16<14:03:54,  1.94s/it]
Outer training loop:  48%|####7     | 23926/50000 [14:47:17<13:44:31,  1.90s/it]
Outer training loop:  48%|####7     | 23927/50000 [14:47:19<13:25:17,  1.85s/it]
Outer training loop:  48%|####7     | 23928/50000 [14:47:21<13:13:06,  1.83s/it]
Outer training loop:  48%|####7     | 23929/50000 [14:47:23<13:52:24,  1.92s/it]
Outer training loop:  48%|####7     | 23930/50000 [14:47:25<13:50:34,  1.91s/it]
Outer training loop:  48%|####7     | 23931/50000 [14:47:27<13:40:50,  1.89s/it]
Outer training loop:  48%|####7     | 23932/50000 [14:47:28<13:15:24,  1.83s/it]
Outer training loop:  48%|####7     | 23933/50000 [14:47:30<13:46:38,  1.90s/it]
Outer training loop:  48%|####7     | 23934/50000 [14:47:32<13:43:44,  1.90s/it]
Outer training loop:  48%|####7     | 23935/50000 [14:47:34<13:44:56,  1.90s/it]
Outer training loop:  48%|####7     | 23936/50000 [14:47:36<13:56:45,  1.93s/it]
Outer training loop:  48%|####7     | 23937/50000 [14:47:38<13:51:17,  1.91s/it]
Outer training loop:  48%|####7     | 23938/50000 [14:47:40<13:33:54,  1.87s/it]
Outer training loop:  48%|####7     | 23939/50000 [14:47:42<13:18:34,  1.84s/it]
Outer training loop:  48%|####7     | 23940/50000 [14:47:44<13:42:52,  1.89s/it]
Outer training loop:  48%|####7     | 23941/50000 [14:47:46<13:44:17,  1.90s/it]
Outer training loop:  48%|####7     | 23942/50000 [14:47:47<13:21:08,  1.84s/it]
Outer training loop:  48%|####7     | 23943/50000 [14:47:49<13:41:31,  1.89s/it]
Outer training loop:  48%|####7     | 23944/50000 [14:47:51<13:35:05,  1.88s/it]
Outer training loop:  48%|####7     | 23945/50000 [14:47:53<13:19:29,  1.84s/it]
Outer training loop:  48%|####7     | 23946/50000 [14:47:55<13:10:59,  1.82s/it]
Outer training loop:  48%|####7     | 23947/50000 [14:47:57<13:32:56,  1.87s/it]
Outer training loop:  48%|####7     | 23948/50000 [14:47:59<13:29:52,  1.87s/it]
Outer training loop:  48%|####7     | 23949/50000 [14:48:00<13:21:09,  1.85s/it]
Outer training loop:  48%|####7     | 23950/50000 [14:48:02<13:24:57,  1.85s/it]
Outer training loop:  48%|####7     | 23951/50000 [14:48:04<13:38:00,  1.88s/it]
Outer training loop:  48%|####7     | 23952/50000 [14:48:06<13:28:07,  1.86s/it]
Outer training loop:  48%|####7     | 23953/50000 [14:48:08<13:18:22,  1.84s/it]
Outer training loop:  48%|####7     | 23954/50000 [14:48:10<13:44:03,  1.90s/it]
Outer training loop:  48%|####7     | 23955/50000 [14:48:12<13:51:06,  1.91s/it]
Outer training loop:  48%|####7     | 23956/50000 [14:48:14<14:00:42,  1.94s/it]
Outer training loop:  48%|####7     | 23957/50000 [14:48:16<13:55:28,  1.92s/it]
Outer training loop:  48%|####7     | 23958/50000 [14:48:17<13:40:16,  1.89s/it]
Outer training loop:  48%|####7     | 23959/50000 [14:48:19<13:27:14,  1.86s/it]
Outer training loop:  48%|####7     | 23960/50000 [14:48:21<13:46:49,  1.91s/it]
Outer training loop:  48%|####7     | 23961/50000 [14:48:23<13:31:49,  1.87s/it]
Outer training loop:  48%|####7     | 23962/50000 [14:48:25<13:54:46,  1.92s/it]
Outer training loop:  48%|####7     | 23963/50000 [14:48:27<13:33:54,  1.88s/it]
Outer training loop:  48%|####7     | 23964/50000 [14:48:29<14:09:09,  1.96s/it]
Outer training loop:  48%|####7     | 23965/50000 [14:48:31<13:45:42,  1.90s/it]
Outer training loop:  48%|####7     | 23966/50000 [14:48:33<13:48:52,  1.91s/it]
Outer training loop:  48%|####7     | 23967/50000 [14:48:35<13:55:53,  1.93s/it]
Outer training loop:  48%|####7     | 23968/50000 [14:48:37<13:45:38,  1.90s/it]
Outer training loop:  48%|####7     | 23969/50000 [14:48:38<13:26:44,  1.86s/it]
Outer training loop:  48%|####7     | 23970/50000 [14:48:40<13:48:51,  1.91s/it]
Outer training loop:  48%|####7     | 23971/50000 [14:48:42<13:35:12,  1.88s/it]
Outer training loop:  48%|####7     | 23972/50000 [14:48:44<13:18:47,  1.84s/it]
Outer training loop:  48%|####7     | 23973/50000 [14:48:46<13:46:29,  1.91s/it]
Outer training loop:  48%|####7     | 23974/50000 [14:48:48<13:26:20,  1.86s/it]
Outer training loop:  48%|####7     | 23975/50000 [14:48:50<13:24:25,  1.85s/it]
Outer training loop:  48%|####7     | 23976/50000 [14:48:51<13:19:12,  1.84s/it]
Outer training loop:  48%|####7     | 23977/50000 [14:48:53<13:41:21,  1.89s/it]
Outer training loop:  48%|####7     | 23978/50000 [14:48:55<13:30:40,  1.87s/it]
Outer training loop:  48%|####7     | 23979/50000 [14:48:57<13:42:47,  1.90s/it]
Outer training loop:  48%|####7     | 23980/50000 [14:48:59<13:54:47,  1.92s/it]
Outer training loop:  48%|####7     | 23981/50000 [14:49:01<13:26:44,  1.86s/it]
Outer training loop:  48%|####7     | 23982/50000 [14:49:03<13:27:16,  1.86s/it]
Outer training loop:  48%|####7     | 23983/50000 [14:49:05<14:10:51,  1.96s/it]
Outer training loop:  48%|####7     | 23984/50000 [14:49:07<13:45:13,  1.90s/it]
Outer training loop:  48%|####7     | 23985/50000 [14:49:09<13:45:15,  1.90s/it]
Outer training loop:  48%|####7     | 23986/50000 [14:49:11<13:55:45,  1.93s/it]
Outer training loop:  48%|####7     | 23987/50000 [14:49:12<13:38:50,  1.89s/it]
Outer training loop:  48%|####7     | 23988/50000 [14:49:14<13:16:41,  1.84s/it]
Outer training loop:  48%|####7     | 23989/50000 [14:49:16<13:12:20,  1.83s/it]
Outer training loop:  48%|####7     | 23990/50000 [14:49:18<13:41:15,  1.89s/it]
Outer training loop:  48%|####7     | 23991/50000 [14:49:20<13:40:15,  1.89s/it]
Outer training loop:  48%|####7     | 23992/50000 [14:49:22<13:25:44,  1.86s/it]
Outer training loop:  48%|####7     | 23993/50000 [14:49:23<13:22:54,  1.85s/it]
Outer training loop:  48%|####7     | 23994/50000 [14:49:25<13:48:04,  1.91s/it]
Outer training loop:  48%|####7     | 23995/50000 [14:49:27<13:32:48,  1.88s/it]
Outer training loop:  48%|####7     | 23996/50000 [14:49:29<13:34:22,  1.88s/it]
Outer training loop:  48%|####7     | 23997/50000 [14:49:31<13:47:02,  1.91s/it]
Outer training loop:  48%|####7     | 23998/50000 [14:49:33<13:51:16,  1.92s/it]
Outer training loop:  48%|####7     | 23999/50000 [14:49:35<13:43:42,  1.90s/it]
Outer training loop:  48%|####8     | 24000/50000 [14:49:37<13:27:04,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:32,  1.86s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:29,  1.84s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.85s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.87s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:22,  1.86s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.92s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:20,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:15,  1.91s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.87s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:03,  1.96s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.90s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:56,  1.97s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:55,  1.98s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.93s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.94s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:42,  1.89s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:44,  1.98s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:43,  1.98s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.92s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:32,  1.96s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:22,  1.87s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:21,  1.98s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:17,  1.93s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:10,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:07,  1.81s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:05,  1.81s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.81s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:05,  1.92s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.87s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:52,  1.94s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.92s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.90s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:37,  1.96s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:34,  1.89s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.82s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.81s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:23,  1.82s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:21,  1.80s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:21,  1.92s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.94s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.86s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.84s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.90s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:02,  2.04s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.94s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:26,  1.78s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:24,  1.80s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:25,  1.85s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:21,  1.82s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:20,  1.82s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:20,  1.85s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:19,  1.86s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:31,  2.04s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:26,  2.00s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:23,  1.99s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:17,  1.97s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:13,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:12,  1.95s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.93s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:12,  2.00s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:08,  1.97s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.92s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:00,  1.95s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:56,  1.91s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:55,  1.92s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:51,  1.96s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.92s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.91s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:38,  1.87s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:35,  1.84s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.93s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:32,  1.89s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:29,  1.98s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:25,  1.94s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:23,  1.95s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:20,  1.93s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:21,  1.98s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:18,  1.97s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:20,  2.07s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:17,  2.04s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:14,  2.01s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:10,  1.96s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:07,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:04,  1.89s/it]

Evaluating policy:  60%|######    | 50/83 [01:36<01:01,  1.87s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<01:04,  2.01s/it]

Evaluating policy:  63%|######2   | 52/83 [01:40<01:01,  1.98s/it]

Evaluating policy:  64%|######3   | 53/83 [01:42<00:57,  1.91s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:53,  1.89s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:47,  1.99s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:44,  1.92s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:06<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:29,  1.86s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:29,  1.99s/it]

Evaluating policy:  83%|########3 | 69/83 [02:12<00:27,  1.99s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:25,  1.93s/it]

Evaluating policy:  86%|########5 | 71/83 [02:16<00:22,  1.88s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:19<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:12,  1.82s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:11,  1.97s/it]

Evaluating policy:  94%|#########3| 78/83 [02:29<00:09,  1.90s/it]

Evaluating policy:  95%|#########5| 79/83 [02:31<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:33<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.85s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.86s/it]

Outer training loop:  48%|####8     | 24001/50000 [14:55:00<710:09:23, 98.33s/it]
Outer training loop:  48%|####8     | 24002/50000 [14:55:02<500:54:25, 69.36s/it]
Outer training loop:  48%|####8     | 24003/50000 [14:55:04<354:44:35, 49.12s/it]
Outer training loop:  48%|####8     | 24004/50000 [14:55:06<253:31:50, 35.11s/it]
Outer training loop:  48%|####8     | 24005/50000 [14:55:08<181:32:11, 25.14s/it]
Outer training loop:  48%|####8     | 24006/50000 [14:55:10<131:09:04, 18.16s/it]
Outer training loop:  48%|####8     | 24007/50000 [14:55:12<95:41:33, 13.25s/it] 
Outer training loop:  48%|####8     | 24008/50000 [14:55:14<71:40:39,  9.93s/it]
Outer training loop:  48%|####8     | 24009/50000 [14:55:16<53:50:32,  7.46s/it]
Outer training loop:  48%|####8     | 24010/50000 [14:55:18<41:57:17,  5.81s/it]
Outer training loop:  48%|####8     | 24011/50000 [14:55:20<34:01:25,  4.71s/it]
Outer training loop:  48%|####8     | 24012/50000 [14:55:22<27:47:00,  3.85s/it]
Outer training loop:  48%|####8     | 24013/50000 [14:55:23<23:26:31,  3.25s/it]
Outer training loop:  48%|####8     | 24014/50000 [14:55:25<20:33:48,  2.85s/it]
Outer training loop:  48%|####8     | 24015/50000 [14:55:27<18:48:48,  2.61s/it]
Outer training loop:  48%|####8     | 24016/50000 [14:55:29<17:05:03,  2.37s/it]
Outer training loop:  48%|####8     | 24017/50000 [14:55:31<15:55:15,  2.21s/it]
Outer training loop:  48%|####8     | 24018/50000 [14:55:33<15:29:22,  2.15s/it]
Outer training loop:  48%|####8     | 24019/50000 [14:55:35<15:00:18,  2.08s/it]
Outer training loop:  48%|####8     | 24020/50000 [14:55:37<14:08:47,  1.96s/it]
Outer training loop:  48%|####8     | 24021/50000 [14:55:39<14:27:08,  2.00s/it]
Outer training loop:  48%|####8     | 24022/50000 [14:55:41<14:18:11,  1.98s/it]
Outer training loop:  48%|####8     | 24023/50000 [14:55:43<14:07:40,  1.96s/it]
Outer training loop:  48%|####8     | 24024/50000 [14:55:45<14:08:09,  1.96s/it]
Outer training loop:  48%|####8     | 24025/50000 [14:55:46<13:49:54,  1.92s/it]
Outer training loop:  48%|####8     | 24026/50000 [14:55:48<13:43:30,  1.90s/it]
Outer training loop:  48%|####8     | 24027/50000 [14:55:50<13:45:33,  1.91s/it]
Outer training loop:  48%|####8     | 24028/50000 [14:55:52<14:04:11,  1.95s/it]
Outer training loop:  48%|####8     | 24029/50000 [14:55:54<13:52:55,  1.92s/it]
Outer training loop:  48%|####8     | 24030/50000 [14:55:56<13:43:59,  1.90s/it]
Outer training loop:  48%|####8     | 24031/50000 [14:55:58<14:06:37,  1.96s/it]
Outer training loop:  48%|####8     | 24032/50000 [14:56:00<13:41:26,  1.90s/it]
Outer training loop:  48%|####8     | 24033/50000 [14:56:01<13:15:12,  1.84s/it]
Outer training loop:  48%|####8     | 24034/50000 [14:56:03<13:12:36,  1.83s/it]
Outer training loop:  48%|####8     | 24035/50000 [14:56:05<13:50:14,  1.92s/it]
Outer training loop:  48%|####8     | 24036/50000 [14:56:07<13:35:08,  1.88s/it]
Outer training loop:  48%|####8     | 24037/50000 [14:56:09<13:19:03,  1.85s/it]
Outer training loop:  48%|####8     | 24038/50000 [14:56:11<13:58:22,  1.94s/it]
Outer training loop:  48%|####8     | 24039/50000 [14:56:13<13:39:03,  1.89s/it]
Outer training loop:  48%|####8     | 24040/50000 [14:56:15<13:18:46,  1.85s/it]
Outer training loop:  48%|####8     | 24041/50000 [14:56:16<13:09:13,  1.82s/it]
Outer training loop:  48%|####8     | 24042/50000 [14:56:18<13:18:13,  1.85s/it]
Outer training loop:  48%|####8     | 24043/50000 [14:56:20<13:16:06,  1.84s/it]
Outer training loop:  48%|####8     | 24044/50000 [14:56:22<13:07:58,  1.82s/it]
Outer training loop:  48%|####8     | 24045/50000 [14:56:24<13:31:18,  1.88s/it]
Outer training loop:  48%|####8     | 24046/50000 [14:56:26<13:37:00,  1.89s/it]
Outer training loop:  48%|####8     | 24047/50000 [14:56:28<13:20:24,  1.85s/it]
Outer training loop:  48%|####8     | 24048/50000 [14:56:30<13:36:34,  1.89s/it]
Outer training loop:  48%|####8     | 24049/50000 [14:56:31<13:21:53,  1.85s/it]
Outer training loop:  48%|####8     | 24050/50000 [14:56:33<13:13:48,  1.84s/it]
Outer training loop:  48%|####8     | 24051/50000 [14:56:35<13:11:50,  1.83s/it]
Outer training loop:  48%|####8     | 24052/50000 [14:56:37<13:27:23,  1.87s/it]
Outer training loop:  48%|####8     | 24053/50000 [14:56:39<13:03:45,  1.81s/it]
Outer training loop:  48%|####8     | 24054/50000 [14:56:40<13:03:35,  1.81s/it]
Outer training loop:  48%|####8     | 24055/50000 [14:56:42<13:41:15,  1.90s/it]
Outer training loop:  48%|####8     | 24056/50000 [14:56:44<13:36:56,  1.89s/it]
Outer training loop:  48%|####8     | 24057/50000 [14:56:46<13:28:44,  1.87s/it]
Outer training loop:  48%|####8     | 24058/50000 [14:56:48<13:11:40,  1.83s/it]
Outer training loop:  48%|####8     | 24059/50000 [14:56:50<13:35:37,  1.89s/it]
Outer training loop:  48%|####8     | 24060/50000 [14:56:52<13:43:17,  1.90s/it]
Outer training loop:  48%|####8     | 24061/50000 [14:56:54<13:40:02,  1.90s/it]
Outer training loop:  48%|####8     | 24062/50000 [14:56:56<14:02:40,  1.95s/it]
Outer training loop:  48%|####8     | 24063/50000 [14:56:58<13:41:15,  1.90s/it]
Outer training loop:  48%|####8     | 24064/50000 [14:56:59<13:28:08,  1.87s/it]
Outer training loop:  48%|####8     | 24065/50000 [14:57:01<13:15:31,  1.84s/it]
Outer training loop:  48%|####8     | 24066/50000 [14:57:03<13:41:35,  1.90s/it]
Outer training loop:  48%|####8     | 24067/50000 [14:57:05<13:30:17,  1.87s/it]
Outer training loop:  48%|####8     | 24068/50000 [14:57:07<13:07:53,  1.82s/it]
Outer training loop:  48%|####8     | 24069/50000 [14:57:09<13:28:03,  1.87s/it]
Outer training loop:  48%|####8     | 24070/50000 [14:57:11<13:17:09,  1.84s/it]
Outer training loop:  48%|####8     | 24071/50000 [14:57:12<13:32:29,  1.88s/it]
Outer training loop:  48%|####8     | 24072/50000 [14:57:14<13:48:10,  1.92s/it]
Outer training loop:  48%|####8     | 24073/50000 [14:57:16<13:26:42,  1.87s/it]
Outer training loop:  48%|####8     | 24074/50000 [14:57:18<13:25:15,  1.86s/it]
Outer training loop:  48%|####8     | 24075/50000 [14:57:20<13:30:24,  1.88s/it]
Outer training loop:  48%|####8     | 24076/50000 [14:57:22<13:41:36,  1.90s/it]
Outer training loop:  48%|####8     | 24077/50000 [14:57:24<13:34:05,  1.88s/it]
Outer training loop:  48%|####8     | 24078/50000 [14:57:26<13:38:57,  1.90s/it]
Outer training loop:  48%|####8     | 24079/50000 [14:57:28<13:54:33,  1.93s/it]
Outer training loop:  48%|####8     | 24080/50000 [14:57:29<13:28:05,  1.87s/it]
Outer training loop:  48%|####8     | 24081/50000 [14:57:31<13:02:05,  1.81s/it]
Outer training loop:  48%|####8     | 24082/50000 [14:57:33<13:48:45,  1.92s/it]
Outer training loop:  48%|####8     | 24083/50000 [14:57:35<13:47:33,  1.92s/it]
Outer training loop:  48%|####8     | 24084/50000 [14:57:37<13:31:09,  1.88s/it]
Outer training loop:  48%|####8     | 24085/50000 [14:57:39<13:50:42,  1.92s/it]
Outer training loop:  48%|####8     | 24086/50000 [14:57:41<13:18:17,  1.85s/it]
Outer training loop:  48%|####8     | 24087/50000 [14:57:42<13:11:29,  1.83s/it]
Outer training loop:  48%|####8     | 24088/50000 [14:57:44<13:00:57,  1.81s/it]
Outer training loop:  48%|####8     | 24089/50000 [14:57:46<13:18:44,  1.85s/it]
Outer training loop:  48%|####8     | 24090/50000 [14:57:48<12:55:13,  1.80s/it]
Outer training loop:  48%|####8     | 24091/50000 [14:57:50<13:13:30,  1.84s/it]
Outer training loop:  48%|####8     | 24092/50000 [14:57:52<13:01:31,  1.81s/it]
Outer training loop:  48%|####8     | 24093/50000 [14:57:54<13:32:54,  1.88s/it]
Outer training loop:  48%|####8     | 24094/50000 [14:57:55<13:27:55,  1.87s/it]
Outer training loop:  48%|####8     | 24095/50000 [14:57:57<13:24:51,  1.86s/it]
Outer training loop:  48%|####8     | 24096/50000 [14:57:59<14:00:52,  1.95s/it]
Outer training loop:  48%|####8     | 24097/50000 [14:58:01<13:37:33,  1.89s/it]
Outer training loop:  48%|####8     | 24098/50000 [14:58:03<13:33:48,  1.89s/it]
Outer training loop:  48%|####8     | 24099/50000 [14:58:05<13:47:57,  1.92s/it]
Outer training loop:  48%|####8     | 24100/50000 [14:58:07<13:24:05,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.96s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  48%|####8     | 24101/50000 [14:58:14<24:57:48,  3.47s/it]
Outer training loop:  48%|####8     | 24102/50000 [14:58:16<21:17:55,  2.96s/it]
Outer training loop:  48%|####8     | 24103/50000 [14:58:18<19:27:21,  2.70s/it]
Outer training loop:  48%|####8     | 24104/50000 [14:58:20<17:52:30,  2.48s/it]
Outer training loop:  48%|####8     | 24105/50000 [14:58:22<16:33:10,  2.30s/it]
Outer training loop:  48%|####8     | 24106/50000 [14:58:24<15:26:19,  2.15s/it]
Outer training loop:  48%|####8     | 24107/50000 [14:58:26<15:55:14,  2.21s/it]
Outer training loop:  48%|####8     | 24108/50000 [14:58:28<14:58:10,  2.08s/it]
Outer training loop:  48%|####8     | 24109/50000 [14:58:29<14:21:43,  2.00s/it]
Outer training loop:  48%|####8     | 24110/50000 [14:58:31<14:24:37,  2.00s/it]
Outer training loop:  48%|####8     | 24111/50000 [14:58:33<14:12:21,  1.98s/it]
Outer training loop:  48%|####8     | 24112/50000 [14:58:35<14:17:11,  1.99s/it]
Outer training loop:  48%|####8     | 24113/50000 [14:58:37<14:14:38,  1.98s/it]
Outer training loop:  48%|####8     | 24114/50000 [14:58:39<13:51:51,  1.93s/it]
Outer training loop:  48%|####8     | 24115/50000 [14:58:41<13:32:43,  1.88s/it]
Outer training loop:  48%|####8     | 24116/50000 [14:58:43<13:20:35,  1.86s/it]
Outer training loop:  48%|####8     | 24117/50000 [14:58:45<13:35:33,  1.89s/it]
Outer training loop:  48%|####8     | 24118/50000 [14:58:47<13:22:12,  1.86s/it]
Outer training loop:  48%|####8     | 24119/50000 [14:58:48<13:09:42,  1.83s/it]
Outer training loop:  48%|####8     | 24120/50000 [14:58:50<13:56:25,  1.94s/it]
Outer training loop:  48%|####8     | 24121/50000 [14:58:52<13:45:31,  1.91s/it]
Outer training loop:  48%|####8     | 24122/50000 [14:58:54<13:32:36,  1.88s/it]
Outer training loop:  48%|####8     | 24123/50000 [14:58:56<13:54:16,  1.93s/it]
Outer training loop:  48%|####8     | 24124/50000 [14:58:58<13:48:30,  1.92s/it]
Outer training loop:  48%|####8     | 24125/50000 [14:59:00<13:36:25,  1.89s/it]
Outer training loop:  48%|####8     | 24126/50000 [14:59:02<13:25:31,  1.87s/it]
Outer training loop:  48%|####8     | 24127/50000 [14:59:04<13:44:16,  1.91s/it]
Outer training loop:  48%|####8     | 24128/50000 [14:59:06<13:44:59,  1.91s/it]
Outer training loop:  48%|####8     | 24129/50000 [14:59:07<13:26:53,  1.87s/it]
Outer training loop:  48%|####8     | 24130/50000 [14:59:09<13:46:27,  1.92s/it]
Outer training loop:  48%|####8     | 24131/50000 [14:59:11<13:29:58,  1.88s/it]
Outer training loop:  48%|####8     | 24132/50000 [14:59:13<13:15:30,  1.85s/it]
Outer training loop:  48%|####8     | 24133/50000 [14:59:15<13:20:37,  1.86s/it]
Outer training loop:  48%|####8     | 24134/50000 [14:59:17<13:27:57,  1.87s/it]
Outer training loop:  48%|####8     | 24135/50000 [14:59:19<13:21:03,  1.86s/it]
Outer training loop:  48%|####8     | 24136/50000 [14:59:20<13:23:03,  1.86s/it]
Outer training loop:  48%|####8     | 24137/50000 [14:59:22<13:05:31,  1.82s/it]
Outer training loop:  48%|####8     | 24138/50000 [14:59:24<13:22:13,  1.86s/it]
Outer training loop:  48%|####8     | 24139/50000 [14:59:26<13:14:42,  1.84s/it]
Outer training loop:  48%|####8     | 24140/50000 [14:59:28<13:40:23,  1.90s/it]
Outer training loop:  48%|####8     | 24141/50000 [14:59:30<13:57:36,  1.94s/it]
Outer training loop:  48%|####8     | 24142/50000 [14:59:32<13:47:34,  1.92s/it]
Outer training loop:  48%|####8     | 24143/50000 [14:59:34<13:22:58,  1.86s/it]
Outer training loop:  48%|####8     | 24144/50000 [14:59:36<13:45:32,  1.92s/it]
Outer training loop:  48%|####8     | 24145/50000 [14:59:38<13:45:12,  1.91s/it]
Outer training loop:  48%|####8     | 24146/50000 [14:59:39<13:34:50,  1.89s/it]
Outer training loop:  48%|####8     | 24147/50000 [14:59:41<13:22:39,  1.86s/it]
Outer training loop:  48%|####8     | 24148/50000 [14:59:43<13:55:08,  1.94s/it]
Outer training loop:  48%|####8     | 24149/50000 [14:59:45<13:37:37,  1.90s/it]
Outer training loop:  48%|####8     | 24150/50000 [14:59:47<13:51:51,  1.93s/it]
Outer training loop:  48%|####8     | 24151/50000 [14:59:49<14:04:52,  1.96s/it]
Outer training loop:  48%|####8     | 24152/50000 [14:59:51<13:34:04,  1.89s/it]
Outer training loop:  48%|####8     | 24153/50000 [14:59:53<13:29:56,  1.88s/it]
Outer training loop:  48%|####8     | 24154/50000 [14:59:55<13:24:12,  1.87s/it]
Outer training loop:  48%|####8     | 24155/50000 [14:59:57<13:39:31,  1.90s/it]
Outer training loop:  48%|####8     | 24156/50000 [14:59:58<13:20:40,  1.86s/it]
Outer training loop:  48%|####8     | 24157/50000 [15:00:00<13:35:14,  1.89s/it]
Outer training loop:  48%|####8     | 24158/50000 [15:00:02<13:42:17,  1.91s/it]
Outer training loop:  48%|####8     | 24159/50000 [15:00:04<13:23:03,  1.86s/it]
Outer training loop:  48%|####8     | 24160/50000 [15:00:06<13:22:07,  1.86s/it]
Outer training loop:  48%|####8     | 24161/50000 [15:00:08<13:41:43,  1.91s/it]
Outer training loop:  48%|####8     | 24162/50000 [15:00:10<13:16:58,  1.85s/it]
Outer training loop:  48%|####8     | 24163/50000 [15:00:12<13:22:55,  1.86s/it]
Outer training loop:  48%|####8     | 24164/50000 [15:00:13<13:23:50,  1.87s/it]
Outer training loop:  48%|####8     | 24165/50000 [15:00:15<13:36:42,  1.90s/it]
Outer training loop:  48%|####8     | 24166/50000 [15:00:17<13:30:54,  1.88s/it]
Outer training loop:  48%|####8     | 24167/50000 [15:00:19<13:27:46,  1.88s/it]
Outer training loop:  48%|####8     | 24168/50000 [15:00:21<13:35:58,  1.90s/it]
Outer training loop:  48%|####8     | 24169/50000 [15:00:23<13:31:48,  1.89s/it]
Outer training loop:  48%|####8     | 24170/50000 [15:00:25<13:30:01,  1.88s/it]
Outer training loop:  48%|####8     | 24171/50000 [15:00:27<13:52:44,  1.93s/it]
Outer training loop:  48%|####8     | 24172/50000 [15:00:29<13:38:49,  1.90s/it]
Outer training loop:  48%|####8     | 24173/50000 [15:00:30<13:16:13,  1.85s/it]
Outer training loop:  48%|####8     | 24174/50000 [15:00:32<13:03:26,  1.82s/it]
Outer training loop:  48%|####8     | 24175/50000 [15:00:34<13:39:02,  1.90s/it]
Outer training loop:  48%|####8     | 24176/50000 [15:00:36<13:28:39,  1.88s/it]
Outer training loop:  48%|####8     | 24177/50000 [15:00:38<13:33:13,  1.89s/it]
Outer training loop:  48%|####8     | 24178/50000 [15:00:40<13:49:00,  1.93s/it]
Outer training loop:  48%|####8     | 24179/50000 [15:00:42<13:37:44,  1.90s/it]
Outer training loop:  48%|####8     | 24180/50000 [15:00:44<13:17:49,  1.85s/it]
Outer training loop:  48%|####8     | 24181/50000 [15:00:46<13:46:24,  1.92s/it]
Outer training loop:  48%|####8     | 24182/50000 [15:00:47<13:42:31,  1.91s/it]
Outer training loop:  48%|####8     | 24183/50000 [15:00:49<13:31:46,  1.89s/it]
Outer training loop:  48%|####8     | 24184/50000 [15:00:51<13:40:57,  1.91s/it]
Outer training loop:  48%|####8     | 24185/50000 [15:00:53<13:16:42,  1.85s/it]
Outer training loop:  48%|####8     | 24186/50000 [15:00:55<13:24:19,  1.87s/it]
Outer training loop:  48%|####8     | 24187/50000 [15:00:57<13:18:32,  1.86s/it]
Outer training loop:  48%|####8     | 24188/50000 [15:00:59<13:36:24,  1.90s/it]
Outer training loop:  48%|####8     | 24189/50000 [15:01:00<13:12:27,  1.84s/it]
Outer training loop:  48%|####8     | 24190/50000 [15:01:02<13:07:41,  1.83s/it]
Outer training loop:  48%|####8     | 24191/50000 [15:01:04<13:37:53,  1.90s/it]
Outer training loop:  48%|####8     | 24192/50000 [15:01:06<13:19:00,  1.86s/it]
Outer training loop:  48%|####8     | 24193/50000 [15:01:08<13:00:29,  1.81s/it]
Outer training loop:  48%|####8     | 24194/50000 [15:01:10<13:06:33,  1.83s/it]
Outer training loop:  48%|####8     | 24195/50000 [15:01:12<13:30:23,  1.88s/it]
Outer training loop:  48%|####8     | 24196/50000 [15:01:13<12:58:54,  1.81s/it]
Outer training loop:  48%|####8     | 24197/50000 [15:01:15<12:53:01,  1.80s/it]
Outer training loop:  48%|####8     | 24198/50000 [15:01:17<13:26:42,  1.88s/it]
Outer training loop:  48%|####8     | 24199/50000 [15:01:19<13:11:05,  1.84s/it]
Outer training loop:  48%|####8     | 24200/50000 [15:01:21<13:15:08,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  48%|####8     | 24201/50000 [15:01:28<25:08:58,  3.51s/it]
Outer training loop:  48%|####8     | 24202/50000 [15:01:30<21:35:26,  3.01s/it]
Outer training loop:  48%|####8     | 24203/50000 [15:01:32<19:09:07,  2.67s/it]
Outer training loop:  48%|####8     | 24204/50000 [15:01:34<17:39:05,  2.46s/it]
Outer training loop:  48%|####8     | 24205/50000 [15:01:36<16:44:18,  2.34s/it]
Outer training loop:  48%|####8     | 24206/50000 [15:01:38<15:33:35,  2.17s/it]
Outer training loop:  48%|####8     | 24207/50000 [15:01:39<14:45:02,  2.06s/it]
Outer training loop:  48%|####8     | 24208/50000 [15:01:41<13:53:45,  1.94s/it]
Outer training loop:  48%|####8     | 24209/50000 [15:01:43<14:10:57,  1.98s/it]
Outer training loop:  48%|####8     | 24210/50000 [15:01:45<13:51:34,  1.93s/it]
Outer training loop:  48%|####8     | 24211/50000 [15:01:47<13:39:18,  1.91s/it]
Outer training loop:  48%|####8     | 24212/50000 [15:01:49<13:25:46,  1.87s/it]
Outer training loop:  48%|####8     | 24213/50000 [15:01:51<13:39:57,  1.91s/it]
Outer training loop:  48%|####8     | 24214/50000 [15:01:52<13:25:32,  1.87s/it]
Outer training loop:  48%|####8     | 24215/50000 [15:01:54<13:17:47,  1.86s/it]
Outer training loop:  48%|####8     | 24216/50000 [15:01:56<13:33:38,  1.89s/it]
Outer training loop:  48%|####8     | 24217/50000 [15:01:58<13:29:09,  1.88s/it]
Outer training loop:  48%|####8     | 24218/50000 [15:02:00<13:19:34,  1.86s/it]
Outer training loop:  48%|####8     | 24219/50000 [15:02:02<13:18:57,  1.86s/it]
Outer training loop:  48%|####8     | 24220/50000 [15:02:04<13:38:22,  1.90s/it]
Outer training loop:  48%|####8     | 24221/50000 [15:02:06<13:24:29,  1.87s/it]
Outer training loop:  48%|####8     | 24222/50000 [15:02:07<13:24:13,  1.87s/it]
Outer training loop:  48%|####8     | 24223/50000 [15:02:09<13:40:24,  1.91s/it]
Outer training loop:  48%|####8     | 24224/50000 [15:02:11<13:15:46,  1.85s/it]
Outer training loop:  48%|####8     | 24225/50000 [15:02:13<13:17:56,  1.86s/it]
Outer training loop:  48%|####8     | 24226/50000 [15:02:15<13:11:22,  1.84s/it]
Outer training loop:  48%|####8     | 24227/50000 [15:02:17<13:44:58,  1.92s/it]
Outer training loop:  48%|####8     | 24228/50000 [15:02:19<13:39:54,  1.91s/it]
Outer training loop:  48%|####8     | 24229/50000 [15:02:21<13:45:32,  1.92s/it]
Outer training loop:  48%|####8     | 24230/50000 [15:02:23<13:49:49,  1.93s/it]
Outer training loop:  48%|####8     | 24231/50000 [15:02:25<13:31:49,  1.89s/it]
Outer training loop:  48%|####8     | 24232/50000 [15:02:26<13:23:54,  1.87s/it]
Outer training loop:  48%|####8     | 24233/50000 [15:02:28<13:30:18,  1.89s/it]
Outer training loop:  48%|####8     | 24234/50000 [15:02:30<13:35:28,  1.90s/it]
Outer training loop:  48%|####8     | 24235/50000 [15:02:32<13:26:48,  1.88s/it]
Outer training loop:  48%|####8     | 24236/50000 [15:02:34<13:37:54,  1.90s/it]
Outer training loop:  48%|####8     | 24237/50000 [15:02:36<13:33:34,  1.89s/it]
Outer training loop:  48%|####8     | 24238/50000 [15:02:38<13:09:57,  1.84s/it]
Outer training loop:  48%|####8     | 24239/50000 [15:02:39<13:09:18,  1.84s/it]
Outer training loop:  48%|####8     | 24240/50000 [15:02:41<13:21:59,  1.87s/it]
Outer training loop:  48%|####8     | 24241/50000 [15:02:43<13:12:59,  1.85s/it]
Outer training loop:  48%|####8     | 24242/50000 [15:02:45<13:02:13,  1.82s/it]
Outer training loop:  48%|####8     | 24243/50000 [15:02:47<12:58:51,  1.81s/it]
Outer training loop:  48%|####8     | 24244/50000 [15:02:49<13:29:45,  1.89s/it]
Outer training loop:  48%|####8     | 24245/50000 [15:02:51<13:27:47,  1.88s/it]
Outer training loop:  48%|####8     | 24246/50000 [15:02:52<13:18:25,  1.86s/it]
Outer training loop:  48%|####8     | 24247/50000 [15:02:54<13:27:42,  1.88s/it]
Outer training loop:  48%|####8     | 24248/50000 [15:02:56<13:20:05,  1.86s/it]
Outer training loop:  48%|####8     | 24249/50000 [15:02:58<13:22:52,  1.87s/it]
Outer training loop:  48%|####8     | 24250/50000 [15:03:00<13:46:45,  1.93s/it]
Outer training loop:  49%|####8     | 24251/50000 [15:03:02<13:28:17,  1.88s/it]
Outer training loop:  49%|####8     | 24252/50000 [15:03:04<13:13:46,  1.85s/it]
Outer training loop:  49%|####8     | 24253/50000 [15:03:06<13:23:04,  1.87s/it]
Outer training loop:  49%|####8     | 24254/50000 [15:03:08<14:24:27,  2.01s/it]
Outer training loop:  49%|####8     | 24255/50000 [15:03:10<14:00:10,  1.96s/it]
Outer training loop:  49%|####8     | 24256/50000 [15:03:12<13:44:43,  1.92s/it]
Outer training loop:  49%|####8     | 24257/50000 [15:03:14<14:00:59,  1.96s/it]
Outer training loop:  49%|####8     | 24258/50000 [15:03:15<13:40:25,  1.91s/it]
Outer training loop:  49%|####8     | 24259/50000 [15:03:17<13:27:41,  1.88s/it]
Outer training loop:  49%|####8     | 24260/50000 [15:03:19<13:22:03,  1.87s/it]
Outer training loop:  49%|####8     | 24261/50000 [15:03:21<13:33:37,  1.90s/it]
Outer training loop:  49%|####8     | 24262/50000 [15:03:23<13:23:17,  1.87s/it]
Outer training loop:  49%|####8     | 24263/50000 [15:03:25<13:13:48,  1.85s/it]
Outer training loop:  49%|####8     | 24264/50000 [15:03:27<13:28:52,  1.89s/it]
Outer training loop:  49%|####8     | 24265/50000 [15:03:29<13:19:45,  1.86s/it]
Outer training loop:  49%|####8     | 24266/50000 [15:03:30<13:12:43,  1.85s/it]
Outer training loop:  49%|####8     | 24267/50000 [15:03:32<13:27:57,  1.88s/it]
Outer training loop:  49%|####8     | 24268/50000 [15:03:34<13:22:59,  1.87s/it]
Outer training loop:  49%|####8     | 24269/50000 [15:03:36<13:26:51,  1.88s/it]
Outer training loop:  49%|####8     | 24270/50000 [15:03:38<13:55:44,  1.95s/it]
Outer training loop:  49%|####8     | 24271/50000 [15:03:40<13:31:48,  1.89s/it]
Outer training loop:  49%|####8     | 24272/50000 [15:03:42<13:14:51,  1.85s/it]
Outer training loop:  49%|####8     | 24273/50000 [15:03:43<13:07:26,  1.84s/it]
Outer training loop:  49%|####8     | 24274/50000 [15:03:46<13:36:26,  1.90s/it]
Outer training loop:  49%|####8     | 24275/50000 [15:03:47<13:21:31,  1.87s/it]
Outer training loop:  49%|####8     | 24276/50000 [15:03:49<13:21:31,  1.87s/it]
Outer training loop:  49%|####8     | 24277/50000 [15:03:51<13:47:26,  1.93s/it]
Outer training loop:  49%|####8     | 24278/50000 [15:03:53<13:34:30,  1.90s/it]
Outer training loop:  49%|####8     | 24279/50000 [15:03:55<13:29:33,  1.89s/it]
Outer training loop:  49%|####8     | 24280/50000 [15:03:57<13:40:04,  1.91s/it]
Outer training loop:  49%|####8     | 24281/50000 [15:03:59<13:19:30,  1.87s/it]
Outer training loop:  49%|####8     | 24282/50000 [15:04:00<13:08:02,  1.84s/it]
Outer training loop:  49%|####8     | 24283/50000 [15:04:02<13:09:44,  1.84s/it]
Outer training loop:  49%|####8     | 24284/50000 [15:04:04<13:35:37,  1.90s/it]
Outer training loop:  49%|####8     | 24285/50000 [15:04:06<13:30:51,  1.89s/it]
Outer training loop:  49%|####8     | 24286/50000 [15:04:08<13:16:58,  1.86s/it]
Outer training loop:  49%|####8     | 24287/50000 [15:04:10<13:28:25,  1.89s/it]
Outer training loop:  49%|####8     | 24288/50000 [15:04:12<13:22:14,  1.87s/it]
Outer training loop:  49%|####8     | 24289/50000 [15:04:14<13:16:11,  1.86s/it]
Outer training loop:  49%|####8     | 24290/50000 [15:04:16<13:26:55,  1.88s/it]
Outer training loop:  49%|####8     | 24291/50000 [15:04:17<13:16:52,  1.86s/it]
Outer training loop:  49%|####8     | 24292/50000 [15:04:19<13:22:30,  1.87s/it]
Outer training loop:  49%|####8     | 24293/50000 [15:04:21<13:16:01,  1.86s/it]
Outer training loop:  49%|####8     | 24294/50000 [15:04:23<13:44:36,  1.92s/it]
Outer training loop:  49%|####8     | 24295/50000 [15:04:25<13:25:39,  1.88s/it]
Outer training loop:  49%|####8     | 24296/50000 [15:04:27<13:15:11,  1.86s/it]
Outer training loop:  49%|####8     | 24297/50000 [15:04:29<13:22:58,  1.87s/it]
Outer training loop:  49%|####8     | 24298/50000 [15:04:30<13:11:55,  1.85s/it]
Outer training loop:  49%|####8     | 24299/50000 [15:04:32<13:12:08,  1.85s/it]
Outer training loop:  49%|####8     | 24300/50000 [15:04:34<13:30:29,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  49%|####8     | 24301/50000 [15:04:42<26:05:12,  3.65s/it]
Outer training loop:  49%|####8     | 24302/50000 [15:04:44<22:00:04,  3.08s/it]
Outer training loop:  49%|####8     | 24303/50000 [15:04:46<19:13:01,  2.69s/it]
Outer training loop:  49%|####8     | 24304/50000 [15:04:47<17:19:25,  2.43s/it]
Outer training loop:  49%|####8     | 24305/50000 [15:04:49<16:38:35,  2.33s/it]
Outer training loop:  49%|####8     | 24306/50000 [15:04:51<15:35:52,  2.19s/it]
Outer training loop:  49%|####8     | 24307/50000 [15:04:53<15:06:50,  2.12s/it]
Outer training loop:  49%|####8     | 24308/50000 [15:04:55<14:33:21,  2.04s/it]
Outer training loop:  49%|####8     | 24309/50000 [15:04:57<14:41:10,  2.06s/it]
Outer training loop:  49%|####8     | 24310/50000 [15:04:59<14:10:21,  1.99s/it]
Outer training loop:  49%|####8     | 24311/50000 [15:05:01<13:41:56,  1.92s/it]
Outer training loop:  49%|####8     | 24312/50000 [15:05:03<13:48:20,  1.93s/it]
Outer training loop:  49%|####8     | 24313/50000 [15:05:05<13:39:25,  1.91s/it]
Outer training loop:  49%|####8     | 24314/50000 [15:05:06<13:26:57,  1.88s/it]
Outer training loop:  49%|####8     | 24315/50000 [15:05:08<13:37:02,  1.91s/it]
Outer training loop:  49%|####8     | 24316/50000 [15:05:10<13:34:41,  1.90s/it]
Outer training loop:  49%|####8     | 24317/50000 [15:05:12<13:29:54,  1.89s/it]
Outer training loop:  49%|####8     | 24318/50000 [15:05:14<13:27:36,  1.89s/it]
Outer training loop:  49%|####8     | 24319/50000 [15:05:16<13:47:42,  1.93s/it]
Outer training loop:  49%|####8     | 24320/50000 [15:05:18<13:37:01,  1.91s/it]
Outer training loop:  49%|####8     | 24321/50000 [15:05:20<13:33:29,  1.90s/it]
Outer training loop:  49%|####8     | 24322/50000 [15:05:22<13:46:40,  1.93s/it]
Outer training loop:  49%|####8     | 24323/50000 [15:05:24<13:49:15,  1.94s/it]
Outer training loop:  49%|####8     | 24324/50000 [15:05:26<13:42:39,  1.92s/it]
Outer training loop:  49%|####8     | 24325/50000 [15:05:28<14:01:25,  1.97s/it]
Outer training loop:  49%|####8     | 24326/50000 [15:05:29<13:30:41,  1.89s/it]
Outer training loop:  49%|####8     | 24327/50000 [15:05:31<13:23:53,  1.88s/it]
Outer training loop:  49%|####8     | 24328/50000 [15:05:33<13:42:18,  1.92s/it]
Outer training loop:  49%|####8     | 24329/50000 [15:05:35<13:42:37,  1.92s/it]
Outer training loop:  49%|####8     | 24330/50000 [15:05:37<13:13:06,  1.85s/it]
Outer training loop:  49%|####8     | 24331/50000 [15:05:39<12:51:01,  1.80s/it]
Outer training loop:  49%|####8     | 24332/50000 [15:05:41<13:40:39,  1.92s/it]
Outer training loop:  49%|####8     | 24333/50000 [15:05:43<13:25:20,  1.88s/it]
Outer training loop:  49%|####8     | 24334/50000 [15:05:44<12:58:27,  1.82s/it]
Outer training loop:  49%|####8     | 24335/50000 [15:05:46<13:22:35,  1.88s/it]
Outer training loop:  49%|####8     | 24336/50000 [15:05:48<13:12:06,  1.85s/it]
Outer training loop:  49%|####8     | 24337/50000 [15:05:50<13:00:23,  1.82s/it]
Outer training loop:  49%|####8     | 24338/50000 [15:05:52<12:49:33,  1.80s/it]
Outer training loop:  49%|####8     | 24339/50000 [15:05:54<13:08:04,  1.84s/it]
Outer training loop:  49%|####8     | 24340/50000 [15:05:55<13:03:41,  1.83s/it]
Outer training loop:  49%|####8     | 24341/50000 [15:05:57<13:19:31,  1.87s/it]
Outer training loop:  49%|####8     | 24342/50000 [15:05:59<13:27:54,  1.89s/it]
Outer training loop:  49%|####8     | 24343/50000 [15:06:01<13:15:31,  1.86s/it]
Outer training loop:  49%|####8     | 24344/50000 [15:06:03<13:21:34,  1.87s/it]
Outer training loop:  49%|####8     | 24345/50000 [15:06:05<13:46:31,  1.93s/it]
Outer training loop:  49%|####8     | 24346/50000 [15:06:07<13:26:37,  1.89s/it]
Outer training loop:  49%|####8     | 24347/50000 [15:06:09<13:03:01,  1.83s/it]
Outer training loop:  49%|####8     | 24348/50000 [15:06:10<13:10:17,  1.85s/it]
Outer training loop:  49%|####8     | 24349/50000 [15:06:13<13:41:39,  1.92s/it]
Outer training loop:  49%|####8     | 24350/50000 [15:06:14<13:43:43,  1.93s/it]
Outer training loop:  49%|####8     | 24351/50000 [15:06:16<14:00:04,  1.97s/it]
Outer training loop:  49%|####8     | 24352/50000 [15:06:18<13:59:32,  1.96s/it]
Outer training loop:  49%|####8     | 24353/50000 [15:06:20<13:52:46,  1.95s/it]
Outer training loop:  49%|####8     | 24354/50000 [15:06:22<13:41:52,  1.92s/it]
Outer training loop:  49%|####8     | 24355/50000 [15:06:24<14:09:11,  1.99s/it]
Outer training loop:  49%|####8     | 24356/50000 [15:06:26<13:49:52,  1.94s/it]
Outer training loop:  49%|####8     | 24357/50000 [15:06:28<13:41:24,  1.92s/it]
Outer training loop:  49%|####8     | 24358/50000 [15:06:30<13:18:10,  1.87s/it]
Outer training loop:  49%|####8     | 24359/50000 [15:06:32<13:50:31,  1.94s/it]
Outer training loop:  49%|####8     | 24360/50000 [15:06:34<13:35:52,  1.91s/it]
Outer training loop:  49%|####8     | 24361/50000 [15:06:36<13:51:08,  1.95s/it]
Outer training loop:  49%|####8     | 24362/50000 [15:06:38<14:07:25,  1.98s/it]
Outer training loop:  49%|####8     | 24363/50000 [15:06:40<14:02:40,  1.97s/it]
Outer training loop:  49%|####8     | 24364/50000 [15:06:42<13:50:27,  1.94s/it]
Outer training loop:  49%|####8     | 24365/50000 [15:06:44<14:17:38,  2.01s/it]
Outer training loop:  49%|####8     | 24366/50000 [15:06:46<14:11:41,  1.99s/it]
Outer training loop:  49%|####8     | 24367/50000 [15:06:48<13:52:25,  1.95s/it]
Outer training loop:  49%|####8     | 24368/50000 [15:06:50<13:40:19,  1.92s/it]
Outer training loop:  49%|####8     | 24369/50000 [15:06:52<13:58:05,  1.96s/it]
Outer training loop:  49%|####8     | 24370/50000 [15:06:53<13:49:16,  1.94s/it]
Outer training loop:  49%|####8     | 24371/50000 [15:06:55<13:25:58,  1.89s/it]
Outer training loop:  49%|####8     | 24372/50000 [15:06:57<13:20:39,  1.87s/it]
Outer training loop:  49%|####8     | 24373/50000 [15:06:59<13:36:11,  1.91s/it]
Outer training loop:  49%|####8     | 24374/50000 [15:07:01<13:40:17,  1.92s/it]
Outer training loop:  49%|####8     | 24375/50000 [15:07:03<13:13:57,  1.86s/it]
Outer training loop:  49%|####8     | 24376/50000 [15:07:05<13:37:05,  1.91s/it]
Outer training loop:  49%|####8     | 24377/50000 [15:07:06<13:05:49,  1.84s/it]
Outer training loop:  49%|####8     | 24378/50000 [15:07:08<12:58:01,  1.82s/it]
Outer training loop:  49%|####8     | 24379/50000 [15:07:10<13:02:36,  1.83s/it]
Outer training loop:  49%|####8     | 24380/50000 [15:07:12<13:15:45,  1.86s/it]
Outer training loop:  49%|####8     | 24381/50000 [15:07:14<13:24:15,  1.88s/it]
Outer training loop:  49%|####8     | 24382/50000 [15:07:16<13:20:49,  1.88s/it]
Outer training loop:  49%|####8     | 24383/50000 [15:07:18<13:38:51,  1.92s/it]
Outer training loop:  49%|####8     | 24384/50000 [15:07:20<14:02:48,  1.97s/it]
Outer training loop:  49%|####8     | 24385/50000 [15:07:22<13:38:22,  1.92s/it]
Outer training loop:  49%|####8     | 24386/50000 [15:07:24<13:24:50,  1.89s/it]
Outer training loop:  49%|####8     | 24387/50000 [15:07:26<13:40:39,  1.92s/it]
Outer training loop:  49%|####8     | 24388/50000 [15:07:27<13:42:03,  1.93s/it]
Outer training loop:  49%|####8     | 24389/50000 [15:07:29<13:30:13,  1.90s/it]
Outer training loop:  49%|####8     | 24390/50000 [15:07:31<13:41:01,  1.92s/it]
Outer training loop:  49%|####8     | 24391/50000 [15:07:33<13:38:59,  1.92s/it]
Outer training loop:  49%|####8     | 24392/50000 [15:07:35<13:52:32,  1.95s/it]
Outer training loop:  49%|####8     | 24393/50000 [15:07:37<13:47:44,  1.94s/it]
Outer training loop:  49%|####8     | 24394/50000 [15:07:39<14:10:42,  1.99s/it]
Outer training loop:  49%|####8     | 24395/50000 [15:07:41<14:07:37,  1.99s/it]
Outer training loop:  49%|####8     | 24396/50000 [15:07:43<13:44:10,  1.93s/it]
Outer training loop:  49%|####8     | 24397/50000 [15:07:45<13:47:00,  1.94s/it]
Outer training loop:  49%|####8     | 24398/50000 [15:07:47<13:39:54,  1.92s/it]
Outer training loop:  49%|####8     | 24399/50000 [15:07:49<13:46:19,  1.94s/it]
Outer training loop:  49%|####8     | 24400/50000 [15:07:51<14:06:46,  1.98s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.93s/it]

Outer training loop:  49%|####8     | 24401/50000 [15:07:59<26:42:06,  3.76s/it]
Outer training loop:  49%|####8     | 24402/50000 [15:08:01<22:49:34,  3.21s/it]
Outer training loop:  49%|####8     | 24403/50000 [15:08:03<19:57:47,  2.81s/it]
Outer training loop:  49%|####8     | 24404/50000 [15:08:05<18:10:30,  2.56s/it]
Outer training loop:  49%|####8     | 24405/50000 [15:08:07<17:02:52,  2.40s/it]
Outer training loop:  49%|####8     | 24406/50000 [15:08:09<16:00:01,  2.25s/it]
Outer training loop:  49%|####8     | 24407/50000 [15:08:10<15:11:40,  2.14s/it]
Outer training loop:  49%|####8     | 24408/50000 [15:08:13<15:13:37,  2.14s/it]
Outer training loop:  49%|####8     | 24409/50000 [15:08:14<14:46:39,  2.08s/it]
Outer training loop:  49%|####8     | 24410/50000 [15:08:16<14:26:39,  2.03s/it]
Outer training loop:  49%|####8     | 24411/50000 [15:08:18<14:20:58,  2.02s/it]
Outer training loop:  49%|####8     | 24412/50000 [15:08:20<14:16:56,  2.01s/it]
Outer training loop:  49%|####8     | 24413/50000 [15:08:22<14:11:15,  2.00s/it]
Outer training loop:  49%|####8     | 24414/50000 [15:08:24<13:49:51,  1.95s/it]
Outer training loop:  49%|####8     | 24415/50000 [15:08:26<14:06:06,  1.98s/it]
Outer training loop:  49%|####8     | 24416/50000 [15:08:28<13:44:41,  1.93s/it]
Outer training loop:  49%|####8     | 24417/50000 [15:08:30<13:30:47,  1.90s/it]
Outer training loop:  49%|####8     | 24418/50000 [15:08:32<14:10:23,  1.99s/it]
Outer training loop:  49%|####8     | 24419/50000 [15:08:34<13:58:41,  1.97s/it]
Outer training loop:  49%|####8     | 24420/50000 [15:08:36<13:47:53,  1.94s/it]
Outer training loop:  49%|####8     | 24421/50000 [15:08:38<13:55:44,  1.96s/it]
Outer training loop:  49%|####8     | 24422/50000 [15:08:40<13:42:17,  1.93s/it]
Outer training loop:  49%|####8     | 24423/50000 [15:08:41<13:21:07,  1.88s/it]
Outer training loop:  49%|####8     | 24424/50000 [15:08:43<13:06:25,  1.84s/it]
Outer training loop:  49%|####8     | 24425/50000 [15:08:45<13:41:54,  1.93s/it]
Outer training loop:  49%|####8     | 24426/50000 [15:08:47<13:18:53,  1.87s/it]
Outer training loop:  49%|####8     | 24427/50000 [15:08:49<13:08:50,  1.85s/it]
Outer training loop:  49%|####8     | 24428/50000 [15:08:51<13:43:30,  1.93s/it]
Outer training loop:  49%|####8     | 24429/50000 [15:08:53<13:31:24,  1.90s/it]
Outer training loop:  49%|####8     | 24430/50000 [15:08:55<13:32:42,  1.91s/it]
Outer training loop:  49%|####8     | 24431/50000 [15:08:57<13:18:14,  1.87s/it]
Outer training loop:  49%|####8     | 24432/50000 [15:08:59<13:36:34,  1.92s/it]
Outer training loop:  49%|####8     | 24433/50000 [15:09:00<13:26:54,  1.89s/it]
Outer training loop:  49%|####8     | 24434/50000 [15:09:02<13:08:25,  1.85s/it]
Outer training loop:  49%|####8     | 24435/50000 [15:09:04<13:22:25,  1.88s/it]
Outer training loop:  49%|####8     | 24436/50000 [15:09:06<13:15:42,  1.87s/it]
Outer training loop:  49%|####8     | 24437/50000 [15:09:08<13:11:36,  1.86s/it]
Outer training loop:  49%|####8     | 24438/50000 [15:09:10<13:10:06,  1.85s/it]
Outer training loop:  49%|####8     | 24439/50000 [15:09:12<13:21:50,  1.88s/it]
Outer training loop:  49%|####8     | 24440/50000 [15:09:13<13:19:35,  1.88s/it]
Outer training loop:  49%|####8     | 24441/50000 [15:09:15<13:08:30,  1.85s/it]
Outer training loop:  49%|####8     | 24442/50000 [15:09:17<13:05:22,  1.84s/it]
Outer training loop:  49%|####8     | 24443/50000 [15:09:19<13:41:05,  1.93s/it]
Outer training loop:  49%|####8     | 24444/50000 [15:09:21<13:18:23,  1.87s/it]
Outer training loop:  49%|####8     | 24445/50000 [15:09:23<13:20:39,  1.88s/it]
Outer training loop:  49%|####8     | 24446/50000 [15:09:25<13:39:13,  1.92s/it]
Outer training loop:  49%|####8     | 24447/50000 [15:09:27<13:28:35,  1.90s/it]
Outer training loop:  49%|####8     | 24448/50000 [15:09:29<14:01:13,  1.98s/it]
Outer training loop:  49%|####8     | 24449/50000 [15:09:31<14:12:26,  2.00s/it]
Outer training loop:  49%|####8     | 24450/50000 [15:09:33<13:38:40,  1.92s/it]
Outer training loop:  49%|####8     | 24451/50000 [15:09:34<13:22:15,  1.88s/it]
Outer training loop:  49%|####8     | 24452/50000 [15:09:36<13:23:37,  1.89s/it]
Outer training loop:  49%|####8     | 24453/50000 [15:09:38<13:44:49,  1.94s/it]
Outer training loop:  49%|####8     | 24454/50000 [15:09:40<13:41:44,  1.93s/it]
Outer training loop:  49%|####8     | 24455/50000 [15:09:42<13:31:45,  1.91s/it]
Outer training loop:  49%|####8     | 24456/50000 [15:09:44<14:00:21,  1.97s/it]
Outer training loop:  49%|####8     | 24457/50000 [15:09:46<13:53:47,  1.96s/it]
Outer training loop:  49%|####8     | 24458/50000 [15:09:48<13:38:01,  1.92s/it]
Outer training loop:  49%|####8     | 24459/50000 [15:09:50<13:39:16,  1.92s/it]
Outer training loop:  49%|####8     | 24460/50000 [15:09:52<14:06:33,  1.99s/it]
Outer training loop:  49%|####8     | 24461/50000 [15:09:54<14:01:26,  1.98s/it]
Outer training loop:  49%|####8     | 24462/50000 [15:09:56<13:46:32,  1.94s/it]
Outer training loop:  49%|####8     | 24463/50000 [15:09:58<14:27:16,  2.04s/it]
Outer training loop:  49%|####8     | 24464/50000 [15:10:00<13:54:05,  1.96s/it]
Outer training loop:  49%|####8     | 24465/50000 [15:10:02<13:30:39,  1.90s/it]
Outer training loop:  49%|####8     | 24466/50000 [15:10:04<13:32:49,  1.91s/it]
Outer training loop:  49%|####8     | 24467/50000 [15:10:06<13:32:40,  1.91s/it]
Outer training loop:  49%|####8     | 24468/50000 [15:10:07<13:27:35,  1.90s/it]
Outer training loop:  49%|####8     | 24469/50000 [15:10:10<13:53:58,  1.96s/it]
Outer training loop:  49%|####8     | 24470/50000 [15:10:11<13:43:32,  1.94s/it]
Outer training loop:  49%|####8     | 24471/50000 [15:10:13<13:35:18,  1.92s/it]
Outer training loop:  49%|####8     | 24472/50000 [15:10:15<13:27:21,  1.90s/it]
Outer training loop:  49%|####8     | 24473/50000 [15:10:17<13:45:29,  1.94s/it]
Outer training loop:  49%|####8     | 24474/50000 [15:10:19<13:47:02,  1.94s/it]
Outer training loop:  49%|####8     | 24475/50000 [15:10:21<13:46:06,  1.94s/it]
Outer training loop:  49%|####8     | 24476/50000 [15:10:23<14:23:49,  2.03s/it]
Outer training loop:  49%|####8     | 24477/50000 [15:10:25<14:08:28,  1.99s/it]
Outer training loop:  49%|####8     | 24478/50000 [15:10:27<14:14:36,  2.01s/it]
Outer training loop:  49%|####8     | 24479/50000 [15:10:29<14:22:50,  2.03s/it]
Outer training loop:  49%|####8     | 24480/50000 [15:10:31<14:27:39,  2.04s/it]
Outer training loop:  49%|####8     | 24481/50000 [15:10:33<14:14:15,  2.01s/it]
Outer training loop:  49%|####8     | 24482/50000 [15:10:35<14:11:03,  2.00s/it]
Outer training loop:  49%|####8     | 24483/50000 [15:10:37<14:22:11,  2.03s/it]
Outer training loop:  49%|####8     | 24484/50000 [15:10:39<14:23:00,  2.03s/it]
Outer training loop:  49%|####8     | 24485/50000 [15:10:41<14:10:35,  2.00s/it]
Outer training loop:  49%|####8     | 24486/50000 [15:10:44<14:41:48,  2.07s/it]
Outer training loop:  49%|####8     | 24487/50000 [15:10:46<14:17:45,  2.02s/it]
Outer training loop:  49%|####8     | 24488/50000 [15:10:47<14:02:57,  1.98s/it]
Outer training loop:  49%|####8     | 24489/50000 [15:10:49<14:09:52,  2.00s/it]
Outer training loop:  49%|####8     | 24490/50000 [15:10:51<14:07:12,  1.99s/it]
Outer training loop:  49%|####8     | 24491/50000 [15:10:53<14:00:51,  1.98s/it]
Outer training loop:  49%|####8     | 24492/50000 [15:10:55<13:45:44,  1.94s/it]
Outer training loop:  49%|####8     | 24493/50000 [15:10:57<14:19:00,  2.02s/it]
Outer training loop:  49%|####8     | 24494/50000 [15:10:59<13:59:47,  1.98s/it]
Outer training loop:  49%|####8     | 24495/50000 [15:11:01<13:54:39,  1.96s/it]
Outer training loop:  49%|####8     | 24496/50000 [15:11:03<14:00:11,  1.98s/it]
Outer training loop:  49%|####8     | 24497/50000 [15:11:05<13:46:43,  1.94s/it]
Outer training loop:  49%|####8     | 24498/50000 [15:11:07<14:12:34,  2.01s/it]
Outer training loop:  49%|####8     | 24499/50000 [15:11:09<14:16:47,  2.02s/it]
Outer training loop:  49%|####9     | 24500/50000 [15:11:11<14:01:30,  1.98s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.07s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:02,  2.02s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.96s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:40,  1.95s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:39,  1.97s/it]

Evaluating policy:   4%|3         | 3/83 [00:06<02:40,  2.00s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:32,  1.93s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:30,  1.93s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:33,  1.99s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:29,  1.97s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:26,  1.96s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:27,  2.00s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:27,  2.02s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:24,  2.01s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:22,  2.01s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:16,  1.95s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:18,  2.01s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:12,  1.95s/it]

Evaluating policy:  19%|#9        | 16/83 [00:31<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:33<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:35<02:07,  1.97s/it]

Evaluating policy:  23%|##2       | 19/83 [00:37<02:03,  1.94s/it]

Evaluating policy:  24%|##4       | 20/83 [00:39<02:02,  1.94s/it]

Evaluating policy:  25%|##5       | 21/83 [00:41<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:54,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:45<01:58,  1.98s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:55,  1.96s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:52,  1.93s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:48,  1.93s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:51,  2.03s/it]

Evaluating policy:  35%|###4      | 29/83 [00:56<01:45,  1.95s/it]

Evaluating policy:  36%|###6      | 30/83 [00:58<01:43,  1.95s/it]

Evaluating policy:  37%|###7      | 31/83 [01:00<01:39,  1.92s/it]

Evaluating policy:  39%|###8      | 32/83 [01:02<01:36,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:04<01:33,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:06<01:35,  1.95s/it]

Evaluating policy:  42%|####2     | 35/83 [01:08<01:32,  1.92s/it]

Evaluating policy:  43%|####3     | 36/83 [01:10<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:25,  1.86s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:22,  1.88s/it]

Evaluating policy:  48%|####8     | 40/83 [01:17<01:23,  1.95s/it]

Evaluating policy:  49%|####9     | 41/83 [01:19<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:21<01:17,  1.90s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:23<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:25<01:14,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:10,  1.87s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:29<01:11,  1.94s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:08,  1.91s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:06,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:04,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:36<01:02,  1.88s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:40<00:58,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:55,  1.84s/it]

Evaluating policy:  65%|######5   | 54/83 [01:44<00:56,  1.95s/it]

Evaluating policy:  66%|######6   | 55/83 [01:46<00:54,  1.94s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:51,  1.91s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:49,  1.91s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:42,  1.86s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:41,  1.87s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:41,  2.00s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:01<00:39,  1.95s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:03<00:35,  1.89s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:33,  1.87s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:06<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:28,  1.88s/it]

Evaluating policy:  83%|########3 | 69/83 [02:12<00:25,  1.83s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:26,  2.02s/it]

Evaluating policy:  86%|########5 | 71/83 [02:16<00:23,  1.96s/it]

Evaluating policy:  87%|########6 | 72/83 [02:18<00:20,  1.89s/it]

Evaluating policy:  88%|########7 | 73/83 [02:20<00:18,  1.86s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:16,  1.88s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:15,  1.90s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:29<00:09,  1.87s/it]

Evaluating policy:  95%|#########5| 79/83 [02:32<00:08,  2.08s/it]

Evaluating policy:  96%|#########6| 80/83 [02:33<00:06,  2.02s/it]

Evaluating policy:  98%|#########7| 81/83 [02:35<00:03,  1.96s/it]

Evaluating policy:  99%|#########8| 82/83 [02:37<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 83/83 [02:39<00:00,  1.86s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:18,  1.69s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:19,  1.72s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:17,  1.72s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:20,  1.77s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.76s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:33,  1.99s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:28,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:19,  1.89s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:20,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:23,  2.00s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:19,  1.96s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:16,  1.98s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.94s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:05,  1.97s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.93s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:59,  1.99s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:54,  1.94s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.91s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.90s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.88s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:42,  1.86s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:44,  1.93s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.85s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:32,  1.85s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:35,  1.94s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:33,  1.94s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.92s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:23,  1.86s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:23,  1.95s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.89s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:14,  2.01s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:11,  1.99s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.89s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:56,  1.83s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:54,  1.82s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:56,  1.93s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:53,  1.90s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:44,  1.87s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:45,  2.06s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:42,  2.00s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:39,  1.95s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.92s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.87s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.85s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:25,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:25,  1.99s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:23,  1.93s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.83s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:12,  1.82s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.84s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.80s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.88s/it]

Outer training loop:  49%|####9     | 24501/50000 [15:16:36<699:41:33, 98.78s/it]
Outer training loop:  49%|####9     | 24502/50000 [15:16:38<493:39:31, 69.70s/it]
Outer training loop:  49%|####9     | 24503/50000 [15:16:40<349:39:31, 49.37s/it]
Outer training loop:  49%|####9     | 24504/50000 [15:16:42<248:45:28, 35.12s/it]
Outer training loop:  49%|####9     | 24505/50000 [15:16:43<178:07:57, 25.15s/it]
Outer training loop:  49%|####9     | 24506/50000 [15:16:45<128:36:44, 18.16s/it]
Outer training loop:  49%|####9     | 24507/50000 [15:16:48<95:12:05, 13.44s/it] 
Outer training loop:  49%|####9     | 24508/50000 [15:16:50<70:39:48,  9.98s/it]
Outer training loop:  49%|####9     | 24509/50000 [15:16:51<53:14:15,  7.52s/it]
Outer training loop:  49%|####9     | 24510/50000 [15:16:53<41:25:37,  5.85s/it]
Outer training loop:  49%|####9     | 24511/50000 [15:16:55<32:59:31,  4.66s/it]
Outer training loop:  49%|####9     | 24512/50000 [15:16:57<27:03:19,  3.82s/it]
Outer training loop:  49%|####9     | 24513/50000 [15:16:59<22:47:44,  3.22s/it]
Outer training loop:  49%|####9     | 24514/50000 [15:17:01<19:55:07,  2.81s/it]
Outer training loop:  49%|####9     | 24515/50000 [15:17:03<17:34:07,  2.48s/it]
Outer training loop:  49%|####9     | 24516/50000 [15:17:04<16:13:38,  2.29s/it]
Outer training loop:  49%|####9     | 24517/50000 [15:17:06<15:41:30,  2.22s/it]
Outer training loop:  49%|####9     | 24518/50000 [15:17:08<14:43:26,  2.08s/it]
Outer training loop:  49%|####9     | 24519/50000 [15:17:10<14:18:45,  2.02s/it]
Outer training loop:  49%|####9     | 24520/50000 [15:17:12<14:17:53,  2.02s/it]
Outer training loop:  49%|####9     | 24521/50000 [15:17:14<14:03:19,  1.99s/it]
Outer training loop:  49%|####9     | 24522/50000 [15:17:16<13:41:10,  1.93s/it]
Outer training loop:  49%|####9     | 24523/50000 [15:17:18<13:28:38,  1.90s/it]
Outer training loop:  49%|####9     | 24524/50000 [15:17:20<13:35:36,  1.92s/it]
Outer training loop:  49%|####9     | 24525/50000 [15:17:21<13:20:38,  1.89s/it]
Outer training loop:  49%|####9     | 24526/50000 [15:17:23<13:22:12,  1.89s/it]
Outer training loop:  49%|####9     | 24527/50000 [15:17:25<13:29:02,  1.91s/it]
Outer training loop:  49%|####9     | 24528/50000 [15:17:27<13:13:30,  1.87s/it]
Outer training loop:  49%|####9     | 24529/50000 [15:17:29<13:33:44,  1.92s/it]
Outer training loop:  49%|####9     | 24530/50000 [15:17:31<13:36:31,  1.92s/it]
Outer training loop:  49%|####9     | 24531/50000 [15:17:33<13:09:21,  1.86s/it]
Outer training loop:  49%|####9     | 24532/50000 [15:17:35<13:08:38,  1.86s/it]
Outer training loop:  49%|####9     | 24533/50000 [15:17:36<12:48:45,  1.81s/it]
Outer training loop:  49%|####9     | 24534/50000 [15:17:38<13:19:46,  1.88s/it]
Outer training loop:  49%|####9     | 24535/50000 [15:17:40<12:53:10,  1.82s/it]
Outer training loop:  49%|####9     | 24536/50000 [15:17:42<12:54:31,  1.82s/it]
Outer training loop:  49%|####9     | 24537/50000 [15:17:44<13:35:09,  1.92s/it]
Outer training loop:  49%|####9     | 24538/50000 [15:17:46<13:11:27,  1.87s/it]
Outer training loop:  49%|####9     | 24539/50000 [15:17:47<13:02:46,  1.84s/it]
Outer training loop:  49%|####9     | 24540/50000 [15:17:49<13:24:10,  1.90s/it]
Outer training loop:  49%|####9     | 24541/50000 [15:17:51<12:56:46,  1.83s/it]
Outer training loop:  49%|####9     | 24542/50000 [15:17:53<12:40:19,  1.79s/it]
Outer training loop:  49%|####9     | 24543/50000 [15:17:55<12:56:33,  1.83s/it]
Outer training loop:  49%|####9     | 24544/50000 [15:17:57<13:23:41,  1.89s/it]
Outer training loop:  49%|####9     | 24545/50000 [15:17:59<13:08:06,  1.86s/it]
Outer training loop:  49%|####9     | 24546/50000 [15:18:00<12:52:26,  1.82s/it]
Outer training loop:  49%|####9     | 24547/50000 [15:18:02<13:22:46,  1.89s/it]
Outer training loop:  49%|####9     | 24548/50000 [15:18:04<13:14:23,  1.87s/it]
Outer training loop:  49%|####9     | 24549/50000 [15:18:06<12:51:36,  1.82s/it]
Outer training loop:  49%|####9     | 24550/50000 [15:18:08<13:25:38,  1.90s/it]
Outer training loop:  49%|####9     | 24551/50000 [15:18:10<13:26:43,  1.90s/it]
Outer training loop:  49%|####9     | 24552/50000 [15:18:12<13:22:16,  1.89s/it]
Outer training loop:  49%|####9     | 24553/50000 [15:18:14<13:16:36,  1.88s/it]
Outer training loop:  49%|####9     | 24554/50000 [15:18:16<13:20:14,  1.89s/it]
Outer training loop:  49%|####9     | 24555/50000 [15:18:17<13:20:51,  1.89s/it]
Outer training loop:  49%|####9     | 24556/50000 [15:18:19<13:02:05,  1.84s/it]
Outer training loop:  49%|####9     | 24557/50000 [15:18:21<13:09:35,  1.86s/it]
Outer training loop:  49%|####9     | 24558/50000 [15:18:23<13:00:29,  1.84s/it]
Outer training loop:  49%|####9     | 24559/50000 [15:18:25<12:47:01,  1.81s/it]
Outer training loop:  49%|####9     | 24560/50000 [15:18:26<12:48:18,  1.81s/it]
Outer training loop:  49%|####9     | 24561/50000 [15:18:28<13:18:09,  1.88s/it]
Outer training loop:  49%|####9     | 24562/50000 [15:18:30<12:54:33,  1.83s/it]
Outer training loop:  49%|####9     | 24563/50000 [15:18:32<12:37:18,  1.79s/it]
Outer training loop:  49%|####9     | 24564/50000 [15:18:34<13:06:23,  1.85s/it]
Outer training loop:  49%|####9     | 24565/50000 [15:18:36<13:08:14,  1.86s/it]
Outer training loop:  49%|####9     | 24566/50000 [15:18:38<13:00:43,  1.84s/it]
Outer training loop:  49%|####9     | 24567/50000 [15:18:40<13:34:13,  1.92s/it]
Outer training loop:  49%|####9     | 24568/50000 [15:18:41<13:14:02,  1.87s/it]
Outer training loop:  49%|####9     | 24569/50000 [15:18:43<13:03:33,  1.85s/it]
Outer training loop:  49%|####9     | 24570/50000 [15:18:45<12:45:37,  1.81s/it]
Outer training loop:  49%|####9     | 24571/50000 [15:18:47<12:54:48,  1.83s/it]
Outer training loop:  49%|####9     | 24572/50000 [15:18:49<12:40:52,  1.80s/it]
Outer training loop:  49%|####9     | 24573/50000 [15:18:50<12:43:31,  1.80s/it]
Outer training loop:  49%|####9     | 24574/50000 [15:18:52<12:47:46,  1.81s/it]
Outer training loop:  49%|####9     | 24575/50000 [15:18:54<12:40:03,  1.79s/it]
Outer training loop:  49%|####9     | 24576/50000 [15:18:56<12:42:05,  1.80s/it]
Outer training loop:  49%|####9     | 24577/50000 [15:18:57<12:39:31,  1.79s/it]
Outer training loop:  49%|####9     | 24578/50000 [15:18:59<13:00:02,  1.84s/it]
Outer training loop:  49%|####9     | 24579/50000 [15:19:01<13:12:10,  1.87s/it]
Outer training loop:  49%|####9     | 24580/50000 [15:19:03<12:55:04,  1.83s/it]
Outer training loop:  49%|####9     | 24581/50000 [15:19:05<13:36:35,  1.93s/it]
Outer training loop:  49%|####9     | 24582/50000 [15:19:07<13:16:40,  1.88s/it]
Outer training loop:  49%|####9     | 24583/50000 [15:19:09<12:58:47,  1.84s/it]
Outer training loop:  49%|####9     | 24584/50000 [15:19:11<13:18:54,  1.89s/it]
Outer training loop:  49%|####9     | 24585/50000 [15:19:13<13:51:14,  1.96s/it]
Outer training loop:  49%|####9     | 24586/50000 [15:19:15<13:39:04,  1.93s/it]
Outer training loop:  49%|####9     | 24587/50000 [15:19:17<13:20:20,  1.89s/it]
Outer training loop:  49%|####9     | 24588/50000 [15:19:19<13:28:16,  1.91s/it]
Outer training loop:  49%|####9     | 24589/50000 [15:19:20<13:26:20,  1.90s/it]
Outer training loop:  49%|####9     | 24590/50000 [15:19:22<12:59:57,  1.84s/it]
Outer training loop:  49%|####9     | 24591/50000 [15:19:24<13:23:25,  1.90s/it]
Outer training loop:  49%|####9     | 24592/50000 [15:19:26<13:33:19,  1.92s/it]
Outer training loop:  49%|####9     | 24593/50000 [15:19:28<13:30:15,  1.91s/it]
Outer training loop:  49%|####9     | 24594/50000 [15:19:30<13:05:26,  1.85s/it]
Outer training loop:  49%|####9     | 24595/50000 [15:19:32<13:23:17,  1.90s/it]
Outer training loop:  49%|####9     | 24596/50000 [15:19:34<13:10:07,  1.87s/it]
Outer training loop:  49%|####9     | 24597/50000 [15:19:35<13:16:36,  1.88s/it]
Outer training loop:  49%|####9     | 24598/50000 [15:19:37<13:30:17,  1.91s/it]
Outer training loop:  49%|####9     | 24599/50000 [15:19:39<13:23:32,  1.90s/it]
Outer training loop:  49%|####9     | 24600/50000 [15:19:41<13:07:59,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  49%|####9     | 24601/50000 [15:19:48<24:28:07,  3.47s/it]
Outer training loop:  49%|####9     | 24602/50000 [15:19:50<21:07:42,  2.99s/it]
Outer training loop:  49%|####9     | 24603/50000 [15:19:52<18:39:37,  2.65s/it]
Outer training loop:  49%|####9     | 24604/50000 [15:19:54<16:46:19,  2.38s/it]
Outer training loop:  49%|####9     | 24605/50000 [15:19:56<15:56:27,  2.26s/it]
Outer training loop:  49%|####9     | 24606/50000 [15:19:58<14:55:00,  2.11s/it]
Outer training loop:  49%|####9     | 24607/50000 [15:19:59<14:06:22,  2.00s/it]
Outer training loop:  49%|####9     | 24608/50000 [15:20:01<13:33:57,  1.92s/it]
Outer training loop:  49%|####9     | 24609/50000 [15:20:03<13:40:06,  1.94s/it]
Outer training loop:  49%|####9     | 24610/50000 [15:20:05<13:30:24,  1.92s/it]
Outer training loop:  49%|####9     | 24611/50000 [15:20:07<13:21:48,  1.89s/it]
Outer training loop:  49%|####9     | 24612/50000 [15:20:09<13:27:11,  1.91s/it]
Outer training loop:  49%|####9     | 24613/50000 [15:20:10<13:18:49,  1.89s/it]
Outer training loop:  49%|####9     | 24614/50000 [15:20:12<13:12:36,  1.87s/it]
Outer training loop:  49%|####9     | 24615/50000 [15:20:14<12:54:28,  1.83s/it]
Outer training loop:  49%|####9     | 24616/50000 [15:20:16<13:05:09,  1.86s/it]
Outer training loop:  49%|####9     | 24617/50000 [15:20:18<13:03:55,  1.85s/it]
Outer training loop:  49%|####9     | 24618/50000 [15:20:20<13:07:44,  1.86s/it]
Outer training loop:  49%|####9     | 24619/50000 [15:20:21<12:44:28,  1.81s/it]
Outer training loop:  49%|####9     | 24620/50000 [15:20:23<13:04:46,  1.86s/it]
Outer training loop:  49%|####9     | 24621/50000 [15:20:25<12:58:46,  1.84s/it]
Outer training loop:  49%|####9     | 24622/50000 [15:20:27<12:58:03,  1.84s/it]
Outer training loop:  49%|####9     | 24623/50000 [15:20:29<13:08:00,  1.86s/it]
Outer training loop:  49%|####9     | 24624/50000 [15:20:31<13:06:32,  1.86s/it]
Outer training loop:  49%|####9     | 24625/50000 [15:20:33<13:06:58,  1.86s/it]
Outer training loop:  49%|####9     | 24626/50000 [15:20:35<13:55:17,  1.98s/it]
Outer training loop:  49%|####9     | 24627/50000 [15:20:37<13:44:45,  1.95s/it]
Outer training loop:  49%|####9     | 24628/50000 [15:20:39<13:23:55,  1.90s/it]
Outer training loop:  49%|####9     | 24629/50000 [15:20:40<13:13:43,  1.88s/it]
Outer training loop:  49%|####9     | 24630/50000 [15:20:42<13:26:17,  1.91s/it]
Outer training loop:  49%|####9     | 24631/50000 [15:20:44<13:23:02,  1.90s/it]
Outer training loop:  49%|####9     | 24632/50000 [15:20:46<13:20:15,  1.89s/it]
Outer training loop:  49%|####9     | 24633/50000 [15:20:48<13:25:27,  1.91s/it]
Outer training loop:  49%|####9     | 24634/50000 [15:20:50<13:26:59,  1.91s/it]
Outer training loop:  49%|####9     | 24635/50000 [15:20:52<13:05:28,  1.86s/it]
Outer training loop:  49%|####9     | 24636/50000 [15:20:53<12:55:27,  1.83s/it]
Outer training loop:  49%|####9     | 24637/50000 [15:20:56<13:24:01,  1.90s/it]
Outer training loop:  49%|####9     | 24638/50000 [15:20:57<13:14:37,  1.88s/it]
Outer training loop:  49%|####9     | 24639/50000 [15:20:59<13:09:05,  1.87s/it]
Outer training loop:  49%|####9     | 24640/50000 [15:21:01<13:37:13,  1.93s/it]
Outer training loop:  49%|####9     | 24641/50000 [15:21:03<13:02:16,  1.85s/it]
Outer training loop:  49%|####9     | 24642/50000 [15:21:05<13:13:02,  1.88s/it]
Outer training loop:  49%|####9     | 24643/50000 [15:21:07<13:28:07,  1.91s/it]
Outer training loop:  49%|####9     | 24644/50000 [15:21:09<13:20:14,  1.89s/it]
Outer training loop:  49%|####9     | 24645/50000 [15:21:10<12:59:47,  1.85s/it]
Outer training loop:  49%|####9     | 24646/50000 [15:21:12<13:00:02,  1.85s/it]
Outer training loop:  49%|####9     | 24647/50000 [15:21:14<13:13:35,  1.88s/it]
Outer training loop:  49%|####9     | 24648/50000 [15:21:16<13:20:16,  1.89s/it]
Outer training loop:  49%|####9     | 24649/50000 [15:21:18<13:18:46,  1.89s/it]
Outer training loop:  49%|####9     | 24650/50000 [15:21:20<13:40:32,  1.94s/it]
Outer training loop:  49%|####9     | 24651/50000 [15:21:22<13:30:21,  1.92s/it]
Outer training loop:  49%|####9     | 24652/50000 [15:21:24<13:16:15,  1.88s/it]
Outer training loop:  49%|####9     | 24653/50000 [15:21:26<13:56:07,  1.98s/it]
Outer training loop:  49%|####9     | 24654/50000 [15:21:28<13:37:26,  1.94s/it]
Outer training loop:  49%|####9     | 24655/50000 [15:21:30<13:18:26,  1.89s/it]
Outer training loop:  49%|####9     | 24656/50000 [15:21:31<13:17:00,  1.89s/it]
Outer training loop:  49%|####9     | 24657/50000 [15:21:34<13:35:44,  1.93s/it]
Outer training loop:  49%|####9     | 24658/50000 [15:21:35<13:30:19,  1.92s/it]
Outer training loop:  49%|####9     | 24659/50000 [15:21:37<13:08:58,  1.87s/it]
Outer training loop:  49%|####9     | 24660/50000 [15:21:39<13:29:07,  1.92s/it]
Outer training loop:  49%|####9     | 24661/50000 [15:21:41<13:07:55,  1.87s/it]
Outer training loop:  49%|####9     | 24662/50000 [15:21:43<13:15:05,  1.88s/it]
Outer training loop:  49%|####9     | 24663/50000 [15:21:45<12:56:49,  1.84s/it]
Outer training loop:  49%|####9     | 24664/50000 [15:21:47<13:19:26,  1.89s/it]
Outer training loop:  49%|####9     | 24665/50000 [15:21:48<13:03:34,  1.86s/it]
Outer training loop:  49%|####9     | 24666/50000 [15:21:50<12:59:27,  1.85s/it]
Outer training loop:  49%|####9     | 24667/50000 [15:21:52<13:29:38,  1.92s/it]
Outer training loop:  49%|####9     | 24668/50000 [15:21:54<13:15:22,  1.88s/it]
Outer training loop:  49%|####9     | 24669/50000 [15:21:56<12:59:47,  1.85s/it]
Outer training loop:  49%|####9     | 24670/50000 [15:21:58<13:09:11,  1.87s/it]
Outer training loop:  49%|####9     | 24671/50000 [15:22:00<13:40:09,  1.94s/it]
Outer training loop:  49%|####9     | 24672/50000 [15:22:02<13:26:43,  1.91s/it]
Outer training loop:  49%|####9     | 24673/50000 [15:22:04<13:13:11,  1.88s/it]
Outer training loop:  49%|####9     | 24674/50000 [15:22:06<14:01:10,  1.99s/it]
Outer training loop:  49%|####9     | 24675/50000 [15:22:08<13:38:22,  1.94s/it]
Outer training loop:  49%|####9     | 24676/50000 [15:22:10<13:55:34,  1.98s/it]
Outer training loop:  49%|####9     | 24677/50000 [15:22:12<13:41:30,  1.95s/it]
Outer training loop:  49%|####9     | 24678/50000 [15:22:13<13:39:28,  1.94s/it]
Outer training loop:  49%|####9     | 24679/50000 [15:22:15<13:24:10,  1.91s/it]
Outer training loop:  49%|####9     | 24680/50000 [15:22:17<13:26:41,  1.91s/it]
Outer training loop:  49%|####9     | 24681/50000 [15:22:19<13:54:27,  1.98s/it]
Outer training loop:  49%|####9     | 24682/50000 [15:22:21<13:32:23,  1.93s/it]
Outer training loop:  49%|####9     | 24683/50000 [15:22:23<12:59:14,  1.85s/it]
Outer training loop:  49%|####9     | 24684/50000 [15:22:25<13:13:22,  1.88s/it]
Outer training loop:  49%|####9     | 24685/50000 [15:22:27<13:39:43,  1.94s/it]
Outer training loop:  49%|####9     | 24686/50000 [15:22:29<13:40:16,  1.94s/it]
Outer training loop:  49%|####9     | 24687/50000 [15:22:31<13:32:31,  1.93s/it]
Outer training loop:  49%|####9     | 24688/50000 [15:22:33<13:42:55,  1.95s/it]
Outer training loop:  49%|####9     | 24689/50000 [15:22:35<13:27:07,  1.91s/it]
Outer training loop:  49%|####9     | 24690/50000 [15:22:36<13:13:15,  1.88s/it]
Outer training loop:  49%|####9     | 24691/50000 [15:22:38<13:24:13,  1.91s/it]
Outer training loop:  49%|####9     | 24692/50000 [15:22:40<13:29:57,  1.92s/it]
Outer training loop:  49%|####9     | 24693/50000 [15:22:42<13:15:36,  1.89s/it]
Outer training loop:  49%|####9     | 24694/50000 [15:22:44<13:22:25,  1.90s/it]
Outer training loop:  49%|####9     | 24695/50000 [15:22:46<13:37:46,  1.94s/it]
Outer training loop:  49%|####9     | 24696/50000 [15:22:48<13:21:53,  1.90s/it]
Outer training loop:  49%|####9     | 24697/50000 [15:22:50<13:20:47,  1.90s/it]
Outer training loop:  49%|####9     | 24698/50000 [15:22:52<13:23:07,  1.90s/it]
Outer training loop:  49%|####9     | 24699/50000 [15:22:53<13:05:41,  1.86s/it]
Outer training loop:  49%|####9     | 24700/50000 [15:22:55<13:02:23,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.99s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  49%|####9     | 24701/50000 [15:23:03<24:28:38,  3.48s/it]
Outer training loop:  49%|####9     | 24702/50000 [15:23:05<21:31:09,  3.06s/it]
Outer training loop:  49%|####9     | 24703/50000 [15:23:07<19:04:48,  2.72s/it]
Outer training loop:  49%|####9     | 24704/50000 [15:23:08<17:05:38,  2.43s/it]
Outer training loop:  49%|####9     | 24705/50000 [15:23:10<15:41:05,  2.23s/it]
Outer training loop:  49%|####9     | 24706/50000 [15:23:12<15:05:26,  2.15s/it]
Outer training loop:  49%|####9     | 24707/50000 [15:23:14<14:28:35,  2.06s/it]
Outer training loop:  49%|####9     | 24708/50000 [15:23:16<13:52:19,  1.97s/it]
Outer training loop:  49%|####9     | 24709/50000 [15:23:18<13:53:34,  1.98s/it]
Outer training loop:  49%|####9     | 24710/50000 [15:23:20<13:44:30,  1.96s/it]
Outer training loop:  49%|####9     | 24711/50000 [15:23:21<13:28:38,  1.92s/it]
Outer training loop:  49%|####9     | 24712/50000 [15:23:23<13:20:22,  1.90s/it]
Outer training loop:  49%|####9     | 24713/50000 [15:23:25<13:39:23,  1.94s/it]
Outer training loop:  49%|####9     | 24714/50000 [15:23:27<13:31:20,  1.93s/it]
Outer training loop:  49%|####9     | 24715/50000 [15:23:29<13:21:08,  1.90s/it]
Outer training loop:  49%|####9     | 24716/50000 [15:23:31<13:35:22,  1.93s/it]
Outer training loop:  49%|####9     | 24717/50000 [15:23:33<13:10:43,  1.88s/it]
Outer training loop:  49%|####9     | 24718/50000 [15:23:35<13:04:44,  1.86s/it]
Outer training loop:  49%|####9     | 24719/50000 [15:23:36<13:00:32,  1.85s/it]
Outer training loop:  49%|####9     | 24720/50000 [15:23:38<13:20:17,  1.90s/it]
Outer training loop:  49%|####9     | 24721/50000 [15:23:40<13:18:03,  1.89s/it]
Outer training loop:  49%|####9     | 24722/50000 [15:23:42<13:15:18,  1.89s/it]
Outer training loop:  49%|####9     | 24723/50000 [15:23:44<13:55:01,  1.98s/it]
Outer training loop:  49%|####9     | 24724/50000 [15:23:46<13:33:56,  1.93s/it]
Outer training loop:  49%|####9     | 24725/50000 [15:23:48<13:27:16,  1.92s/it]
Outer training loop:  49%|####9     | 24726/50000 [15:23:50<13:45:04,  1.96s/it]
Outer training loop:  49%|####9     | 24727/50000 [15:23:52<13:27:29,  1.92s/it]
Outer training loop:  49%|####9     | 24728/50000 [15:23:54<13:03:35,  1.86s/it]
Outer training loop:  49%|####9     | 24729/50000 [15:23:56<13:02:25,  1.86s/it]
Outer training loop:  49%|####9     | 24730/50000 [15:23:58<13:21:04,  1.90s/it]
Outer training loop:  49%|####9     | 24731/50000 [15:23:59<12:56:32,  1.84s/it]
Outer training loop:  49%|####9     | 24732/50000 [15:24:01<12:48:28,  1.82s/it]
Outer training loop:  49%|####9     | 24733/50000 [15:24:03<13:08:32,  1.87s/it]
Outer training loop:  49%|####9     | 24734/50000 [15:24:05<13:13:29,  1.88s/it]
Outer training loop:  49%|####9     | 24735/50000 [15:24:07<12:57:34,  1.85s/it]
Outer training loop:  49%|####9     | 24736/50000 [15:24:09<13:26:56,  1.92s/it]
Outer training loop:  49%|####9     | 24737/50000 [15:24:11<13:08:07,  1.87s/it]
Outer training loop:  49%|####9     | 24738/50000 [15:24:12<12:57:49,  1.85s/it]
Outer training loop:  49%|####9     | 24739/50000 [15:24:14<12:52:51,  1.84s/it]
Outer training loop:  49%|####9     | 24740/50000 [15:24:16<13:09:12,  1.87s/it]
Outer training loop:  49%|####9     | 24741/50000 [15:24:18<13:00:47,  1.85s/it]
Outer training loop:  49%|####9     | 24742/50000 [15:24:20<12:42:40,  1.81s/it]
Outer training loop:  49%|####9     | 24743/50000 [15:24:22<13:09:54,  1.88s/it]
Outer training loop:  49%|####9     | 24744/50000 [15:24:23<12:50:00,  1.83s/it]
Outer training loop:  49%|####9     | 24745/50000 [15:24:25<13:06:02,  1.87s/it]
Outer training loop:  49%|####9     | 24746/50000 [15:24:27<12:57:53,  1.85s/it]
Outer training loop:  49%|####9     | 24747/50000 [15:24:29<13:12:25,  1.88s/it]
Outer training loop:  49%|####9     | 24748/50000 [15:24:31<13:10:43,  1.88s/it]
Outer training loop:  49%|####9     | 24749/50000 [15:24:33<12:47:33,  1.82s/it]
Outer training loop:  50%|####9     | 24750/50000 [15:24:35<12:57:07,  1.85s/it]
Outer training loop:  50%|####9     | 24751/50000 [15:24:36<12:51:42,  1.83s/it]
Outer training loop:  50%|####9     | 24752/50000 [15:24:38<12:39:29,  1.80s/it]
Outer training loop:  50%|####9     | 24753/50000 [15:24:40<13:05:41,  1.87s/it]
Outer training loop:  50%|####9     | 24754/50000 [15:24:42<12:49:48,  1.83s/it]
Outer training loop:  50%|####9     | 24755/50000 [15:24:44<12:50:20,  1.83s/it]
Outer training loop:  50%|####9     | 24756/50000 [15:24:45<12:46:44,  1.82s/it]
Outer training loop:  50%|####9     | 24757/50000 [15:24:47<12:59:38,  1.85s/it]
Outer training loop:  50%|####9     | 24758/50000 [15:24:49<12:51:30,  1.83s/it]
Outer training loop:  50%|####9     | 24759/50000 [15:24:51<12:48:11,  1.83s/it]
Outer training loop:  50%|####9     | 24760/50000 [15:24:53<12:34:22,  1.79s/it]
Outer training loop:  50%|####9     | 24761/50000 [15:24:55<13:05:46,  1.87s/it]
Outer training loop:  50%|####9     | 24762/50000 [15:24:57<12:54:38,  1.84s/it]
Outer training loop:  50%|####9     | 24763/50000 [15:24:58<13:02:42,  1.86s/it]
Outer training loop:  50%|####9     | 24764/50000 [15:25:00<13:11:06,  1.88s/it]
Outer training loop:  50%|####9     | 24765/50000 [15:25:02<13:00:53,  1.86s/it]
Outer training loop:  50%|####9     | 24766/50000 [15:25:04<13:11:03,  1.88s/it]
Outer training loop:  50%|####9     | 24767/50000 [15:25:06<13:25:00,  1.91s/it]
Outer training loop:  50%|####9     | 24768/50000 [15:25:08<13:16:46,  1.89s/it]
Outer training loop:  50%|####9     | 24769/50000 [15:25:10<13:00:02,  1.85s/it]
Outer training loop:  50%|####9     | 24770/50000 [15:25:12<12:56:56,  1.85s/it]
Outer training loop:  50%|####9     | 24771/50000 [15:25:14<13:33:35,  1.93s/it]
Outer training loop:  50%|####9     | 24772/50000 [15:25:15<13:15:22,  1.89s/it]
Outer training loop:  50%|####9     | 24773/50000 [15:25:17<13:22:50,  1.91s/it]
Outer training loop:  50%|####9     | 24774/50000 [15:25:19<13:31:10,  1.93s/it]
Outer training loop:  50%|####9     | 24775/50000 [15:25:21<13:28:43,  1.92s/it]
Outer training loop:  50%|####9     | 24776/50000 [15:25:23<13:08:52,  1.88s/it]
Outer training loop:  50%|####9     | 24777/50000 [15:25:25<12:59:11,  1.85s/it]
Outer training loop:  50%|####9     | 24778/50000 [15:25:27<13:18:24,  1.90s/it]
Outer training loop:  50%|####9     | 24779/50000 [15:25:29<13:19:09,  1.90s/it]
Outer training loop:  50%|####9     | 24780/50000 [15:25:31<13:06:31,  1.87s/it]
Outer training loop:  50%|####9     | 24781/50000 [15:25:33<13:28:23,  1.92s/it]
Outer training loop:  50%|####9     | 24782/50000 [15:25:34<13:16:51,  1.90s/it]
Outer training loop:  50%|####9     | 24783/50000 [15:25:36<13:08:14,  1.88s/it]
Outer training loop:  50%|####9     | 24784/50000 [15:25:38<13:18:51,  1.90s/it]
Outer training loop:  50%|####9     | 24785/50000 [15:25:40<13:22:54,  1.91s/it]
Outer training loop:  50%|####9     | 24786/50000 [15:25:42<13:09:47,  1.88s/it]
Outer training loop:  50%|####9     | 24787/50000 [15:25:44<13:47:54,  1.97s/it]
Outer training loop:  50%|####9     | 24788/50000 [15:25:46<13:31:52,  1.93s/it]
Outer training loop:  50%|####9     | 24789/50000 [15:25:48<13:11:04,  1.88s/it]
Outer training loop:  50%|####9     | 24790/50000 [15:25:50<13:47:02,  1.97s/it]
Outer training loop:  50%|####9     | 24791/50000 [15:25:52<13:08:45,  1.88s/it]
Outer training loop:  50%|####9     | 24792/50000 [15:25:53<13:04:51,  1.87s/it]
Outer training loop:  50%|####9     | 24793/50000 [15:25:55<12:48:14,  1.83s/it]
Outer training loop:  50%|####9     | 24794/50000 [15:25:57<13:03:14,  1.86s/it]
Outer training loop:  50%|####9     | 24795/50000 [15:25:59<12:56:20,  1.85s/it]
Outer training loop:  50%|####9     | 24796/50000 [15:26:01<12:58:55,  1.85s/it]
Outer training loop:  50%|####9     | 24797/50000 [15:26:03<13:18:29,  1.90s/it]
Outer training loop:  50%|####9     | 24798/50000 [15:26:05<13:14:26,  1.89s/it]
Outer training loop:  50%|####9     | 24799/50000 [15:26:06<12:59:03,  1.85s/it]
Outer training loop:  50%|####9     | 24800/50000 [15:26:08<12:55:48,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  50%|####9     | 24801/50000 [15:26:16<25:23:24,  3.63s/it]
Outer training loop:  50%|####9     | 24802/50000 [15:26:18<21:40:12,  3.10s/it]
Outer training loop:  50%|####9     | 24803/50000 [15:26:20<19:09:32,  2.74s/it]
Outer training loop:  50%|####9     | 24804/50000 [15:26:22<17:18:27,  2.47s/it]
Outer training loop:  50%|####9     | 24805/50000 [15:26:24<16:25:17,  2.35s/it]
Outer training loop:  50%|####9     | 24806/50000 [15:26:26<15:32:03,  2.22s/it]
Outer training loop:  50%|####9     | 24807/50000 [15:26:28<14:53:24,  2.13s/it]
Outer training loop:  50%|####9     | 24808/50000 [15:26:30<14:35:30,  2.09s/it]
Outer training loop:  50%|####9     | 24809/50000 [15:26:32<14:19:05,  2.05s/it]
Outer training loop:  50%|####9     | 24810/50000 [15:26:33<14:02:45,  2.01s/it]
Outer training loop:  50%|####9     | 24811/50000 [15:26:35<14:04:43,  2.01s/it]
Outer training loop:  50%|####9     | 24812/50000 [15:26:38<14:18:56,  2.05s/it]
Outer training loop:  50%|####9     | 24813/50000 [15:26:39<13:57:23,  1.99s/it]
Outer training loop:  50%|####9     | 24814/50000 [15:26:41<13:35:40,  1.94s/it]
Outer training loop:  50%|####9     | 24815/50000 [15:26:43<13:47:13,  1.97s/it]
Outer training loop:  50%|####9     | 24816/50000 [15:26:45<13:33:40,  1.94s/it]
Outer training loop:  50%|####9     | 24817/50000 [15:26:47<13:21:32,  1.91s/it]
Outer training loop:  50%|####9     | 24818/50000 [15:26:49<13:22:00,  1.91s/it]
Outer training loop:  50%|####9     | 24819/50000 [15:26:51<13:31:44,  1.93s/it]
Outer training loop:  50%|####9     | 24820/50000 [15:26:53<14:06:19,  2.02s/it]
Outer training loop:  50%|####9     | 24821/50000 [15:26:55<14:00:37,  2.00s/it]
Outer training loop:  50%|####9     | 24822/50000 [15:26:57<13:53:32,  1.99s/it]
Outer training loop:  50%|####9     | 24823/50000 [15:26:59<14:11:31,  2.03s/it]
Outer training loop:  50%|####9     | 24824/50000 [15:27:01<13:55:34,  1.99s/it]
Outer training loop:  50%|####9     | 24825/50000 [15:27:03<13:30:38,  1.93s/it]
Outer training loop:  50%|####9     | 24826/50000 [15:27:05<13:55:27,  1.99s/it]
Outer training loop:  50%|####9     | 24827/50000 [15:27:07<13:31:31,  1.93s/it]
Outer training loop:  50%|####9     | 24828/50000 [15:27:09<13:24:35,  1.92s/it]
Outer training loop:  50%|####9     | 24829/50000 [15:27:11<13:47:30,  1.97s/it]
Outer training loop:  50%|####9     | 24830/50000 [15:27:13<13:35:30,  1.94s/it]
Outer training loop:  50%|####9     | 24831/50000 [15:27:15<13:39:37,  1.95s/it]
Outer training loop:  50%|####9     | 24832/50000 [15:27:16<13:25:31,  1.92s/it]
Outer training loop:  50%|####9     | 24833/50000 [15:27:19<13:44:43,  1.97s/it]
Outer training loop:  50%|####9     | 24834/50000 [15:27:20<13:29:13,  1.93s/it]
Outer training loop:  50%|####9     | 24835/50000 [15:27:22<13:36:45,  1.95s/it]
Outer training loop:  50%|####9     | 24836/50000 [15:27:24<13:44:21,  1.97s/it]
Outer training loop:  50%|####9     | 24837/50000 [15:27:26<13:31:22,  1.93s/it]
Outer training loop:  50%|####9     | 24838/50000 [15:27:28<13:48:43,  1.98s/it]
Outer training loop:  50%|####9     | 24839/50000 [15:27:30<13:57:15,  2.00s/it]
Outer training loop:  50%|####9     | 24840/50000 [15:27:32<13:33:47,  1.94s/it]
Outer training loop:  50%|####9     | 24841/50000 [15:27:34<13:34:02,  1.94s/it]
Outer training loop:  50%|####9     | 24842/50000 [15:27:36<14:00:36,  2.00s/it]
Outer training loop:  50%|####9     | 24843/50000 [15:27:38<13:55:54,  1.99s/it]
Outer training loop:  50%|####9     | 24844/50000 [15:27:40<13:51:51,  1.98s/it]
Outer training loop:  50%|####9     | 24845/50000 [15:27:42<13:48:08,  1.98s/it]
Outer training loop:  50%|####9     | 24846/50000 [15:27:44<14:09:06,  2.03s/it]
Outer training loop:  50%|####9     | 24847/50000 [15:27:46<13:48:19,  1.98s/it]
Outer training loop:  50%|####9     | 24848/50000 [15:27:48<13:41:59,  1.96s/it]
Outer training loop:  50%|####9     | 24849/50000 [15:27:50<13:58:17,  2.00s/it]
Outer training loop:  50%|####9     | 24850/50000 [15:27:52<13:30:23,  1.93s/it]
Outer training loop:  50%|####9     | 24851/50000 [15:27:54<13:32:53,  1.94s/it]
Outer training loop:  50%|####9     | 24852/50000 [15:27:56<13:21:33,  1.91s/it]
Outer training loop:  50%|####9     | 24853/50000 [15:27:58<13:46:22,  1.97s/it]
Outer training loop:  50%|####9     | 24854/50000 [15:28:00<13:33:56,  1.94s/it]
Outer training loop:  50%|####9     | 24855/50000 [15:28:02<13:30:44,  1.93s/it]
Outer training loop:  50%|####9     | 24856/50000 [15:28:04<13:52:40,  1.99s/it]
Outer training loop:  50%|####9     | 24857/50000 [15:28:06<13:31:03,  1.94s/it]
Outer training loop:  50%|####9     | 24858/50000 [15:28:08<13:38:21,  1.95s/it]
Outer training loop:  50%|####9     | 24859/50000 [15:28:09<13:11:26,  1.89s/it]
Outer training loop:  50%|####9     | 24860/50000 [15:28:11<13:39:04,  1.95s/it]
Outer training loop:  50%|####9     | 24861/50000 [15:28:13<13:34:36,  1.94s/it]
Outer training loop:  50%|####9     | 24862/50000 [15:28:15<13:13:30,  1.89s/it]
Outer training loop:  50%|####9     | 24863/50000 [15:28:17<13:20:09,  1.91s/it]
Outer training loop:  50%|####9     | 24864/50000 [15:28:19<13:14:35,  1.90s/it]
Outer training loop:  50%|####9     | 24865/50000 [15:28:21<13:00:23,  1.86s/it]
Outer training loop:  50%|####9     | 24866/50000 [15:28:22<12:47:05,  1.83s/it]
Outer training loop:  50%|####9     | 24867/50000 [15:28:25<13:26:55,  1.93s/it]
Outer training loop:  50%|####9     | 24868/50000 [15:28:27<13:38:13,  1.95s/it]
Outer training loop:  50%|####9     | 24869/50000 [15:28:29<13:33:55,  1.94s/it]
Outer training loop:  50%|####9     | 24870/50000 [15:28:31<14:06:52,  2.02s/it]
Outer training loop:  50%|####9     | 24871/50000 [15:28:33<13:50:07,  1.98s/it]
Outer training loop:  50%|####9     | 24872/50000 [15:28:35<13:49:11,  1.98s/it]
Outer training loop:  50%|####9     | 24873/50000 [15:28:37<13:59:26,  2.00s/it]
Outer training loop:  50%|####9     | 24874/50000 [15:28:38<13:32:24,  1.94s/it]
Outer training loop:  50%|####9     | 24875/50000 [15:28:40<13:17:22,  1.90s/it]
Outer training loop:  50%|####9     | 24876/50000 [15:28:42<13:17:47,  1.91s/it]
Outer training loop:  50%|####9     | 24877/50000 [15:28:44<13:33:16,  1.94s/it]
Outer training loop:  50%|####9     | 24878/50000 [15:28:46<13:38:42,  1.96s/it]
Outer training loop:  50%|####9     | 24879/50000 [15:28:48<13:50:59,  1.98s/it]
Outer training loop:  50%|####9     | 24880/50000 [15:28:50<14:12:47,  2.04s/it]
Outer training loop:  50%|####9     | 24881/50000 [15:28:52<13:27:21,  1.93s/it]
Outer training loop:  50%|####9     | 24882/50000 [15:28:54<13:15:57,  1.90s/it]
Outer training loop:  50%|####9     | 24883/50000 [15:28:56<13:43:20,  1.97s/it]
Outer training loop:  50%|####9     | 24884/50000 [15:28:58<13:23:48,  1.92s/it]
Outer training loop:  50%|####9     | 24885/50000 [15:29:00<13:05:50,  1.88s/it]
Outer training loop:  50%|####9     | 24886/50000 [15:29:01<12:56:09,  1.85s/it]
Outer training loop:  50%|####9     | 24887/50000 [15:29:04<13:30:09,  1.94s/it]
Outer training loop:  50%|####9     | 24888/50000 [15:29:06<13:29:40,  1.93s/it]
Outer training loop:  50%|####9     | 24889/50000 [15:29:07<13:09:05,  1.89s/it]
Outer training loop:  50%|####9     | 24890/50000 [15:29:10<13:54:50,  1.99s/it]
Outer training loop:  50%|####9     | 24891/50000 [15:29:11<13:39:14,  1.96s/it]
Outer training loop:  50%|####9     | 24892/50000 [15:29:13<13:46:10,  1.97s/it]
Outer training loop:  50%|####9     | 24893/50000 [15:29:15<13:28:23,  1.93s/it]
Outer training loop:  50%|####9     | 24894/50000 [15:29:17<13:49:02,  1.98s/it]
Outer training loop:  50%|####9     | 24895/50000 [15:29:19<13:40:25,  1.96s/it]
Outer training loop:  50%|####9     | 24896/50000 [15:29:21<13:22:43,  1.92s/it]
Outer training loop:  50%|####9     | 24897/50000 [15:29:23<13:36:15,  1.95s/it]
Outer training loop:  50%|####9     | 24898/50000 [15:29:25<13:28:51,  1.93s/it]
Outer training loop:  50%|####9     | 24899/50000 [15:29:27<13:21:00,  1.91s/it]
Outer training loop:  50%|####9     | 24900/50000 [15:29:29<13:40:41,  1.96s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.79s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  50%|####9     | 24901/50000 [15:29:36<24:55:24,  3.57s/it]
Outer training loop:  50%|####9     | 24902/50000 [15:29:38<21:24:01,  3.07s/it]
Outer training loop:  50%|####9     | 24903/50000 [15:29:40<18:58:36,  2.72s/it]
Outer training loop:  50%|####9     | 24904/50000 [15:29:42<17:29:33,  2.51s/it]
Outer training loop:  50%|####9     | 24905/50000 [15:29:44<15:59:54,  2.30s/it]
Outer training loop:  50%|####9     | 24906/50000 [15:29:46<15:05:25,  2.16s/it]
Outer training loop:  50%|####9     | 24907/50000 [15:29:48<14:48:12,  2.12s/it]
Outer training loop:  50%|####9     | 24908/50000 [15:29:50<14:09:18,  2.03s/it]
Outer training loop:  50%|####9     | 24909/50000 [15:29:51<13:46:58,  1.98s/it]
Outer training loop:  50%|####9     | 24910/50000 [15:29:53<13:50:14,  1.99s/it]
Outer training loop:  50%|####9     | 24911/50000 [15:29:55<13:40:10,  1.96s/it]
Outer training loop:  50%|####9     | 24912/50000 [15:29:57<13:14:40,  1.90s/it]
Outer training loop:  50%|####9     | 24913/50000 [15:29:59<13:12:10,  1.89s/it]
Outer training loop:  50%|####9     | 24914/50000 [15:30:01<13:27:33,  1.93s/it]
Outer training loop:  50%|####9     | 24915/50000 [15:30:03<13:01:19,  1.87s/it]
Outer training loop:  50%|####9     | 24916/50000 [15:30:05<12:55:13,  1.85s/it]
Outer training loop:  50%|####9     | 24917/50000 [15:30:07<13:29:24,  1.94s/it]
Outer training loop:  50%|####9     | 24918/50000 [15:30:09<13:13:57,  1.90s/it]
Outer training loop:  50%|####9     | 24919/50000 [15:30:10<13:09:53,  1.89s/it]
Outer training loop:  50%|####9     | 24920/50000 [15:30:13<13:50:32,  1.99s/it]
Outer training loop:  50%|####9     | 24921/50000 [15:30:14<13:37:36,  1.96s/it]
Outer training loop:  50%|####9     | 24922/50000 [15:30:16<13:44:13,  1.97s/it]
Outer training loop:  50%|####9     | 24923/50000 [15:30:19<13:59:30,  2.01s/it]
Outer training loop:  50%|####9     | 24924/50000 [15:30:21<13:54:13,  2.00s/it]
Outer training loop:  50%|####9     | 24925/50000 [15:30:22<13:30:43,  1.94s/it]
Outer training loop:  50%|####9     | 24926/50000 [15:30:24<13:14:57,  1.90s/it]
Outer training loop:  50%|####9     | 24927/50000 [15:30:26<13:32:22,  1.94s/it]
Outer training loop:  50%|####9     | 24928/50000 [15:30:28<13:32:39,  1.94s/it]
Outer training loop:  50%|####9     | 24929/50000 [15:30:30<13:15:29,  1.90s/it]
Outer training loop:  50%|####9     | 24930/50000 [15:30:32<13:30:32,  1.94s/it]
Outer training loop:  50%|####9     | 24931/50000 [15:30:34<13:30:49,  1.94s/it]
Outer training loop:  50%|####9     | 24932/50000 [15:30:36<13:27:50,  1.93s/it]
Outer training loop:  50%|####9     | 24933/50000 [15:30:38<13:27:11,  1.93s/it]
Outer training loop:  50%|####9     | 24934/50000 [15:30:40<13:24:58,  1.93s/it]
Outer training loop:  50%|####9     | 24935/50000 [15:30:42<13:10:23,  1.89s/it]
Outer training loop:  50%|####9     | 24936/50000 [15:30:44<13:24:25,  1.93s/it]
Outer training loop:  50%|####9     | 24937/50000 [15:30:45<13:02:51,  1.87s/it]
Outer training loop:  50%|####9     | 24938/50000 [15:30:47<12:51:08,  1.85s/it]
Outer training loop:  50%|####9     | 24939/50000 [15:30:49<13:25:19,  1.93s/it]
Outer training loop:  50%|####9     | 24940/50000 [15:30:51<13:27:23,  1.93s/it]
Outer training loop:  50%|####9     | 24941/50000 [15:30:53<13:16:03,  1.91s/it]
Outer training loop:  50%|####9     | 24942/50000 [15:30:55<13:38:18,  1.96s/it]
Outer training loop:  50%|####9     | 24943/50000 [15:30:57<13:21:18,  1.92s/it]
Outer training loop:  50%|####9     | 24944/50000 [15:30:59<13:08:35,  1.89s/it]
Outer training loop:  50%|####9     | 24945/50000 [15:31:01<13:32:38,  1.95s/it]
Outer training loop:  50%|####9     | 24946/50000 [15:31:03<13:11:39,  1.90s/it]
Outer training loop:  50%|####9     | 24947/50000 [15:31:04<13:14:39,  1.90s/it]
Outer training loop:  50%|####9     | 24948/50000 [15:31:06<13:10:51,  1.89s/it]
Outer training loop:  50%|####9     | 24949/50000 [15:31:08<13:13:48,  1.90s/it]
Outer training loop:  50%|####9     | 24950/50000 [15:31:10<13:15:51,  1.91s/it]
Outer training loop:  50%|####9     | 24951/50000 [15:31:12<13:11:25,  1.90s/it]
Outer training loop:  50%|####9     | 24952/50000 [15:31:14<13:24:12,  1.93s/it]
Outer training loop:  50%|####9     | 24953/50000 [15:31:16<13:08:58,  1.89s/it]
Outer training loop:  50%|####9     | 24954/50000 [15:31:18<12:56:32,  1.86s/it]
Outer training loop:  50%|####9     | 24955/50000 [15:31:20<13:06:35,  1.88s/it]
Outer training loop:  50%|####9     | 24956/50000 [15:31:21<12:57:50,  1.86s/it]
Outer training loop:  50%|####9     | 24957/50000 [15:31:23<12:52:16,  1.85s/it]
Outer training loop:  50%|####9     | 24958/50000 [15:31:25<13:00:59,  1.87s/it]
Outer training loop:  50%|####9     | 24959/50000 [15:31:27<13:24:16,  1.93s/it]
Outer training loop:  50%|####9     | 24960/50000 [15:31:29<12:57:54,  1.86s/it]
Outer training loop:  50%|####9     | 24961/50000 [15:31:31<12:49:44,  1.84s/it]
Outer training loop:  50%|####9     | 24962/50000 [15:31:32<12:44:35,  1.83s/it]
Outer training loop:  50%|####9     | 24963/50000 [15:31:35<13:12:25,  1.90s/it]
Outer training loop:  50%|####9     | 24964/50000 [15:31:37<13:25:09,  1.93s/it]
Outer training loop:  50%|####9     | 24965/50000 [15:31:38<13:15:14,  1.91s/it]
Outer training loop:  50%|####9     | 24966/50000 [15:31:40<13:25:09,  1.93s/it]
Outer training loop:  50%|####9     | 24967/50000 [15:31:42<13:47:00,  1.98s/it]
Outer training loop:  50%|####9     | 24968/50000 [15:31:44<13:30:27,  1.94s/it]
Outer training loop:  50%|####9     | 24969/50000 [15:31:46<13:46:14,  1.98s/it]
Outer training loop:  50%|####9     | 24970/50000 [15:31:48<13:25:59,  1.93s/it]
Outer training loop:  50%|####9     | 24971/50000 [15:31:50<13:14:51,  1.91s/it]
Outer training loop:  50%|####9     | 24972/50000 [15:31:52<13:30:48,  1.94s/it]
Outer training loop:  50%|####9     | 24973/50000 [15:31:54<13:08:43,  1.89s/it]
Outer training loop:  50%|####9     | 24974/50000 [15:31:56<13:05:39,  1.88s/it]
Outer training loop:  50%|####9     | 24975/50000 [15:31:58<13:04:29,  1.88s/it]
Outer training loop:  50%|####9     | 24976/50000 [15:32:00<13:18:56,  1.92s/it]
Outer training loop:  50%|####9     | 24977/50000 [15:32:01<12:57:51,  1.87s/it]
Outer training loop:  50%|####9     | 24978/50000 [15:32:03<12:26:00,  1.79s/it]
Outer training loop:  50%|####9     | 24979/50000 [15:32:05<12:53:07,  1.85s/it]
Outer training loop:  50%|####9     | 24980/50000 [15:32:07<13:00:20,  1.87s/it]
Outer training loop:  50%|####9     | 24981/50000 [15:32:09<13:00:36,  1.87s/it]
Outer training loop:  50%|####9     | 24982/50000 [15:32:11<12:59:16,  1.87s/it]
Outer training loop:  50%|####9     | 24983/50000 [15:32:13<13:09:04,  1.89s/it]
Outer training loop:  50%|####9     | 24984/50000 [15:32:14<12:51:18,  1.85s/it]
Outer training loop:  50%|####9     | 24985/50000 [15:32:16<12:58:13,  1.87s/it]
Outer training loop:  50%|####9     | 24986/50000 [15:32:18<13:31:23,  1.95s/it]
Outer training loop:  50%|####9     | 24987/50000 [15:32:20<13:36:18,  1.96s/it]
Outer training loop:  50%|####9     | 24988/50000 [15:32:22<13:19:20,  1.92s/it]
Outer training loop:  50%|####9     | 24989/50000 [15:32:24<13:21:18,  1.92s/it]
Outer training loop:  50%|####9     | 24990/50000 [15:32:26<13:14:45,  1.91s/it]
Outer training loop:  50%|####9     | 24991/50000 [15:32:28<13:15:24,  1.91s/it]
Outer training loop:  50%|####9     | 24992/50000 [15:32:30<12:59:40,  1.87s/it]
Outer training loop:  50%|####9     | 24993/50000 [15:32:32<13:21:00,  1.92s/it]
Outer training loop:  50%|####9     | 24994/50000 [15:32:33<12:59:11,  1.87s/it]
Outer training loop:  50%|####9     | 24995/50000 [15:32:35<13:15:06,  1.91s/it]
Outer training loop:  50%|####9     | 24996/50000 [15:32:37<13:11:27,  1.90s/it]
Outer training loop:  50%|####9     | 24997/50000 [15:32:39<13:36:32,  1.96s/it]
Outer training loop:  50%|####9     | 24998/50000 [15:32:41<13:29:09,  1.94s/it]
Outer training loop:  50%|####9     | 24999/50000 [15:32:43<13:23:40,  1.93s/it]
Outer training loop:  50%|#####     | 25000/50000 [15:32:45<13:36:05,  1.96s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:27,  1.79s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.83s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:33,  1.92s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.89s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.89s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.87s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:28,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.90s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.93s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:22,  1.96s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:15,  1.91s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:15,  1.94s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:18,  2.00s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.90s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:04,  1.94s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:54,  1.85s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.86s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:58,  1.97s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:54,  1.94s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:51,  1.93s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.89s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:48,  1.97s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:42,  1.90s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:41,  1.92s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:40,  1.93s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:34,  1.90s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:35,  1.95s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:22,  1.83s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:19,  1.81s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:21,  1.90s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:17,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:16,  1.87s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:11,  1.84s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:10,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:07,  1.83s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:09,  1.94s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.90s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:03,  1.87s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.85s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:58,  1.83s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:54,  1.82s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:54,  1.94s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.91s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.92s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:43,  1.83s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:41,  1.81s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:39,  1.81s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.95s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.95s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.90s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.88s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.84s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.99s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:21,  1.93s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.90s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.91s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.91s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.90s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.89s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:08,  2.05s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:06,  2.01s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.95s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.84s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:27,  1.80s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:24,  1.79s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:25,  1.81s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.82s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:18,  1.78s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:31,  1.97s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:29,  1.96s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:24,  1.97s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:20,  1.95s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:14,  1.95s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:03,  1.87s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:06,  1.95s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:02,  1.91s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:51,  1.82s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:56,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:45,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:44,  1.86s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.94s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.92s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:44,  1.98s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:40,  1.93s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:38,  2.02s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:34,  1.96s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:32,  1.96s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:29,  1.94s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:25,  1.89s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:24,  1.96s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:21,  1.95s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:19,  1.94s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:16,  1.91s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:14,  1.91s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:14,  2.00s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:10,  1.96s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.90s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.89s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:57,  1.85s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:54,  1.82s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:56,  1.94s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.89s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.84s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:41,  1.97s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:38,  1.92s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.90s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.88s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.89s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:26,  2.01s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:23,  1.95s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.88s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.86s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.84s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:12,  1.85s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:10,  1.83s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.81s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.98s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.92s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.90s/it]

Outer training loop:  50%|#####     | 25001/50000 [15:38:08<681:39:42, 98.16s/it]
Outer training loop:  50%|#####     | 25002/50000 [15:38:10<481:10:21, 69.29s/it]
Outer training loop:  50%|#####     | 25003/50000 [15:38:12<340:32:44, 49.04s/it]
Outer training loop:  50%|#####     | 25004/50000 [15:38:14<242:21:49, 34.91s/it]
Outer training loop:  50%|#####     | 25005/50000 [15:38:15<173:12:27, 24.95s/it]
Outer training loop:  50%|#####     | 25006/50000 [15:38:17<124:56:32, 18.00s/it]
Outer training loop:  50%|#####     | 25007/50000 [15:38:19<92:31:38, 13.33s/it] 
Outer training loop:  50%|#####     | 25008/50000 [15:38:21<68:30:49,  9.87s/it]
Outer training loop:  50%|#####     | 25009/50000 [15:38:23<51:55:19,  7.48s/it]
Outer training loop:  50%|#####     | 25010/50000 [15:38:25<41:08:25,  5.93s/it]
Outer training loop:  50%|#####     | 25011/50000 [15:38:27<32:52:00,  4.73s/it]
Outer training loop:  50%|#####     | 25012/50000 [15:38:29<26:53:11,  3.87s/it]
Outer training loop:  50%|#####     | 25013/50000 [15:38:31<22:50:40,  3.29s/it]
Outer training loop:  50%|#####     | 25014/50000 [15:38:33<20:00:51,  2.88s/it]
Outer training loop:  50%|#####     | 25015/50000 [15:38:35<17:42:39,  2.55s/it]
Outer training loop:  50%|#####     | 25016/50000 [15:38:37<16:17:23,  2.35s/it]
Outer training loop:  50%|#####     | 25017/50000 [15:38:39<15:43:46,  2.27s/it]
Outer training loop:  50%|#####     | 25018/50000 [15:38:41<14:49:28,  2.14s/it]
Outer training loop:  50%|#####     | 25019/50000 [15:38:43<14:18:39,  2.06s/it]
Outer training loop:  50%|#####     | 25020/50000 [15:38:44<13:43:19,  1.98s/it]
Outer training loop:  50%|#####     | 25021/50000 [15:38:46<13:59:57,  2.02s/it]
Outer training loop:  50%|#####     | 25022/50000 [15:38:48<13:42:09,  1.97s/it]
Outer training loop:  50%|#####     | 25023/50000 [15:38:50<13:27:49,  1.94s/it]
Outer training loop:  50%|#####     | 25024/50000 [15:38:52<13:38:53,  1.97s/it]
Outer training loop:  50%|#####     | 25025/50000 [15:38:54<13:18:19,  1.92s/it]
Outer training loop:  50%|#####     | 25026/50000 [15:38:56<13:00:57,  1.88s/it]
Outer training loop:  50%|#####     | 25027/50000 [15:38:58<13:26:51,  1.94s/it]
Outer training loop:  50%|#####     | 25028/50000 [15:39:00<13:11:29,  1.90s/it]
Outer training loop:  50%|#####     | 25029/50000 [15:39:02<13:13:39,  1.91s/it]
Outer training loop:  50%|#####     | 25030/50000 [15:39:04<13:20:33,  1.92s/it]
Outer training loop:  50%|#####     | 25031/50000 [15:39:05<13:08:51,  1.90s/it]
Outer training loop:  50%|#####     | 25032/50000 [15:39:07<12:49:38,  1.85s/it]
Outer training loop:  50%|#####     | 25033/50000 [15:39:09<12:40:53,  1.83s/it]
Outer training loop:  50%|#####     | 25034/50000 [15:39:11<13:04:36,  1.89s/it]
Outer training loop:  50%|#####     | 25035/50000 [15:39:13<13:10:46,  1.90s/it]
Outer training loop:  50%|#####     | 25036/50000 [15:39:15<13:00:30,  1.88s/it]
Outer training loop:  50%|#####     | 25037/50000 [15:39:17<13:08:32,  1.90s/it]
Outer training loop:  50%|#####     | 25038/50000 [15:39:18<12:48:16,  1.85s/it]
Outer training loop:  50%|#####     | 25039/50000 [15:39:20<12:41:51,  1.83s/it]
Outer training loop:  50%|#####     | 25040/50000 [15:39:22<12:35:04,  1.82s/it]
Outer training loop:  50%|#####     | 25041/50000 [15:39:24<12:40:41,  1.83s/it]
Outer training loop:  50%|#####     | 25042/50000 [15:39:26<12:37:09,  1.82s/it]
Outer training loop:  50%|#####     | 25043/50000 [15:39:28<12:49:54,  1.85s/it]
Outer training loop:  50%|#####     | 25044/50000 [15:39:29<12:49:02,  1.85s/it]
Outer training loop:  50%|#####     | 25045/50000 [15:39:31<13:13:47,  1.91s/it]
Outer training loop:  50%|#####     | 25046/50000 [15:39:33<12:47:42,  1.85s/it]
Outer training loop:  50%|#####     | 25047/50000 [15:39:35<13:03:43,  1.88s/it]
Outer training loop:  50%|#####     | 25048/50000 [15:39:37<13:46:10,  1.99s/it]
Outer training loop:  50%|#####     | 25049/50000 [15:39:39<13:21:00,  1.93s/it]
Outer training loop:  50%|#####     | 25050/50000 [15:39:41<13:07:24,  1.89s/it]
Outer training loop:  50%|#####     | 25051/50000 [15:39:43<13:24:06,  1.93s/it]
Outer training loop:  50%|#####     | 25052/50000 [15:39:45<12:59:38,  1.88s/it]
Outer training loop:  50%|#####     | 25053/50000 [15:39:47<12:59:44,  1.88s/it]
Outer training loop:  50%|#####     | 25054/50000 [15:39:49<13:28:16,  1.94s/it]
Outer training loop:  50%|#####     | 25055/50000 [15:39:51<13:14:35,  1.91s/it]
Outer training loop:  50%|#####     | 25056/50000 [15:39:52<13:04:07,  1.89s/it]
Outer training loop:  50%|#####     | 25057/50000 [15:39:54<12:44:39,  1.84s/it]
Outer training loop:  50%|#####     | 25058/50000 [15:39:56<13:19:07,  1.92s/it]
Outer training loop:  50%|#####     | 25059/50000 [15:39:58<13:10:09,  1.90s/it]
Outer training loop:  50%|#####     | 25060/50000 [15:40:00<12:57:21,  1.87s/it]
Outer training loop:  50%|#####     | 25061/50000 [15:40:02<13:15:04,  1.91s/it]
Outer training loop:  50%|#####     | 25062/50000 [15:40:04<13:14:36,  1.91s/it]
Outer training loop:  50%|#####     | 25063/50000 [15:40:06<13:18:27,  1.92s/it]
Outer training loop:  50%|#####     | 25064/50000 [15:40:08<13:28:15,  1.94s/it]
Outer training loop:  50%|#####     | 25065/50000 [15:40:10<13:34:18,  1.96s/it]
Outer training loop:  50%|#####     | 25066/50000 [15:40:12<13:10:59,  1.90s/it]
Outer training loop:  50%|#####     | 25067/50000 [15:40:13<13:02:53,  1.88s/it]
Outer training loop:  50%|#####     | 25068/50000 [15:40:15<13:24:11,  1.94s/it]
Outer training loop:  50%|#####     | 25069/50000 [15:40:17<12:55:41,  1.87s/it]
Outer training loop:  50%|#####     | 25070/50000 [15:40:19<13:02:43,  1.88s/it]
Outer training loop:  50%|#####     | 25071/50000 [15:40:21<13:20:08,  1.93s/it]
Outer training loop:  50%|#####     | 25072/50000 [15:40:23<13:22:50,  1.93s/it]
Outer training loop:  50%|#####     | 25073/50000 [15:40:25<13:20:26,  1.93s/it]
Outer training loop:  50%|#####     | 25074/50000 [15:40:27<13:37:36,  1.97s/it]
Outer training loop:  50%|#####     | 25075/50000 [15:40:29<13:16:58,  1.92s/it]
Outer training loop:  50%|#####     | 25076/50000 [15:40:31<13:12:20,  1.91s/it]
Outer training loop:  50%|#####     | 25077/50000 [15:40:32<12:59:04,  1.88s/it]
Outer training loop:  50%|#####     | 25078/50000 [15:40:34<13:14:35,  1.91s/it]
Outer training loop:  50%|#####     | 25079/50000 [15:40:36<13:08:27,  1.90s/it]
Outer training loop:  50%|#####     | 25080/50000 [15:40:38<12:53:16,  1.86s/it]
Outer training loop:  50%|#####     | 25081/50000 [15:40:40<13:14:01,  1.91s/it]
Outer training loop:  50%|#####     | 25082/50000 [15:40:42<13:07:21,  1.90s/it]
Outer training loop:  50%|#####     | 25083/50000 [15:40:44<13:06:58,  1.90s/it]
Outer training loop:  50%|#####     | 25084/50000 [15:40:46<12:58:18,  1.87s/it]
Outer training loop:  50%|#####     | 25085/50000 [15:40:48<13:16:09,  1.92s/it]
Outer training loop:  50%|#####     | 25086/50000 [15:40:50<13:19:55,  1.93s/it]
Outer training loop:  50%|#####     | 25087/50000 [15:40:52<13:11:48,  1.91s/it]
Outer training loop:  50%|#####     | 25088/50000 [15:40:54<13:29:54,  1.95s/it]
Outer training loop:  50%|#####     | 25089/50000 [15:40:55<13:16:41,  1.92s/it]
Outer training loop:  50%|#####     | 25090/50000 [15:40:57<13:04:37,  1.89s/it]
Outer training loop:  50%|#####     | 25091/50000 [15:40:59<13:18:58,  1.92s/it]
Outer training loop:  50%|#####     | 25092/50000 [15:41:01<12:59:31,  1.88s/it]
Outer training loop:  50%|#####     | 25093/50000 [15:41:03<13:04:30,  1.89s/it]
Outer training loop:  50%|#####     | 25094/50000 [15:41:05<13:18:55,  1.92s/it]
Outer training loop:  50%|#####     | 25095/50000 [15:41:07<12:52:13,  1.86s/it]
Outer training loop:  50%|#####     | 25096/50000 [15:41:09<12:50:33,  1.86s/it]
Outer training loop:  50%|#####     | 25097/50000 [15:41:10<12:37:37,  1.83s/it]
Outer training loop:  50%|#####     | 25098/50000 [15:41:13<13:31:27,  1.96s/it]
Outer training loop:  50%|#####     | 25099/50000 [15:41:14<13:20:24,  1.93s/it]
Outer training loop:  50%|#####     | 25100/50000 [15:41:16<12:53:32,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.87s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  50%|#####     | 25101/50000 [15:41:24<24:25:25,  3.53s/it]
Outer training loop:  50%|#####     | 25102/50000 [15:41:26<21:22:20,  3.09s/it]
Outer training loop:  50%|#####     | 25103/50000 [15:41:27<18:32:25,  2.68s/it]
Outer training loop:  50%|#####     | 25104/50000 [15:41:29<16:51:32,  2.44s/it]
Outer training loop:  50%|#####     | 25105/50000 [15:41:31<16:01:48,  2.32s/it]
Outer training loop:  50%|#####     | 25106/50000 [15:41:33<14:59:21,  2.17s/it]
Outer training loop:  50%|#####     | 25107/50000 [15:41:35<14:20:12,  2.07s/it]
Outer training loop:  50%|#####     | 25108/50000 [15:41:37<14:20:51,  2.08s/it]
Outer training loop:  50%|#####     | 25109/50000 [15:41:39<13:34:58,  1.96s/it]
Outer training loop:  50%|#####     | 25110/50000 [15:41:41<13:35:58,  1.97s/it]
Outer training loop:  50%|#####     | 25111/50000 [15:41:43<13:32:09,  1.96s/it]
Outer training loop:  50%|#####     | 25112/50000 [15:41:44<13:21:38,  1.93s/it]
Outer training loop:  50%|#####     | 25113/50000 [15:41:46<12:56:25,  1.87s/it]
Outer training loop:  50%|#####     | 25114/50000 [15:41:48<12:44:31,  1.84s/it]
Outer training loop:  50%|#####     | 25115/50000 [15:41:50<13:03:10,  1.89s/it]
Outer training loop:  50%|#####     | 25116/50000 [15:41:52<12:52:16,  1.86s/it]
Outer training loop:  50%|#####     | 25117/50000 [15:41:54<12:59:56,  1.88s/it]
Outer training loop:  50%|#####     | 25118/50000 [15:41:56<13:30:52,  1.96s/it]
Outer training loop:  50%|#####     | 25119/50000 [15:41:58<13:14:41,  1.92s/it]
Outer training loop:  50%|#####     | 25120/50000 [15:41:59<12:46:44,  1.85s/it]
Outer training loop:  50%|#####     | 25121/50000 [15:42:01<12:52:12,  1.86s/it]
Outer training loop:  50%|#####     | 25122/50000 [15:42:03<13:17:27,  1.92s/it]
Outer training loop:  50%|#####     | 25123/50000 [15:42:05<12:57:20,  1.87s/it]
Outer training loop:  50%|#####     | 25124/50000 [15:42:07<12:59:21,  1.88s/it]
Outer training loop:  50%|#####     | 25125/50000 [15:42:09<12:46:37,  1.85s/it]
Outer training loop:  50%|#####     | 25126/50000 [15:42:11<13:02:32,  1.89s/it]
Outer training loop:  50%|#####     | 25127/50000 [15:42:13<13:15:21,  1.92s/it]
Outer training loop:  50%|#####     | 25128/50000 [15:42:15<13:09:23,  1.90s/it]
Outer training loop:  50%|#####     | 25129/50000 [15:42:17<13:27:38,  1.95s/it]
Outer training loop:  50%|#####     | 25130/50000 [15:42:18<13:10:37,  1.91s/it]
Outer training loop:  50%|#####     | 25131/50000 [15:42:20<12:55:40,  1.87s/it]
Outer training loop:  50%|#####     | 25132/50000 [15:42:22<13:16:47,  1.92s/it]
Outer training loop:  50%|#####     | 25133/50000 [15:42:24<13:07:58,  1.90s/it]
Outer training loop:  50%|#####     | 25134/50000 [15:42:26<12:48:32,  1.85s/it]
Outer training loop:  50%|#####     | 25135/50000 [15:42:28<13:05:51,  1.90s/it]
Outer training loop:  50%|#####     | 25136/50000 [15:42:30<12:57:59,  1.88s/it]
Outer training loop:  50%|#####     | 25137/50000 [15:42:32<12:53:32,  1.87s/it]
Outer training loop:  50%|#####     | 25138/50000 [15:42:34<13:29:00,  1.95s/it]
Outer training loop:  50%|#####     | 25139/50000 [15:42:36<13:15:26,  1.92s/it]
Outer training loop:  50%|#####     | 25140/50000 [15:42:37<13:07:12,  1.90s/it]
Outer training loop:  50%|#####     | 25141/50000 [15:42:39<13:01:02,  1.89s/it]
Outer training loop:  50%|#####     | 25142/50000 [15:42:41<13:27:54,  1.95s/it]
Outer training loop:  50%|#####     | 25143/50000 [15:42:43<13:03:35,  1.89s/it]
Outer training loop:  50%|#####     | 25144/50000 [15:42:45<12:46:32,  1.85s/it]
Outer training loop:  50%|#####     | 25145/50000 [15:42:47<13:31:10,  1.96s/it]
Outer training loop:  50%|#####     | 25146/50000 [15:42:49<13:34:47,  1.97s/it]
Outer training loop:  50%|#####     | 25147/50000 [15:42:51<13:17:22,  1.93s/it]
Outer training loop:  50%|#####     | 25148/50000 [15:42:53<13:35:59,  1.97s/it]
Outer training loop:  50%|#####     | 25149/50000 [15:42:55<13:41:54,  1.98s/it]
Outer training loop:  50%|#####     | 25150/50000 [15:42:57<13:29:14,  1.95s/it]
Outer training loop:  50%|#####     | 25151/50000 [15:42:59<13:24:07,  1.94s/it]
Outer training loop:  50%|#####     | 25152/50000 [15:43:01<13:36:39,  1.97s/it]
Outer training loop:  50%|#####     | 25153/50000 [15:43:03<13:21:57,  1.94s/it]
Outer training loop:  50%|#####     | 25154/50000 [15:43:04<13:10:22,  1.91s/it]
Outer training loop:  50%|#####     | 25155/50000 [15:43:07<13:37:49,  1.98s/it]
Outer training loop:  50%|#####     | 25156/50000 [15:43:08<13:22:47,  1.94s/it]
Outer training loop:  50%|#####     | 25157/50000 [15:43:10<13:11:12,  1.91s/it]
Outer training loop:  50%|#####     | 25158/50000 [15:43:13<13:45:26,  1.99s/it]
Outer training loop:  50%|#####     | 25159/50000 [15:43:14<13:28:22,  1.95s/it]
Outer training loop:  50%|#####     | 25160/50000 [15:43:16<13:13:12,  1.92s/it]
Outer training loop:  50%|#####     | 25161/50000 [15:43:18<12:58:21,  1.88s/it]
Outer training loop:  50%|#####     | 25162/50000 [15:43:20<13:41:53,  1.99s/it]
Outer training loop:  50%|#####     | 25163/50000 [15:43:22<13:31:15,  1.96s/it]
Outer training loop:  50%|#####     | 25164/50000 [15:43:24<13:22:24,  1.94s/it]
Outer training loop:  50%|#####     | 25165/50000 [15:43:26<13:55:58,  2.02s/it]
Outer training loop:  50%|#####     | 25166/50000 [15:43:28<13:57:01,  2.02s/it]
Outer training loop:  50%|#####     | 25167/50000 [15:43:30<13:29:26,  1.96s/it]
Outer training loop:  50%|#####     | 25168/50000 [15:43:32<13:28:14,  1.95s/it]
Outer training loop:  50%|#####     | 25169/50000 [15:43:34<13:55:40,  2.02s/it]
Outer training loop:  50%|#####     | 25170/50000 [15:43:36<13:43:16,  1.99s/it]
Outer training loop:  50%|#####     | 25171/50000 [15:43:38<13:21:45,  1.94s/it]
Outer training loop:  50%|#####     | 25172/50000 [15:43:40<13:16:28,  1.92s/it]
Outer training loop:  50%|#####     | 25173/50000 [15:43:42<13:11:03,  1.91s/it]
Outer training loop:  50%|#####     | 25174/50000 [15:43:44<13:06:03,  1.90s/it]
Outer training loop:  50%|#####     | 25175/50000 [15:43:45<12:53:25,  1.87s/it]
Outer training loop:  50%|#####     | 25176/50000 [15:43:47<13:01:56,  1.89s/it]
Outer training loop:  50%|#####     | 25177/50000 [15:43:49<12:54:40,  1.87s/it]
Outer training loop:  50%|#####     | 25178/50000 [15:43:51<12:49:32,  1.86s/it]
Outer training loop:  50%|#####     | 25179/50000 [15:43:53<13:10:02,  1.91s/it]
Outer training loop:  50%|#####     | 25180/50000 [15:43:55<12:55:14,  1.87s/it]
Outer training loop:  50%|#####     | 25181/50000 [15:43:57<12:42:24,  1.84s/it]
Outer training loop:  50%|#####     | 25182/50000 [15:43:59<13:08:43,  1.91s/it]
Outer training loop:  50%|#####     | 25183/50000 [15:44:00<12:55:43,  1.88s/it]
Outer training loop:  50%|#####     | 25184/50000 [15:44:02<12:53:42,  1.87s/it]
Outer training loop:  50%|#####     | 25185/50000 [15:44:04<12:56:34,  1.88s/it]
Outer training loop:  50%|#####     | 25186/50000 [15:44:06<13:17:56,  1.93s/it]
Outer training loop:  50%|#####     | 25187/50000 [15:44:08<13:04:14,  1.90s/it]
Outer training loop:  50%|#####     | 25188/50000 [15:44:10<12:55:02,  1.87s/it]
Outer training loop:  50%|#####     | 25189/50000 [15:44:12<13:06:11,  1.90s/it]
Outer training loop:  50%|#####     | 25190/50000 [15:44:14<12:57:26,  1.88s/it]
Outer training loop:  50%|#####     | 25191/50000 [15:44:15<12:48:06,  1.86s/it]
Outer training loop:  50%|#####     | 25192/50000 [15:44:17<13:06:18,  1.90s/it]
Outer training loop:  50%|#####     | 25193/50000 [15:44:19<12:57:48,  1.88s/it]
Outer training loop:  50%|#####     | 25194/50000 [15:44:21<12:50:31,  1.86s/it]
Outer training loop:  50%|#####     | 25195/50000 [15:44:23<13:07:14,  1.90s/it]
Outer training loop:  50%|#####     | 25196/50000 [15:44:25<13:40:27,  1.98s/it]
Outer training loop:  50%|#####     | 25197/50000 [15:44:27<13:06:20,  1.90s/it]
Outer training loop:  50%|#####     | 25198/50000 [15:44:29<13:03:41,  1.90s/it]
Outer training loop:  50%|#####     | 25199/50000 [15:44:31<13:15:47,  1.93s/it]
Outer training loop:  50%|#####     | 25200/50000 [15:44:33<12:59:46,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.79s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  50%|#####     | 25201/50000 [15:44:40<24:22:55,  3.54s/it]
Outer training loop:  50%|#####     | 25202/50000 [15:44:42<20:39:44,  3.00s/it]
Outer training loop:  50%|#####     | 25203/50000 [15:44:44<18:38:15,  2.71s/it]
Outer training loop:  50%|#####     | 25204/50000 [15:44:46<16:57:44,  2.46s/it]
Outer training loop:  50%|#####     | 25205/50000 [15:44:47<15:34:36,  2.26s/it]
Outer training loop:  50%|#####     | 25206/50000 [15:44:49<14:40:31,  2.13s/it]
Outer training loop:  50%|#####     | 25207/50000 [15:44:51<14:16:34,  2.07s/it]
Outer training loop:  50%|#####     | 25208/50000 [15:44:53<13:43:15,  1.99s/it]
Outer training loop:  50%|#####     | 25209/50000 [15:44:55<13:16:46,  1.93s/it]
Outer training loop:  50%|#####     | 25210/50000 [15:44:57<13:21:54,  1.94s/it]
Outer training loop:  50%|#####     | 25211/50000 [15:44:59<13:18:53,  1.93s/it]
Outer training loop:  50%|#####     | 25212/50000 [15:45:00<12:54:02,  1.87s/it]
Outer training loop:  50%|#####     | 25213/50000 [15:45:02<12:41:23,  1.84s/it]
Outer training loop:  50%|#####     | 25214/50000 [15:45:04<13:04:17,  1.90s/it]
Outer training loop:  50%|#####     | 25215/50000 [15:45:06<13:04:49,  1.90s/it]
Outer training loop:  50%|#####     | 25216/50000 [15:45:08<12:52:48,  1.87s/it]
Outer training loop:  50%|#####     | 25217/50000 [15:45:10<13:15:47,  1.93s/it]
Outer training loop:  50%|#####     | 25218/50000 [15:45:12<12:53:56,  1.87s/it]
Outer training loop:  50%|#####     | 25219/50000 [15:45:14<12:53:38,  1.87s/it]
Outer training loop:  50%|#####     | 25220/50000 [15:45:16<13:23:26,  1.95s/it]
Outer training loop:  50%|#####     | 25221/50000 [15:45:18<13:02:38,  1.90s/it]
Outer training loop:  50%|#####     | 25222/50000 [15:45:19<12:53:53,  1.87s/it]
Outer training loop:  50%|#####     | 25223/50000 [15:45:21<12:51:36,  1.87s/it]
Outer training loop:  50%|#####     | 25224/50000 [15:45:23<13:10:16,  1.91s/it]
Outer training loop:  50%|#####     | 25225/50000 [15:45:25<13:13:24,  1.92s/it]
Outer training loop:  50%|#####     | 25226/50000 [15:45:27<12:59:10,  1.89s/it]
Outer training loop:  50%|#####     | 25227/50000 [15:45:29<13:19:48,  1.94s/it]
Outer training loop:  50%|#####     | 25228/50000 [15:45:31<12:57:43,  1.88s/it]
Outer training loop:  50%|#####     | 25229/50000 [15:45:33<12:47:23,  1.86s/it]
Outer training loop:  50%|#####     | 25230/50000 [15:45:35<13:22:17,  1.94s/it]
Outer training loop:  50%|#####     | 25231/50000 [15:45:37<13:16:52,  1.93s/it]
Outer training loop:  50%|#####     | 25232/50000 [15:45:38<12:52:41,  1.87s/it]
Outer training loop:  50%|#####     | 25233/50000 [15:45:40<13:00:32,  1.89s/it]
Outer training loop:  50%|#####     | 25234/50000 [15:45:42<13:16:25,  1.93s/it]
Outer training loop:  50%|#####     | 25235/50000 [15:45:44<13:23:01,  1.95s/it]
Outer training loop:  50%|#####     | 25236/50000 [15:45:46<13:10:16,  1.91s/it]
Outer training loop:  50%|#####     | 25237/50000 [15:45:48<13:22:22,  1.94s/it]
Outer training loop:  50%|#####     | 25238/50000 [15:45:50<13:07:57,  1.91s/it]
Outer training loop:  50%|#####     | 25239/50000 [15:45:52<12:53:57,  1.88s/it]
Outer training loop:  50%|#####     | 25240/50000 [15:45:54<13:17:01,  1.93s/it]
Outer training loop:  50%|#####     | 25241/50000 [15:45:56<13:00:36,  1.89s/it]
Outer training loop:  50%|#####     | 25242/50000 [15:45:58<13:28:49,  1.96s/it]
Outer training loop:  50%|#####     | 25243/50000 [15:46:00<13:16:46,  1.93s/it]
Outer training loop:  50%|#####     | 25244/50000 [15:46:02<13:33:11,  1.97s/it]
Outer training loop:  50%|#####     | 25245/50000 [15:46:04<13:13:38,  1.92s/it]
Outer training loop:  50%|#####     | 25246/50000 [15:46:05<13:02:35,  1.90s/it]
Outer training loop:  50%|#####     | 25247/50000 [15:46:07<13:14:06,  1.92s/it]
Outer training loop:  50%|#####     | 25248/50000 [15:46:09<12:56:24,  1.88s/it]
Outer training loop:  50%|#####     | 25249/50000 [15:46:11<12:52:14,  1.87s/it]
Outer training loop:  50%|#####     | 25250/50000 [15:46:13<12:42:43,  1.85s/it]
Outer training loop:  51%|#####     | 25251/50000 [15:46:15<13:07:31,  1.91s/it]
Outer training loop:  51%|#####     | 25252/50000 [15:46:17<12:56:02,  1.88s/it]
Outer training loop:  51%|#####     | 25253/50000 [15:46:18<12:46:43,  1.86s/it]
Outer training loop:  51%|#####     | 25254/50000 [15:46:21<13:24:13,  1.95s/it]
Outer training loop:  51%|#####     | 25255/50000 [15:46:22<13:12:14,  1.92s/it]
Outer training loop:  51%|#####     | 25256/50000 [15:46:24<13:01:59,  1.90s/it]
Outer training loop:  51%|#####     | 25257/50000 [15:46:26<13:19:43,  1.94s/it]
Outer training loop:  51%|#####     | 25258/50000 [15:46:28<13:01:34,  1.90s/it]
Outer training loop:  51%|#####     | 25259/50000 [15:46:30<12:35:35,  1.83s/it]
Outer training loop:  51%|#####     | 25260/50000 [15:46:32<12:39:12,  1.84s/it]
Outer training loop:  51%|#####     | 25261/50000 [15:46:34<13:06:13,  1.91s/it]
Outer training loop:  51%|#####     | 25262/50000 [15:46:36<12:56:15,  1.88s/it]
Outer training loop:  51%|#####     | 25263/50000 [15:46:37<12:49:02,  1.87s/it]
Outer training loop:  51%|#####     | 25264/50000 [15:46:39<13:09:15,  1.91s/it]
Outer training loop:  51%|#####     | 25265/50000 [15:46:41<12:49:29,  1.87s/it]
Outer training loop:  51%|#####     | 25266/50000 [15:46:43<12:54:06,  1.88s/it]
Outer training loop:  51%|#####     | 25267/50000 [15:46:45<13:09:48,  1.92s/it]
Outer training loop:  51%|#####     | 25268/50000 [15:46:47<12:59:43,  1.89s/it]
Outer training loop:  51%|#####     | 25269/50000 [15:46:49<13:19:04,  1.94s/it]
Outer training loop:  51%|#####     | 25270/50000 [15:46:51<13:10:37,  1.92s/it]
Outer training loop:  51%|#####     | 25271/50000 [15:46:53<13:12:15,  1.92s/it]
Outer training loop:  51%|#####     | 25272/50000 [15:46:55<12:57:20,  1.89s/it]
Outer training loop:  51%|#####     | 25273/50000 [15:46:56<13:00:22,  1.89s/it]
Outer training loop:  51%|#####     | 25274/50000 [15:46:59<13:26:04,  1.96s/it]
Outer training loop:  51%|#####     | 25275/50000 [15:47:00<13:03:29,  1.90s/it]
Outer training loop:  51%|#####     | 25276/50000 [15:47:02<13:10:26,  1.92s/it]
Outer training loop:  51%|#####     | 25277/50000 [15:47:04<13:34:48,  1.98s/it]
Outer training loop:  51%|#####     | 25278/50000 [15:47:06<13:31:45,  1.97s/it]
Outer training loop:  51%|#####     | 25279/50000 [15:47:08<13:18:51,  1.94s/it]
Outer training loop:  51%|#####     | 25280/50000 [15:47:10<13:25:00,  1.95s/it]
Outer training loop:  51%|#####     | 25281/50000 [15:47:12<13:05:12,  1.91s/it]
Outer training loop:  51%|#####     | 25282/50000 [15:47:14<12:47:14,  1.86s/it]
Outer training loop:  51%|#####     | 25283/50000 [15:47:16<12:35:32,  1.83s/it]
Outer training loop:  51%|#####     | 25284/50000 [15:47:18<13:06:31,  1.91s/it]
Outer training loop:  51%|#####     | 25285/50000 [15:47:19<12:54:40,  1.88s/it]
Outer training loop:  51%|#####     | 25286/50000 [15:47:21<12:43:08,  1.85s/it]
Outer training loop:  51%|#####     | 25287/50000 [15:47:23<12:53:19,  1.88s/it]
Outer training loop:  51%|#####     | 25288/50000 [15:47:25<12:46:57,  1.86s/it]
Outer training loop:  51%|#####     | 25289/50000 [15:47:27<13:03:12,  1.90s/it]
Outer training loop:  51%|#####     | 25290/50000 [15:47:29<12:56:41,  1.89s/it]
Outer training loop:  51%|#####     | 25291/50000 [15:47:31<13:06:59,  1.91s/it]
Outer training loop:  51%|#####     | 25292/50000 [15:47:33<13:09:52,  1.92s/it]
Outer training loop:  51%|#####     | 25293/50000 [15:47:35<12:50:06,  1.87s/it]
Outer training loop:  51%|#####     | 25294/50000 [15:47:37<13:14:31,  1.93s/it]
Outer training loop:  51%|#####     | 25295/50000 [15:47:38<13:05:52,  1.91s/it]
Outer training loop:  51%|#####     | 25296/50000 [15:47:40<13:03:40,  1.90s/it]
Outer training loop:  51%|#####     | 25297/50000 [15:47:42<13:32:15,  1.97s/it]
Outer training loop:  51%|#####     | 25298/50000 [15:47:44<13:05:59,  1.91s/it]
Outer training loop:  51%|#####     | 25299/50000 [15:47:46<13:01:02,  1.90s/it]
Outer training loop:  51%|#####     | 25300/50000 [15:47:48<13:12:27,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  51%|#####     | 25301/50000 [15:47:56<24:46:08,  3.61s/it]
Outer training loop:  51%|#####     | 25302/50000 [15:47:58<21:12:05,  3.09s/it]
Outer training loop:  51%|#####     | 25303/50000 [15:47:59<18:33:44,  2.71s/it]
Outer training loop:  51%|#####     | 25304/50000 [15:48:01<17:08:26,  2.50s/it]
Outer training loop:  51%|#####     | 25305/50000 [15:48:03<15:44:52,  2.30s/it]
Outer training loop:  51%|#####     | 25306/50000 [15:48:05<15:04:01,  2.20s/it]
Outer training loop:  51%|#####     | 25307/50000 [15:48:07<14:29:28,  2.11s/it]
Outer training loop:  51%|#####     | 25308/50000 [15:48:09<14:06:43,  2.06s/it]
Outer training loop:  51%|#####     | 25309/50000 [15:48:11<13:42:50,  2.00s/it]
Outer training loop:  51%|#####     | 25310/50000 [15:48:13<13:40:48,  1.99s/it]
Outer training loop:  51%|#####     | 25311/50000 [15:48:15<13:34:05,  1.98s/it]
Outer training loop:  51%|#####     | 25312/50000 [15:48:17<13:14:57,  1.93s/it]
Outer training loop:  51%|#####     | 25313/50000 [15:48:18<12:51:35,  1.88s/it]
Outer training loop:  51%|#####     | 25314/50000 [15:48:20<13:20:36,  1.95s/it]
Outer training loop:  51%|#####     | 25315/50000 [15:48:22<13:11:12,  1.92s/it]
Outer training loop:  51%|#####     | 25316/50000 [15:48:24<12:58:53,  1.89s/it]
Outer training loop:  51%|#####     | 25317/50000 [15:48:26<12:51:06,  1.87s/it]
Outer training loop:  51%|#####     | 25318/50000 [15:48:28<13:09:29,  1.92s/it]
Outer training loop:  51%|#####     | 25319/50000 [15:48:30<12:52:25,  1.88s/it]
Outer training loop:  51%|#####     | 25320/50000 [15:48:32<12:41:08,  1.85s/it]
Outer training loop:  51%|#####     | 25321/50000 [15:48:34<13:16:05,  1.94s/it]
Outer training loop:  51%|#####     | 25322/50000 [15:48:35<12:49:20,  1.87s/it]
Outer training loop:  51%|#####     | 25323/50000 [15:48:37<12:44:05,  1.86s/it]
Outer training loop:  51%|#####     | 25324/50000 [15:48:39<12:59:11,  1.89s/it]
Outer training loop:  51%|#####     | 25325/50000 [15:48:41<13:08:11,  1.92s/it]
Outer training loop:  51%|#####     | 25326/50000 [15:48:43<13:08:13,  1.92s/it]
Outer training loop:  51%|#####     | 25327/50000 [15:48:45<13:17:11,  1.94s/it]
Outer training loop:  51%|#####     | 25328/50000 [15:48:47<12:56:07,  1.89s/it]
Outer training loop:  51%|#####     | 25329/50000 [15:48:49<12:54:08,  1.88s/it]
Outer training loop:  51%|#####     | 25330/50000 [15:48:51<12:58:54,  1.89s/it]
Outer training loop:  51%|#####     | 25331/50000 [15:48:52<12:52:14,  1.88s/it]
Outer training loop:  51%|#####     | 25332/50000 [15:48:54<12:37:53,  1.84s/it]
Outer training loop:  51%|#####     | 25333/50000 [15:48:56<12:44:58,  1.86s/it]
Outer training loop:  51%|#####     | 25334/50000 [15:48:58<13:03:11,  1.91s/it]
Outer training loop:  51%|#####     | 25335/50000 [15:49:00<13:03:21,  1.91s/it]
Outer training loop:  51%|#####     | 25336/50000 [15:49:02<12:57:44,  1.89s/it]
Outer training loop:  51%|#####     | 25337/50000 [15:49:04<12:58:02,  1.89s/it]
Outer training loop:  51%|#####     | 25338/50000 [15:49:06<13:22:30,  1.95s/it]
Outer training loop:  51%|#####     | 25339/50000 [15:49:08<13:32:09,  1.98s/it]
Outer training loop:  51%|#####     | 25340/50000 [15:49:10<13:18:15,  1.94s/it]
Outer training loop:  51%|#####     | 25341/50000 [15:49:12<13:25:52,  1.96s/it]
Outer training loop:  51%|#####     | 25342/50000 [15:49:14<13:20:44,  1.95s/it]
Outer training loop:  51%|#####     | 25343/50000 [15:49:15<12:52:28,  1.88s/it]
Outer training loop:  51%|#####     | 25344/50000 [15:49:17<13:12:34,  1.93s/it]
Outer training loop:  51%|#####     | 25345/50000 [15:49:19<13:05:00,  1.91s/it]
Outer training loop:  51%|#####     | 25346/50000 [15:49:21<12:56:05,  1.89s/it]
Outer training loop:  51%|#####     | 25347/50000 [15:49:23<12:48:01,  1.87s/it]
Outer training loop:  51%|#####     | 25348/50000 [15:49:25<13:03:24,  1.91s/it]
Outer training loop:  51%|#####     | 25349/50000 [15:49:27<12:41:17,  1.85s/it]
Outer training loop:  51%|#####     | 25350/50000 [15:49:29<12:47:56,  1.87s/it]
Outer training loop:  51%|#####     | 25351/50000 [15:49:31<12:51:37,  1.88s/it]
Outer training loop:  51%|#####     | 25352/50000 [15:49:32<12:33:06,  1.83s/it]
Outer training loop:  51%|#####     | 25353/50000 [15:49:34<12:32:07,  1.83s/it]
Outer training loop:  51%|#####     | 25354/50000 [15:49:36<12:53:31,  1.88s/it]
Outer training loop:  51%|#####     | 25355/50000 [15:49:38<13:04:45,  1.91s/it]
Outer training loop:  51%|#####     | 25356/50000 [15:49:40<13:02:36,  1.91s/it]
Outer training loop:  51%|#####     | 25357/50000 [15:49:42<13:09:11,  1.92s/it]
Outer training loop:  51%|#####     | 25358/50000 [15:49:44<13:27:49,  1.97s/it]
Outer training loop:  51%|#####     | 25359/50000 [15:49:46<13:11:29,  1.93s/it]
Outer training loop:  51%|#####     | 25360/50000 [15:49:48<12:51:28,  1.88s/it]
Outer training loop:  51%|#####     | 25361/50000 [15:49:50<13:11:40,  1.93s/it]
Outer training loop:  51%|#####     | 25362/50000 [15:49:51<12:53:31,  1.88s/it]
Outer training loop:  51%|#####     | 25363/50000 [15:49:53<12:37:41,  1.85s/it]
Outer training loop:  51%|#####     | 25364/50000 [15:49:55<13:09:21,  1.92s/it]
Outer training loop:  51%|#####     | 25365/50000 [15:49:57<12:52:12,  1.88s/it]
Outer training loop:  51%|#####     | 25366/50000 [15:49:59<13:01:13,  1.90s/it]
Outer training loop:  51%|#####     | 25367/50000 [15:50:01<12:54:43,  1.89s/it]
Outer training loop:  51%|#####     | 25368/50000 [15:50:03<12:56:58,  1.89s/it]
Outer training loop:  51%|#####     | 25369/50000 [15:50:05<12:55:04,  1.89s/it]
Outer training loop:  51%|#####     | 25370/50000 [15:50:06<12:49:46,  1.88s/it]
Outer training loop:  51%|#####     | 25371/50000 [15:50:09<13:07:55,  1.92s/it]
Outer training loop:  51%|#####     | 25372/50000 [15:50:10<12:56:21,  1.89s/it]
Outer training loop:  51%|#####     | 25373/50000 [15:50:12<12:54:37,  1.89s/it]
Outer training loop:  51%|#####     | 25374/50000 [15:50:14<12:37:58,  1.85s/it]
Outer training loop:  51%|#####     | 25375/50000 [15:50:16<13:06:07,  1.92s/it]
Outer training loop:  51%|#####     | 25376/50000 [15:50:18<12:59:03,  1.90s/it]
Outer training loop:  51%|#####     | 25377/50000 [15:50:20<12:54:30,  1.89s/it]
Outer training loop:  51%|#####     | 25378/50000 [15:50:22<13:06:29,  1.92s/it]
Outer training loop:  51%|#####     | 25379/50000 [15:50:24<12:48:20,  1.87s/it]
Outer training loop:  51%|#####     | 25380/50000 [15:50:25<12:48:23,  1.87s/it]
Outer training loop:  51%|#####     | 25381/50000 [15:50:27<13:15:27,  1.94s/it]
Outer training loop:  51%|#####     | 25382/50000 [15:50:29<13:05:20,  1.91s/it]
Outer training loop:  51%|#####     | 25383/50000 [15:50:31<12:52:39,  1.88s/it]
Outer training loop:  51%|#####     | 25384/50000 [15:50:33<12:50:48,  1.88s/it]
Outer training loop:  51%|#####     | 25385/50000 [15:50:35<13:14:50,  1.94s/it]
Outer training loop:  51%|#####     | 25386/50000 [15:50:37<12:58:02,  1.90s/it]
Outer training loop:  51%|#####     | 25387/50000 [15:50:39<12:54:10,  1.89s/it]
Outer training loop:  51%|#####     | 25388/50000 [15:50:41<13:11:32,  1.93s/it]
Outer training loop:  51%|#####     | 25389/50000 [15:50:43<13:44:43,  2.01s/it]
Outer training loop:  51%|#####     | 25390/50000 [15:50:45<13:19:11,  1.95s/it]
Outer training loop:  51%|#####     | 25391/50000 [15:50:46<12:47:08,  1.87s/it]
Outer training loop:  51%|#####     | 25392/50000 [15:50:48<12:49:27,  1.88s/it]
Outer training loop:  51%|#####     | 25393/50000 [15:50:50<12:57:46,  1.90s/it]
Outer training loop:  51%|#####     | 25394/50000 [15:50:52<12:41:17,  1.86s/it]
Outer training loop:  51%|#####     | 25395/50000 [15:50:54<12:28:20,  1.82s/it]
Outer training loop:  51%|#####     | 25396/50000 [15:50:56<12:59:45,  1.90s/it]
Outer training loop:  51%|#####     | 25397/50000 [15:50:58<13:05:47,  1.92s/it]
Outer training loop:  51%|#####     | 25398/50000 [15:51:00<12:59:30,  1.90s/it]
Outer training loop:  51%|#####     | 25399/50000 [15:51:02<13:04:56,  1.91s/it]
Outer training loop:  51%|#####     | 25400/50000 [15:51:04<13:10:52,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.79s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  51%|#####     | 25401/50000 [15:51:11<24:26:59,  3.58s/it]
Outer training loop:  51%|#####     | 25402/50000 [15:51:13<21:13:45,  3.11s/it]
Outer training loop:  51%|#####     | 25403/50000 [15:51:15<18:38:57,  2.73s/it]
Outer training loop:  51%|#####     | 25404/50000 [15:51:17<17:26:31,  2.55s/it]
Outer training loop:  51%|#####     | 25405/50000 [15:51:19<16:03:54,  2.35s/it]
Outer training loop:  51%|#####     | 25406/50000 [15:51:21<15:07:51,  2.21s/it]
Outer training loop:  51%|#####     | 25407/50000 [15:51:23<14:50:05,  2.17s/it]
Outer training loop:  51%|#####     | 25408/50000 [15:51:25<14:05:49,  2.06s/it]
Outer training loop:  51%|#####     | 25409/50000 [15:51:27<13:38:18,  2.00s/it]
Outer training loop:  51%|#####     | 25410/50000 [15:51:28<13:13:06,  1.94s/it]
Outer training loop:  51%|#####     | 25411/50000 [15:51:30<13:19:22,  1.95s/it]
Outer training loop:  51%|#####     | 25412/50000 [15:51:32<13:12:38,  1.93s/it]
Outer training loop:  51%|#####     | 25413/50000 [15:51:34<12:37:11,  1.85s/it]
Outer training loop:  51%|#####     | 25414/50000 [15:51:36<13:01:40,  1.91s/it]
Outer training loop:  51%|#####     | 25415/50000 [15:51:38<12:52:59,  1.89s/it]
Outer training loop:  51%|#####     | 25416/50000 [15:51:40<12:59:01,  1.90s/it]
Outer training loop:  51%|#####     | 25417/50000 [15:51:42<13:11:48,  1.93s/it]
Outer training loop:  51%|#####     | 25418/50000 [15:51:44<13:10:35,  1.93s/it]
Outer training loop:  51%|#####     | 25419/50000 [15:51:46<13:05:31,  1.92s/it]
Outer training loop:  51%|#####     | 25420/50000 [15:51:48<13:22:46,  1.96s/it]
Outer training loop:  51%|#####     | 25421/50000 [15:51:49<12:58:30,  1.90s/it]
Outer training loop:  51%|#####     | 25422/50000 [15:51:51<12:44:31,  1.87s/it]
Outer training loop:  51%|#####     | 25423/50000 [15:51:53<13:05:01,  1.92s/it]
Outer training loop:  51%|#####     | 25424/50000 [15:51:55<12:53:32,  1.89s/it]
Outer training loop:  51%|#####     | 25425/50000 [15:51:57<12:41:15,  1.86s/it]
Outer training loop:  51%|#####     | 25426/50000 [15:51:59<12:35:36,  1.84s/it]
Outer training loop:  51%|#####     | 25427/50000 [15:52:01<13:07:22,  1.92s/it]
Outer training loop:  51%|#####     | 25428/50000 [15:52:02<12:49:33,  1.88s/it]
Outer training loop:  51%|#####     | 25429/50000 [15:52:04<12:31:34,  1.84s/it]
Outer training loop:  51%|#####     | 25430/50000 [15:52:06<12:59:42,  1.90s/it]
Outer training loop:  51%|#####     | 25431/50000 [15:52:08<12:59:22,  1.90s/it]
Outer training loop:  51%|#####     | 25432/50000 [15:52:10<12:51:34,  1.88s/it]
Outer training loop:  51%|#####     | 25433/50000 [15:52:12<12:54:03,  1.89s/it]
Outer training loop:  51%|#####     | 25434/50000 [15:52:14<12:52:43,  1.89s/it]
Outer training loop:  51%|#####     | 25435/50000 [15:52:16<12:51:24,  1.88s/it]
Outer training loop:  51%|#####     | 25436/50000 [15:52:18<13:11:19,  1.93s/it]
Outer training loop:  51%|#####     | 25437/50000 [15:52:20<13:19:59,  1.95s/it]
Outer training loop:  51%|#####     | 25438/50000 [15:52:22<13:11:45,  1.93s/it]
Outer training loop:  51%|#####     | 25439/50000 [15:52:23<12:46:56,  1.87s/it]
Outer training loop:  51%|#####     | 25440/50000 [15:52:25<13:08:23,  1.93s/it]
Outer training loop:  51%|#####     | 25441/50000 [15:52:27<13:03:06,  1.91s/it]
Outer training loop:  51%|#####     | 25442/50000 [15:52:29<13:13:35,  1.94s/it]
Outer training loop:  51%|#####     | 25443/50000 [15:52:31<13:20:21,  1.96s/it]
Outer training loop:  51%|#####     | 25444/50000 [15:52:33<12:53:10,  1.89s/it]
Outer training loop:  51%|#####     | 25445/50000 [15:52:35<12:44:25,  1.87s/it]
Outer training loop:  51%|#####     | 25446/50000 [15:52:37<13:14:32,  1.94s/it]
Outer training loop:  51%|#####     | 25447/50000 [15:52:39<12:50:12,  1.88s/it]
Outer training loop:  51%|#####     | 25448/50000 [15:52:40<12:42:37,  1.86s/it]
Outer training loop:  51%|#####     | 25449/50000 [15:52:42<12:50:22,  1.88s/it]
Outer training loop:  51%|#####     | 25450/50000 [15:52:45<13:15:22,  1.94s/it]
Outer training loop:  51%|#####     | 25451/50000 [15:52:46<13:03:21,  1.91s/it]
Outer training loop:  51%|#####     | 25452/50000 [15:52:48<12:50:04,  1.88s/it]
Outer training loop:  51%|#####     | 25453/50000 [15:52:50<13:22:54,  1.96s/it]
Outer training loop:  51%|#####     | 25454/50000 [15:52:52<12:51:51,  1.89s/it]
Outer training loop:  51%|#####     | 25455/50000 [15:52:54<12:44:52,  1.87s/it]
Outer training loop:  51%|#####     | 25456/50000 [15:52:56<12:34:19,  1.84s/it]
Outer training loop:  51%|#####     | 25457/50000 [15:52:58<13:03:55,  1.92s/it]
Outer training loop:  51%|#####     | 25458/50000 [15:53:00<13:02:01,  1.91s/it]
Outer training loop:  51%|#####     | 25459/50000 [15:53:01<12:49:31,  1.88s/it]
Outer training loop:  51%|#####     | 25460/50000 [15:53:03<13:04:31,  1.92s/it]
Outer training loop:  51%|#####     | 25461/50000 [15:53:05<13:00:16,  1.91s/it]
Outer training loop:  51%|#####     | 25462/50000 [15:53:07<12:55:42,  1.90s/it]
Outer training loop:  51%|#####     | 25463/50000 [15:53:09<13:04:49,  1.92s/it]
Outer training loop:  51%|#####     | 25464/50000 [15:53:11<12:48:19,  1.88s/it]
Outer training loop:  51%|#####     | 25465/50000 [15:53:13<12:44:04,  1.87s/it]
Outer training loop:  51%|#####     | 25466/50000 [15:53:14<12:23:37,  1.82s/it]
Outer training loop:  51%|#####     | 25467/50000 [15:53:16<12:45:01,  1.87s/it]
Outer training loop:  51%|#####     | 25468/50000 [15:53:18<12:33:29,  1.84s/it]
Outer training loop:  51%|#####     | 25469/50000 [15:53:20<12:44:34,  1.87s/it]
Outer training loop:  51%|#####     | 25470/50000 [15:53:22<13:04:39,  1.92s/it]
Outer training loop:  51%|#####     | 25471/50000 [15:53:24<12:55:02,  1.90s/it]
Outer training loop:  51%|#####     | 25472/50000 [15:53:26<12:46:31,  1.88s/it]
Outer training loop:  51%|#####     | 25473/50000 [15:53:28<12:35:32,  1.85s/it]
Outer training loop:  51%|#####     | 25474/50000 [15:53:30<13:04:19,  1.92s/it]
Outer training loop:  51%|#####     | 25475/50000 [15:53:32<12:46:44,  1.88s/it]
Outer training loop:  51%|#####     | 25476/50000 [15:53:33<12:30:40,  1.84s/it]
Outer training loop:  51%|#####     | 25477/50000 [15:53:35<12:50:48,  1.89s/it]
Outer training loop:  51%|#####     | 25478/50000 [15:53:37<12:28:54,  1.83s/it]
Outer training loop:  51%|#####     | 25479/50000 [15:53:39<12:10:37,  1.79s/it]
Outer training loop:  51%|#####     | 25480/50000 [15:53:41<12:17:17,  1.80s/it]
Outer training loop:  51%|#####     | 25481/50000 [15:53:43<12:52:33,  1.89s/it]
Outer training loop:  51%|#####     | 25482/50000 [15:53:44<12:34:12,  1.85s/it]
Outer training loop:  51%|#####     | 25483/50000 [15:53:46<12:25:55,  1.83s/it]
Outer training loop:  51%|#####     | 25484/50000 [15:53:48<12:47:51,  1.88s/it]
Outer training loop:  51%|#####     | 25485/50000 [15:53:50<12:53:11,  1.89s/it]
Outer training loop:  51%|#####     | 25486/50000 [15:53:52<12:58:45,  1.91s/it]
Outer training loop:  51%|#####     | 25487/50000 [15:53:54<12:59:27,  1.91s/it]
Outer training loop:  51%|#####     | 25488/50000 [15:53:56<12:49:26,  1.88s/it]
Outer training loop:  51%|#####     | 25489/50000 [15:53:58<12:44:33,  1.87s/it]
Outer training loop:  51%|#####     | 25490/50000 [15:53:59<12:30:20,  1.84s/it]
Outer training loop:  51%|#####     | 25491/50000 [15:54:01<13:07:16,  1.93s/it]
Outer training loop:  51%|#####     | 25492/50000 [15:54:03<13:02:41,  1.92s/it]
Outer training loop:  51%|#####     | 25493/50000 [15:54:05<12:48:16,  1.88s/it]
Outer training loop:  51%|#####     | 25494/50000 [15:54:07<13:12:09,  1.94s/it]
Outer training loop:  51%|#####     | 25495/50000 [15:54:09<12:53:21,  1.89s/it]
Outer training loop:  51%|#####     | 25496/50000 [15:54:11<12:41:26,  1.86s/it]
Outer training loop:  51%|#####     | 25497/50000 [15:54:13<12:50:21,  1.89s/it]
Outer training loop:  51%|#####     | 25498/50000 [15:54:15<13:11:43,  1.94s/it]
Outer training loop:  51%|#####     | 25499/50000 [15:54:17<13:03:12,  1.92s/it]
Outer training loop:  51%|#####1    | 25500/50000 [15:54:18<12:40:49,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:46,  2.04s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:40,  1.99s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:34,  1.94s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:35,  1.97s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:30,  1.93s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.90s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:26,  1.95s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.95s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:17,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:16,  1.89s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:17,  1.93s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:04,  1.83s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<01:59,  1.84s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:57,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:50,  1.93s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.87s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:41,  1.99s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:37,  1.96s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:33,  1.90s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.89s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:25,  1.86s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:26,  1.93s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:24,  1.93s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.85s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.85s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.86s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:15,  1.94s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.91s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:08,  1.84s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.84s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<00:59,  1.81s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:58,  1.83s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<01:00,  1.96s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:57,  1.91s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:54,  1.87s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:47,  1.96s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:44,  1.93s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.81s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:31,  1.96s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.91s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.80s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:14,  1.83s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.99s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.95s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.90s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.94s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.86s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:28,  1.81s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:24,  1.79s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:37,  1.97s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:34,  1.96s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:29,  1.91s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.94s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:18,  1.90s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:14,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:14,  1.97s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.94s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.93s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.91s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:52,  1.84s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:50,  1.84s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:46,  1.87s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:41,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:40,  1.84s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:38,  1.83s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:41,  1.92s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:35,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:33,  1.96s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:31,  1.94s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:28,  1.92s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:23,  1.86s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:19,  1.80s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.82s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:19,  1.94s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:08,  1.85s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:07,  1.93s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:07,  1.99s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.92s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.90s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:52,  1.81s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:54,  1.94s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.91s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.83s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.83s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:38,  1.82s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.95s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.91s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.96s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.88s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.88s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.87s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.85s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.85s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.81s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.79s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.80s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.95s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.89s/it]

Outer training loop:  51%|#####1    | 25501/50000 [15:59:39<663:14:37, 97.46s/it]
Outer training loop:  51%|#####1    | 25502/50000 [15:59:41<468:01:55, 68.78s/it]
Outer training loop:  51%|#####1    | 25503/50000 [15:59:43<331:29:07, 48.71s/it]
Outer training loop:  51%|#####1    | 25504/50000 [15:59:44<235:37:25, 34.63s/it]
Outer training loop:  51%|#####1    | 25505/50000 [15:59:46<168:43:47, 24.80s/it]
Outer training loop:  51%|#####1    | 25506/50000 [15:59:48<121:46:42, 17.90s/it]
Outer training loop:  51%|#####1    | 25507/50000 [15:59:50<89:11:17, 13.11s/it] 
Outer training loop:  51%|#####1    | 25508/50000 [15:59:52<66:17:17,  9.74s/it]
Outer training loop:  51%|#####1    | 25509/50000 [15:59:54<51:14:41,  7.53s/it]
Outer training loop:  51%|#####1    | 25510/50000 [15:59:56<39:23:10,  5.79s/it]
Outer training loop:  51%|#####1    | 25511/50000 [15:59:58<31:11:18,  4.58s/it]
Outer training loop:  51%|#####1    | 25512/50000 [16:00:00<25:31:05,  3.75s/it]
Outer training loop:  51%|#####1    | 25513/50000 [16:00:02<22:02:23,  3.24s/it]
Outer training loop:  51%|#####1    | 25514/50000 [16:00:04<19:10:03,  2.82s/it]
Outer training loop:  51%|#####1    | 25515/50000 [16:00:05<17:08:32,  2.52s/it]
Outer training loop:  51%|#####1    | 25516/50000 [16:00:07<16:13:33,  2.39s/it]
Outer training loop:  51%|#####1    | 25517/50000 [16:00:09<15:26:28,  2.27s/it]
Outer training loop:  51%|#####1    | 25518/50000 [16:00:11<14:48:55,  2.18s/it]
Outer training loop:  51%|#####1    | 25519/50000 [16:00:13<14:32:53,  2.14s/it]
Outer training loop:  51%|#####1    | 25520/50000 [16:00:15<13:52:45,  2.04s/it]
Outer training loop:  51%|#####1    | 25521/50000 [16:00:17<13:30:17,  1.99s/it]
Outer training loop:  51%|#####1    | 25522/50000 [16:00:19<13:52:40,  2.04s/it]
Outer training loop:  51%|#####1    | 25523/50000 [16:00:21<13:29:18,  1.98s/it]
Outer training loop:  51%|#####1    | 25524/50000 [16:00:23<12:53:07,  1.90s/it]
Outer training loop:  51%|#####1    | 25525/50000 [16:00:25<12:54:20,  1.90s/it]
Outer training loop:  51%|#####1    | 25526/50000 [16:00:27<13:03:54,  1.92s/it]
Outer training loop:  51%|#####1    | 25527/50000 [16:00:29<13:02:20,  1.92s/it]
Outer training loop:  51%|#####1    | 25528/50000 [16:00:30<12:51:21,  1.89s/it]
Outer training loop:  51%|#####1    | 25529/50000 [16:00:32<13:11:41,  1.94s/it]
Outer training loop:  51%|#####1    | 25530/50000 [16:00:34<12:57:44,  1.91s/it]
Outer training loop:  51%|#####1    | 25531/50000 [16:00:36<13:02:56,  1.92s/it]
Outer training loop:  51%|#####1    | 25532/50000 [16:00:38<12:58:52,  1.91s/it]
Outer training loop:  51%|#####1    | 25533/50000 [16:00:40<13:21:32,  1.97s/it]
Outer training loop:  51%|#####1    | 25534/50000 [16:00:42<13:09:11,  1.94s/it]
Outer training loop:  51%|#####1    | 25535/50000 [16:00:44<12:50:04,  1.89s/it]
Outer training loop:  51%|#####1    | 25536/50000 [16:00:46<13:00:06,  1.91s/it]
Outer training loop:  51%|#####1    | 25537/50000 [16:00:48<12:54:40,  1.90s/it]
Outer training loop:  51%|#####1    | 25538/50000 [16:00:49<12:35:28,  1.85s/it]
Outer training loop:  51%|#####1    | 25539/50000 [16:00:52<12:59:32,  1.91s/it]
Outer training loop:  51%|#####1    | 25540/50000 [16:00:53<12:50:58,  1.89s/it]
Outer training loop:  51%|#####1    | 25541/50000 [16:00:55<12:57:25,  1.91s/it]
Outer training loop:  51%|#####1    | 25542/50000 [16:00:57<12:41:44,  1.87s/it]
Outer training loop:  51%|#####1    | 25543/50000 [16:00:59<12:54:58,  1.90s/it]
Outer training loop:  51%|#####1    | 25544/50000 [16:01:01<12:48:04,  1.88s/it]
Outer training loop:  51%|#####1    | 25545/50000 [16:01:03<12:44:48,  1.88s/it]
Outer training loop:  51%|#####1    | 25546/50000 [16:01:05<13:07:37,  1.93s/it]
Outer training loop:  51%|#####1    | 25547/50000 [16:01:07<12:54:03,  1.90s/it]
Outer training loop:  51%|#####1    | 25548/50000 [16:01:09<12:49:50,  1.89s/it]
Outer training loop:  51%|#####1    | 25549/50000 [16:01:10<12:55:50,  1.90s/it]
Outer training loop:  51%|#####1    | 25550/50000 [16:01:12<12:55:09,  1.90s/it]
Outer training loop:  51%|#####1    | 25551/50000 [16:01:14<12:38:21,  1.86s/it]
Outer training loop:  51%|#####1    | 25552/50000 [16:01:16<12:44:34,  1.88s/it]
Outer training loop:  51%|#####1    | 25553/50000 [16:01:18<12:43:25,  1.87s/it]
Outer training loop:  51%|#####1    | 25554/50000 [16:01:20<12:30:06,  1.84s/it]
Outer training loop:  51%|#####1    | 25555/50000 [16:01:21<12:26:31,  1.83s/it]
Outer training loop:  51%|#####1    | 25556/50000 [16:01:24<12:54:03,  1.90s/it]
Outer training loop:  51%|#####1    | 25557/50000 [16:01:25<12:36:08,  1.86s/it]
Outer training loop:  51%|#####1    | 25558/50000 [16:01:27<12:34:05,  1.85s/it]
Outer training loop:  51%|#####1    | 25559/50000 [16:01:29<12:59:32,  1.91s/it]
Outer training loop:  51%|#####1    | 25560/50000 [16:01:31<12:47:08,  1.88s/it]
Outer training loop:  51%|#####1    | 25561/50000 [16:01:33<12:34:14,  1.85s/it]
Outer training loop:  51%|#####1    | 25562/50000 [16:01:35<12:30:17,  1.84s/it]
Outer training loop:  51%|#####1    | 25563/50000 [16:01:37<12:57:49,  1.91s/it]
Outer training loop:  51%|#####1    | 25564/50000 [16:01:38<12:39:14,  1.86s/it]
Outer training loop:  51%|#####1    | 25565/50000 [16:01:40<12:37:03,  1.86s/it]
Outer training loop:  51%|#####1    | 25566/50000 [16:01:42<12:39:14,  1.86s/it]
Outer training loop:  51%|#####1    | 25567/50000 [16:01:44<13:35:22,  2.00s/it]
Outer training loop:  51%|#####1    | 25568/50000 [16:01:46<13:20:48,  1.97s/it]
Outer training loop:  51%|#####1    | 25569/50000 [16:01:48<12:52:32,  1.90s/it]
Outer training loop:  51%|#####1    | 25570/50000 [16:01:50<13:15:08,  1.95s/it]
Outer training loop:  51%|#####1    | 25571/50000 [16:01:52<13:07:07,  1.93s/it]
Outer training loop:  51%|#####1    | 25572/50000 [16:01:54<13:03:00,  1.92s/it]
Outer training loop:  51%|#####1    | 25573/50000 [16:01:56<12:51:44,  1.90s/it]
Outer training loop:  51%|#####1    | 25574/50000 [16:01:58<13:02:03,  1.92s/it]
Outer training loop:  51%|#####1    | 25575/50000 [16:02:00<12:41:45,  1.87s/it]
Outer training loop:  51%|#####1    | 25576/50000 [16:02:01<12:34:41,  1.85s/it]
Outer training loop:  51%|#####1    | 25577/50000 [16:02:03<12:52:20,  1.90s/it]
Outer training loop:  51%|#####1    | 25578/50000 [16:02:05<12:35:02,  1.85s/it]
Outer training loop:  51%|#####1    | 25579/50000 [16:02:07<12:31:40,  1.85s/it]
Outer training loop:  51%|#####1    | 25580/50000 [16:02:09<12:59:31,  1.92s/it]
Outer training loop:  51%|#####1    | 25581/50000 [16:02:11<12:55:03,  1.90s/it]
Outer training loop:  51%|#####1    | 25582/50000 [16:02:13<12:43:49,  1.88s/it]
Outer training loop:  51%|#####1    | 25583/50000 [16:02:15<12:42:46,  1.87s/it]
Outer training loop:  51%|#####1    | 25584/50000 [16:02:17<13:01:59,  1.92s/it]
Outer training loop:  51%|#####1    | 25585/50000 [16:02:19<13:03:06,  1.92s/it]
Outer training loop:  51%|#####1    | 25586/50000 [16:02:20<12:55:50,  1.91s/it]
Outer training loop:  51%|#####1    | 25587/50000 [16:02:22<12:48:30,  1.89s/it]
Outer training loop:  51%|#####1    | 25588/50000 [16:02:24<12:46:40,  1.88s/it]
Outer training loop:  51%|#####1    | 25589/50000 [16:02:26<12:53:31,  1.90s/it]
Outer training loop:  51%|#####1    | 25590/50000 [16:02:28<13:21:26,  1.97s/it]
Outer training loop:  51%|#####1    | 25591/50000 [16:02:30<13:02:15,  1.92s/it]
Outer training loop:  51%|#####1    | 25592/50000 [16:02:32<12:51:27,  1.90s/it]
Outer training loop:  51%|#####1    | 25593/50000 [16:02:34<13:14:57,  1.95s/it]
Outer training loop:  51%|#####1    | 25594/50000 [16:02:36<12:46:45,  1.89s/it]
Outer training loop:  51%|#####1    | 25595/50000 [16:02:37<12:43:03,  1.88s/it]
Outer training loop:  51%|#####1    | 25596/50000 [16:02:39<12:40:54,  1.87s/it]
Outer training loop:  51%|#####1    | 25597/50000 [16:02:41<12:59:21,  1.92s/it]
Outer training loop:  51%|#####1    | 25598/50000 [16:02:43<12:55:48,  1.91s/it]
Outer training loop:  51%|#####1    | 25599/50000 [16:02:45<12:44:47,  1.88s/it]
Outer training loop:  51%|#####1    | 25600/50000 [16:02:47<13:19:07,  1.97s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  51%|#####1    | 25601/50000 [16:02:55<24:12:21,  3.57s/it]
Outer training loop:  51%|#####1    | 25602/50000 [16:02:56<20:37:37,  3.04s/it]
Outer training loop:  51%|#####1    | 25603/50000 [16:02:58<18:24:58,  2.72s/it]
Outer training loop:  51%|#####1    | 25604/50000 [16:03:00<16:37:53,  2.45s/it]
Outer training loop:  51%|#####1    | 25605/50000 [16:03:02<15:39:55,  2.31s/it]
Outer training loop:  51%|#####1    | 25606/50000 [16:03:04<14:24:52,  2.13s/it]
Outer training loop:  51%|#####1    | 25607/50000 [16:03:06<14:00:49,  2.07s/it]
Outer training loop:  51%|#####1    | 25608/50000 [16:03:08<13:42:19,  2.02s/it]
Outer training loop:  51%|#####1    | 25609/50000 [16:03:09<13:05:50,  1.93s/it]
Outer training loop:  51%|#####1    | 25610/50000 [16:03:11<13:06:31,  1.93s/it]
Outer training loop:  51%|#####1    | 25611/50000 [16:03:13<13:11:00,  1.95s/it]
Outer training loop:  51%|#####1    | 25612/50000 [16:03:15<12:58:20,  1.91s/it]
Outer training loop:  51%|#####1    | 25613/50000 [16:03:17<12:39:33,  1.87s/it]
Outer training loop:  51%|#####1    | 25614/50000 [16:03:19<13:32:59,  2.00s/it]
Outer training loop:  51%|#####1    | 25615/50000 [16:03:21<13:19:59,  1.97s/it]
Outer training loop:  51%|#####1    | 25616/50000 [16:03:23<12:58:26,  1.92s/it]
Outer training loop:  51%|#####1    | 25617/50000 [16:03:25<13:45:16,  2.03s/it]
Outer training loop:  51%|#####1    | 25618/50000 [16:03:27<13:05:46,  1.93s/it]
Outer training loop:  51%|#####1    | 25619/50000 [16:03:29<12:49:49,  1.89s/it]
Outer training loop:  51%|#####1    | 25620/50000 [16:03:31<12:56:46,  1.91s/it]
Outer training loop:  51%|#####1    | 25621/50000 [16:03:33<13:13:23,  1.95s/it]
Outer training loop:  51%|#####1    | 25622/50000 [16:03:35<13:03:37,  1.93s/it]
Outer training loop:  51%|#####1    | 25623/50000 [16:03:36<12:51:17,  1.90s/it]
Outer training loop:  51%|#####1    | 25624/50000 [16:03:38<13:04:05,  1.93s/it]
Outer training loop:  51%|#####1    | 25625/50000 [16:03:40<12:47:10,  1.89s/it]
Outer training loop:  51%|#####1    | 25626/50000 [16:03:42<12:33:38,  1.86s/it]
Outer training loop:  51%|#####1    | 25627/50000 [16:03:44<13:19:29,  1.97s/it]
Outer training loop:  51%|#####1    | 25628/50000 [16:03:46<13:05:36,  1.93s/it]
Outer training loop:  51%|#####1    | 25629/50000 [16:03:48<12:51:18,  1.90s/it]
Outer training loop:  51%|#####1    | 25630/50000 [16:03:50<12:52:25,  1.90s/it]
Outer training loop:  51%|#####1    | 25631/50000 [16:03:52<13:10:49,  1.95s/it]
Outer training loop:  51%|#####1    | 25632/50000 [16:03:54<13:09:40,  1.94s/it]
Outer training loop:  51%|#####1    | 25633/50000 [16:03:56<13:02:15,  1.93s/it]
Outer training loop:  51%|#####1    | 25634/50000 [16:03:58<13:11:29,  1.95s/it]
Outer training loop:  51%|#####1    | 25635/50000 [16:03:59<12:45:30,  1.89s/it]
Outer training loop:  51%|#####1    | 25636/50000 [16:04:01<12:35:28,  1.86s/it]
Outer training loop:  51%|#####1    | 25637/50000 [16:04:03<12:48:19,  1.89s/it]
Outer training loop:  51%|#####1    | 25638/50000 [16:04:05<13:05:13,  1.93s/it]
Outer training loop:  51%|#####1    | 25639/50000 [16:04:07<12:44:30,  1.88s/it]
Outer training loop:  51%|#####1    | 25640/50000 [16:04:09<12:40:46,  1.87s/it]
Outer training loop:  51%|#####1    | 25641/50000 [16:04:11<13:14:10,  1.96s/it]
Outer training loop:  51%|#####1    | 25642/50000 [16:04:13<12:59:08,  1.92s/it]
Outer training loop:  51%|#####1    | 25643/50000 [16:04:15<12:39:46,  1.87s/it]
Outer training loop:  51%|#####1    | 25644/50000 [16:04:16<12:42:56,  1.88s/it]
Outer training loop:  51%|#####1    | 25645/50000 [16:04:18<12:55:08,  1.91s/it]
Outer training loop:  51%|#####1    | 25646/50000 [16:04:20<12:51:32,  1.90s/it]
Outer training loop:  51%|#####1    | 25647/50000 [16:04:22<12:40:14,  1.87s/it]
Outer training loop:  51%|#####1    | 25648/50000 [16:04:24<12:49:15,  1.90s/it]
Outer training loop:  51%|#####1    | 25649/50000 [16:04:26<12:38:37,  1.87s/it]
Outer training loop:  51%|#####1    | 25650/50000 [16:04:28<12:36:27,  1.86s/it]
Outer training loop:  51%|#####1    | 25651/50000 [16:04:30<12:48:48,  1.89s/it]
Outer training loop:  51%|#####1    | 25652/50000 [16:04:31<12:30:17,  1.85s/it]
Outer training loop:  51%|#####1    | 25653/50000 [16:04:33<12:11:47,  1.80s/it]
Outer training loop:  51%|#####1    | 25654/50000 [16:04:35<12:08:37,  1.80s/it]
Outer training loop:  51%|#####1    | 25655/50000 [16:04:37<12:23:45,  1.83s/it]
Outer training loop:  51%|#####1    | 25656/50000 [16:04:39<12:13:39,  1.81s/it]
Outer training loop:  51%|#####1    | 25657/50000 [16:04:41<12:29:48,  1.85s/it]
Outer training loop:  51%|#####1    | 25658/50000 [16:04:42<12:38:07,  1.87s/it]
Outer training loop:  51%|#####1    | 25659/50000 [16:04:44<12:38:50,  1.87s/it]
Outer training loop:  51%|#####1    | 25660/50000 [16:04:46<12:30:55,  1.85s/it]
Outer training loop:  51%|#####1    | 25661/50000 [16:04:48<12:18:07,  1.82s/it]
Outer training loop:  51%|#####1    | 25662/50000 [16:04:50<13:07:33,  1.94s/it]
Outer training loop:  51%|#####1    | 25663/50000 [16:04:52<12:46:53,  1.89s/it]
Outer training loop:  51%|#####1    | 25664/50000 [16:04:54<13:01:46,  1.93s/it]
Outer training loop:  51%|#####1    | 25665/50000 [16:04:56<13:12:33,  1.95s/it]
Outer training loop:  51%|#####1    | 25666/50000 [16:04:58<13:02:31,  1.93s/it]
Outer training loop:  51%|#####1    | 25667/50000 [16:05:00<12:53:07,  1.91s/it]
Outer training loop:  51%|#####1    | 25668/50000 [16:05:01<12:42:09,  1.88s/it]
Outer training loop:  51%|#####1    | 25669/50000 [16:05:03<12:59:20,  1.92s/it]
Outer training loop:  51%|#####1    | 25670/50000 [16:05:05<13:08:38,  1.94s/it]
Outer training loop:  51%|#####1    | 25671/50000 [16:05:07<13:01:27,  1.93s/it]
Outer training loop:  51%|#####1    | 25672/50000 [16:05:09<13:09:11,  1.95s/it]
Outer training loop:  51%|#####1    | 25673/50000 [16:05:11<12:52:54,  1.91s/it]
Outer training loop:  51%|#####1    | 25674/50000 [16:05:13<13:20:29,  1.97s/it]
Outer training loop:  51%|#####1    | 25675/50000 [16:05:15<13:34:29,  2.01s/it]
Outer training loop:  51%|#####1    | 25676/50000 [16:05:17<13:08:14,  1.94s/it]
Outer training loop:  51%|#####1    | 25677/50000 [16:05:19<13:05:30,  1.94s/it]
Outer training loop:  51%|#####1    | 25678/50000 [16:05:21<12:53:30,  1.91s/it]
Outer training loop:  51%|#####1    | 25679/50000 [16:05:23<12:52:14,  1.91s/it]
Outer training loop:  51%|#####1    | 25680/50000 [16:05:25<12:46:38,  1.89s/it]
Outer training loop:  51%|#####1    | 25681/50000 [16:05:27<12:40:41,  1.88s/it]
Outer training loop:  51%|#####1    | 25682/50000 [16:05:29<12:57:32,  1.92s/it]
Outer training loop:  51%|#####1    | 25683/50000 [16:05:30<12:58:46,  1.92s/it]
Outer training loop:  51%|#####1    | 25684/50000 [16:05:32<12:47:50,  1.89s/it]
Outer training loop:  51%|#####1    | 25685/50000 [16:05:34<12:57:05,  1.92s/it]
Outer training loop:  51%|#####1    | 25686/50000 [16:05:36<12:56:03,  1.92s/it]
Outer training loop:  51%|#####1    | 25687/50000 [16:05:38<12:48:20,  1.90s/it]
Outer training loop:  51%|#####1    | 25688/50000 [16:05:40<12:57:14,  1.92s/it]
Outer training loop:  51%|#####1    | 25689/50000 [16:05:42<13:13:23,  1.96s/it]
Outer training loop:  51%|#####1    | 25690/50000 [16:05:44<13:01:32,  1.93s/it]
Outer training loop:  51%|#####1    | 25691/50000 [16:05:46<12:31:39,  1.86s/it]
Outer training loop:  51%|#####1    | 25692/50000 [16:05:47<12:30:56,  1.85s/it]
Outer training loop:  51%|#####1    | 25693/50000 [16:05:49<12:38:52,  1.87s/it]
Outer training loop:  51%|#####1    | 25694/50000 [16:05:51<12:41:31,  1.88s/it]
Outer training loop:  51%|#####1    | 25695/50000 [16:05:53<12:30:07,  1.85s/it]
Outer training loop:  51%|#####1    | 25696/50000 [16:05:55<12:53:05,  1.91s/it]
Outer training loop:  51%|#####1    | 25697/50000 [16:05:57<12:44:02,  1.89s/it]
Outer training loop:  51%|#####1    | 25698/50000 [16:05:59<12:49:03,  1.90s/it]
Outer training loop:  51%|#####1    | 25699/50000 [16:06:01<13:01:08,  1.93s/it]
Outer training loop:  51%|#####1    | 25700/50000 [16:06:03<12:42:38,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.01s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  51%|#####1    | 25701/50000 [16:06:10<24:06:14,  3.57s/it]
Outer training loop:  51%|#####1    | 25702/50000 [16:06:12<20:47:34,  3.08s/it]
Outer training loop:  51%|#####1    | 25703/50000 [16:06:14<18:31:50,  2.75s/it]
Outer training loop:  51%|#####1    | 25704/50000 [16:06:16<16:33:21,  2.45s/it]
Outer training loop:  51%|#####1    | 25705/50000 [16:06:18<15:06:36,  2.24s/it]
Outer training loop:  51%|#####1    | 25706/50000 [16:06:19<14:17:01,  2.12s/it]
Outer training loop:  51%|#####1    | 25707/50000 [16:06:21<14:02:42,  2.08s/it]
Outer training loop:  51%|#####1    | 25708/50000 [16:06:23<13:23:10,  1.98s/it]
Outer training loop:  51%|#####1    | 25709/50000 [16:06:25<12:50:09,  1.90s/it]
Outer training loop:  51%|#####1    | 25710/50000 [16:06:27<13:08:14,  1.95s/it]
Outer training loop:  51%|#####1    | 25711/50000 [16:06:29<13:18:35,  1.97s/it]
Outer training loop:  51%|#####1    | 25712/50000 [16:06:31<12:52:45,  1.91s/it]
Outer training loop:  51%|#####1    | 25713/50000 [16:06:33<12:45:41,  1.89s/it]
Outer training loop:  51%|#####1    | 25714/50000 [16:06:34<12:32:57,  1.86s/it]
Outer training loop:  51%|#####1    | 25715/50000 [16:06:36<12:31:14,  1.86s/it]
Outer training loop:  51%|#####1    | 25716/50000 [16:06:38<12:55:10,  1.92s/it]
Outer training loop:  51%|#####1    | 25717/50000 [16:06:40<12:55:38,  1.92s/it]
Outer training loop:  51%|#####1    | 25718/50000 [16:06:42<12:44:32,  1.89s/it]
Outer training loop:  51%|#####1    | 25719/50000 [16:06:44<12:56:07,  1.92s/it]
Outer training loop:  51%|#####1    | 25720/50000 [16:06:46<13:12:29,  1.96s/it]
Outer training loop:  51%|#####1    | 25721/50000 [16:06:48<13:04:25,  1.94s/it]
Outer training loop:  51%|#####1    | 25722/50000 [16:06:50<13:09:44,  1.95s/it]
Outer training loop:  51%|#####1    | 25723/50000 [16:06:52<13:12:43,  1.96s/it]
Outer training loop:  51%|#####1    | 25724/50000 [16:06:54<12:50:28,  1.90s/it]
Outer training loop:  51%|#####1    | 25725/50000 [16:06:55<12:33:06,  1.86s/it]
Outer training loop:  51%|#####1    | 25726/50000 [16:06:57<12:45:28,  1.89s/it]
Outer training loop:  51%|#####1    | 25727/50000 [16:06:59<12:38:20,  1.87s/it]
Outer training loop:  51%|#####1    | 25728/50000 [16:07:01<12:28:39,  1.85s/it]
Outer training loop:  51%|#####1    | 25729/50000 [16:07:03<12:16:50,  1.82s/it]
Outer training loop:  51%|#####1    | 25730/50000 [16:07:05<12:44:56,  1.89s/it]
Outer training loop:  51%|#####1    | 25731/50000 [16:07:07<12:31:29,  1.86s/it]
Outer training loop:  51%|#####1    | 25732/50000 [16:07:08<12:23:35,  1.84s/it]
Outer training loop:  51%|#####1    | 25733/50000 [16:07:10<12:49:13,  1.90s/it]
Outer training loop:  51%|#####1    | 25734/50000 [16:07:12<12:29:58,  1.85s/it]
Outer training loop:  51%|#####1    | 25735/50000 [16:07:14<12:33:40,  1.86s/it]
Outer training loop:  51%|#####1    | 25736/50000 [16:07:16<12:28:10,  1.85s/it]
Outer training loop:  51%|#####1    | 25737/50000 [16:07:18<12:46:48,  1.90s/it]
Outer training loop:  51%|#####1    | 25738/50000 [16:07:20<12:38:11,  1.87s/it]
Outer training loop:  51%|#####1    | 25739/50000 [16:07:22<12:40:55,  1.88s/it]
Outer training loop:  51%|#####1    | 25740/50000 [16:07:24<13:06:31,  1.95s/it]
Outer training loop:  51%|#####1    | 25741/50000 [16:07:26<12:57:54,  1.92s/it]
Outer training loop:  51%|#####1    | 25742/50000 [16:07:27<12:44:08,  1.89s/it]
Outer training loop:  51%|#####1    | 25743/50000 [16:07:29<12:37:32,  1.87s/it]
Outer training loop:  51%|#####1    | 25744/50000 [16:07:31<12:43:52,  1.89s/it]
Outer training loop:  51%|#####1    | 25745/50000 [16:07:33<12:31:46,  1.86s/it]
Outer training loop:  51%|#####1    | 25746/50000 [16:07:35<12:35:35,  1.87s/it]
Outer training loop:  51%|#####1    | 25747/50000 [16:07:37<13:01:07,  1.93s/it]
Outer training loop:  51%|#####1    | 25748/50000 [16:07:39<12:36:29,  1.87s/it]
Outer training loop:  51%|#####1    | 25749/50000 [16:07:40<12:30:59,  1.86s/it]
Outer training loop:  52%|#####1    | 25750/50000 [16:07:42<12:47:37,  1.90s/it]
Outer training loop:  52%|#####1    | 25751/50000 [16:07:44<12:56:27,  1.92s/it]
Outer training loop:  52%|#####1    | 25752/50000 [16:07:46<12:36:19,  1.87s/it]
Outer training loop:  52%|#####1    | 25753/50000 [16:07:48<12:27:27,  1.85s/it]
Outer training loop:  52%|#####1    | 25754/50000 [16:07:50<12:39:27,  1.88s/it]
Outer training loop:  52%|#####1    | 25755/50000 [16:07:52<12:28:17,  1.85s/it]
Outer training loop:  52%|#####1    | 25756/50000 [16:07:54<12:24:01,  1.84s/it]
Outer training loop:  52%|#####1    | 25757/50000 [16:07:56<12:46:21,  1.90s/it]
Outer training loop:  52%|#####1    | 25758/50000 [16:07:57<12:36:56,  1.87s/it]
Outer training loop:  52%|#####1    | 25759/50000 [16:07:59<12:24:53,  1.84s/it]
Outer training loop:  52%|#####1    | 25760/50000 [16:08:01<12:47:34,  1.90s/it]
Outer training loop:  52%|#####1    | 25761/50000 [16:08:03<12:54:31,  1.92s/it]
Outer training loop:  52%|#####1    | 25762/50000 [16:08:05<12:59:12,  1.93s/it]
Outer training loop:  52%|#####1    | 25763/50000 [16:08:07<12:53:59,  1.92s/it]
Outer training loop:  52%|#####1    | 25764/50000 [16:08:09<12:59:35,  1.93s/it]
Outer training loop:  52%|#####1    | 25765/50000 [16:08:11<12:56:14,  1.92s/it]
Outer training loop:  52%|#####1    | 25766/50000 [16:08:13<12:53:14,  1.91s/it]
Outer training loop:  52%|#####1    | 25767/50000 [16:08:15<13:15:28,  1.97s/it]
Outer training loop:  52%|#####1    | 25768/50000 [16:08:17<12:43:25,  1.89s/it]
Outer training loop:  52%|#####1    | 25769/50000 [16:08:18<12:36:40,  1.87s/it]
Outer training loop:  52%|#####1    | 25770/50000 [16:08:20<12:38:23,  1.88s/it]
Outer training loop:  52%|#####1    | 25771/50000 [16:08:22<12:44:16,  1.89s/it]
Outer training loop:  52%|#####1    | 25772/50000 [16:08:24<12:30:58,  1.86s/it]
Outer training loop:  52%|#####1    | 25773/50000 [16:08:26<12:34:39,  1.87s/it]
Outer training loop:  52%|#####1    | 25774/50000 [16:08:28<13:15:21,  1.97s/it]
Outer training loop:  52%|#####1    | 25775/50000 [16:08:30<13:02:19,  1.94s/it]
Outer training loop:  52%|#####1    | 25776/50000 [16:08:32<12:46:20,  1.90s/it]
Outer training loop:  52%|#####1    | 25777/50000 [16:08:34<13:01:25,  1.94s/it]
Outer training loop:  52%|#####1    | 25778/50000 [16:08:36<12:40:39,  1.88s/it]
Outer training loop:  52%|#####1    | 25779/50000 [16:08:38<12:50:16,  1.91s/it]
Outer training loop:  52%|#####1    | 25780/50000 [16:08:39<12:58:54,  1.93s/it]
Outer training loop:  52%|#####1    | 25781/50000 [16:08:41<12:52:55,  1.91s/it]
Outer training loop:  52%|#####1    | 25782/50000 [16:08:43<12:53:57,  1.92s/it]
Outer training loop:  52%|#####1    | 25783/50000 [16:08:45<13:04:22,  1.94s/it]
Outer training loop:  52%|#####1    | 25784/50000 [16:08:47<13:10:38,  1.96s/it]
Outer training loop:  52%|#####1    | 25785/50000 [16:08:49<13:01:31,  1.94s/it]
Outer training loop:  52%|#####1    | 25786/50000 [16:08:51<13:08:24,  1.95s/it]
Outer training loop:  52%|#####1    | 25787/50000 [16:08:53<13:01:06,  1.94s/it]
Outer training loop:  52%|#####1    | 25788/50000 [16:08:55<12:42:39,  1.89s/it]
Outer training loop:  52%|#####1    | 25789/50000 [16:08:57<12:32:24,  1.86s/it]
Outer training loop:  52%|#####1    | 25790/50000 [16:08:59<13:04:25,  1.94s/it]
Outer training loop:  52%|#####1    | 25791/50000 [16:09:00<12:36:10,  1.87s/it]
Outer training loop:  52%|#####1    | 25792/50000 [16:09:02<12:21:13,  1.84s/it]
Outer training loop:  52%|#####1    | 25793/50000 [16:09:04<12:46:04,  1.90s/it]
Outer training loop:  52%|#####1    | 25794/50000 [16:09:06<12:22:17,  1.84s/it]
Outer training loop:  52%|#####1    | 25795/50000 [16:09:08<12:16:11,  1.82s/it]
Outer training loop:  52%|#####1    | 25796/50000 [16:09:10<12:15:01,  1.82s/it]
Outer training loop:  52%|#####1    | 25797/50000 [16:09:12<12:38:09,  1.88s/it]
Outer training loop:  52%|#####1    | 25798/50000 [16:09:13<12:37:05,  1.88s/it]
Outer training loop:  52%|#####1    | 25799/50000 [16:09:15<12:28:13,  1.86s/it]
Outer training loop:  52%|#####1    | 25800/50000 [16:09:17<12:55:17,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  52%|#####1    | 25801/50000 [16:09:25<24:18:07,  3.62s/it]
Outer training loop:  52%|#####1    | 25802/50000 [16:09:27<20:30:51,  3.05s/it]
Outer training loop:  52%|#####1    | 25803/50000 [16:09:29<18:07:24,  2.70s/it]
Outer training loop:  52%|#####1    | 25804/50000 [16:09:31<16:46:42,  2.50s/it]
Outer training loop:  52%|#####1    | 25805/50000 [16:09:32<15:13:59,  2.27s/it]
Outer training loop:  52%|#####1    | 25806/50000 [16:09:34<14:25:17,  2.15s/it]
Outer training loop:  52%|#####1    | 25807/50000 [16:09:36<14:14:27,  2.12s/it]
Outer training loop:  52%|#####1    | 25808/50000 [16:09:38<13:52:31,  2.06s/it]
Outer training loop:  52%|#####1    | 25809/50000 [16:09:40<13:29:01,  2.01s/it]
Outer training loop:  52%|#####1    | 25810/50000 [16:09:42<13:11:49,  1.96s/it]
Outer training loop:  52%|#####1    | 25811/50000 [16:09:44<13:37:20,  2.03s/it]
Outer training loop:  52%|#####1    | 25812/50000 [16:09:46<13:25:52,  2.00s/it]
Outer training loop:  52%|#####1    | 25813/50000 [16:09:48<13:16:28,  1.98s/it]
Outer training loop:  52%|#####1    | 25814/50000 [16:09:50<13:52:17,  2.06s/it]
Outer training loop:  52%|#####1    | 25815/50000 [16:09:52<13:24:04,  1.99s/it]
Outer training loop:  52%|#####1    | 25816/50000 [16:09:54<12:55:08,  1.92s/it]
Outer training loop:  52%|#####1    | 25817/50000 [16:09:56<12:57:57,  1.93s/it]
Outer training loop:  52%|#####1    | 25818/50000 [16:09:58<13:08:30,  1.96s/it]
Outer training loop:  52%|#####1    | 25819/50000 [16:10:00<12:44:53,  1.90s/it]
Outer training loop:  52%|#####1    | 25820/50000 [16:10:01<12:39:23,  1.88s/it]
Outer training loop:  52%|#####1    | 25821/50000 [16:10:03<12:52:37,  1.92s/it]
Outer training loop:  52%|#####1    | 25822/50000 [16:10:05<12:48:12,  1.91s/it]
Outer training loop:  52%|#####1    | 25823/50000 [16:10:07<12:59:03,  1.93s/it]
Outer training loop:  52%|#####1    | 25824/50000 [16:10:09<13:13:49,  1.97s/it]
Outer training loop:  52%|#####1    | 25825/50000 [16:10:11<12:54:03,  1.92s/it]
Outer training loop:  52%|#####1    | 25826/50000 [16:10:13<12:49:45,  1.91s/it]
Outer training loop:  52%|#####1    | 25827/50000 [16:10:15<12:28:50,  1.86s/it]
Outer training loop:  52%|#####1    | 25828/50000 [16:10:17<12:46:38,  1.90s/it]
Outer training loop:  52%|#####1    | 25829/50000 [16:10:18<12:29:34,  1.86s/it]
Outer training loop:  52%|#####1    | 25830/50000 [16:10:20<12:25:00,  1.85s/it]
Outer training loop:  52%|#####1    | 25831/50000 [16:10:23<13:09:11,  1.96s/it]
Outer training loop:  52%|#####1    | 25832/50000 [16:10:24<13:04:16,  1.95s/it]
Outer training loop:  52%|#####1    | 25833/50000 [16:10:26<13:01:02,  1.94s/it]
Outer training loop:  52%|#####1    | 25834/50000 [16:10:28<12:51:18,  1.92s/it]
Outer training loop:  52%|#####1    | 25835/50000 [16:10:30<13:00:11,  1.94s/it]
Outer training loop:  52%|#####1    | 25836/50000 [16:10:32<12:38:25,  1.88s/it]
Outer training loop:  52%|#####1    | 25837/50000 [16:10:34<12:40:39,  1.89s/it]
Outer training loop:  52%|#####1    | 25838/50000 [16:10:36<13:24:25,  2.00s/it]
Outer training loop:  52%|#####1    | 25839/50000 [16:10:38<13:13:32,  1.97s/it]
Outer training loop:  52%|#####1    | 25840/50000 [16:10:40<12:58:48,  1.93s/it]
Outer training loop:  52%|#####1    | 25841/50000 [16:10:42<12:55:12,  1.93s/it]
Outer training loop:  52%|#####1    | 25842/50000 [16:10:44<13:10:30,  1.96s/it]
Outer training loop:  52%|#####1    | 25843/50000 [16:10:46<12:47:58,  1.91s/it]
Outer training loop:  52%|#####1    | 25844/50000 [16:10:47<12:43:42,  1.90s/it]
Outer training loop:  52%|#####1    | 25845/50000 [16:10:49<12:56:52,  1.93s/it]
Outer training loop:  52%|#####1    | 25846/50000 [16:10:51<13:03:51,  1.95s/it]
Outer training loop:  52%|#####1    | 25847/50000 [16:10:53<12:53:55,  1.92s/it]
Outer training loop:  52%|#####1    | 25848/50000 [16:10:55<12:47:25,  1.91s/it]
Outer training loop:  52%|#####1    | 25849/50000 [16:10:57<12:51:40,  1.92s/it]
Outer training loop:  52%|#####1    | 25850/50000 [16:10:59<12:33:30,  1.87s/it]
Outer training loop:  52%|#####1    | 25851/50000 [16:11:01<12:32:04,  1.87s/it]
Outer training loop:  52%|#####1    | 25852/50000 [16:11:03<12:47:55,  1.91s/it]
Outer training loop:  52%|#####1    | 25853/50000 [16:11:05<12:49:55,  1.91s/it]
Outer training loop:  52%|#####1    | 25854/50000 [16:11:07<12:47:04,  1.91s/it]
Outer training loop:  52%|#####1    | 25855/50000 [16:11:08<12:19:32,  1.84s/it]
Outer training loop:  52%|#####1    | 25856/50000 [16:11:10<12:47:32,  1.91s/it]
Outer training loop:  52%|#####1    | 25857/50000 [16:11:12<12:41:00,  1.89s/it]
Outer training loop:  52%|#####1    | 25858/50000 [16:11:14<13:14:05,  1.97s/it]
Outer training loop:  52%|#####1    | 25859/50000 [16:11:16<13:21:22,  1.99s/it]
Outer training loop:  52%|#####1    | 25860/50000 [16:11:18<12:49:35,  1.91s/it]
Outer training loop:  52%|#####1    | 25861/50000 [16:11:20<12:33:19,  1.87s/it]
Outer training loop:  52%|#####1    | 25862/50000 [16:11:22<12:30:46,  1.87s/it]
Outer training loop:  52%|#####1    | 25863/50000 [16:11:24<12:45:31,  1.90s/it]
Outer training loop:  52%|#####1    | 25864/50000 [16:11:26<12:38:05,  1.88s/it]
Outer training loop:  52%|#####1    | 25865/50000 [16:11:27<12:41:21,  1.89s/it]
Outer training loop:  52%|#####1    | 25866/50000 [16:11:29<12:25:14,  1.85s/it]
Outer training loop:  52%|#####1    | 25867/50000 [16:11:31<12:46:24,  1.91s/it]
Outer training loop:  52%|#####1    | 25868/50000 [16:11:33<12:43:52,  1.90s/it]
Outer training loop:  52%|#####1    | 25869/50000 [16:11:35<12:52:06,  1.92s/it]
Outer training loop:  52%|#####1    | 25870/50000 [16:11:37<13:06:41,  1.96s/it]
Outer training loop:  52%|#####1    | 25871/50000 [16:11:39<12:53:09,  1.92s/it]
Outer training loop:  52%|#####1    | 25872/50000 [16:11:41<12:47:43,  1.91s/it]
Outer training loop:  52%|#####1    | 25873/50000 [16:11:43<12:54:59,  1.93s/it]
Outer training loop:  52%|#####1    | 25874/50000 [16:11:45<12:28:34,  1.86s/it]
Outer training loop:  52%|#####1    | 25875/50000 [16:11:46<12:14:13,  1.83s/it]
Outer training loop:  52%|#####1    | 25876/50000 [16:11:48<12:38:48,  1.89s/it]
Outer training loop:  52%|#####1    | 25877/50000 [16:11:50<12:47:35,  1.91s/it]
Outer training loop:  52%|#####1    | 25878/50000 [16:11:52<12:24:02,  1.85s/it]
Outer training loop:  52%|#####1    | 25879/50000 [16:11:54<12:22:41,  1.85s/it]
Outer training loop:  52%|#####1    | 25880/50000 [16:11:56<12:38:29,  1.89s/it]
Outer training loop:  52%|#####1    | 25881/50000 [16:11:58<12:43:47,  1.90s/it]
Outer training loop:  52%|#####1    | 25882/50000 [16:12:00<12:28:56,  1.86s/it]
Outer training loop:  52%|#####1    | 25883/50000 [16:12:02<12:41:58,  1.90s/it]
Outer training loop:  52%|#####1    | 25884/50000 [16:12:03<12:14:04,  1.83s/it]
Outer training loop:  52%|#####1    | 25885/50000 [16:12:05<12:14:06,  1.83s/it]
Outer training loop:  52%|#####1    | 25886/50000 [16:12:07<12:00:43,  1.79s/it]
Outer training loop:  52%|#####1    | 25887/50000 [16:12:09<12:41:07,  1.89s/it]
Outer training loop:  52%|#####1    | 25888/50000 [16:12:11<12:32:35,  1.87s/it]
Outer training loop:  52%|#####1    | 25889/50000 [16:12:13<12:39:04,  1.89s/it]
Outer training loop:  52%|#####1    | 25890/50000 [16:12:15<12:52:53,  1.92s/it]
Outer training loop:  52%|#####1    | 25891/50000 [16:12:16<12:44:32,  1.90s/it]
Outer training loop:  52%|#####1    | 25892/50000 [16:12:18<12:20:47,  1.84s/it]
Outer training loop:  52%|#####1    | 25893/50000 [16:12:20<12:30:51,  1.87s/it]
Outer training loop:  52%|#####1    | 25894/50000 [16:12:22<12:40:45,  1.89s/it]
Outer training loop:  52%|#####1    | 25895/50000 [16:12:24<12:38:11,  1.89s/it]
Outer training loop:  52%|#####1    | 25896/50000 [16:12:26<12:40:22,  1.89s/it]
Outer training loop:  52%|#####1    | 25897/50000 [16:12:28<13:02:12,  1.95s/it]
Outer training loop:  52%|#####1    | 25898/50000 [16:12:30<12:39:24,  1.89s/it]
Outer training loop:  52%|#####1    | 25899/50000 [16:12:31<12:31:13,  1.87s/it]
Outer training loop:  52%|#####1    | 25900/50000 [16:12:34<12:49:42,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  52%|#####1    | 25901/50000 [16:12:41<24:01:27,  3.59s/it]
Outer training loop:  52%|#####1    | 25902/50000 [16:12:43<20:40:04,  3.09s/it]
Outer training loop:  52%|#####1    | 25903/50000 [16:12:45<18:10:47,  2.72s/it]
Outer training loop:  52%|#####1    | 25904/50000 [16:12:47<16:50:03,  2.52s/it]
Outer training loop:  52%|#####1    | 25905/50000 [16:12:49<16:04:57,  2.40s/it]
Outer training loop:  52%|#####1    | 25906/50000 [16:12:51<15:03:54,  2.25s/it]
Outer training loop:  52%|#####1    | 25907/50000 [16:12:53<14:38:47,  2.19s/it]
Outer training loop:  52%|#####1    | 25908/50000 [16:12:55<14:09:46,  2.12s/it]
Outer training loop:  52%|#####1    | 25909/50000 [16:12:57<13:39:52,  2.04s/it]
Outer training loop:  52%|#####1    | 25910/50000 [16:12:59<13:39:35,  2.04s/it]
Outer training loop:  52%|#####1    | 25911/50000 [16:13:01<13:19:12,  1.99s/it]
Outer training loop:  52%|#####1    | 25912/50000 [16:13:02<12:59:27,  1.94s/it]
Outer training loop:  52%|#####1    | 25913/50000 [16:13:04<12:53:05,  1.93s/it]
Outer training loop:  52%|#####1    | 25914/50000 [16:13:06<13:04:38,  1.95s/it]
Outer training loop:  52%|#####1    | 25915/50000 [16:13:08<12:54:55,  1.93s/it]
Outer training loop:  52%|#####1    | 25916/50000 [16:13:10<12:45:13,  1.91s/it]
Outer training loop:  52%|#####1    | 25917/50000 [16:13:12<13:07:33,  1.96s/it]
Outer training loop:  52%|#####1    | 25918/50000 [16:13:14<12:49:59,  1.92s/it]
Outer training loop:  52%|#####1    | 25919/50000 [16:13:16<12:34:00,  1.88s/it]
Outer training loop:  52%|#####1    | 25920/50000 [16:13:18<12:34:43,  1.88s/it]
Outer training loop:  52%|#####1    | 25921/50000 [16:13:20<13:04:40,  1.96s/it]
Outer training loop:  52%|#####1    | 25922/50000 [16:13:22<12:42:14,  1.90s/it]
Outer training loop:  52%|#####1    | 25923/50000 [16:13:23<12:41:33,  1.90s/it]
Outer training loop:  52%|#####1    | 25924/50000 [16:13:25<12:54:27,  1.93s/it]
Outer training loop:  52%|#####1    | 25925/50000 [16:13:27<12:41:06,  1.90s/it]
Outer training loop:  52%|#####1    | 25926/50000 [16:13:29<12:26:26,  1.86s/it]
Outer training loop:  52%|#####1    | 25927/50000 [16:13:31<12:32:22,  1.88s/it]
Outer training loop:  52%|#####1    | 25928/50000 [16:13:33<12:49:25,  1.92s/it]
Outer training loop:  52%|#####1    | 25929/50000 [16:13:35<12:48:21,  1.92s/it]
Outer training loop:  52%|#####1    | 25930/50000 [16:13:37<12:37:12,  1.89s/it]
Outer training loop:  52%|#####1    | 25931/50000 [16:13:39<12:48:53,  1.92s/it]
Outer training loop:  52%|#####1    | 25932/50000 [16:13:41<12:38:19,  1.89s/it]
Outer training loop:  52%|#####1    | 25933/50000 [16:13:42<12:26:42,  1.86s/it]
Outer training loop:  52%|#####1    | 25934/50000 [16:13:44<12:08:42,  1.82s/it]
Outer training loop:  52%|#####1    | 25935/50000 [16:13:46<12:31:30,  1.87s/it]
Outer training loop:  52%|#####1    | 25936/50000 [16:13:48<12:32:31,  1.88s/it]
Outer training loop:  52%|#####1    | 25937/50000 [16:13:50<12:32:17,  1.88s/it]
Outer training loop:  52%|#####1    | 25938/50000 [16:13:52<12:47:33,  1.91s/it]
Outer training loop:  52%|#####1    | 25939/50000 [16:13:54<12:36:36,  1.89s/it]
Outer training loop:  52%|#####1    | 25940/50000 [16:13:56<12:52:52,  1.93s/it]
Outer training loop:  52%|#####1    | 25941/50000 [16:13:58<13:08:16,  1.97s/it]
Outer training loop:  52%|#####1    | 25942/50000 [16:14:00<12:58:15,  1.94s/it]
Outer training loop:  52%|#####1    | 25943/50000 [16:14:01<12:44:20,  1.91s/it]
Outer training loop:  52%|#####1    | 25944/50000 [16:14:03<12:30:16,  1.87s/it]
Outer training loop:  52%|#####1    | 25945/50000 [16:14:05<12:40:46,  1.90s/it]
Outer training loop:  52%|#####1    | 25946/50000 [16:14:07<12:22:20,  1.85s/it]
Outer training loop:  52%|#####1    | 25947/50000 [16:14:09<12:29:30,  1.87s/it]
Outer training loop:  52%|#####1    | 25948/50000 [16:14:11<12:48:00,  1.92s/it]
Outer training loop:  52%|#####1    | 25949/50000 [16:14:13<12:28:45,  1.87s/it]
Outer training loop:  52%|#####1    | 25950/50000 [16:14:14<12:16:11,  1.84s/it]
Outer training loop:  52%|#####1    | 25951/50000 [16:14:16<12:28:44,  1.87s/it]
Outer training loop:  52%|#####1    | 25952/50000 [16:14:18<12:14:44,  1.83s/it]
Outer training loop:  52%|#####1    | 25953/50000 [16:14:20<12:18:05,  1.84s/it]
Outer training loop:  52%|#####1    | 25954/50000 [16:14:22<12:15:36,  1.84s/it]
Outer training loop:  52%|#####1    | 25955/50000 [16:14:24<12:37:17,  1.89s/it]
Outer training loop:  52%|#####1    | 25956/50000 [16:14:26<12:36:14,  1.89s/it]
Outer training loop:  52%|#####1    | 25957/50000 [16:14:28<12:34:52,  1.88s/it]
Outer training loop:  52%|#####1    | 25958/50000 [16:14:29<12:38:07,  1.89s/it]
Outer training loop:  52%|#####1    | 25959/50000 [16:14:31<12:25:38,  1.86s/it]
Outer training loop:  52%|#####1    | 25960/50000 [16:14:33<12:14:29,  1.83s/it]
Outer training loop:  52%|#####1    | 25961/50000 [16:14:35<12:11:45,  1.83s/it]
Outer training loop:  52%|#####1    | 25962/50000 [16:14:37<12:35:36,  1.89s/it]
Outer training loop:  52%|#####1    | 25963/50000 [16:14:39<12:25:46,  1.86s/it]
Outer training loop:  52%|#####1    | 25964/50000 [16:14:40<12:26:08,  1.86s/it]
Outer training loop:  52%|#####1    | 25965/50000 [16:14:42<12:33:41,  1.88s/it]
Outer training loop:  52%|#####1    | 25966/50000 [16:14:44<12:35:15,  1.89s/it]
Outer training loop:  52%|#####1    | 25967/50000 [16:14:46<12:27:18,  1.87s/it]
Outer training loop:  52%|#####1    | 25968/50000 [16:14:48<13:02:34,  1.95s/it]
Outer training loop:  52%|#####1    | 25969/50000 [16:14:51<13:35:42,  2.04s/it]
Outer training loop:  52%|#####1    | 25970/50000 [16:14:52<12:58:54,  1.94s/it]
Outer training loop:  52%|#####1    | 25971/50000 [16:14:54<12:25:46,  1.86s/it]
Outer training loop:  52%|#####1    | 25972/50000 [16:14:56<12:48:31,  1.92s/it]
Outer training loop:  52%|#####1    | 25973/50000 [16:14:58<12:44:13,  1.91s/it]
Outer training loop:  52%|#####1    | 25974/50000 [16:15:00<12:38:43,  1.89s/it]
Outer training loop:  52%|#####1    | 25975/50000 [16:15:02<12:44:23,  1.91s/it]
Outer training loop:  52%|#####1    | 25976/50000 [16:15:04<12:41:10,  1.90s/it]
Outer training loop:  52%|#####1    | 25977/50000 [16:15:05<12:40:37,  1.90s/it]
Outer training loop:  52%|#####1    | 25978/50000 [16:15:07<12:44:35,  1.91s/it]
Outer training loop:  52%|#####1    | 25979/50000 [16:15:09<12:55:11,  1.94s/it]
Outer training loop:  52%|#####1    | 25980/50000 [16:15:11<12:48:56,  1.92s/it]
Outer training loop:  52%|#####1    | 25981/50000 [16:15:13<12:31:56,  1.88s/it]
Outer training loop:  52%|#####1    | 25982/50000 [16:15:15<12:14:39,  1.84s/it]
Outer training loop:  52%|#####1    | 25983/50000 [16:15:17<12:45:54,  1.91s/it]
Outer training loop:  52%|#####1    | 25984/50000 [16:15:19<12:42:20,  1.90s/it]
Outer training loop:  52%|#####1    | 25985/50000 [16:15:21<12:45:28,  1.91s/it]
Outer training loop:  52%|#####1    | 25986/50000 [16:15:23<12:45:44,  1.91s/it]
Outer training loop:  52%|#####1    | 25987/50000 [16:15:24<12:37:51,  1.89s/it]
Outer training loop:  52%|#####1    | 25988/50000 [16:15:26<12:25:34,  1.86s/it]
Outer training loop:  52%|#####1    | 25989/50000 [16:15:28<12:51:48,  1.93s/it]
Outer training loop:  52%|#####1    | 25990/50000 [16:15:30<12:51:36,  1.93s/it]
Outer training loop:  52%|#####1    | 25991/50000 [16:15:32<12:30:15,  1.87s/it]
Outer training loop:  52%|#####1    | 25992/50000 [16:15:34<12:51:01,  1.93s/it]
Outer training loop:  52%|#####1    | 25993/50000 [16:15:36<12:57:41,  1.94s/it]
Outer training loop:  52%|#####1    | 25994/50000 [16:15:38<12:37:37,  1.89s/it]
Outer training loop:  52%|#####1    | 25995/50000 [16:15:40<12:21:45,  1.85s/it]
Outer training loop:  52%|#####1    | 25996/50000 [16:15:42<12:46:32,  1.92s/it]
Outer training loop:  52%|#####1    | 25997/50000 [16:15:43<12:41:23,  1.90s/it]
Outer training loop:  52%|#####1    | 25998/50000 [16:15:45<12:21:13,  1.85s/it]
Outer training loop:  52%|#####1    | 25999/50000 [16:15:47<12:42:11,  1.91s/it]
Outer training loop:  52%|#####2    | 26000/50000 [16:15:49<12:32:49,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:46,  2.03s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:44,  2.03s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:38,  1.98s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:33,  1.95s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:32,  1.95s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:30,  1.95s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:25,  1.92s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:26,  1.98s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:21,  1.94s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:12,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:13,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:11,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:05,  1.93s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:02,  1.91s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:54,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:57,  1.96s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:54,  1.94s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:52,  1.93s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:45,  1.95s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:39,  1.91s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:36,  1.89s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:21,  1.82s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:20,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:17,  1.80s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:19,  1.89s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:06,  1.79s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:04,  1.80s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:07,  1.93s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.87s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.91s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.88s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:54,  1.95s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:52,  1.96s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.92s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.92s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.89s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:42,  1.91s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.99s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.94s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:34,  1.90s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.84s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:21,  2.00s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.94s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.91s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.91s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.86s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.90s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:04,  2.04s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.99s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.90s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.78s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.75s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:20,  1.76s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.79s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:18,  1.77s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:17,  1.79s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:28,  2.01s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:22,  1.95s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:14,  1.92s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.91s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:03,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:11,  2.02s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:05,  1.96s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:00,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:56,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:58,  1.94s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.88s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:46,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.93s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:45,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.86s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:35,  1.84s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:38,  1.97s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:34,  1.92s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.92s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:24,  1.92s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:23,  1.94s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:16,  1.91s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:15,  1.94s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:17,  2.04s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:12,  1.96s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:09,  1.92s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.86s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<01:02,  2.03s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:59,  1.97s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.89s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.90s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.87s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:46,  2.01s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:43,  1.96s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.92s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.89s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.90s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:30,  2.02s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:27,  1.96s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.91s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.84s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:12,  1.85s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.99s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.90s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.88s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.85s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.82s/it]

Outer training loop:  52%|#####2    | 26001/50000 [16:21:12<653:38:25, 98.05s/it]
Outer training loop:  52%|#####2    | 26002/50000 [16:21:13<461:00:18, 69.16s/it]
Outer training loop:  52%|#####2    | 26003/50000 [16:21:15<326:15:23, 48.94s/it]
Outer training loop:  52%|#####2    | 26004/50000 [16:21:17<232:54:08, 34.94s/it]
Outer training loop:  52%|#####2    | 26005/50000 [16:21:19<166:42:14, 25.01s/it]
Outer training loop:  52%|#####2    | 26006/50000 [16:21:21<120:34:44, 18.09s/it]
Outer training loop:  52%|#####2    | 26007/50000 [16:21:23<87:49:16, 13.18s/it] 
Outer training loop:  52%|#####2    | 26008/50000 [16:21:25<65:39:17,  9.85s/it]
Outer training loop:  52%|#####2    | 26009/50000 [16:21:27<49:29:14,  7.43s/it]
Outer training loop:  52%|#####2    | 26010/50000 [16:21:28<38:11:43,  5.73s/it]
Outer training loop:  52%|#####2    | 26011/50000 [16:21:30<30:37:33,  4.60s/it]
Outer training loop:  52%|#####2    | 26012/50000 [16:21:32<25:03:06,  3.76s/it]
Outer training loop:  52%|#####2    | 26013/50000 [16:21:34<21:05:53,  3.17s/it]
Outer training loop:  52%|#####2    | 26014/50000 [16:21:36<18:23:51,  2.76s/it]
Outer training loop:  52%|#####2    | 26015/50000 [16:21:38<16:39:37,  2.50s/it]
Outer training loop:  52%|#####2    | 26016/50000 [16:21:40<15:17:54,  2.30s/it]
Outer training loop:  52%|#####2    | 26017/50000 [16:21:41<14:03:47,  2.11s/it]
Outer training loop:  52%|#####2    | 26018/50000 [16:21:43<13:11:19,  1.98s/it]
Outer training loop:  52%|#####2    | 26019/50000 [16:21:45<13:04:51,  1.96s/it]
Outer training loop:  52%|#####2    | 26020/50000 [16:21:47<12:44:30,  1.91s/it]
Outer training loop:  52%|#####2    | 26021/50000 [16:21:48<12:34:34,  1.89s/it]
Outer training loop:  52%|#####2    | 26022/50000 [16:21:51<12:58:49,  1.95s/it]
Outer training loop:  52%|#####2    | 26023/50000 [16:21:52<12:28:33,  1.87s/it]
Outer training loop:  52%|#####2    | 26024/50000 [16:21:54<12:12:41,  1.83s/it]
Outer training loop:  52%|#####2    | 26025/50000 [16:21:56<12:22:56,  1.86s/it]
Outer training loop:  52%|#####2    | 26026/50000 [16:21:58<12:13:31,  1.84s/it]
Outer training loop:  52%|#####2    | 26027/50000 [16:21:59<12:05:51,  1.82s/it]
Outer training loop:  52%|#####2    | 26028/50000 [16:22:01<12:00:28,  1.80s/it]
Outer training loop:  52%|#####2    | 26029/50000 [16:22:03<12:14:07,  1.84s/it]
Outer training loop:  52%|#####2    | 26030/50000 [16:22:05<12:06:04,  1.82s/it]
Outer training loop:  52%|#####2    | 26031/50000 [16:22:07<12:11:51,  1.83s/it]
Outer training loop:  52%|#####2    | 26032/50000 [16:22:09<12:33:48,  1.89s/it]
Outer training loop:  52%|#####2    | 26033/50000 [16:22:11<12:36:31,  1.89s/it]
Outer training loop:  52%|#####2    | 26034/50000 [16:22:12<12:27:23,  1.87s/it]
Outer training loop:  52%|#####2    | 26035/50000 [16:22:15<12:47:49,  1.92s/it]
Outer training loop:  52%|#####2    | 26036/50000 [16:22:17<13:00:44,  1.95s/it]
Outer training loop:  52%|#####2    | 26037/50000 [16:22:18<12:23:31,  1.86s/it]
Outer training loop:  52%|#####2    | 26038/50000 [16:22:20<12:20:15,  1.85s/it]
Outer training loop:  52%|#####2    | 26039/50000 [16:22:22<12:50:31,  1.93s/it]
Outer training loop:  52%|#####2    | 26040/50000 [16:22:24<12:27:43,  1.87s/it]
Outer training loop:  52%|#####2    | 26041/50000 [16:22:26<12:29:16,  1.88s/it]
Outer training loop:  52%|#####2    | 26042/50000 [16:22:28<12:43:34,  1.91s/it]
Outer training loop:  52%|#####2    | 26043/50000 [16:22:30<12:36:42,  1.90s/it]
Outer training loop:  52%|#####2    | 26044/50000 [16:22:31<12:12:45,  1.84s/it]
Outer training loop:  52%|#####2    | 26045/50000 [16:22:33<11:54:54,  1.79s/it]
Outer training loop:  52%|#####2    | 26046/50000 [16:22:35<12:20:53,  1.86s/it]
Outer training loop:  52%|#####2    | 26047/50000 [16:22:37<11:56:06,  1.79s/it]
Outer training loop:  52%|#####2    | 26048/50000 [16:22:38<12:01:28,  1.81s/it]
Outer training loop:  52%|#####2    | 26049/50000 [16:22:40<12:21:51,  1.86s/it]
Outer training loop:  52%|#####2    | 26050/50000 [16:22:42<12:19:29,  1.85s/it]
Outer training loop:  52%|#####2    | 26051/50000 [16:22:44<12:16:16,  1.84s/it]
Outer training loop:  52%|#####2    | 26052/50000 [16:22:46<12:03:24,  1.81s/it]
Outer training loop:  52%|#####2    | 26053/50000 [16:22:48<12:19:53,  1.85s/it]
Outer training loop:  52%|#####2    | 26054/50000 [16:22:50<12:25:23,  1.87s/it]
Outer training loop:  52%|#####2    | 26055/50000 [16:22:51<12:11:34,  1.83s/it]
Outer training loop:  52%|#####2    | 26056/50000 [16:22:53<12:21:36,  1.86s/it]
Outer training loop:  52%|#####2    | 26057/50000 [16:22:55<12:04:40,  1.82s/it]
Outer training loop:  52%|#####2    | 26058/50000 [16:22:57<11:49:08,  1.78s/it]
Outer training loop:  52%|#####2    | 26059/50000 [16:22:59<12:12:31,  1.84s/it]
Outer training loop:  52%|#####2    | 26060/50000 [16:23:01<12:23:37,  1.86s/it]
Outer training loop:  52%|#####2    | 26061/50000 [16:23:02<12:09:42,  1.83s/it]
Outer training loop:  52%|#####2    | 26062/50000 [16:23:04<12:19:59,  1.85s/it]
Outer training loop:  52%|#####2    | 26063/50000 [16:23:06<12:22:43,  1.86s/it]
Outer training loop:  52%|#####2    | 26064/50000 [16:23:08<12:09:01,  1.83s/it]
Outer training loop:  52%|#####2    | 26065/50000 [16:23:10<12:03:00,  1.81s/it]
Outer training loop:  52%|#####2    | 26066/50000 [16:23:12<12:16:53,  1.85s/it]
Outer training loop:  52%|#####2    | 26067/50000 [16:23:13<12:06:26,  1.82s/it]
Outer training loop:  52%|#####2    | 26068/50000 [16:23:15<12:05:21,  1.82s/it]
Outer training loop:  52%|#####2    | 26069/50000 [16:23:17<11:59:14,  1.80s/it]
Outer training loop:  52%|#####2    | 26070/50000 [16:23:19<12:14:54,  1.84s/it]
Outer training loop:  52%|#####2    | 26071/50000 [16:23:21<12:02:14,  1.81s/it]
Outer training loop:  52%|#####2    | 26072/50000 [16:23:22<11:54:48,  1.79s/it]
Outer training loop:  52%|#####2    | 26073/50000 [16:23:24<12:21:01,  1.86s/it]
Outer training loop:  52%|#####2    | 26074/50000 [16:23:26<12:28:13,  1.88s/it]
Outer training loop:  52%|#####2    | 26075/50000 [16:23:28<12:18:09,  1.85s/it]
Outer training loop:  52%|#####2    | 26076/50000 [16:23:30<12:20:56,  1.86s/it]
Outer training loop:  52%|#####2    | 26077/50000 [16:23:32<12:10:23,  1.83s/it]
Outer training loop:  52%|#####2    | 26078/50000 [16:23:34<12:18:45,  1.85s/it]
Outer training loop:  52%|#####2    | 26079/50000 [16:23:36<12:10:58,  1.83s/it]
Outer training loop:  52%|#####2    | 26080/50000 [16:23:38<12:30:31,  1.88s/it]
Outer training loop:  52%|#####2    | 26081/50000 [16:23:39<12:26:17,  1.87s/it]
Outer training loop:  52%|#####2    | 26082/50000 [16:23:41<12:25:27,  1.87s/it]
Outer training loop:  52%|#####2    | 26083/50000 [16:23:43<12:30:14,  1.88s/it]
Outer training loop:  52%|#####2    | 26084/50000 [16:23:45<12:16:24,  1.85s/it]
Outer training loop:  52%|#####2    | 26085/50000 [16:23:47<12:01:07,  1.81s/it]
Outer training loop:  52%|#####2    | 26086/50000 [16:23:49<12:39:57,  1.91s/it]
Outer training loop:  52%|#####2    | 26087/50000 [16:23:51<12:29:46,  1.88s/it]
Outer training loop:  52%|#####2    | 26088/50000 [16:23:52<12:23:02,  1.86s/it]
Outer training loop:  52%|#####2    | 26089/50000 [16:23:54<12:41:51,  1.91s/it]
Outer training loop:  52%|#####2    | 26090/50000 [16:23:56<12:16:25,  1.85s/it]
Outer training loop:  52%|#####2    | 26091/50000 [16:23:58<12:02:05,  1.81s/it]
Outer training loop:  52%|#####2    | 26092/50000 [16:24:00<12:09:09,  1.83s/it]
Outer training loop:  52%|#####2    | 26093/50000 [16:24:02<12:05:30,  1.82s/it]
Outer training loop:  52%|#####2    | 26094/50000 [16:24:03<11:58:11,  1.80s/it]
Outer training loop:  52%|#####2    | 26095/50000 [16:24:05<11:44:15,  1.77s/it]
Outer training loop:  52%|#####2    | 26096/50000 [16:24:07<12:08:33,  1.83s/it]
Outer training loop:  52%|#####2    | 26097/50000 [16:24:09<12:06:08,  1.82s/it]
Outer training loop:  52%|#####2    | 26098/50000 [16:24:11<12:11:12,  1.84s/it]
Outer training loop:  52%|#####2    | 26099/50000 [16:24:13<12:34:06,  1.89s/it]
Outer training loop:  52%|#####2    | 26100/50000 [16:24:15<12:30:12,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  52%|#####2    | 26101/50000 [16:24:22<23:28:47,  3.54s/it]
Outer training loop:  52%|#####2    | 26102/50000 [16:24:24<19:51:54,  2.99s/it]
Outer training loop:  52%|#####2    | 26103/50000 [16:24:26<18:03:05,  2.72s/it]
Outer training loop:  52%|#####2    | 26104/50000 [16:24:28<16:19:19,  2.46s/it]
Outer training loop:  52%|#####2    | 26105/50000 [16:24:29<15:05:37,  2.27s/it]
Outer training loop:  52%|#####2    | 26106/50000 [16:24:31<14:11:20,  2.14s/it]
Outer training loop:  52%|#####2    | 26107/50000 [16:24:33<13:44:16,  2.07s/it]
Outer training loop:  52%|#####2    | 26108/50000 [16:24:35<13:13:28,  1.99s/it]
Outer training loop:  52%|#####2    | 26109/50000 [16:24:37<12:57:00,  1.95s/it]
Outer training loop:  52%|#####2    | 26110/50000 [16:24:39<12:59:13,  1.96s/it]
Outer training loop:  52%|#####2    | 26111/50000 [16:24:41<12:45:03,  1.92s/it]
Outer training loop:  52%|#####2    | 26112/50000 [16:24:42<12:25:39,  1.87s/it]
Outer training loop:  52%|#####2    | 26113/50000 [16:24:44<12:37:48,  1.90s/it]
Outer training loop:  52%|#####2    | 26114/50000 [16:24:46<12:34:59,  1.90s/it]
Outer training loop:  52%|#####2    | 26115/50000 [16:24:48<12:22:15,  1.86s/it]
Outer training loop:  52%|#####2    | 26116/50000 [16:24:50<12:23:27,  1.87s/it]
Outer training loop:  52%|#####2    | 26117/50000 [16:24:52<12:32:21,  1.89s/it]
Outer training loop:  52%|#####2    | 26118/50000 [16:24:54<12:19:32,  1.86s/it]
Outer training loop:  52%|#####2    | 26119/50000 [16:24:55<12:21:03,  1.86s/it]
Outer training loop:  52%|#####2    | 26120/50000 [16:24:58<12:48:50,  1.93s/it]
Outer training loop:  52%|#####2    | 26121/50000 [16:24:59<12:36:21,  1.90s/it]
Outer training loop:  52%|#####2    | 26122/50000 [16:25:01<12:44:13,  1.92s/it]
Outer training loop:  52%|#####2    | 26123/50000 [16:25:03<12:48:35,  1.93s/it]
Outer training loop:  52%|#####2    | 26124/50000 [16:25:05<12:45:33,  1.92s/it]
Outer training loop:  52%|#####2    | 26125/50000 [16:25:07<12:43:26,  1.92s/it]
Outer training loop:  52%|#####2    | 26126/50000 [16:25:09<12:32:37,  1.89s/it]
Outer training loop:  52%|#####2    | 26127/50000 [16:25:11<12:41:35,  1.91s/it]
Outer training loop:  52%|#####2    | 26128/50000 [16:25:13<12:29:17,  1.88s/it]
Outer training loop:  52%|#####2    | 26129/50000 [16:25:15<12:20:32,  1.86s/it]
Outer training loop:  52%|#####2    | 26130/50000 [16:25:17<12:36:48,  1.90s/it]
Outer training loop:  52%|#####2    | 26131/50000 [16:25:18<12:14:51,  1.85s/it]
Outer training loop:  52%|#####2    | 26132/50000 [16:25:20<12:19:27,  1.86s/it]
Outer training loop:  52%|#####2    | 26133/50000 [16:25:22<12:56:32,  1.95s/it]
Outer training loop:  52%|#####2    | 26134/50000 [16:25:24<12:26:20,  1.88s/it]
Outer training loop:  52%|#####2    | 26135/50000 [16:25:26<12:22:52,  1.87s/it]
Outer training loop:  52%|#####2    | 26136/50000 [16:25:28<12:44:33,  1.92s/it]
Outer training loop:  52%|#####2    | 26137/50000 [16:25:30<12:31:27,  1.89s/it]
Outer training loop:  52%|#####2    | 26138/50000 [16:25:31<12:15:22,  1.85s/it]
Outer training loop:  52%|#####2    | 26139/50000 [16:25:33<12:02:17,  1.82s/it]
Outer training loop:  52%|#####2    | 26140/50000 [16:25:35<12:46:49,  1.93s/it]
Outer training loop:  52%|#####2    | 26141/50000 [16:25:37<12:46:00,  1.93s/it]
Outer training loop:  52%|#####2    | 26142/50000 [16:25:39<12:24:09,  1.87s/it]
Outer training loop:  52%|#####2    | 26143/50000 [16:25:41<12:46:07,  1.93s/it]
Outer training loop:  52%|#####2    | 26144/50000 [16:25:43<12:34:01,  1.90s/it]
Outer training loop:  52%|#####2    | 26145/50000 [16:25:45<12:28:58,  1.88s/it]
Outer training loop:  52%|#####2    | 26146/50000 [16:25:47<12:42:44,  1.92s/it]
Outer training loop:  52%|#####2    | 26147/50000 [16:25:49<12:34:38,  1.90s/it]
Outer training loop:  52%|#####2    | 26148/50000 [16:25:50<12:23:48,  1.87s/it]
Outer training loop:  52%|#####2    | 26149/50000 [16:25:52<12:18:25,  1.86s/it]
Outer training loop:  52%|#####2    | 26150/50000 [16:25:54<12:39:46,  1.91s/it]
Outer training loop:  52%|#####2    | 26151/50000 [16:25:56<12:36:19,  1.90s/it]
Outer training loop:  52%|#####2    | 26152/50000 [16:25:58<12:19:43,  1.86s/it]
Outer training loop:  52%|#####2    | 26153/50000 [16:26:00<12:37:39,  1.91s/it]
Outer training loop:  52%|#####2    | 26154/50000 [16:26:02<12:18:26,  1.86s/it]
Outer training loop:  52%|#####2    | 26155/50000 [16:26:04<12:19:41,  1.86s/it]
Outer training loop:  52%|#####2    | 26156/50000 [16:26:06<12:49:08,  1.94s/it]
Outer training loop:  52%|#####2    | 26157/50000 [16:26:07<12:27:36,  1.88s/it]
Outer training loop:  52%|#####2    | 26158/50000 [16:26:09<12:29:58,  1.89s/it]
Outer training loop:  52%|#####2    | 26159/50000 [16:26:11<12:15:02,  1.85s/it]
Outer training loop:  52%|#####2    | 26160/50000 [16:26:13<12:37:19,  1.91s/it]
Outer training loop:  52%|#####2    | 26161/50000 [16:26:15<12:23:13,  1.87s/it]
Outer training loop:  52%|#####2    | 26162/50000 [16:26:17<12:00:34,  1.81s/it]
Outer training loop:  52%|#####2    | 26163/50000 [16:26:19<12:21:59,  1.87s/it]
Outer training loop:  52%|#####2    | 26164/50000 [16:26:21<12:33:53,  1.90s/it]
Outer training loop:  52%|#####2    | 26165/50000 [16:26:22<12:20:28,  1.86s/it]
Outer training loop:  52%|#####2    | 26166/50000 [16:26:24<12:22:41,  1.87s/it]
Outer training loop:  52%|#####2    | 26167/50000 [16:26:26<12:13:17,  1.85s/it]
Outer training loop:  52%|#####2    | 26168/50000 [16:26:28<12:17:13,  1.86s/it]
Outer training loop:  52%|#####2    | 26169/50000 [16:26:30<12:16:08,  1.85s/it]
Outer training loop:  52%|#####2    | 26170/50000 [16:26:32<12:41:25,  1.92s/it]
Outer training loop:  52%|#####2    | 26171/50000 [16:26:34<12:32:59,  1.90s/it]
Outer training loop:  52%|#####2    | 26172/50000 [16:26:36<12:23:11,  1.87s/it]
Outer training loop:  52%|#####2    | 26173/50000 [16:26:37<12:30:04,  1.89s/it]
Outer training loop:  52%|#####2    | 26174/50000 [16:26:39<12:27:28,  1.88s/it]
Outer training loop:  52%|#####2    | 26175/50000 [16:26:41<12:17:04,  1.86s/it]
Outer training loop:  52%|#####2    | 26176/50000 [16:26:43<12:55:57,  1.95s/it]
Outer training loop:  52%|#####2    | 26177/50000 [16:26:45<12:59:52,  1.96s/it]
Outer training loop:  52%|#####2    | 26178/50000 [16:26:47<12:39:49,  1.91s/it]
Outer training loop:  52%|#####2    | 26179/50000 [16:26:49<12:50:29,  1.94s/it]
Outer training loop:  52%|#####2    | 26180/50000 [16:26:51<13:07:17,  1.98s/it]
Outer training loop:  52%|#####2    | 26181/50000 [16:26:53<12:32:28,  1.90s/it]
Outer training loop:  52%|#####2    | 26182/50000 [16:26:55<12:17:09,  1.86s/it]
Outer training loop:  52%|#####2    | 26183/50000 [16:26:57<12:46:56,  1.93s/it]
Outer training loop:  52%|#####2    | 26184/50000 [16:26:58<12:26:59,  1.88s/it]
Outer training loop:  52%|#####2    | 26185/50000 [16:27:00<12:08:12,  1.83s/it]
Outer training loop:  52%|#####2    | 26186/50000 [16:27:02<12:37:08,  1.91s/it]
Outer training loop:  52%|#####2    | 26187/50000 [16:27:04<12:29:26,  1.89s/it]
Outer training loop:  52%|#####2    | 26188/50000 [16:27:06<12:19:57,  1.86s/it]
Outer training loop:  52%|#####2    | 26189/50000 [16:27:08<12:10:25,  1.84s/it]
Outer training loop:  52%|#####2    | 26190/50000 [16:27:10<12:34:24,  1.90s/it]
Outer training loop:  52%|#####2    | 26191/50000 [16:27:12<12:21:54,  1.87s/it]
Outer training loop:  52%|#####2    | 26192/50000 [16:27:13<12:19:36,  1.86s/it]
Outer training loop:  52%|#####2    | 26193/50000 [16:27:15<12:40:34,  1.92s/it]
Outer training loop:  52%|#####2    | 26194/50000 [16:27:17<12:23:28,  1.87s/it]
Outer training loop:  52%|#####2    | 26195/50000 [16:27:19<12:19:23,  1.86s/it]
Outer training loop:  52%|#####2    | 26196/50000 [16:27:21<12:48:49,  1.94s/it]
Outer training loop:  52%|#####2    | 26197/50000 [16:27:23<12:40:04,  1.92s/it]
Outer training loop:  52%|#####2    | 26198/50000 [16:27:25<12:42:19,  1.92s/it]
Outer training loop:  52%|#####2    | 26199/50000 [16:27:27<12:31:38,  1.89s/it]
Outer training loop:  52%|#####2    | 26200/50000 [16:27:29<13:11:09,  1.99s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  52%|#####2    | 26201/50000 [16:27:37<24:12:12,  3.66s/it]
Outer training loop:  52%|#####2    | 26202/50000 [16:27:39<20:49:37,  3.15s/it]
Outer training loop:  52%|#####2    | 26203/50000 [16:27:40<18:18:43,  2.77s/it]
Outer training loop:  52%|#####2    | 26204/50000 [16:27:43<17:09:39,  2.60s/it]
Outer training loop:  52%|#####2    | 26205/50000 [16:27:45<16:01:08,  2.42s/it]
Outer training loop:  52%|#####2    | 26206/50000 [16:27:47<15:10:39,  2.30s/it]
Outer training loop:  52%|#####2    | 26207/50000 [16:27:49<14:30:21,  2.19s/it]
Outer training loop:  52%|#####2    | 26208/50000 [16:27:51<13:58:31,  2.11s/it]
Outer training loop:  52%|#####2    | 26209/50000 [16:27:52<13:29:08,  2.04s/it]
Outer training loop:  52%|#####2    | 26210/50000 [16:27:55<13:35:50,  2.06s/it]
Outer training loop:  52%|#####2    | 26211/50000 [16:27:56<13:22:52,  2.03s/it]
Outer training loop:  52%|#####2    | 26212/50000 [16:27:58<12:59:29,  1.97s/it]
Outer training loop:  52%|#####2    | 26213/50000 [16:28:00<12:57:34,  1.96s/it]
Outer training loop:  52%|#####2    | 26214/50000 [16:28:02<12:59:31,  1.97s/it]
Outer training loop:  52%|#####2    | 26215/50000 [16:28:04<12:48:41,  1.94s/it]
Outer training loop:  52%|#####2    | 26216/50000 [16:28:06<12:45:11,  1.93s/it]
Outer training loop:  52%|#####2    | 26217/50000 [16:28:08<12:57:30,  1.96s/it]
Outer training loop:  52%|#####2    | 26218/50000 [16:28:10<12:50:27,  1.94s/it]
Outer training loop:  52%|#####2    | 26219/50000 [16:28:12<12:30:23,  1.89s/it]
Outer training loop:  52%|#####2    | 26220/50000 [16:28:14<12:41:56,  1.92s/it]
Outer training loop:  52%|#####2    | 26221/50000 [16:28:16<12:33:11,  1.90s/it]
Outer training loop:  52%|#####2    | 26222/50000 [16:28:17<12:21:05,  1.87s/it]
Outer training loop:  52%|#####2    | 26223/50000 [16:28:19<12:27:25,  1.89s/it]
Outer training loop:  52%|#####2    | 26224/50000 [16:28:21<12:58:08,  1.96s/it]
Outer training loop:  52%|#####2    | 26225/50000 [16:28:23<12:48:14,  1.94s/it]
Outer training loop:  52%|#####2    | 26226/50000 [16:28:25<12:46:29,  1.93s/it]
Outer training loop:  52%|#####2    | 26227/50000 [16:28:27<12:53:36,  1.95s/it]
Outer training loop:  52%|#####2    | 26228/50000 [16:28:29<12:33:03,  1.90s/it]
Outer training loop:  52%|#####2    | 26229/50000 [16:28:31<12:17:57,  1.86s/it]
Outer training loop:  52%|#####2    | 26230/50000 [16:28:33<13:17:41,  2.01s/it]
Outer training loop:  52%|#####2    | 26231/50000 [16:28:35<13:06:03,  1.98s/it]
Outer training loop:  52%|#####2    | 26232/50000 [16:28:37<12:57:42,  1.96s/it]
Outer training loop:  52%|#####2    | 26233/50000 [16:28:39<12:38:26,  1.91s/it]
Outer training loop:  52%|#####2    | 26234/50000 [16:28:41<13:07:09,  1.99s/it]
Outer training loop:  52%|#####2    | 26235/50000 [16:28:43<13:06:08,  1.98s/it]
Outer training loop:  52%|#####2    | 26236/50000 [16:28:45<12:50:10,  1.94s/it]
Outer training loop:  52%|#####2    | 26237/50000 [16:28:47<12:40:46,  1.92s/it]
Outer training loop:  52%|#####2    | 26238/50000 [16:28:49<12:48:14,  1.94s/it]
Outer training loop:  52%|#####2    | 26239/50000 [16:28:51<12:46:19,  1.94s/it]
Outer training loop:  52%|#####2    | 26240/50000 [16:28:52<12:32:15,  1.90s/it]
Outer training loop:  52%|#####2    | 26241/50000 [16:28:54<13:00:25,  1.97s/it]
Outer training loop:  52%|#####2    | 26242/50000 [16:28:56<12:42:35,  1.93s/it]
Outer training loop:  52%|#####2    | 26243/50000 [16:28:58<12:41:30,  1.92s/it]
Outer training loop:  52%|#####2    | 26244/50000 [16:29:00<12:58:15,  1.97s/it]
Outer training loop:  52%|#####2    | 26245/50000 [16:29:02<13:05:17,  1.98s/it]
Outer training loop:  52%|#####2    | 26246/50000 [16:29:04<13:08:17,  1.99s/it]
Outer training loop:  52%|#####2    | 26247/50000 [16:29:06<13:23:35,  2.03s/it]
Outer training loop:  52%|#####2    | 26248/50000 [16:29:08<13:01:59,  1.98s/it]
Outer training loop:  52%|#####2    | 26249/50000 [16:29:10<12:48:55,  1.94s/it]
Outer training loop:  52%|#####2    | 26250/50000 [16:29:12<13:15:32,  2.01s/it]
Outer training loop:  53%|#####2    | 26251/50000 [16:29:14<12:49:27,  1.94s/it]
Outer training loop:  53%|#####2    | 26252/50000 [16:29:16<12:50:20,  1.95s/it]
Outer training loop:  53%|#####2    | 26253/50000 [16:29:18<12:25:52,  1.88s/it]
Outer training loop:  53%|#####2    | 26254/50000 [16:29:20<12:41:22,  1.92s/it]
Outer training loop:  53%|#####2    | 26255/50000 [16:29:22<12:39:30,  1.92s/it]
Outer training loop:  53%|#####2    | 26256/50000 [16:29:24<12:28:23,  1.89s/it]
Outer training loop:  53%|#####2    | 26257/50000 [16:29:26<12:55:13,  1.96s/it]
Outer training loop:  53%|#####2    | 26258/50000 [16:29:28<12:44:13,  1.93s/it]
Outer training loop:  53%|#####2    | 26259/50000 [16:29:29<12:27:49,  1.89s/it]
Outer training loop:  53%|#####2    | 26260/50000 [16:29:31<12:19:28,  1.87s/it]
Outer training loop:  53%|#####2    | 26261/50000 [16:29:33<12:43:48,  1.93s/it]
Outer training loop:  53%|#####2    | 26262/50000 [16:29:35<12:40:27,  1.92s/it]
Outer training loop:  53%|#####2    | 26263/50000 [16:29:37<12:31:46,  1.90s/it]
Outer training loop:  53%|#####2    | 26264/50000 [16:29:39<12:40:28,  1.92s/it]
Outer training loop:  53%|#####2    | 26265/50000 [16:29:41<12:25:51,  1.89s/it]
Outer training loop:  53%|#####2    | 26266/50000 [16:29:43<12:17:51,  1.87s/it]
Outer training loop:  53%|#####2    | 26267/50000 [16:29:44<12:04:27,  1.83s/it]
Outer training loop:  53%|#####2    | 26268/50000 [16:29:46<12:21:50,  1.88s/it]
Outer training loop:  53%|#####2    | 26269/50000 [16:29:48<12:21:49,  1.88s/it]
Outer training loop:  53%|#####2    | 26270/50000 [16:29:50<12:15:00,  1.86s/it]
Outer training loop:  53%|#####2    | 26271/50000 [16:29:52<12:32:17,  1.90s/it]
Outer training loop:  53%|#####2    | 26272/50000 [16:29:54<12:19:24,  1.87s/it]
Outer training loop:  53%|#####2    | 26273/50000 [16:29:56<12:20:43,  1.87s/it]
Outer training loop:  53%|#####2    | 26274/50000 [16:29:58<12:44:36,  1.93s/it]
Outer training loop:  53%|#####2    | 26275/50000 [16:30:00<12:47:38,  1.94s/it]
Outer training loop:  53%|#####2    | 26276/50000 [16:30:02<12:30:51,  1.90s/it]
Outer training loop:  53%|#####2    | 26277/50000 [16:30:04<12:45:46,  1.94s/it]
Outer training loop:  53%|#####2    | 26278/50000 [16:30:05<12:48:36,  1.94s/it]
Outer training loop:  53%|#####2    | 26279/50000 [16:30:07<12:38:13,  1.92s/it]
Outer training loop:  53%|#####2    | 26280/50000 [16:30:09<12:56:03,  1.96s/it]
Outer training loop:  53%|#####2    | 26281/50000 [16:30:11<12:54:09,  1.96s/it]
Outer training loop:  53%|#####2    | 26282/50000 [16:30:13<12:41:56,  1.93s/it]
Outer training loop:  53%|#####2    | 26283/50000 [16:30:15<12:15:25,  1.86s/it]
Outer training loop:  53%|#####2    | 26284/50000 [16:30:17<12:38:14,  1.92s/it]
Outer training loop:  53%|#####2    | 26285/50000 [16:30:19<12:25:44,  1.89s/it]
Outer training loop:  53%|#####2    | 26286/50000 [16:30:21<12:39:12,  1.92s/it]
Outer training loop:  53%|#####2    | 26287/50000 [16:30:23<12:39:04,  1.92s/it]
Outer training loop:  53%|#####2    | 26288/50000 [16:30:25<12:27:52,  1.89s/it]
Outer training loop:  53%|#####2    | 26289/50000 [16:30:26<12:30:58,  1.90s/it]
Outer training loop:  53%|#####2    | 26290/50000 [16:30:28<12:38:25,  1.92s/it]
Outer training loop:  53%|#####2    | 26291/50000 [16:30:30<12:12:12,  1.85s/it]
Outer training loop:  53%|#####2    | 26292/50000 [16:30:32<12:06:37,  1.84s/it]
Outer training loop:  53%|#####2    | 26293/50000 [16:30:34<12:12:44,  1.85s/it]
Outer training loop:  53%|#####2    | 26294/50000 [16:30:36<12:24:59,  1.89s/it]
Outer training loop:  53%|#####2    | 26295/50000 [16:30:38<12:22:53,  1.88s/it]
Outer training loop:  53%|#####2    | 26296/50000 [16:30:39<12:06:38,  1.84s/it]
Outer training loop:  53%|#####2    | 26297/50000 [16:30:41<12:32:07,  1.90s/it]
Outer training loop:  53%|#####2    | 26298/50000 [16:30:43<12:26:12,  1.89s/it]
Outer training loop:  53%|#####2    | 26299/50000 [16:30:45<12:00:15,  1.82s/it]
Outer training loop:  53%|#####2    | 26300/50000 [16:30:47<12:15:26,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  53%|#####2    | 26301/50000 [16:30:54<23:29:52,  3.57s/it]
Outer training loop:  53%|#####2    | 26302/50000 [16:30:56<19:47:17,  3.01s/it]
Outer training loop:  53%|#####2    | 26303/50000 [16:30:58<17:41:03,  2.69s/it]
Outer training loop:  53%|#####2    | 26304/50000 [16:31:00<16:26:24,  2.50s/it]
Outer training loop:  53%|#####2    | 26305/50000 [16:31:02<15:00:58,  2.28s/it]
Outer training loop:  53%|#####2    | 26306/50000 [16:31:04<14:16:42,  2.17s/it]
Outer training loop:  53%|#####2    | 26307/50000 [16:31:06<13:58:12,  2.12s/it]
Outer training loop:  53%|#####2    | 26308/50000 [16:31:08<13:20:15,  2.03s/it]
Outer training loop:  53%|#####2    | 26309/50000 [16:31:10<13:17:37,  2.02s/it]
Outer training loop:  53%|#####2    | 26310/50000 [16:31:12<13:14:13,  2.01s/it]
Outer training loop:  53%|#####2    | 26311/50000 [16:31:13<12:50:29,  1.95s/it]
Outer training loop:  53%|#####2    | 26312/50000 [16:31:15<12:29:13,  1.90s/it]
Outer training loop:  53%|#####2    | 26313/50000 [16:31:17<12:40:10,  1.93s/it]
Outer training loop:  53%|#####2    | 26314/50000 [16:31:19<12:27:35,  1.89s/it]
Outer training loop:  53%|#####2    | 26315/50000 [16:31:21<12:25:13,  1.89s/it]
Outer training loop:  53%|#####2    | 26316/50000 [16:31:23<12:19:16,  1.87s/it]
Outer training loop:  53%|#####2    | 26317/50000 [16:31:25<12:29:01,  1.90s/it]
Outer training loop:  53%|#####2    | 26318/50000 [16:31:26<12:11:28,  1.85s/it]
Outer training loop:  53%|#####2    | 26319/50000 [16:31:28<12:11:16,  1.85s/it]
Outer training loop:  53%|#####2    | 26320/50000 [16:31:30<12:27:51,  1.89s/it]
Outer training loop:  53%|#####2    | 26321/50000 [16:31:32<12:01:51,  1.83s/it]
Outer training loop:  53%|#####2    | 26322/50000 [16:31:34<12:09:41,  1.85s/it]
Outer training loop:  53%|#####2    | 26323/50000 [16:31:36<12:19:02,  1.87s/it]
Outer training loop:  53%|#####2    | 26324/50000 [16:31:38<12:38:34,  1.92s/it]
Outer training loop:  53%|#####2    | 26325/50000 [16:31:40<12:21:53,  1.88s/it]
Outer training loop:  53%|#####2    | 26326/50000 [16:31:42<12:24:51,  1.89s/it]
Outer training loop:  53%|#####2    | 26327/50000 [16:31:44<13:03:24,  1.99s/it]
Outer training loop:  53%|#####2    | 26328/50000 [16:31:46<12:47:26,  1.95s/it]
Outer training loop:  53%|#####2    | 26329/50000 [16:31:47<12:20:33,  1.88s/it]
Outer training loop:  53%|#####2    | 26330/50000 [16:31:49<12:11:32,  1.85s/it]
Outer training loop:  53%|#####2    | 26331/50000 [16:31:51<12:20:55,  1.88s/it]
Outer training loop:  53%|#####2    | 26332/50000 [16:31:53<12:14:50,  1.86s/it]
Outer training loop:  53%|#####2    | 26333/50000 [16:31:55<12:37:03,  1.92s/it]
Outer training loop:  53%|#####2    | 26334/50000 [16:31:57<12:20:46,  1.88s/it]
Outer training loop:  53%|#####2    | 26335/50000 [16:31:59<12:38:42,  1.92s/it]
Outer training loop:  53%|#####2    | 26336/50000 [16:32:01<12:29:18,  1.90s/it]
Outer training loop:  53%|#####2    | 26337/50000 [16:32:02<12:13:19,  1.86s/it]
Outer training loop:  53%|#####2    | 26338/50000 [16:32:04<12:17:55,  1.87s/it]
Outer training loop:  53%|#####2    | 26339/50000 [16:32:06<12:09:11,  1.85s/it]
Outer training loop:  53%|#####2    | 26340/50000 [16:32:08<12:04:37,  1.84s/it]
Outer training loop:  53%|#####2    | 26341/50000 [16:32:10<12:07:43,  1.85s/it]
Outer training loop:  53%|#####2    | 26342/50000 [16:32:12<12:29:55,  1.90s/it]
Outer training loop:  53%|#####2    | 26343/50000 [16:32:14<12:41:21,  1.93s/it]
Outer training loop:  53%|#####2    | 26344/50000 [16:32:15<12:18:54,  1.87s/it]
Outer training loop:  53%|#####2    | 26345/50000 [16:32:17<12:27:43,  1.90s/it]
Outer training loop:  53%|#####2    | 26346/50000 [16:32:19<12:27:26,  1.90s/it]
Outer training loop:  53%|#####2    | 26347/50000 [16:32:21<12:19:37,  1.88s/it]
Outer training loop:  53%|#####2    | 26348/50000 [16:32:23<12:06:01,  1.84s/it]
Outer training loop:  53%|#####2    | 26349/50000 [16:32:25<12:32:53,  1.91s/it]
Outer training loop:  53%|#####2    | 26350/50000 [16:32:27<12:17:35,  1.87s/it]
Outer training loop:  53%|#####2    | 26351/50000 [16:32:29<12:18:21,  1.87s/it]
Outer training loop:  53%|#####2    | 26352/50000 [16:32:31<12:48:41,  1.95s/it]
Outer training loop:  53%|#####2    | 26353/50000 [16:32:33<12:24:07,  1.89s/it]
Outer training loop:  53%|#####2    | 26354/50000 [16:32:34<12:18:04,  1.87s/it]
Outer training loop:  53%|#####2    | 26355/50000 [16:32:36<12:06:57,  1.84s/it]
Outer training loop:  53%|#####2    | 26356/50000 [16:32:38<12:34:24,  1.91s/it]
Outer training loop:  53%|#####2    | 26357/50000 [16:32:40<12:28:17,  1.90s/it]
Outer training loop:  53%|#####2    | 26358/50000 [16:32:42<12:19:51,  1.88s/it]
Outer training loop:  53%|#####2    | 26359/50000 [16:32:44<12:48:03,  1.95s/it]
Outer training loop:  53%|#####2    | 26360/50000 [16:32:46<12:42:45,  1.94s/it]
Outer training loop:  53%|#####2    | 26361/50000 [16:32:48<12:25:44,  1.89s/it]
Outer training loop:  53%|#####2    | 26362/50000 [16:32:50<12:57:12,  1.97s/it]
Outer training loop:  53%|#####2    | 26363/50000 [16:32:52<12:57:15,  1.97s/it]
Outer training loop:  53%|#####2    | 26364/50000 [16:32:54<12:49:53,  1.95s/it]
Outer training loop:  53%|#####2    | 26365/50000 [16:32:56<13:03:43,  1.99s/it]
Outer training loop:  53%|#####2    | 26366/50000 [16:32:58<13:02:04,  1.99s/it]
Outer training loop:  53%|#####2    | 26367/50000 [16:33:00<12:43:28,  1.94s/it]
Outer training loop:  53%|#####2    | 26368/50000 [16:33:01<12:27:41,  1.90s/it]
Outer training loop:  53%|#####2    | 26369/50000 [16:33:03<12:30:55,  1.91s/it]
Outer training loop:  53%|#####2    | 26370/50000 [16:33:05<12:17:30,  1.87s/it]
Outer training loop:  53%|#####2    | 26371/50000 [16:33:07<12:12:36,  1.86s/it]
Outer training loop:  53%|#####2    | 26372/50000 [16:33:09<12:18:23,  1.88s/it]
Outer training loop:  53%|#####2    | 26373/50000 [16:33:11<12:16:59,  1.87s/it]
Outer training loop:  53%|#####2    | 26374/50000 [16:33:13<12:22:02,  1.88s/it]
Outer training loop:  53%|#####2    | 26375/50000 [16:33:15<12:44:48,  1.94s/it]
Outer training loop:  53%|#####2    | 26376/50000 [16:33:17<12:28:17,  1.90s/it]
Outer training loop:  53%|#####2    | 26377/50000 [16:33:19<12:56:40,  1.97s/it]
Outer training loop:  53%|#####2    | 26378/50000 [16:33:21<13:07:37,  2.00s/it]
Outer training loop:  53%|#####2    | 26379/50000 [16:33:23<12:56:45,  1.97s/it]
Outer training loop:  53%|#####2    | 26380/50000 [16:33:25<12:57:24,  1.97s/it]
Outer training loop:  53%|#####2    | 26381/50000 [16:33:27<13:01:39,  1.99s/it]
Outer training loop:  53%|#####2    | 26382/50000 [16:33:28<12:36:33,  1.92s/it]
Outer training loop:  53%|#####2    | 26383/50000 [16:33:30<12:28:15,  1.90s/it]
Outer training loop:  53%|#####2    | 26384/50000 [16:33:32<12:49:08,  1.95s/it]
Outer training loop:  53%|#####2    | 26385/50000 [16:33:34<12:41:33,  1.93s/it]
Outer training loop:  53%|#####2    | 26386/50000 [16:33:36<12:39:29,  1.93s/it]
Outer training loop:  53%|#####2    | 26387/50000 [16:33:38<12:38:17,  1.93s/it]
Outer training loop:  53%|#####2    | 26388/50000 [16:33:40<12:58:11,  1.98s/it]
Outer training loop:  53%|#####2    | 26389/50000 [16:33:42<12:45:00,  1.94s/it]
Outer training loop:  53%|#####2    | 26390/50000 [16:33:44<12:40:13,  1.93s/it]
Outer training loop:  53%|#####2    | 26391/50000 [16:33:46<12:41:18,  1.93s/it]
Outer training loop:  53%|#####2    | 26392/50000 [16:33:48<12:24:19,  1.89s/it]
Outer training loop:  53%|#####2    | 26393/50000 [16:33:50<12:25:42,  1.90s/it]
Outer training loop:  53%|#####2    | 26394/50000 [16:33:51<12:13:37,  1.86s/it]
Outer training loop:  53%|#####2    | 26395/50000 [16:33:53<12:38:53,  1.93s/it]
Outer training loop:  53%|#####2    | 26396/50000 [16:33:55<12:28:20,  1.90s/it]
Outer training loop:  53%|#####2    | 26397/50000 [16:33:57<12:18:05,  1.88s/it]
Outer training loop:  53%|#####2    | 26398/50000 [16:33:59<12:44:10,  1.94s/it]
Outer training loop:  53%|#####2    | 26399/50000 [16:34:01<12:40:50,  1.93s/it]
Outer training loop:  53%|#####2    | 26400/50000 [16:34:03<12:19:48,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  53%|#####2    | 26401/50000 [16:34:11<23:43:32,  3.62s/it]
Outer training loop:  53%|#####2    | 26402/50000 [16:34:13<20:45:41,  3.17s/it]
Outer training loop:  53%|#####2    | 26403/50000 [16:34:15<18:22:44,  2.80s/it]
Outer training loop:  53%|#####2    | 26404/50000 [16:34:17<16:33:07,  2.53s/it]
Outer training loop:  53%|#####2    | 26405/50000 [16:34:19<15:56:42,  2.43s/it]
Outer training loop:  53%|#####2    | 26406/50000 [16:34:21<15:05:35,  2.30s/it]
Outer training loop:  53%|#####2    | 26407/50000 [16:34:23<14:06:59,  2.15s/it]
Outer training loop:  53%|#####2    | 26408/50000 [16:34:25<13:58:56,  2.13s/it]
Outer training loop:  53%|#####2    | 26409/50000 [16:34:27<13:37:39,  2.08s/it]
Outer training loop:  53%|#####2    | 26410/50000 [16:34:28<13:06:02,  2.00s/it]
Outer training loop:  53%|#####2    | 26411/50000 [16:34:30<12:45:10,  1.95s/it]
Outer training loop:  53%|#####2    | 26412/50000 [16:34:32<12:52:26,  1.96s/it]
Outer training loop:  53%|#####2    | 26413/50000 [16:34:34<12:44:23,  1.94s/it]
Outer training loop:  53%|#####2    | 26414/50000 [16:34:36<12:48:12,  1.95s/it]
Outer training loop:  53%|#####2    | 26415/50000 [16:34:38<12:54:33,  1.97s/it]
Outer training loop:  53%|#####2    | 26416/50000 [16:34:40<12:48:21,  1.95s/it]
Outer training loop:  53%|#####2    | 26417/50000 [16:34:42<12:42:56,  1.94s/it]
Outer training loop:  53%|#####2    | 26418/50000 [16:34:44<12:57:14,  1.98s/it]
Outer training loop:  53%|#####2    | 26419/50000 [16:34:46<12:46:47,  1.95s/it]
Outer training loop:  53%|#####2    | 26420/50000 [16:34:48<12:39:39,  1.93s/it]
Outer training loop:  53%|#####2    | 26421/50000 [16:34:50<12:35:29,  1.92s/it]
Outer training loop:  53%|#####2    | 26422/50000 [16:34:52<12:58:10,  1.98s/it]
Outer training loop:  53%|#####2    | 26423/50000 [16:34:54<12:31:30,  1.91s/it]
Outer training loop:  53%|#####2    | 26424/50000 [16:34:56<12:47:46,  1.95s/it]
Outer training loop:  53%|#####2    | 26425/50000 [16:34:58<12:51:09,  1.96s/it]
Outer training loop:  53%|#####2    | 26426/50000 [16:34:59<12:44:49,  1.95s/it]
Outer training loop:  53%|#####2    | 26427/50000 [16:35:01<12:42:30,  1.94s/it]
Outer training loop:  53%|#####2    | 26428/50000 [16:35:03<12:55:43,  1.97s/it]
Outer training loop:  53%|#####2    | 26429/50000 [16:35:05<12:56:32,  1.98s/it]
Outer training loop:  53%|#####2    | 26430/50000 [16:35:07<12:43:40,  1.94s/it]
Outer training loop:  53%|#####2    | 26431/50000 [16:35:09<13:03:56,  2.00s/it]
Outer training loop:  53%|#####2    | 26432/50000 [16:35:11<12:51:40,  1.96s/it]
Outer training loop:  53%|#####2    | 26433/50000 [16:35:13<12:43:02,  1.94s/it]
Outer training loop:  53%|#####2    | 26434/50000 [16:35:15<12:29:12,  1.91s/it]
Outer training loop:  53%|#####2    | 26435/50000 [16:35:17<12:52:37,  1.97s/it]
Outer training loop:  53%|#####2    | 26436/50000 [16:35:19<12:48:42,  1.96s/it]
Outer training loop:  53%|#####2    | 26437/50000 [16:35:21<12:36:26,  1.93s/it]
Outer training loop:  53%|#####2    | 26438/50000 [16:35:23<12:44:36,  1.95s/it]
Outer training loop:  53%|#####2    | 26439/50000 [16:35:25<12:28:49,  1.91s/it]
Outer training loop:  53%|#####2    | 26440/50000 [16:35:27<12:19:25,  1.88s/it]
Outer training loop:  53%|#####2    | 26441/50000 [16:35:29<12:24:08,  1.90s/it]
Outer training loop:  53%|#####2    | 26442/50000 [16:35:31<12:36:27,  1.93s/it]
Outer training loop:  53%|#####2    | 26443/50000 [16:35:32<12:26:38,  1.90s/it]
Outer training loop:  53%|#####2    | 26444/50000 [16:35:34<12:23:11,  1.89s/it]
Outer training loop:  53%|#####2    | 26445/50000 [16:35:36<12:35:52,  1.93s/it]
Outer training loop:  53%|#####2    | 26446/50000 [16:35:38<12:32:04,  1.92s/it]
Outer training loop:  53%|#####2    | 26447/50000 [16:35:40<12:39:40,  1.94s/it]
Outer training loop:  53%|#####2    | 26448/50000 [16:35:42<12:44:29,  1.95s/it]
Outer training loop:  53%|#####2    | 26449/50000 [16:35:44<12:20:22,  1.89s/it]
Outer training loop:  53%|#####2    | 26450/50000 [16:35:46<12:13:56,  1.87s/it]
Outer training loop:  53%|#####2    | 26451/50000 [16:35:47<12:06:09,  1.85s/it]
Outer training loop:  53%|#####2    | 26452/50000 [16:35:49<12:29:12,  1.91s/it]
Outer training loop:  53%|#####2    | 26453/50000 [16:35:51<12:22:21,  1.89s/it]
Outer training loop:  53%|#####2    | 26454/50000 [16:35:53<11:59:27,  1.83s/it]
Outer training loop:  53%|#####2    | 26455/50000 [16:35:55<12:18:30,  1.88s/it]
Outer training loop:  53%|#####2    | 26456/50000 [16:35:57<12:11:59,  1.87s/it]
Outer training loop:  53%|#####2    | 26457/50000 [16:35:59<12:06:45,  1.85s/it]
Outer training loop:  53%|#####2    | 26458/50000 [16:36:00<11:55:25,  1.82s/it]
Outer training loop:  53%|#####2    | 26459/50000 [16:36:03<12:26:21,  1.90s/it]
Outer training loop:  53%|#####2    | 26460/50000 [16:36:04<12:10:43,  1.86s/it]
Outer training loop:  53%|#####2    | 26461/50000 [16:36:06<12:11:07,  1.86s/it]
Outer training loop:  53%|#####2    | 26462/50000 [16:36:08<12:27:04,  1.90s/it]
Outer training loop:  53%|#####2    | 26463/50000 [16:36:10<12:25:15,  1.90s/it]
Outer training loop:  53%|#####2    | 26464/50000 [16:36:12<12:18:11,  1.88s/it]
Outer training loop:  53%|#####2    | 26465/50000 [16:36:14<12:05:23,  1.85s/it]
Outer training loop:  53%|#####2    | 26466/50000 [16:36:16<12:24:27,  1.90s/it]
Outer training loop:  53%|#####2    | 26467/50000 [16:36:17<12:06:04,  1.85s/it]
Outer training loop:  53%|#####2    | 26468/50000 [16:36:19<12:11:59,  1.87s/it]
Outer training loop:  53%|#####2    | 26469/50000 [16:36:21<12:22:14,  1.89s/it]
Outer training loop:  53%|#####2    | 26470/50000 [16:36:23<12:18:51,  1.88s/it]
Outer training loop:  53%|#####2    | 26471/50000 [16:36:25<12:16:02,  1.88s/it]
Outer training loop:  53%|#####2    | 26472/50000 [16:36:27<12:18:46,  1.88s/it]
Outer training loop:  53%|#####2    | 26473/50000 [16:36:29<12:37:00,  1.93s/it]
Outer training loop:  53%|#####2    | 26474/50000 [16:36:31<12:42:05,  1.94s/it]
Outer training loop:  53%|#####2    | 26475/50000 [16:36:33<12:26:27,  1.90s/it]
Outer training loop:  53%|#####2    | 26476/50000 [16:36:35<12:45:56,  1.95s/it]
Outer training loop:  53%|#####2    | 26477/50000 [16:36:37<12:38:45,  1.94s/it]
Outer training loop:  53%|#####2    | 26478/50000 [16:36:39<12:24:17,  1.90s/it]
Outer training loop:  53%|#####2    | 26479/50000 [16:36:41<12:36:42,  1.93s/it]
Outer training loop:  53%|#####2    | 26480/50000 [16:36:42<12:14:38,  1.87s/it]
Outer training loop:  53%|#####2    | 26481/50000 [16:36:44<12:00:04,  1.84s/it]
Outer training loop:  53%|#####2    | 26482/50000 [16:36:46<12:22:45,  1.89s/it]
Outer training loop:  53%|#####2    | 26483/50000 [16:36:48<12:09:17,  1.86s/it]
Outer training loop:  53%|#####2    | 26484/50000 [16:36:50<11:58:01,  1.83s/it]
Outer training loop:  53%|#####2    | 26485/50000 [16:36:51<12:06:18,  1.85s/it]
Outer training loop:  53%|#####2    | 26486/50000 [16:36:53<12:20:50,  1.89s/it]
Outer training loop:  53%|#####2    | 26487/50000 [16:36:55<12:09:17,  1.86s/it]
Outer training loop:  53%|#####2    | 26488/50000 [16:36:57<11:59:29,  1.84s/it]
Outer training loop:  53%|#####2    | 26489/50000 [16:36:59<12:16:25,  1.88s/it]
Outer training loop:  53%|#####2    | 26490/50000 [16:37:01<12:03:15,  1.85s/it]
Outer training loop:  53%|#####2    | 26491/50000 [16:37:02<11:45:47,  1.80s/it]
Outer training loop:  53%|#####2    | 26492/50000 [16:37:04<12:03:18,  1.85s/it]
Outer training loop:  53%|#####2    | 26493/50000 [16:37:06<12:03:22,  1.85s/it]
Outer training loop:  53%|#####2    | 26494/50000 [16:37:08<11:56:00,  1.83s/it]
Outer training loop:  53%|#####2    | 26495/50000 [16:37:10<12:03:20,  1.85s/it]
Outer training loop:  53%|#####2    | 26496/50000 [16:37:12<12:32:49,  1.92s/it]
Outer training loop:  53%|#####2    | 26497/50000 [16:37:14<12:24:44,  1.90s/it]
Outer training loop:  53%|#####2    | 26498/50000 [16:37:16<12:13:38,  1.87s/it]
Outer training loop:  53%|#####2    | 26499/50000 [16:37:18<12:27:35,  1.91s/it]
Outer training loop:  53%|#####3    | 26500/50000 [16:37:19<12:07:30,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.90s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:28,  1.81s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:30,  1.86s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.86s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.82s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:24,  1.85s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.89s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:19,  1.86s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:19,  1.88s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.88s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:10,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:02,  1.82s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:01,  1.83s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:03,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:00,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:03,  1.96s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:58,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.86s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:56,  1.94s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:45,  1.85s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:44,  1.86s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:47,  1.96s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:41,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.87s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:25,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:22,  1.82s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:20,  1.83s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:20,  1.87s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.85s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:16,  1.86s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:09,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:07,  1.83s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:09,  1.93s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:06,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.81s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:56,  1.84s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:56,  1.94s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:53,  1.92s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:51,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:48,  1.88s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:43,  1.81s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:42,  1.83s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:39,  1.98s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.89s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:31,  1.84s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:28,  1.79s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.81s/it]

Evaluating policy:  83%|########3 | 69/83 [02:08<00:25,  1.80s/it]

Evaluating policy:  84%|########4 | 70/83 [02:10<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:23,  1.99s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:21,  1.93s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.88s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.90s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:06,  2.07s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.99s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.75s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:18,  1.73s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.79s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:18,  1.78s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:17,  1.79s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:31,  2.02s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:27,  1.99s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:22,  1.95s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:20,  1.98s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:16,  1.95s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:10,  1.94s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.92s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.90s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:01,  1.93s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.86s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.90s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:43,  1.84s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:41,  1.85s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:38,  1.83s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:41,  1.91s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:27,  1.83s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:32,  1.97s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:32,  2.00s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:26,  1.93s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:20,  1.88s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.88s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.90s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:20,  2.02s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:17,  1.98s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:14,  1.96s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:09,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:07,  1.88s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:05,  1.87s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:08,  2.02s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.94s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.94s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.89s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.86s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:51,  1.97s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:48,  1.95s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:35,  1.98s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:32,  1.91s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:30,  1.90s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.96s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.92s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.92s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.87s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.83s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.86s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.84s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.81s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  2.00s/it]

Outer training loop:  53%|#####3    | 26501/50000 [16:42:40<636:32:50, 97.52s/it]
Outer training loop:  53%|#####3    | 26502/50000 [16:42:42<448:57:41, 68.78s/it]
Outer training loop:  53%|#####3    | 26503/50000 [16:42:44<317:58:47, 48.72s/it]
Outer training loop:  53%|#####3    | 26504/50000 [16:42:46<226:05:24, 34.64s/it]
Outer training loop:  53%|#####3    | 26505/50000 [16:42:48<162:14:41, 24.86s/it]
Outer training loop:  53%|#####3    | 26506/50000 [16:42:50<117:20:38, 17.98s/it]
Outer training loop:  53%|#####3    | 26507/50000 [16:42:51<85:49:55, 13.15s/it] 
Outer training loop:  53%|#####3    | 26508/50000 [16:42:53<63:35:02,  9.74s/it]
Outer training loop:  53%|#####3    | 26509/50000 [16:42:55<48:18:23,  7.40s/it]
Outer training loop:  53%|#####3    | 26510/50000 [16:42:57<37:19:56,  5.72s/it]
Outer training loop:  53%|#####3    | 26511/50000 [16:42:59<30:53:34,  4.73s/it]
Outer training loop:  53%|#####3    | 26512/50000 [16:43:01<25:19:40,  3.88s/it]
Outer training loop:  53%|#####3    | 26513/50000 [16:43:03<21:10:53,  3.25s/it]
Outer training loop:  53%|#####3    | 26514/50000 [16:43:05<18:44:44,  2.87s/it]
Outer training loop:  53%|#####3    | 26515/50000 [16:43:07<16:59:42,  2.61s/it]
Outer training loop:  53%|#####3    | 26516/50000 [16:43:09<15:32:32,  2.38s/it]
Outer training loop:  53%|#####3    | 26517/50000 [16:43:11<14:32:10,  2.23s/it]
Outer training loop:  53%|#####3    | 26518/50000 [16:43:13<14:12:40,  2.18s/it]
Outer training loop:  53%|#####3    | 26519/50000 [16:43:15<13:26:18,  2.06s/it]
Outer training loop:  53%|#####3    | 26520/50000 [16:43:16<13:00:59,  2.00s/it]
Outer training loop:  53%|#####3    | 26521/50000 [16:43:18<12:53:23,  1.98s/it]
Outer training loop:  53%|#####3    | 26522/50000 [16:43:20<12:58:34,  1.99s/it]
Outer training loop:  53%|#####3    | 26523/50000 [16:43:22<12:34:10,  1.93s/it]
Outer training loop:  53%|#####3    | 26524/50000 [16:43:24<12:42:13,  1.95s/it]
Outer training loop:  53%|#####3    | 26525/50000 [16:43:26<12:30:04,  1.92s/it]
Outer training loop:  53%|#####3    | 26526/50000 [16:43:28<12:21:57,  1.90s/it]
Outer training loop:  53%|#####3    | 26527/50000 [16:43:30<12:42:00,  1.95s/it]
Outer training loop:  53%|#####3    | 26528/50000 [16:43:32<12:33:14,  1.93s/it]
Outer training loop:  53%|#####3    | 26529/50000 [16:43:34<12:24:57,  1.90s/it]
Outer training loop:  53%|#####3    | 26530/50000 [16:43:36<12:16:34,  1.88s/it]
Outer training loop:  53%|#####3    | 26531/50000 [16:43:38<12:31:02,  1.92s/it]
Outer training loop:  53%|#####3    | 26532/50000 [16:43:39<12:29:28,  1.92s/it]
Outer training loop:  53%|#####3    | 26533/50000 [16:43:41<12:21:10,  1.90s/it]
Outer training loop:  53%|#####3    | 26534/50000 [16:43:43<12:11:03,  1.87s/it]
Outer training loop:  53%|#####3    | 26535/50000 [16:43:45<12:35:09,  1.93s/it]
Outer training loop:  53%|#####3    | 26536/50000 [16:43:47<12:24:51,  1.90s/it]
Outer training loop:  53%|#####3    | 26537/50000 [16:43:49<12:12:56,  1.87s/it]
Outer training loop:  53%|#####3    | 26538/50000 [16:43:51<12:36:44,  1.94s/it]
Outer training loop:  53%|#####3    | 26539/50000 [16:43:53<12:16:38,  1.88s/it]
Outer training loop:  53%|#####3    | 26540/50000 [16:43:55<12:25:19,  1.91s/it]
Outer training loop:  53%|#####3    | 26541/50000 [16:43:57<12:37:01,  1.94s/it]
Outer training loop:  53%|#####3    | 26542/50000 [16:43:58<12:26:34,  1.91s/it]
Outer training loop:  53%|#####3    | 26543/50000 [16:44:00<12:25:05,  1.91s/it]
Outer training loop:  53%|#####3    | 26544/50000 [16:44:02<12:37:57,  1.94s/it]
Outer training loop:  53%|#####3    | 26545/50000 [16:44:04<12:24:21,  1.90s/it]
Outer training loop:  53%|#####3    | 26546/50000 [16:44:06<12:11:41,  1.87s/it]
Outer training loop:  53%|#####3    | 26547/50000 [16:44:08<11:56:17,  1.83s/it]
Outer training loop:  53%|#####3    | 26548/50000 [16:44:10<12:20:35,  1.89s/it]
Outer training loop:  53%|#####3    | 26549/50000 [16:44:12<12:15:37,  1.88s/it]
Outer training loop:  53%|#####3    | 26550/50000 [16:44:13<11:58:22,  1.84s/it]
Outer training loop:  53%|#####3    | 26551/50000 [16:44:15<12:14:46,  1.88s/it]
Outer training loop:  53%|#####3    | 26552/50000 [16:44:17<12:09:34,  1.87s/it]
Outer training loop:  53%|#####3    | 26553/50000 [16:44:19<12:14:57,  1.88s/it]
Outer training loop:  53%|#####3    | 26554/50000 [16:44:21<12:26:33,  1.91s/it]
Outer training loop:  53%|#####3    | 26555/50000 [16:44:23<12:45:32,  1.96s/it]
Outer training loop:  53%|#####3    | 26556/50000 [16:44:25<12:56:03,  1.99s/it]
Outer training loop:  53%|#####3    | 26557/50000 [16:44:27<12:35:16,  1.93s/it]
Outer training loop:  53%|#####3    | 26558/50000 [16:44:29<12:48:31,  1.97s/it]
Outer training loop:  53%|#####3    | 26559/50000 [16:44:31<12:27:13,  1.91s/it]
Outer training loop:  53%|#####3    | 26560/50000 [16:44:33<12:08:43,  1.87s/it]
Outer training loop:  53%|#####3    | 26561/50000 [16:44:35<12:24:01,  1.90s/it]
Outer training loop:  53%|#####3    | 26562/50000 [16:44:37<12:29:36,  1.92s/it]
Outer training loop:  53%|#####3    | 26563/50000 [16:44:39<12:35:42,  1.93s/it]
Outer training loop:  53%|#####3    | 26564/50000 [16:44:40<12:15:13,  1.88s/it]
Outer training loop:  53%|#####3    | 26565/50000 [16:44:42<12:23:52,  1.90s/it]
Outer training loop:  53%|#####3    | 26566/50000 [16:44:44<12:21:37,  1.90s/it]
Outer training loop:  53%|#####3    | 26567/50000 [16:44:46<12:05:50,  1.86s/it]
Outer training loop:  53%|#####3    | 26568/50000 [16:44:48<12:21:21,  1.90s/it]
Outer training loop:  53%|#####3    | 26569/50000 [16:44:50<12:18:20,  1.89s/it]
Outer training loop:  53%|#####3    | 26570/50000 [16:44:52<12:14:36,  1.88s/it]
Outer training loop:  53%|#####3    | 26571/50000 [16:44:54<12:32:13,  1.93s/it]
Outer training loop:  53%|#####3    | 26572/50000 [16:44:55<12:19:11,  1.89s/it]
Outer training loop:  53%|#####3    | 26573/50000 [16:44:57<12:00:58,  1.85s/it]
Outer training loop:  53%|#####3    | 26574/50000 [16:44:59<12:17:47,  1.89s/it]
Outer training loop:  53%|#####3    | 26575/50000 [16:45:01<12:09:13,  1.87s/it]
Outer training loop:  53%|#####3    | 26576/50000 [16:45:03<12:09:24,  1.87s/it]
Outer training loop:  53%|#####3    | 26577/50000 [16:45:05<12:14:57,  1.88s/it]
Outer training loop:  53%|#####3    | 26578/50000 [16:45:07<12:29:57,  1.92s/it]
Outer training loop:  53%|#####3    | 26579/50000 [16:45:09<12:16:30,  1.89s/it]
Outer training loop:  53%|#####3    | 26580/50000 [16:45:10<12:18:54,  1.89s/it]
Outer training loop:  53%|#####3    | 26581/50000 [16:45:13<12:54:56,  1.99s/it]
Outer training loop:  53%|#####3    | 26582/50000 [16:45:15<12:34:10,  1.93s/it]
Outer training loop:  53%|#####3    | 26583/50000 [16:45:16<12:25:50,  1.91s/it]
Outer training loop:  53%|#####3    | 26584/50000 [16:45:18<12:35:04,  1.93s/it]
Outer training loop:  53%|#####3    | 26585/50000 [16:45:20<12:16:49,  1.89s/it]
Outer training loop:  53%|#####3    | 26586/50000 [16:45:22<12:11:53,  1.88s/it]
Outer training loop:  53%|#####3    | 26587/50000 [16:45:24<12:06:18,  1.86s/it]
Outer training loop:  53%|#####3    | 26588/50000 [16:45:26<13:00:09,  2.00s/it]
Outer training loop:  53%|#####3    | 26589/50000 [16:45:28<12:49:18,  1.97s/it]
Outer training loop:  53%|#####3    | 26590/50000 [16:45:30<12:30:50,  1.92s/it]
Outer training loop:  53%|#####3    | 26591/50000 [16:45:32<12:39:24,  1.95s/it]
Outer training loop:  53%|#####3    | 26592/50000 [16:45:34<12:28:22,  1.92s/it]
Outer training loop:  53%|#####3    | 26593/50000 [16:45:36<12:31:03,  1.93s/it]
Outer training loop:  53%|#####3    | 26594/50000 [16:45:38<12:36:30,  1.94s/it]
Outer training loop:  53%|#####3    | 26595/50000 [16:45:39<12:29:09,  1.92s/it]
Outer training loop:  53%|#####3    | 26596/50000 [16:45:41<12:21:49,  1.90s/it]
Outer training loop:  53%|#####3    | 26597/50000 [16:45:43<12:14:19,  1.88s/it]
Outer training loop:  53%|#####3    | 26598/50000 [16:45:45<12:35:17,  1.94s/it]
Outer training loop:  53%|#####3    | 26599/50000 [16:45:47<12:19:38,  1.90s/it]
Outer training loop:  53%|#####3    | 26600/50000 [16:45:49<12:09:24,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  53%|#####3    | 26601/50000 [16:45:56<23:01:33,  3.54s/it]
Outer training loop:  53%|#####3    | 26602/50000 [16:45:58<20:05:43,  3.09s/it]
Outer training loop:  53%|#####3    | 26603/50000 [16:46:00<17:43:03,  2.73s/it]
Outer training loop:  53%|#####3    | 26604/50000 [16:46:02<15:50:46,  2.44s/it]
Outer training loop:  53%|#####3    | 26605/50000 [16:46:04<15:08:06,  2.33s/it]
Outer training loop:  53%|#####3    | 26606/50000 [16:46:06<14:08:41,  2.18s/it]
Outer training loop:  53%|#####3    | 26607/50000 [16:46:08<13:21:09,  2.05s/it]
Outer training loop:  53%|#####3    | 26608/50000 [16:46:10<13:15:55,  2.04s/it]
Outer training loop:  53%|#####3    | 26609/50000 [16:46:11<12:49:41,  1.97s/it]
Outer training loop:  53%|#####3    | 26610/50000 [16:46:13<12:33:15,  1.93s/it]
Outer training loop:  53%|#####3    | 26611/50000 [16:46:15<12:37:18,  1.94s/it]
Outer training loop:  53%|#####3    | 26612/50000 [16:46:17<12:27:06,  1.92s/it]
Outer training loop:  53%|#####3    | 26613/50000 [16:46:19<12:24:58,  1.91s/it]
Outer training loop:  53%|#####3    | 26614/50000 [16:46:21<12:13:59,  1.88s/it]
Outer training loop:  53%|#####3    | 26615/50000 [16:46:23<12:31:21,  1.93s/it]
Outer training loop:  53%|#####3    | 26616/50000 [16:46:25<12:27:19,  1.92s/it]
Outer training loop:  53%|#####3    | 26617/50000 [16:46:27<12:08:50,  1.87s/it]
Outer training loop:  53%|#####3    | 26618/50000 [16:46:29<12:27:15,  1.92s/it]
Outer training loop:  53%|#####3    | 26619/50000 [16:46:30<12:19:13,  1.90s/it]
Outer training loop:  53%|#####3    | 26620/50000 [16:46:32<12:10:35,  1.87s/it]
Outer training loop:  53%|#####3    | 26621/50000 [16:46:34<12:08:26,  1.87s/it]
Outer training loop:  53%|#####3    | 26622/50000 [16:46:36<12:34:09,  1.94s/it]
Outer training loop:  53%|#####3    | 26623/50000 [16:46:38<12:19:39,  1.90s/it]
Outer training loop:  53%|#####3    | 26624/50000 [16:46:40<12:13:11,  1.88s/it]
Outer training loop:  53%|#####3    | 26625/50000 [16:46:42<12:28:25,  1.92s/it]
Outer training loop:  53%|#####3    | 26626/50000 [16:46:44<12:25:19,  1.91s/it]
Outer training loop:  53%|#####3    | 26627/50000 [16:46:45<12:04:53,  1.86s/it]
Outer training loop:  53%|#####3    | 26628/50000 [16:46:47<12:07:18,  1.87s/it]
Outer training loop:  53%|#####3    | 26629/50000 [16:46:49<12:01:16,  1.85s/it]
Outer training loop:  53%|#####3    | 26630/50000 [16:46:51<11:46:24,  1.81s/it]
Outer training loop:  53%|#####3    | 26631/50000 [16:46:53<11:43:50,  1.81s/it]
Outer training loop:  53%|#####3    | 26632/50000 [16:46:55<12:12:05,  1.88s/it]
Outer training loop:  53%|#####3    | 26633/50000 [16:46:57<11:57:43,  1.84s/it]
Outer training loop:  53%|#####3    | 26634/50000 [16:46:58<11:52:10,  1.83s/it]
Outer training loop:  53%|#####3    | 26635/50000 [16:47:00<12:05:18,  1.86s/it]
Outer training loop:  53%|#####3    | 26636/50000 [16:47:02<11:53:52,  1.83s/it]
Outer training loop:  53%|#####3    | 26637/50000 [16:47:04<11:57:41,  1.84s/it]
Outer training loop:  53%|#####3    | 26638/50000 [16:47:06<12:12:37,  1.88s/it]
Outer training loop:  53%|#####3    | 26639/50000 [16:47:08<11:59:17,  1.85s/it]
Outer training loop:  53%|#####3    | 26640/50000 [16:47:09<11:51:00,  1.83s/it]
Outer training loop:  53%|#####3    | 26641/50000 [16:47:11<11:34:26,  1.78s/it]
Outer training loop:  53%|#####3    | 26642/50000 [16:47:13<11:55:41,  1.84s/it]
Outer training loop:  53%|#####3    | 26643/50000 [16:47:15<11:46:28,  1.81s/it]
Outer training loop:  53%|#####3    | 26644/50000 [16:47:17<11:42:20,  1.80s/it]
Outer training loop:  53%|#####3    | 26645/50000 [16:47:19<12:20:18,  1.90s/it]
Outer training loop:  53%|#####3    | 26646/50000 [16:47:21<12:15:56,  1.89s/it]
Outer training loop:  53%|#####3    | 26647/50000 [16:47:22<12:07:30,  1.87s/it]
Outer training loop:  53%|#####3    | 26648/50000 [16:47:24<12:03:27,  1.86s/it]
Outer training loop:  53%|#####3    | 26649/50000 [16:47:26<12:18:53,  1.90s/it]
Outer training loop:  53%|#####3    | 26650/50000 [16:47:28<12:05:25,  1.86s/it]
Outer training loop:  53%|#####3    | 26651/50000 [16:47:30<11:53:51,  1.83s/it]
Outer training loop:  53%|#####3    | 26652/50000 [16:47:32<12:24:25,  1.91s/it]
Outer training loop:  53%|#####3    | 26653/50000 [16:47:34<12:21:59,  1.91s/it]
Outer training loop:  53%|#####3    | 26654/50000 [16:47:36<12:07:40,  1.87s/it]
Outer training loop:  53%|#####3    | 26655/50000 [16:47:38<12:27:37,  1.92s/it]
Outer training loop:  53%|#####3    | 26656/50000 [16:47:40<12:28:50,  1.92s/it]
Outer training loop:  53%|#####3    | 26657/50000 [16:47:41<12:19:39,  1.90s/it]
Outer training loop:  53%|#####3    | 26658/50000 [16:47:43<12:07:05,  1.87s/it]
Outer training loop:  53%|#####3    | 26659/50000 [16:47:45<12:25:06,  1.92s/it]
Outer training loop:  53%|#####3    | 26660/50000 [16:47:47<12:10:05,  1.88s/it]
Outer training loop:  53%|#####3    | 26661/50000 [16:47:49<12:05:28,  1.87s/it]
Outer training loop:  53%|#####3    | 26662/50000 [16:47:51<12:28:14,  1.92s/it]
Outer training loop:  53%|#####3    | 26663/50000 [16:47:53<12:07:10,  1.87s/it]
Outer training loop:  53%|#####3    | 26664/50000 [16:47:54<12:00:58,  1.85s/it]
Outer training loop:  53%|#####3    | 26665/50000 [16:47:56<11:44:33,  1.81s/it]
Outer training loop:  53%|#####3    | 26666/50000 [16:47:58<12:27:23,  1.92s/it]
Outer training loop:  53%|#####3    | 26667/50000 [16:48:00<12:14:00,  1.89s/it]
Outer training loop:  53%|#####3    | 26668/50000 [16:48:02<12:12:59,  1.88s/it]
Outer training loop:  53%|#####3    | 26669/50000 [16:48:04<12:27:11,  1.92s/it]
Outer training loop:  53%|#####3    | 26670/50000 [16:48:06<12:21:14,  1.91s/it]
Outer training loop:  53%|#####3    | 26671/50000 [16:48:08<12:11:32,  1.88s/it]
Outer training loop:  53%|#####3    | 26672/50000 [16:48:10<12:37:18,  1.95s/it]
Outer training loop:  53%|#####3    | 26673/50000 [16:48:12<12:33:57,  1.94s/it]
Outer training loop:  53%|#####3    | 26674/50000 [16:48:14<12:30:23,  1.93s/it]
Outer training loop:  53%|#####3    | 26675/50000 [16:48:16<12:40:27,  1.96s/it]
Outer training loop:  53%|#####3    | 26676/50000 [16:48:17<12:18:21,  1.90s/it]
Outer training loop:  53%|#####3    | 26677/50000 [16:48:19<11:58:02,  1.85s/it]
Outer training loop:  53%|#####3    | 26678/50000 [16:48:21<11:59:42,  1.85s/it]
Outer training loop:  53%|#####3    | 26679/50000 [16:48:23<12:07:34,  1.87s/it]
Outer training loop:  53%|#####3    | 26680/50000 [16:48:25<11:58:28,  1.85s/it]
Outer training loop:  53%|#####3    | 26681/50000 [16:48:27<12:05:02,  1.87s/it]
Outer training loop:  53%|#####3    | 26682/50000 [16:48:29<12:09:37,  1.88s/it]
Outer training loop:  53%|#####3    | 26683/50000 [16:48:31<12:38:09,  1.95s/it]
Outer training loop:  53%|#####3    | 26684/50000 [16:48:32<12:15:56,  1.89s/it]
Outer training loop:  53%|#####3    | 26685/50000 [16:48:34<12:20:40,  1.91s/it]
Outer training loop:  53%|#####3    | 26686/50000 [16:48:36<12:42:30,  1.96s/it]
Outer training loop:  53%|#####3    | 26687/50000 [16:48:38<12:24:08,  1.92s/it]
Outer training loop:  53%|#####3    | 26688/50000 [16:48:40<12:14:08,  1.89s/it]
Outer training loop:  53%|#####3    | 26689/50000 [16:48:42<12:20:08,  1.91s/it]
Outer training loop:  53%|#####3    | 26690/50000 [16:48:44<12:17:40,  1.90s/it]
Outer training loop:  53%|#####3    | 26691/50000 [16:48:46<12:06:22,  1.87s/it]
Outer training loop:  53%|#####3    | 26692/50000 [16:48:48<12:08:59,  1.88s/it]
Outer training loop:  53%|#####3    | 26693/50000 [16:48:49<12:01:47,  1.86s/it]
Outer training loop:  53%|#####3    | 26694/50000 [16:48:51<11:57:33,  1.85s/it]
Outer training loop:  53%|#####3    | 26695/50000 [16:48:53<12:00:41,  1.86s/it]
Outer training loop:  53%|#####3    | 26696/50000 [16:48:55<12:28:41,  1.93s/it]
Outer training loop:  53%|#####3    | 26697/50000 [16:48:57<12:21:57,  1.91s/it]
Outer training loop:  53%|#####3    | 26698/50000 [16:48:59<12:22:34,  1.91s/it]
Outer training loop:  53%|#####3    | 26699/50000 [16:49:01<12:32:58,  1.94s/it]
Outer training loop:  53%|#####3    | 26700/50000 [16:49:03<12:18:24,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.96s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.02s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.91s/it]

Outer training loop:  53%|#####3    | 26701/50000 [16:49:11<23:39:24,  3.66s/it]
Outer training loop:  53%|#####3    | 26702/50000 [16:49:12<20:05:04,  3.10s/it]
Outer training loop:  53%|#####3    | 26703/50000 [16:49:14<17:57:09,  2.77s/it]
Outer training loop:  53%|#####3    | 26704/50000 [16:49:16<16:12:33,  2.50s/it]
Outer training loop:  53%|#####3    | 26705/50000 [16:49:18<15:04:16,  2.33s/it]
Outer training loop:  53%|#####3    | 26706/50000 [16:49:20<14:14:02,  2.20s/it]
Outer training loop:  53%|#####3    | 26707/50000 [16:49:22<13:59:29,  2.16s/it]
Outer training loop:  53%|#####3    | 26708/50000 [16:49:24<13:23:17,  2.07s/it]
Outer training loop:  53%|#####3    | 26709/50000 [16:49:26<12:54:03,  1.99s/it]
Outer training loop:  53%|#####3    | 26710/50000 [16:49:28<13:14:56,  2.05s/it]
Outer training loop:  53%|#####3    | 26711/50000 [16:49:30<12:54:40,  2.00s/it]
Outer training loop:  53%|#####3    | 26712/50000 [16:49:32<12:31:56,  1.94s/it]
Outer training loop:  53%|#####3    | 26713/50000 [16:49:34<12:40:05,  1.96s/it]
Outer training loop:  53%|#####3    | 26714/50000 [16:49:36<12:33:50,  1.94s/it]
Outer training loop:  53%|#####3    | 26715/50000 [16:49:37<12:23:01,  1.91s/it]
Outer training loop:  53%|#####3    | 26716/50000 [16:49:39<12:08:55,  1.88s/it]
Outer training loop:  53%|#####3    | 26717/50000 [16:49:41<12:44:10,  1.97s/it]
Outer training loop:  53%|#####3    | 26718/50000 [16:49:43<12:29:19,  1.93s/it]
Outer training loop:  53%|#####3    | 26719/50000 [16:49:45<12:17:20,  1.90s/it]
Outer training loop:  53%|#####3    | 26720/50000 [16:49:47<12:33:38,  1.94s/it]
Outer training loop:  53%|#####3    | 26721/50000 [16:49:49<12:12:42,  1.89s/it]
Outer training loop:  53%|#####3    | 26722/50000 [16:49:51<12:05:02,  1.87s/it]
Outer training loop:  53%|#####3    | 26723/50000 [16:49:52<11:53:29,  1.84s/it]
Outer training loop:  53%|#####3    | 26724/50000 [16:49:54<12:11:22,  1.89s/it]
Outer training loop:  53%|#####3    | 26725/50000 [16:49:56<11:53:07,  1.84s/it]
Outer training loop:  53%|#####3    | 26726/50000 [16:49:58<11:54:10,  1.84s/it]
Outer training loop:  53%|#####3    | 26727/50000 [16:50:00<12:03:14,  1.86s/it]
Outer training loop:  53%|#####3    | 26728/50000 [16:50:02<11:59:22,  1.85s/it]
Outer training loop:  53%|#####3    | 26729/50000 [16:50:04<11:46:54,  1.82s/it]
Outer training loop:  53%|#####3    | 26730/50000 [16:50:06<12:30:32,  1.94s/it]
Outer training loop:  53%|#####3    | 26731/50000 [16:50:08<12:16:10,  1.90s/it]
Outer training loop:  53%|#####3    | 26732/50000 [16:50:09<12:02:00,  1.86s/it]
Outer training loop:  53%|#####3    | 26733/50000 [16:50:11<12:13:01,  1.89s/it]
Outer training loop:  53%|#####3    | 26734/50000 [16:50:13<12:09:16,  1.88s/it]
Outer training loop:  53%|#####3    | 26735/50000 [16:50:15<12:01:45,  1.86s/it]
Outer training loop:  53%|#####3    | 26736/50000 [16:50:17<11:41:48,  1.81s/it]
Outer training loop:  53%|#####3    | 26737/50000 [16:50:19<12:01:51,  1.86s/it]
Outer training loop:  53%|#####3    | 26738/50000 [16:50:21<12:20:58,  1.91s/it]
Outer training loop:  53%|#####3    | 26739/50000 [16:50:22<12:06:11,  1.87s/it]
Outer training loop:  53%|#####3    | 26740/50000 [16:50:24<11:36:08,  1.80s/it]
Outer training loop:  53%|#####3    | 26741/50000 [16:50:26<12:11:18,  1.89s/it]
Outer training loop:  53%|#####3    | 26742/50000 [16:50:28<12:04:27,  1.87s/it]
Outer training loop:  53%|#####3    | 26743/50000 [16:50:30<12:02:44,  1.86s/it]
Outer training loop:  53%|#####3    | 26744/50000 [16:50:32<12:17:06,  1.90s/it]
Outer training loop:  53%|#####3    | 26745/50000 [16:50:34<12:12:17,  1.89s/it]
Outer training loop:  53%|#####3    | 26746/50000 [16:50:35<11:55:38,  1.85s/it]
Outer training loop:  53%|#####3    | 26747/50000 [16:50:37<12:11:48,  1.89s/it]
Outer training loop:  53%|#####3    | 26748/50000 [16:50:39<11:50:25,  1.83s/it]
Outer training loop:  53%|#####3    | 26749/50000 [16:50:41<12:17:10,  1.90s/it]
Outer training loop:  54%|#####3    | 26750/50000 [16:50:43<12:14:46,  1.90s/it]
Outer training loop:  54%|#####3    | 26751/50000 [16:50:45<12:23:39,  1.92s/it]
Outer training loop:  54%|#####3    | 26752/50000 [16:50:47<12:08:42,  1.88s/it]
Outer training loop:  54%|#####3    | 26753/50000 [16:50:49<12:04:47,  1.87s/it]
Outer training loop:  54%|#####3    | 26754/50000 [16:50:51<12:24:13,  1.92s/it]
Outer training loop:  54%|#####3    | 26755/50000 [16:50:53<12:21:53,  1.91s/it]
Outer training loop:  54%|#####3    | 26756/50000 [16:50:54<12:06:43,  1.88s/it]
Outer training loop:  54%|#####3    | 26757/50000 [16:50:56<11:55:01,  1.85s/it]
Outer training loop:  54%|#####3    | 26758/50000 [16:50:58<12:30:56,  1.94s/it]
Outer training loop:  54%|#####3    | 26759/50000 [16:51:00<12:21:54,  1.92s/it]
Outer training loop:  54%|#####3    | 26760/50000 [16:51:02<12:14:14,  1.90s/it]
Outer training loop:  54%|#####3    | 26761/50000 [16:51:04<12:33:36,  1.95s/it]
Outer training loop:  54%|#####3    | 26762/50000 [16:51:06<12:29:38,  1.94s/it]
Outer training loop:  54%|#####3    | 26763/50000 [16:51:08<12:17:25,  1.90s/it]
Outer training loop:  54%|#####3    | 26764/50000 [16:51:10<12:25:13,  1.92s/it]
Outer training loop:  54%|#####3    | 26765/50000 [16:51:12<12:07:01,  1.88s/it]
Outer training loop:  54%|#####3    | 26766/50000 [16:51:13<12:05:23,  1.87s/it]
Outer training loop:  54%|#####3    | 26767/50000 [16:51:15<11:58:40,  1.86s/it]
Outer training loop:  54%|#####3    | 26768/50000 [16:51:17<12:17:43,  1.91s/it]
Outer training loop:  54%|#####3    | 26769/50000 [16:51:19<12:18:58,  1.91s/it]
Outer training loop:  54%|#####3    | 26770/50000 [16:51:21<12:18:23,  1.91s/it]
Outer training loop:  54%|#####3    | 26771/50000 [16:51:23<12:17:02,  1.90s/it]
Outer training loop:  54%|#####3    | 26772/50000 [16:51:25<12:27:56,  1.93s/it]
Outer training loop:  54%|#####3    | 26773/50000 [16:51:27<12:17:53,  1.91s/it]
Outer training loop:  54%|#####3    | 26774/50000 [16:51:29<12:07:45,  1.88s/it]
Outer training loop:  54%|#####3    | 26775/50000 [16:51:31<12:21:51,  1.92s/it]
Outer training loop:  54%|#####3    | 26776/50000 [16:51:32<11:56:22,  1.85s/it]
Outer training loop:  54%|#####3    | 26777/50000 [16:51:34<11:50:44,  1.84s/it]
Outer training loop:  54%|#####3    | 26778/50000 [16:51:36<12:14:54,  1.90s/it]
Outer training loop:  54%|#####3    | 26779/50000 [16:51:38<11:54:05,  1.85s/it]
Outer training loop:  54%|#####3    | 26780/50000 [16:51:40<11:55:30,  1.85s/it]
Outer training loop:  54%|#####3    | 26781/50000 [16:51:42<11:49:54,  1.83s/it]
Outer training loop:  54%|#####3    | 26782/50000 [16:51:44<12:13:52,  1.90s/it]
Outer training loop:  54%|#####3    | 26783/50000 [16:51:46<12:15:57,  1.90s/it]
Outer training loop:  54%|#####3    | 26784/50000 [16:51:48<12:31:17,  1.94s/it]
Outer training loop:  54%|#####3    | 26785/50000 [16:51:50<12:36:53,  1.96s/it]
Outer training loop:  54%|#####3    | 26786/50000 [16:51:51<12:29:46,  1.94s/it]
Outer training loop:  54%|#####3    | 26787/50000 [16:51:53<12:20:46,  1.91s/it]
Outer training loop:  54%|#####3    | 26788/50000 [16:51:55<12:07:04,  1.88s/it]
Outer training loop:  54%|#####3    | 26789/50000 [16:51:57<12:23:23,  1.92s/it]
Outer training loop:  54%|#####3    | 26790/50000 [16:51:59<12:25:07,  1.93s/it]
Outer training loop:  54%|#####3    | 26791/50000 [16:52:01<12:11:11,  1.89s/it]
Outer training loop:  54%|#####3    | 26792/50000 [16:52:03<12:28:19,  1.93s/it]
Outer training loop:  54%|#####3    | 26793/50000 [16:52:05<12:15:13,  1.90s/it]
Outer training loop:  54%|#####3    | 26794/50000 [16:52:07<12:12:36,  1.89s/it]
Outer training loop:  54%|#####3    | 26795/50000 [16:52:09<12:28:29,  1.94s/it]
Outer training loop:  54%|#####3    | 26796/50000 [16:52:11<12:22:48,  1.92s/it]
Outer training loop:  54%|#####3    | 26797/50000 [16:52:12<12:23:38,  1.92s/it]
Outer training loop:  54%|#####3    | 26798/50000 [16:52:15<12:37:45,  1.96s/it]
Outer training loop:  54%|#####3    | 26799/50000 [16:52:16<12:35:08,  1.95s/it]
Outer training loop:  54%|#####3    | 26800/50000 [16:52:18<12:16:44,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  54%|#####3    | 26801/50000 [16:52:26<22:59:43,  3.57s/it]
Outer training loop:  54%|#####3    | 26802/50000 [16:52:27<19:33:48,  3.04s/it]
Outer training loop:  54%|#####3    | 26803/50000 [16:52:30<17:40:24,  2.74s/it]
Outer training loop:  54%|#####3    | 26804/50000 [16:52:31<16:01:04,  2.49s/it]
Outer training loop:  54%|#####3    | 26805/50000 [16:52:33<14:38:43,  2.27s/it]
Outer training loop:  54%|#####3    | 26806/50000 [16:52:35<14:23:16,  2.23s/it]
Outer training loop:  54%|#####3    | 26807/50000 [16:52:37<13:42:33,  2.13s/it]
Outer training loop:  54%|#####3    | 26808/50000 [16:52:39<13:05:36,  2.03s/it]
Outer training loop:  54%|#####3    | 26809/50000 [16:52:41<12:36:32,  1.96s/it]
Outer training loop:  54%|#####3    | 26810/50000 [16:52:43<12:51:04,  2.00s/it]
Outer training loop:  54%|#####3    | 26811/50000 [16:52:45<12:22:41,  1.92s/it]
Outer training loop:  54%|#####3    | 26812/50000 [16:52:46<11:57:32,  1.86s/it]
Outer training loop:  54%|#####3    | 26813/50000 [16:52:48<11:59:20,  1.86s/it]
Outer training loop:  54%|#####3    | 26814/50000 [16:52:50<11:52:28,  1.84s/it]
Outer training loop:  54%|#####3    | 26815/50000 [16:52:52<11:38:00,  1.81s/it]
Outer training loop:  54%|#####3    | 26816/50000 [16:52:54<12:02:13,  1.87s/it]
Outer training loop:  54%|#####3    | 26817/50000 [16:52:56<11:53:59,  1.85s/it]
Outer training loop:  54%|#####3    | 26818/50000 [16:52:57<12:00:03,  1.86s/it]
Outer training loop:  54%|#####3    | 26819/50000 [16:52:59<11:59:07,  1.86s/it]
Outer training loop:  54%|#####3    | 26820/50000 [16:53:01<12:14:37,  1.90s/it]
Outer training loop:  54%|#####3    | 26821/50000 [16:53:03<12:05:09,  1.88s/it]
Outer training loop:  54%|#####3    | 26822/50000 [16:53:05<12:15:33,  1.90s/it]
Outer training loop:  54%|#####3    | 26823/50000 [16:53:07<12:42:15,  1.97s/it]
Outer training loop:  54%|#####3    | 26824/50000 [16:53:09<12:34:22,  1.95s/it]
Outer training loop:  54%|#####3    | 26825/50000 [16:53:11<12:12:32,  1.90s/it]
Outer training loop:  54%|#####3    | 26826/50000 [16:53:13<12:15:10,  1.90s/it]
Outer training loop:  54%|#####3    | 26827/50000 [16:53:15<12:23:08,  1.92s/it]
Outer training loop:  54%|#####3    | 26828/50000 [16:53:17<12:05:45,  1.88s/it]
Outer training loop:  54%|#####3    | 26829/50000 [16:53:18<12:06:56,  1.88s/it]
Outer training loop:  54%|#####3    | 26830/50000 [16:53:21<12:30:28,  1.94s/it]
Outer training loop:  54%|#####3    | 26831/50000 [16:53:22<12:03:58,  1.87s/it]
Outer training loop:  54%|#####3    | 26832/50000 [16:53:24<11:56:42,  1.86s/it]
Outer training loop:  54%|#####3    | 26833/50000 [16:53:26<11:48:58,  1.84s/it]
Outer training loop:  54%|#####3    | 26834/50000 [16:53:28<12:05:18,  1.88s/it]
Outer training loop:  54%|#####3    | 26835/50000 [16:53:30<11:45:54,  1.83s/it]
Outer training loop:  54%|#####3    | 26836/50000 [16:53:31<11:42:36,  1.82s/it]
Outer training loop:  54%|#####3    | 26837/50000 [16:53:33<11:59:03,  1.86s/it]
Outer training loop:  54%|#####3    | 26838/50000 [16:53:35<12:16:30,  1.91s/it]
Outer training loop:  54%|#####3    | 26839/50000 [16:53:37<11:58:36,  1.86s/it]
Outer training loop:  54%|#####3    | 26840/50000 [16:53:39<12:16:18,  1.91s/it]
Outer training loop:  54%|#####3    | 26841/50000 [16:53:41<12:00:59,  1.87s/it]
Outer training loop:  54%|#####3    | 26842/50000 [16:53:43<11:54:15,  1.85s/it]
Outer training loop:  54%|#####3    | 26843/50000 [16:53:44<11:47:43,  1.83s/it]
Outer training loop:  54%|#####3    | 26844/50000 [16:53:47<12:09:36,  1.89s/it]
Outer training loop:  54%|#####3    | 26845/50000 [16:53:48<12:02:03,  1.87s/it]
Outer training loop:  54%|#####3    | 26846/50000 [16:53:50<12:28:48,  1.94s/it]
Outer training loop:  54%|#####3    | 26847/50000 [16:53:52<12:32:49,  1.95s/it]
Outer training loop:  54%|#####3    | 26848/50000 [16:53:54<12:22:08,  1.92s/it]
Outer training loop:  54%|#####3    | 26849/50000 [16:53:56<11:58:49,  1.86s/it]
Outer training loop:  54%|#####3    | 26850/50000 [16:53:58<12:32:19,  1.95s/it]
Outer training loop:  54%|#####3    | 26851/50000 [16:54:00<12:14:00,  1.90s/it]
Outer training loop:  54%|#####3    | 26852/50000 [16:54:02<12:20:21,  1.92s/it]
Outer training loop:  54%|#####3    | 26853/50000 [16:54:04<12:04:07,  1.88s/it]
Outer training loop:  54%|#####3    | 26854/50000 [16:54:06<12:27:01,  1.94s/it]
Outer training loop:  54%|#####3    | 26855/50000 [16:54:07<11:55:56,  1.86s/it]
Outer training loop:  54%|#####3    | 26856/50000 [16:54:09<12:00:39,  1.87s/it]
Outer training loop:  54%|#####3    | 26857/50000 [16:54:11<12:06:28,  1.88s/it]
Outer training loop:  54%|#####3    | 26858/50000 [16:54:13<11:52:28,  1.85s/it]
Outer training loop:  54%|#####3    | 26859/50000 [16:54:15<11:40:04,  1.82s/it]
Outer training loop:  54%|#####3    | 26860/50000 [16:54:17<12:06:50,  1.88s/it]
Outer training loop:  54%|#####3    | 26861/50000 [16:54:19<12:03:58,  1.88s/it]
Outer training loop:  54%|#####3    | 26862/50000 [16:54:21<12:05:05,  1.88s/it]
Outer training loop:  54%|#####3    | 26863/50000 [16:54:22<11:56:53,  1.86s/it]
Outer training loop:  54%|#####3    | 26864/50000 [16:54:24<12:14:46,  1.91s/it]
Outer training loop:  54%|#####3    | 26865/50000 [16:54:26<12:15:01,  1.91s/it]
Outer training loop:  54%|#####3    | 26866/50000 [16:54:28<12:12:31,  1.90s/it]
Outer training loop:  54%|#####3    | 26867/50000 [16:54:30<12:05:28,  1.88s/it]
Outer training loop:  54%|#####3    | 26868/50000 [16:54:32<12:30:47,  1.95s/it]
Outer training loop:  54%|#####3    | 26869/50000 [16:54:34<12:23:03,  1.93s/it]
Outer training loop:  54%|#####3    | 26870/50000 [16:54:36<12:21:09,  1.92s/it]
Outer training loop:  54%|#####3    | 26871/50000 [16:54:38<12:33:03,  1.95s/it]
Outer training loop:  54%|#####3    | 26872/50000 [16:54:40<12:06:45,  1.89s/it]
Outer training loop:  54%|#####3    | 26873/50000 [16:54:41<11:51:26,  1.85s/it]
Outer training loop:  54%|#####3    | 26874/50000 [16:54:43<12:07:34,  1.89s/it]
Outer training loop:  54%|#####3    | 26875/50000 [16:54:45<12:03:13,  1.88s/it]
Outer training loop:  54%|#####3    | 26876/50000 [16:54:47<11:57:02,  1.86s/it]
Outer training loop:  54%|#####3    | 26877/50000 [16:54:49<12:04:16,  1.88s/it]
Outer training loop:  54%|#####3    | 26878/50000 [16:54:51<12:18:47,  1.92s/it]
Outer training loop:  54%|#####3    | 26879/50000 [16:54:53<12:13:00,  1.90s/it]
Outer training loop:  54%|#####3    | 26880/50000 [16:54:55<12:09:31,  1.89s/it]
Outer training loop:  54%|#####3    | 26881/50000 [16:54:57<12:19:52,  1.92s/it]
Outer training loop:  54%|#####3    | 26882/50000 [16:54:59<12:14:31,  1.91s/it]
Outer training loop:  54%|#####3    | 26883/50000 [16:55:00<12:01:31,  1.87s/it]
Outer training loop:  54%|#####3    | 26884/50000 [16:55:02<11:47:28,  1.84s/it]
Outer training loop:  54%|#####3    | 26885/50000 [16:55:04<12:15:42,  1.91s/it]
Outer training loop:  54%|#####3    | 26886/50000 [16:55:06<12:18:40,  1.92s/it]
Outer training loop:  54%|#####3    | 26887/50000 [16:55:08<12:03:05,  1.88s/it]
Outer training loop:  54%|#####3    | 26888/50000 [16:55:10<12:10:06,  1.90s/it]
Outer training loop:  54%|#####3    | 26889/50000 [16:55:12<11:53:59,  1.85s/it]
Outer training loop:  54%|#####3    | 26890/50000 [16:55:14<11:59:35,  1.87s/it]
Outer training loop:  54%|#####3    | 26891/50000 [16:55:16<12:27:42,  1.94s/it]
Outer training loop:  54%|#####3    | 26892/50000 [16:55:18<12:23:44,  1.93s/it]
Outer training loop:  54%|#####3    | 26893/50000 [16:55:20<12:27:44,  1.94s/it]
Outer training loop:  54%|#####3    | 26894/50000 [16:55:21<12:16:42,  1.91s/it]
Outer training loop:  54%|#####3    | 26895/50000 [16:55:23<12:42:43,  1.98s/it]
Outer training loop:  54%|#####3    | 26896/50000 [16:55:26<12:54:15,  2.01s/it]
Outer training loop:  54%|#####3    | 26897/50000 [16:55:27<12:30:20,  1.95s/it]
Outer training loop:  54%|#####3    | 26898/50000 [16:55:30<12:54:54,  2.01s/it]
Outer training loop:  54%|#####3    | 26899/50000 [16:55:31<12:29:58,  1.95s/it]
Outer training loop:  54%|#####3    | 26900/50000 [16:55:33<12:18:09,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  54%|#####3    | 26901/50000 [16:55:41<23:02:59,  3.59s/it]
Outer training loop:  54%|#####3    | 26902/50000 [16:55:43<19:58:31,  3.11s/it]
Outer training loop:  54%|#####3    | 26903/50000 [16:55:45<17:32:03,  2.73s/it]
Outer training loop:  54%|#####3    | 26904/50000 [16:55:46<15:44:04,  2.45s/it]
Outer training loop:  54%|#####3    | 26905/50000 [16:55:48<14:56:48,  2.33s/it]
Outer training loop:  54%|#####3    | 26906/50000 [16:55:50<14:00:52,  2.18s/it]
Outer training loop:  54%|#####3    | 26907/50000 [16:55:52<13:25:41,  2.09s/it]
Outer training loop:  54%|#####3    | 26908/50000 [16:55:54<12:56:11,  2.02s/it]
Outer training loop:  54%|#####3    | 26909/50000 [16:55:56<12:53:56,  2.01s/it]
Outer training loop:  54%|#####3    | 26910/50000 [16:55:58<12:33:44,  1.96s/it]
Outer training loop:  54%|#####3    | 26911/50000 [16:55:59<12:05:05,  1.88s/it]
Outer training loop:  54%|#####3    | 26912/50000 [16:56:01<12:09:24,  1.90s/it]
Outer training loop:  54%|#####3    | 26913/50000 [16:56:03<11:47:45,  1.84s/it]
Outer training loop:  54%|#####3    | 26914/50000 [16:56:05<11:59:22,  1.87s/it]
Outer training loop:  54%|#####3    | 26915/50000 [16:56:07<12:28:31,  1.95s/it]
Outer training loop:  54%|#####3    | 26916/50000 [16:56:09<12:18:20,  1.92s/it]
Outer training loop:  54%|#####3    | 26917/50000 [16:56:11<12:05:02,  1.88s/it]
Outer training loop:  54%|#####3    | 26918/50000 [16:56:13<12:05:54,  1.89s/it]
Outer training loop:  54%|#####3    | 26919/50000 [16:56:15<12:19:32,  1.92s/it]
Outer training loop:  54%|#####3    | 26920/50000 [16:56:16<11:59:22,  1.87s/it]
Outer training loop:  54%|#####3    | 26921/50000 [16:56:18<11:48:01,  1.84s/it]
Outer training loop:  54%|#####3    | 26922/50000 [16:56:20<12:23:32,  1.93s/it]
Outer training loop:  54%|#####3    | 26923/50000 [16:56:22<12:00:28,  1.87s/it]
Outer training loop:  54%|#####3    | 26924/50000 [16:56:24<12:01:13,  1.88s/it]
Outer training loop:  54%|#####3    | 26925/50000 [16:56:26<12:16:49,  1.92s/it]
Outer training loop:  54%|#####3    | 26926/50000 [16:56:28<12:10:22,  1.90s/it]
Outer training loop:  54%|#####3    | 26927/50000 [16:56:30<11:56:00,  1.86s/it]
Outer training loop:  54%|#####3    | 26928/50000 [16:56:32<11:59:52,  1.87s/it]
Outer training loop:  54%|#####3    | 26929/50000 [16:56:33<12:04:22,  1.88s/it]
Outer training loop:  54%|#####3    | 26930/50000 [16:56:35<12:00:00,  1.87s/it]
Outer training loop:  54%|#####3    | 26931/50000 [16:56:37<11:49:40,  1.85s/it]
Outer training loop:  54%|#####3    | 26932/50000 [16:56:39<11:37:04,  1.81s/it]
Outer training loop:  54%|#####3    | 26933/50000 [16:56:41<12:02:54,  1.88s/it]
Outer training loop:  54%|#####3    | 26934/50000 [16:56:43<11:53:50,  1.86s/it]
Outer training loop:  54%|#####3    | 26935/50000 [16:56:45<12:12:21,  1.91s/it]
Outer training loop:  54%|#####3    | 26936/50000 [16:56:47<12:26:17,  1.94s/it]
Outer training loop:  54%|#####3    | 26937/50000 [16:56:49<12:21:29,  1.93s/it]
Outer training loop:  54%|#####3    | 26938/50000 [16:56:51<12:21:57,  1.93s/it]
Outer training loop:  54%|#####3    | 26939/50000 [16:56:53<12:34:32,  1.96s/it]
Outer training loop:  54%|#####3    | 26940/50000 [16:56:54<12:13:33,  1.91s/it]
Outer training loop:  54%|#####3    | 26941/50000 [16:56:56<11:48:06,  1.84s/it]
Outer training loop:  54%|#####3    | 26942/50000 [16:56:58<11:49:35,  1.85s/it]
Outer training loop:  54%|#####3    | 26943/50000 [16:57:00<12:25:49,  1.94s/it]
Outer training loop:  54%|#####3    | 26944/50000 [16:57:02<12:08:49,  1.90s/it]
Outer training loop:  54%|#####3    | 26945/50000 [16:57:04<12:15:28,  1.91s/it]
Outer training loop:  54%|#####3    | 26946/50000 [16:57:06<12:14:37,  1.91s/it]
Outer training loop:  54%|#####3    | 26947/50000 [16:57:08<12:14:20,  1.91s/it]
Outer training loop:  54%|#####3    | 26948/50000 [16:57:10<12:14:35,  1.91s/it]
Outer training loop:  54%|#####3    | 26949/50000 [16:57:12<12:27:35,  1.95s/it]
Outer training loop:  54%|#####3    | 26950/50000 [16:57:13<12:09:22,  1.90s/it]
Outer training loop:  54%|#####3    | 26951/50000 [16:57:15<12:02:06,  1.88s/it]
Outer training loop:  54%|#####3    | 26952/50000 [16:57:17<12:11:19,  1.90s/it]
Outer training loop:  54%|#####3    | 26953/50000 [16:57:19<12:09:57,  1.90s/it]
Outer training loop:  54%|#####3    | 26954/50000 [16:57:21<11:57:46,  1.87s/it]
Outer training loop:  54%|#####3    | 26955/50000 [16:57:23<12:03:00,  1.88s/it]
Outer training loop:  54%|#####3    | 26956/50000 [16:57:25<12:27:59,  1.95s/it]
Outer training loop:  54%|#####3    | 26957/50000 [16:57:27<12:19:54,  1.93s/it]
Outer training loop:  54%|#####3    | 26958/50000 [16:57:29<12:10:36,  1.90s/it]
Outer training loop:  54%|#####3    | 26959/50000 [16:57:31<12:22:20,  1.93s/it]
Outer training loop:  54%|#####3    | 26960/50000 [16:57:32<11:57:15,  1.87s/it]
Outer training loop:  54%|#####3    | 26961/50000 [16:57:34<11:56:20,  1.87s/it]
Outer training loop:  54%|#####3    | 26962/50000 [16:57:36<12:14:04,  1.91s/it]
Outer training loop:  54%|#####3    | 26963/50000 [16:57:38<11:51:49,  1.85s/it]
Outer training loop:  54%|#####3    | 26964/50000 [16:57:40<11:48:50,  1.85s/it]
Outer training loop:  54%|#####3    | 26965/50000 [16:57:41<11:38:51,  1.82s/it]
Outer training loop:  54%|#####3    | 26966/50000 [16:57:44<12:04:38,  1.89s/it]
Outer training loop:  54%|#####3    | 26967/50000 [16:57:45<12:02:39,  1.88s/it]
Outer training loop:  54%|#####3    | 26968/50000 [16:57:47<11:50:35,  1.85s/it]
Outer training loop:  54%|#####3    | 26969/50000 [16:57:49<12:14:07,  1.91s/it]
Outer training loop:  54%|#####3    | 26970/50000 [16:57:51<12:10:07,  1.90s/it]
Outer training loop:  54%|#####3    | 26971/50000 [16:57:53<11:57:00,  1.87s/it]
Outer training loop:  54%|#####3    | 26972/50000 [16:57:55<12:22:31,  1.93s/it]
Outer training loop:  54%|#####3    | 26973/50000 [16:57:57<12:03:51,  1.89s/it]
Outer training loop:  54%|#####3    | 26974/50000 [16:57:58<11:40:51,  1.83s/it]
Outer training loop:  54%|#####3    | 26975/50000 [16:58:00<12:00:31,  1.88s/it]
Outer training loop:  54%|#####3    | 26976/50000 [16:58:02<11:41:22,  1.83s/it]
Outer training loop:  54%|#####3    | 26977/50000 [16:58:04<11:44:23,  1.84s/it]
Outer training loop:  54%|#####3    | 26978/50000 [16:58:06<12:07:11,  1.90s/it]
Outer training loop:  54%|#####3    | 26979/50000 [16:58:08<11:57:44,  1.87s/it]
Outer training loop:  54%|#####3    | 26980/50000 [16:58:10<11:49:49,  1.85s/it]
Outer training loop:  54%|#####3    | 26981/50000 [16:58:11<11:39:42,  1.82s/it]
Outer training loop:  54%|#####3    | 26982/50000 [16:58:13<11:59:43,  1.88s/it]
Outer training loop:  54%|#####3    | 26983/50000 [16:58:15<11:48:37,  1.85s/it]
Outer training loop:  54%|#####3    | 26984/50000 [16:58:17<11:36:57,  1.82s/it]
Outer training loop:  54%|#####3    | 26985/50000 [16:58:19<11:35:47,  1.81s/it]
Outer training loop:  54%|#####3    | 26986/50000 [16:58:21<11:55:38,  1.87s/it]
Outer training loop:  54%|#####3    | 26987/50000 [16:58:23<11:50:45,  1.85s/it]
Outer training loop:  54%|#####3    | 26988/50000 [16:58:24<11:44:47,  1.84s/it]
Outer training loop:  54%|#####3    | 26989/50000 [16:58:26<12:07:25,  1.90s/it]
Outer training loop:  54%|#####3    | 26990/50000 [16:58:28<11:56:06,  1.87s/it]
Outer training loop:  54%|#####3    | 26991/50000 [16:58:30<11:52:32,  1.86s/it]
Outer training loop:  54%|#####3    | 26992/50000 [16:58:32<12:14:16,  1.91s/it]
Outer training loop:  54%|#####3    | 26993/50000 [16:58:34<12:21:31,  1.93s/it]
Outer training loop:  54%|#####3    | 26994/50000 [16:58:36<12:11:31,  1.91s/it]
Outer training loop:  54%|#####3    | 26995/50000 [16:58:38<12:02:54,  1.89s/it]
Outer training loop:  54%|#####3    | 26996/50000 [16:58:40<12:30:39,  1.96s/it]
Outer training loop:  54%|#####3    | 26997/50000 [16:58:42<12:09:43,  1.90s/it]
Outer training loop:  54%|#####3    | 26998/50000 [16:58:44<12:11:15,  1.91s/it]
Outer training loop:  54%|#####3    | 26999/50000 [16:58:46<12:17:53,  1.92s/it]
Outer training loop:  54%|#####4    | 27000/50000 [16:58:47<12:00:16,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.97s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.78s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:32,  1.88s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.86s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:25,  1.85s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.87s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.89s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:26,  1.98s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:20,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:10,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:14,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:03,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:53,  1.84s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:50,  1.82s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:56,  1.94s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.90s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:48,  1.97s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:42,  1.90s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:31,  1.84s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:32,  1.90s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:27,  1.85s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:23,  1.82s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:21,  1.81s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:23,  1.95s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.90s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.90s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:11,  1.84s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:10,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:07,  1.82s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:08,  1.90s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:06,  1.90s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.87s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.85s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:57,  1.97s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:54,  1.93s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:51,  1.93s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.92s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.90s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:41,  1.99s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:38,  1.92s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:36,  1.90s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.81s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:28,  1.79s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.81s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.98s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.91s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.88s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.87s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.86s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.86s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:08,  2.05s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  2.00s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.97s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.86s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.76s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.78s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.78s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:22,  1.80s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:22,  1.82s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:37,  2.05s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:31,  1.99s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:19,  1.89s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:24,  1.97s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:20,  1.95s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:08,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.92s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.87s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.93s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:57,  1.96s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:45,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:41,  1.81s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:40,  1.83s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:44,  1.93s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:31,  1.84s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:29,  1.83s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:33,  1.94s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:31,  1.94s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:19,  1.80s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:17,  1.81s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:21,  1.95s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:19,  1.93s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:15,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.91s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:09,  1.87s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:10,  1.96s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:06,  1.90s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:04,  1.90s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:54,  1.95s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:53,  1.97s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:50,  1.93s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:48,  1.95s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  2.00s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.93s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:34,  1.90s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.86s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.98s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.90s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.88s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.87s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.87s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.85s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:10,  1.83s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.83s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.86s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.84s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:04,  2.00s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.94s/it]

Outer training loop:  54%|#####4    | 27001/50000 [17:04:09<624:21:57, 97.73s/it]
Outer training loop:  54%|#####4    | 27002/50000 [17:04:11<440:35:02, 68.97s/it]
Outer training loop:  54%|#####4    | 27003/50000 [17:04:12<311:50:14, 48.82s/it]
Outer training loop:  54%|#####4    | 27004/50000 [17:04:14<221:45:40, 34.72s/it]
Outer training loop:  54%|#####4    | 27005/50000 [17:04:16<158:49:58, 24.87s/it]
Outer training loop:  54%|#####4    | 27006/50000 [17:04:18<114:36:43, 17.94s/it]
Outer training loop:  54%|#####4    | 27007/50000 [17:04:20<83:53:24, 13.13s/it] 
Outer training loop:  54%|#####4    | 27008/50000 [17:04:22<62:17:30,  9.75s/it]
Outer training loop:  54%|#####4    | 27009/50000 [17:04:24<48:28:43,  7.59s/it]
Outer training loop:  54%|#####4    | 27010/50000 [17:04:26<37:28:48,  5.87s/it]
Outer training loop:  54%|#####4    | 27011/50000 [17:04:28<29:49:28,  4.67s/it]
Outer training loop:  54%|#####4    | 27012/50000 [17:04:30<24:41:32,  3.87s/it]
Outer training loop:  54%|#####4    | 27013/50000 [17:04:32<20:50:52,  3.26s/it]
Outer training loop:  54%|#####4    | 27014/50000 [17:04:34<18:00:33,  2.82s/it]
Outer training loop:  54%|#####4    | 27015/50000 [17:04:36<16:46:23,  2.63s/it]
Outer training loop:  54%|#####4    | 27016/50000 [17:04:38<15:19:06,  2.40s/it]
Outer training loop:  54%|#####4    | 27017/50000 [17:04:39<14:13:11,  2.23s/it]
Outer training loop:  54%|#####4    | 27018/50000 [17:04:41<13:17:55,  2.08s/it]
Outer training loop:  54%|#####4    | 27019/50000 [17:04:43<13:06:48,  2.05s/it]
Outer training loop:  54%|#####4    | 27020/50000 [17:04:45<12:47:18,  2.00s/it]
Outer training loop:  54%|#####4    | 27021/50000 [17:04:47<12:19:47,  1.93s/it]
Outer training loop:  54%|#####4    | 27022/50000 [17:04:49<12:26:04,  1.95s/it]
Outer training loop:  54%|#####4    | 27023/50000 [17:04:51<12:10:50,  1.91s/it]
Outer training loop:  54%|#####4    | 27024/50000 [17:04:53<12:25:50,  1.95s/it]
Outer training loop:  54%|#####4    | 27025/50000 [17:04:55<12:28:51,  1.96s/it]
Outer training loop:  54%|#####4    | 27026/50000 [17:04:56<12:08:09,  1.90s/it]
Outer training loop:  54%|#####4    | 27027/50000 [17:04:58<12:07:45,  1.90s/it]
Outer training loop:  54%|#####4    | 27028/50000 [17:05:00<11:54:34,  1.87s/it]
Outer training loop:  54%|#####4    | 27029/50000 [17:05:02<12:01:59,  1.89s/it]
Outer training loop:  54%|#####4    | 27030/50000 [17:05:04<12:05:33,  1.90s/it]
Outer training loop:  54%|#####4    | 27031/50000 [17:05:06<11:46:35,  1.85s/it]
Outer training loop:  54%|#####4    | 27032/50000 [17:05:08<12:10:59,  1.91s/it]
Outer training loop:  54%|#####4    | 27033/50000 [17:05:09<11:54:13,  1.87s/it]
Outer training loop:  54%|#####4    | 27034/50000 [17:05:11<11:32:28,  1.81s/it]
Outer training loop:  54%|#####4    | 27035/50000 [17:05:13<11:40:43,  1.83s/it]
Outer training loop:  54%|#####4    | 27036/50000 [17:05:15<11:54:27,  1.87s/it]
Outer training loop:  54%|#####4    | 27037/50000 [17:05:17<12:01:14,  1.88s/it]
Outer training loop:  54%|#####4    | 27038/50000 [17:05:19<11:50:40,  1.86s/it]
Outer training loop:  54%|#####4    | 27039/50000 [17:05:21<12:18:05,  1.93s/it]
Outer training loop:  54%|#####4    | 27040/50000 [17:05:23<12:10:14,  1.91s/it]
Outer training loop:  54%|#####4    | 27041/50000 [17:05:24<11:55:28,  1.87s/it]
Outer training loop:  54%|#####4    | 27042/50000 [17:05:26<11:59:04,  1.88s/it]
Outer training loop:  54%|#####4    | 27043/50000 [17:05:28<12:14:36,  1.92s/it]
Outer training loop:  54%|#####4    | 27044/50000 [17:05:30<12:05:03,  1.90s/it]
Outer training loop:  54%|#####4    | 27045/50000 [17:05:32<11:52:50,  1.86s/it]
Outer training loop:  54%|#####4    | 27046/50000 [17:05:34<12:21:22,  1.94s/it]
Outer training loop:  54%|#####4    | 27047/50000 [17:05:36<12:14:31,  1.92s/it]
Outer training loop:  54%|#####4    | 27048/50000 [17:05:38<12:17:48,  1.93s/it]
Outer training loop:  54%|#####4    | 27049/50000 [17:05:40<12:41:32,  1.99s/it]
Outer training loop:  54%|#####4    | 27050/50000 [17:05:42<12:21:53,  1.94s/it]
Outer training loop:  54%|#####4    | 27051/50000 [17:05:44<12:15:23,  1.92s/it]
Outer training loop:  54%|#####4    | 27052/50000 [17:05:46<12:30:03,  1.96s/it]
Outer training loop:  54%|#####4    | 27053/50000 [17:05:48<12:30:12,  1.96s/it]
Outer training loop:  54%|#####4    | 27054/50000 [17:05:50<12:19:22,  1.93s/it]
Outer training loop:  54%|#####4    | 27055/50000 [17:05:52<12:29:34,  1.96s/it]
Outer training loop:  54%|#####4    | 27056/50000 [17:05:54<12:20:09,  1.94s/it]
Outer training loop:  54%|#####4    | 27057/50000 [17:05:55<12:09:44,  1.91s/it]
Outer training loop:  54%|#####4    | 27058/50000 [17:05:57<12:07:24,  1.90s/it]
Outer training loop:  54%|#####4    | 27059/50000 [17:05:59<12:24:05,  1.95s/it]
Outer training loop:  54%|#####4    | 27060/50000 [17:06:01<12:11:49,  1.91s/it]
Outer training loop:  54%|#####4    | 27061/50000 [17:06:03<11:57:15,  1.88s/it]
Outer training loop:  54%|#####4    | 27062/50000 [17:06:05<12:19:53,  1.94s/it]
Outer training loop:  54%|#####4    | 27063/50000 [17:06:07<11:58:04,  1.88s/it]
Outer training loop:  54%|#####4    | 27064/50000 [17:06:09<11:48:06,  1.85s/it]
Outer training loop:  54%|#####4    | 27065/50000 [17:06:10<11:51:52,  1.86s/it]
Outer training loop:  54%|#####4    | 27066/50000 [17:06:12<12:12:47,  1.92s/it]
Outer training loop:  54%|#####4    | 27067/50000 [17:06:14<11:52:26,  1.86s/it]
Outer training loop:  54%|#####4    | 27068/50000 [17:06:16<11:43:04,  1.84s/it]
Outer training loop:  54%|#####4    | 27069/50000 [17:06:18<12:14:02,  1.92s/it]
Outer training loop:  54%|#####4    | 27070/50000 [17:06:20<12:12:01,  1.92s/it]
Outer training loop:  54%|#####4    | 27071/50000 [17:06:22<11:58:41,  1.88s/it]
Outer training loop:  54%|#####4    | 27072/50000 [17:06:24<12:10:12,  1.91s/it]
Outer training loop:  54%|#####4    | 27073/50000 [17:06:26<12:03:13,  1.89s/it]
Outer training loop:  54%|#####4    | 27074/50000 [17:06:28<12:33:49,  1.97s/it]
Outer training loop:  54%|#####4    | 27075/50000 [17:06:30<12:25:26,  1.95s/it]
Outer training loop:  54%|#####4    | 27076/50000 [17:06:32<12:32:08,  1.97s/it]
Outer training loop:  54%|#####4    | 27077/50000 [17:06:34<12:16:46,  1.93s/it]
Outer training loop:  54%|#####4    | 27078/50000 [17:06:35<12:15:48,  1.93s/it]
Outer training loop:  54%|#####4    | 27079/50000 [17:06:37<12:26:20,  1.95s/it]
Outer training loop:  54%|#####4    | 27080/50000 [17:06:39<11:59:16,  1.88s/it]
Outer training loop:  54%|#####4    | 27081/50000 [17:06:41<12:03:10,  1.89s/it]
Outer training loop:  54%|#####4    | 27082/50000 [17:06:43<11:58:00,  1.88s/it]
Outer training loop:  54%|#####4    | 27083/50000 [17:06:45<12:06:26,  1.90s/it]
Outer training loop:  54%|#####4    | 27084/50000 [17:06:47<12:08:43,  1.91s/it]
Outer training loop:  54%|#####4    | 27085/50000 [17:06:49<12:13:26,  1.92s/it]
Outer training loop:  54%|#####4    | 27086/50000 [17:06:51<12:31:47,  1.97s/it]
Outer training loop:  54%|#####4    | 27087/50000 [17:06:53<12:19:53,  1.94s/it]
Outer training loop:  54%|#####4    | 27088/50000 [17:06:54<12:01:14,  1.89s/it]
Outer training loop:  54%|#####4    | 27089/50000 [17:06:56<11:49:11,  1.86s/it]
Outer training loop:  54%|#####4    | 27090/50000 [17:06:58<12:15:49,  1.93s/it]
Outer training loop:  54%|#####4    | 27091/50000 [17:07:00<12:08:48,  1.91s/it]
Outer training loop:  54%|#####4    | 27092/50000 [17:07:02<12:04:15,  1.90s/it]
Outer training loop:  54%|#####4    | 27093/50000 [17:07:04<12:26:55,  1.96s/it]
Outer training loop:  54%|#####4    | 27094/50000 [17:07:06<12:11:31,  1.92s/it]
Outer training loop:  54%|#####4    | 27095/50000 [17:07:08<11:57:12,  1.88s/it]
Outer training loop:  54%|#####4    | 27096/50000 [17:07:10<12:05:07,  1.90s/it]
Outer training loop:  54%|#####4    | 27097/50000 [17:07:12<11:47:03,  1.85s/it]
Outer training loop:  54%|#####4    | 27098/50000 [17:07:13<11:44:19,  1.85s/it]
Outer training loop:  54%|#####4    | 27099/50000 [17:07:15<11:51:23,  1.86s/it]
Outer training loop:  54%|#####4    | 27100/50000 [17:07:17<12:03:35,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  54%|#####4    | 27101/50000 [17:07:25<22:49:43,  3.59s/it]
Outer training loop:  54%|#####4    | 27102/50000 [17:07:26<19:17:28,  3.03s/it]
Outer training loop:  54%|#####4    | 27103/50000 [17:07:28<17:02:53,  2.68s/it]
Outer training loop:  54%|#####4    | 27104/50000 [17:07:30<15:47:14,  2.48s/it]
Outer training loop:  54%|#####4    | 27105/50000 [17:07:32<14:24:45,  2.27s/it]
Outer training loop:  54%|#####4    | 27106/50000 [17:07:34<13:30:09,  2.12s/it]
Outer training loop:  54%|#####4    | 27107/50000 [17:07:36<12:49:05,  2.02s/it]
Outer training loop:  54%|#####4    | 27108/50000 [17:07:38<12:45:16,  2.01s/it]
Outer training loop:  54%|#####4    | 27109/50000 [17:07:40<12:29:34,  1.96s/it]
Outer training loop:  54%|#####4    | 27110/50000 [17:07:41<12:15:57,  1.93s/it]
Outer training loop:  54%|#####4    | 27111/50000 [17:07:43<12:18:14,  1.94s/it]
Outer training loop:  54%|#####4    | 27112/50000 [17:07:45<12:04:55,  1.90s/it]
Outer training loop:  54%|#####4    | 27113/50000 [17:07:47<11:50:16,  1.86s/it]
Outer training loop:  54%|#####4    | 27114/50000 [17:07:49<11:24:55,  1.80s/it]
Outer training loop:  54%|#####4    | 27115/50000 [17:07:51<12:04:38,  1.90s/it]
Outer training loop:  54%|#####4    | 27116/50000 [17:07:52<11:46:54,  1.85s/it]
Outer training loop:  54%|#####4    | 27117/50000 [17:07:54<11:43:44,  1.85s/it]
Outer training loop:  54%|#####4    | 27118/50000 [17:07:56<12:07:53,  1.91s/it]
Outer training loop:  54%|#####4    | 27119/50000 [17:07:58<11:58:34,  1.88s/it]
Outer training loop:  54%|#####4    | 27120/50000 [17:08:00<11:35:47,  1.82s/it]
Outer training loop:  54%|#####4    | 27121/50000 [17:08:02<12:20:08,  1.94s/it]
Outer training loop:  54%|#####4    | 27122/50000 [17:08:04<12:13:22,  1.92s/it]
Outer training loop:  54%|#####4    | 27123/50000 [17:08:06<12:14:09,  1.93s/it]
Outer training loop:  54%|#####4    | 27124/50000 [17:08:08<11:51:56,  1.87s/it]
Outer training loop:  54%|#####4    | 27125/50000 [17:08:10<12:03:27,  1.90s/it]
Outer training loop:  54%|#####4    | 27126/50000 [17:08:11<12:03:11,  1.90s/it]
Outer training loop:  54%|#####4    | 27127/50000 [17:08:13<12:03:36,  1.90s/it]
Outer training loop:  54%|#####4    | 27128/50000 [17:08:15<12:14:55,  1.93s/it]
Outer training loop:  54%|#####4    | 27129/50000 [17:08:17<12:22:27,  1.95s/it]
Outer training loop:  54%|#####4    | 27130/50000 [17:08:19<12:23:29,  1.95s/it]
Outer training loop:  54%|#####4    | 27131/50000 [17:08:21<12:28:15,  1.96s/it]
Outer training loop:  54%|#####4    | 27132/50000 [17:08:23<12:21:44,  1.95s/it]
Outer training loop:  54%|#####4    | 27133/50000 [17:08:25<12:11:58,  1.92s/it]
Outer training loop:  54%|#####4    | 27134/50000 [17:08:27<12:18:12,  1.94s/it]
Outer training loop:  54%|#####4    | 27135/50000 [17:08:29<12:14:08,  1.93s/it]
Outer training loop:  54%|#####4    | 27136/50000 [17:08:31<11:52:28,  1.87s/it]
Outer training loop:  54%|#####4    | 27137/50000 [17:08:33<12:05:25,  1.90s/it]
Outer training loop:  54%|#####4    | 27138/50000 [17:08:35<11:58:14,  1.89s/it]
Outer training loop:  54%|#####4    | 27139/50000 [17:08:36<11:46:22,  1.85s/it]
Outer training loop:  54%|#####4    | 27140/50000 [17:08:38<11:44:46,  1.85s/it]
Outer training loop:  54%|#####4    | 27141/50000 [17:08:40<11:58:41,  1.89s/it]
Outer training loop:  54%|#####4    | 27142/50000 [17:08:42<11:48:38,  1.86s/it]
Outer training loop:  54%|#####4    | 27143/50000 [17:08:44<11:54:16,  1.87s/it]
Outer training loop:  54%|#####4    | 27144/50000 [17:08:46<11:54:21,  1.88s/it]
Outer training loop:  54%|#####4    | 27145/50000 [17:08:47<11:41:01,  1.84s/it]
Outer training loop:  54%|#####4    | 27146/50000 [17:08:49<11:41:01,  1.84s/it]
Outer training loop:  54%|#####4    | 27147/50000 [17:08:51<11:51:10,  1.87s/it]
Outer training loop:  54%|#####4    | 27148/50000 [17:08:53<11:54:21,  1.88s/it]
Outer training loop:  54%|#####4    | 27149/50000 [17:08:55<11:55:47,  1.88s/it]
Outer training loop:  54%|#####4    | 27150/50000 [17:08:57<12:23:17,  1.95s/it]
Outer training loop:  54%|#####4    | 27151/50000 [17:08:59<12:18:33,  1.94s/it]
Outer training loop:  54%|#####4    | 27152/50000 [17:09:01<12:07:41,  1.91s/it]
Outer training loop:  54%|#####4    | 27153/50000 [17:09:03<12:01:03,  1.89s/it]
Outer training loop:  54%|#####4    | 27154/50000 [17:09:05<12:16:17,  1.93s/it]
Outer training loop:  54%|#####4    | 27155/50000 [17:09:07<12:08:31,  1.91s/it]
Outer training loop:  54%|#####4    | 27156/50000 [17:09:09<12:09:52,  1.92s/it]
Outer training loop:  54%|#####4    | 27157/50000 [17:09:11<12:34:58,  1.98s/it]
Outer training loop:  54%|#####4    | 27158/50000 [17:09:13<12:18:05,  1.94s/it]
Outer training loop:  54%|#####4    | 27159/50000 [17:09:14<12:10:58,  1.92s/it]
Outer training loop:  54%|#####4    | 27160/50000 [17:09:16<12:21:42,  1.95s/it]
Outer training loop:  54%|#####4    | 27161/50000 [17:09:18<12:07:11,  1.91s/it]
Outer training loop:  54%|#####4    | 27162/50000 [17:09:20<12:06:26,  1.91s/it]
Outer training loop:  54%|#####4    | 27163/50000 [17:09:22<12:12:33,  1.92s/it]
Outer training loop:  54%|#####4    | 27164/50000 [17:09:24<12:04:07,  1.90s/it]
Outer training loop:  54%|#####4    | 27165/50000 [17:09:26<11:58:26,  1.89s/it]
Outer training loop:  54%|#####4    | 27166/50000 [17:09:28<11:57:21,  1.88s/it]
Outer training loop:  54%|#####4    | 27167/50000 [17:09:30<12:11:07,  1.92s/it]
Outer training loop:  54%|#####4    | 27168/50000 [17:09:31<11:48:26,  1.86s/it]
Outer training loop:  54%|#####4    | 27169/50000 [17:09:33<11:45:42,  1.85s/it]
Outer training loop:  54%|#####4    | 27170/50000 [17:09:35<12:12:41,  1.93s/it]
Outer training loop:  54%|#####4    | 27171/50000 [17:09:38<12:44:47,  2.01s/it]
Outer training loop:  54%|#####4    | 27172/50000 [17:09:39<12:21:49,  1.95s/it]
Outer training loop:  54%|#####4    | 27173/50000 [17:09:41<12:06:37,  1.91s/it]
Outer training loop:  54%|#####4    | 27174/50000 [17:09:43<12:26:50,  1.96s/it]
Outer training loop:  54%|#####4    | 27175/50000 [17:09:45<12:05:15,  1.91s/it]
Outer training loop:  54%|#####4    | 27176/50000 [17:09:47<12:01:21,  1.90s/it]
Outer training loop:  54%|#####4    | 27177/50000 [17:09:49<12:17:48,  1.94s/it]
Outer training loop:  54%|#####4    | 27178/50000 [17:09:51<12:11:30,  1.92s/it]
Outer training loop:  54%|#####4    | 27179/50000 [17:09:53<11:55:18,  1.88s/it]
Outer training loop:  54%|#####4    | 27180/50000 [17:09:55<12:05:05,  1.91s/it]
Outer training loop:  54%|#####4    | 27181/50000 [17:09:57<12:09:41,  1.92s/it]
Outer training loop:  54%|#####4    | 27182/50000 [17:09:58<12:04:17,  1.90s/it]
Outer training loop:  54%|#####4    | 27183/50000 [17:10:00<12:05:05,  1.91s/it]
Outer training loop:  54%|#####4    | 27184/50000 [17:10:02<12:14:06,  1.93s/it]
Outer training loop:  54%|#####4    | 27185/50000 [17:10:04<12:22:12,  1.95s/it]
Outer training loop:  54%|#####4    | 27186/50000 [17:10:06<12:02:47,  1.90s/it]
Outer training loop:  54%|#####4    | 27187/50000 [17:10:08<12:21:06,  1.95s/it]
Outer training loop:  54%|#####4    | 27188/50000 [17:10:10<12:23:08,  1.95s/it]
Outer training loop:  54%|#####4    | 27189/50000 [17:10:12<11:55:45,  1.88s/it]
Outer training loop:  54%|#####4    | 27190/50000 [17:10:14<12:21:55,  1.95s/it]
Outer training loop:  54%|#####4    | 27191/50000 [17:10:16<12:09:11,  1.92s/it]
Outer training loop:  54%|#####4    | 27192/50000 [17:10:18<11:51:59,  1.87s/it]
Outer training loop:  54%|#####4    | 27193/50000 [17:10:19<11:42:48,  1.85s/it]
Outer training loop:  54%|#####4    | 27194/50000 [17:10:21<11:56:05,  1.88s/it]
Outer training loop:  54%|#####4    | 27195/50000 [17:10:23<11:38:02,  1.84s/it]
Outer training loop:  54%|#####4    | 27196/50000 [17:10:25<11:29:50,  1.82s/it]
Outer training loop:  54%|#####4    | 27197/50000 [17:10:27<11:54:53,  1.88s/it]
Outer training loop:  54%|#####4    | 27198/50000 [17:10:29<11:47:29,  1.86s/it]
Outer training loop:  54%|#####4    | 27199/50000 [17:10:31<11:48:28,  1.86s/it]
Outer training loop:  54%|#####4    | 27200/50000 [17:10:32<11:45:40,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  54%|#####4    | 27201/50000 [17:10:40<22:34:02,  3.56s/it]
Outer training loop:  54%|#####4    | 27202/50000 [17:10:42<18:55:33,  2.99s/it]
Outer training loop:  54%|#####4    | 27203/50000 [17:10:43<16:45:39,  2.65s/it]
Outer training loop:  54%|#####4    | 27204/50000 [17:10:45<15:01:37,  2.37s/it]
Outer training loop:  54%|#####4    | 27205/50000 [17:10:47<14:10:31,  2.24s/it]
Outer training loop:  54%|#####4    | 27206/50000 [17:10:49<13:23:32,  2.12s/it]
Outer training loop:  54%|#####4    | 27207/50000 [17:10:51<12:47:31,  2.02s/it]
Outer training loop:  54%|#####4    | 27208/50000 [17:10:53<12:31:41,  1.98s/it]
Outer training loop:  54%|#####4    | 27209/50000 [17:10:54<12:18:48,  1.94s/it]
Outer training loop:  54%|#####4    | 27210/50000 [17:10:56<12:02:15,  1.90s/it]
Outer training loop:  54%|#####4    | 27211/50000 [17:10:58<12:14:12,  1.93s/it]
Outer training loop:  54%|#####4    | 27212/50000 [17:11:00<11:58:42,  1.89s/it]
Outer training loop:  54%|#####4    | 27213/50000 [17:11:02<11:55:04,  1.88s/it]
Outer training loop:  54%|#####4    | 27214/50000 [17:11:04<12:05:03,  1.91s/it]
Outer training loop:  54%|#####4    | 27215/50000 [17:11:06<11:57:49,  1.89s/it]
Outer training loop:  54%|#####4    | 27216/50000 [17:11:07<11:37:53,  1.84s/it]
Outer training loop:  54%|#####4    | 27217/50000 [17:11:09<11:46:11,  1.86s/it]
Outer training loop:  54%|#####4    | 27218/50000 [17:11:12<12:34:40,  1.99s/it]
Outer training loop:  54%|#####4    | 27219/50000 [17:11:13<12:13:20,  1.93s/it]
Outer training loop:  54%|#####4    | 27220/50000 [17:11:15<11:54:42,  1.88s/it]
Outer training loop:  54%|#####4    | 27221/50000 [17:11:17<12:04:15,  1.91s/it]
Outer training loop:  54%|#####4    | 27222/50000 [17:11:19<11:50:50,  1.87s/it]
Outer training loop:  54%|#####4    | 27223/50000 [17:11:21<11:44:13,  1.86s/it]
Outer training loop:  54%|#####4    | 27224/50000 [17:11:23<11:39:55,  1.84s/it]
Outer training loop:  54%|#####4    | 27225/50000 [17:11:25<12:15:31,  1.94s/it]
Outer training loop:  54%|#####4    | 27226/50000 [17:11:27<11:59:03,  1.89s/it]
Outer training loop:  54%|#####4    | 27227/50000 [17:11:28<11:57:13,  1.89s/it]
Outer training loop:  54%|#####4    | 27228/50000 [17:11:30<12:08:55,  1.92s/it]
Outer training loop:  54%|#####4    | 27229/50000 [17:11:32<11:53:44,  1.88s/it]
Outer training loop:  54%|#####4    | 27230/50000 [17:11:34<11:52:41,  1.88s/it]
Outer training loop:  54%|#####4    | 27231/50000 [17:11:36<12:25:34,  1.96s/it]
Outer training loop:  54%|#####4    | 27232/50000 [17:11:38<12:08:53,  1.92s/it]
Outer training loop:  54%|#####4    | 27233/50000 [17:11:40<11:57:09,  1.89s/it]
Outer training loop:  54%|#####4    | 27234/50000 [17:11:42<11:52:38,  1.88s/it]
Outer training loop:  54%|#####4    | 27235/50000 [17:11:44<12:20:25,  1.95s/it]
Outer training loop:  54%|#####4    | 27236/50000 [17:11:46<11:56:44,  1.89s/it]
Outer training loop:  54%|#####4    | 27237/50000 [17:11:47<11:50:21,  1.87s/it]
Outer training loop:  54%|#####4    | 27238/50000 [17:11:50<12:13:51,  1.93s/it]
Outer training loop:  54%|#####4    | 27239/50000 [17:11:51<12:03:22,  1.91s/it]
Outer training loop:  54%|#####4    | 27240/50000 [17:11:53<11:41:22,  1.85s/it]
Outer training loop:  54%|#####4    | 27241/50000 [17:11:55<12:12:55,  1.93s/it]
Outer training loop:  54%|#####4    | 27242/50000 [17:11:57<11:54:52,  1.88s/it]
Outer training loop:  54%|#####4    | 27243/50000 [17:11:59<11:53:14,  1.88s/it]
Outer training loop:  54%|#####4    | 27244/50000 [17:12:01<11:45:27,  1.86s/it]
Outer training loop:  54%|#####4    | 27245/50000 [17:12:03<11:52:59,  1.88s/it]
Outer training loop:  54%|#####4    | 27246/50000 [17:12:04<11:39:28,  1.84s/it]
Outer training loop:  54%|#####4    | 27247/50000 [17:12:06<11:20:18,  1.79s/it]
Outer training loop:  54%|#####4    | 27248/50000 [17:12:08<11:53:08,  1.88s/it]
Outer training loop:  54%|#####4    | 27249/50000 [17:12:10<11:42:50,  1.85s/it]
Outer training loop:  55%|#####4    | 27250/50000 [17:12:12<11:47:32,  1.87s/it]
Outer training loop:  55%|#####4    | 27251/50000 [17:12:14<11:47:52,  1.87s/it]
Outer training loop:  55%|#####4    | 27252/50000 [17:12:15<11:46:21,  1.86s/it]
Outer training loop:  55%|#####4    | 27253/50000 [17:12:17<11:49:13,  1.87s/it]
Outer training loop:  55%|#####4    | 27254/50000 [17:12:19<11:43:43,  1.86s/it]
Outer training loop:  55%|#####4    | 27255/50000 [17:12:21<11:53:47,  1.88s/it]
Outer training loop:  55%|#####4    | 27256/50000 [17:12:23<11:45:03,  1.86s/it]
Outer training loop:  55%|#####4    | 27257/50000 [17:12:25<11:37:12,  1.84s/it]
Outer training loop:  55%|#####4    | 27258/50000 [17:12:27<12:08:33,  1.92s/it]
Outer training loop:  55%|#####4    | 27259/50000 [17:12:29<12:00:00,  1.90s/it]
Outer training loop:  55%|#####4    | 27260/50000 [17:12:31<11:49:15,  1.87s/it]
Outer training loop:  55%|#####4    | 27261/50000 [17:12:32<11:32:56,  1.83s/it]
Outer training loop:  55%|#####4    | 27262/50000 [17:12:34<11:52:03,  1.88s/it]
Outer training loop:  55%|#####4    | 27263/50000 [17:12:36<11:56:33,  1.89s/it]
Outer training loop:  55%|#####4    | 27264/50000 [17:12:38<11:35:25,  1.84s/it]
Outer training loop:  55%|#####4    | 27265/50000 [17:12:40<11:30:33,  1.82s/it]
Outer training loop:  55%|#####4    | 27266/50000 [17:12:42<12:07:59,  1.92s/it]
Outer training loop:  55%|#####4    | 27267/50000 [17:12:44<11:55:37,  1.89s/it]
Outer training loop:  55%|#####4    | 27268/50000 [17:12:46<12:17:56,  1.95s/it]
Outer training loop:  55%|#####4    | 27269/50000 [17:12:48<12:22:31,  1.96s/it]
Outer training loop:  55%|#####4    | 27270/50000 [17:12:50<12:06:09,  1.92s/it]
Outer training loop:  55%|#####4    | 27271/50000 [17:12:52<12:16:27,  1.94s/it]
Outer training loop:  55%|#####4    | 27272/50000 [17:12:54<12:32:06,  1.99s/it]
Outer training loop:  55%|#####4    | 27273/50000 [17:12:55<12:16:28,  1.94s/it]
Outer training loop:  55%|#####4    | 27274/50000 [17:12:57<12:14:11,  1.94s/it]
Outer training loop:  55%|#####4    | 27275/50000 [17:12:59<12:02:08,  1.91s/it]
Outer training loop:  55%|#####4    | 27276/50000 [17:13:01<12:19:58,  1.95s/it]
Outer training loop:  55%|#####4    | 27277/50000 [17:13:03<12:08:18,  1.92s/it]
Outer training loop:  55%|#####4    | 27278/50000 [17:13:05<12:02:53,  1.91s/it]
Outer training loop:  55%|#####4    | 27279/50000 [17:13:07<12:20:59,  1.96s/it]
Outer training loop:  55%|#####4    | 27280/50000 [17:13:09<11:56:18,  1.89s/it]
Outer training loop:  55%|#####4    | 27281/50000 [17:13:11<11:57:05,  1.89s/it]
Outer training loop:  55%|#####4    | 27282/50000 [17:13:13<12:20:54,  1.96s/it]
Outer training loop:  55%|#####4    | 27283/50000 [17:13:15<12:16:36,  1.95s/it]
Outer training loop:  55%|#####4    | 27284/50000 [17:13:17<12:04:44,  1.91s/it]
Outer training loop:  55%|#####4    | 27285/50000 [17:13:19<12:23:51,  1.96s/it]
Outer training loop:  55%|#####4    | 27286/50000 [17:13:21<12:10:34,  1.93s/it]
Outer training loop:  55%|#####4    | 27287/50000 [17:13:22<11:53:52,  1.89s/it]
Outer training loop:  55%|#####4    | 27288/50000 [17:13:24<12:07:42,  1.92s/it]
Outer training loop:  55%|#####4    | 27289/50000 [17:13:26<12:23:43,  1.96s/it]
Outer training loop:  55%|#####4    | 27290/50000 [17:13:28<12:14:39,  1.94s/it]
Outer training loop:  55%|#####4    | 27291/50000 [17:13:30<12:04:37,  1.91s/it]
Outer training loop:  55%|#####4    | 27292/50000 [17:13:32<12:15:39,  1.94s/it]
Outer training loop:  55%|#####4    | 27293/50000 [17:13:34<12:01:23,  1.91s/it]
Outer training loop:  55%|#####4    | 27294/50000 [17:13:36<11:50:31,  1.88s/it]
Outer training loop:  55%|#####4    | 27295/50000 [17:13:38<11:44:50,  1.86s/it]
Outer training loop:  55%|#####4    | 27296/50000 [17:13:40<12:07:33,  1.92s/it]
Outer training loop:  55%|#####4    | 27297/50000 [17:13:41<11:56:33,  1.89s/it]
Outer training loop:  55%|#####4    | 27298/50000 [17:13:43<11:54:29,  1.89s/it]
Outer training loop:  55%|#####4    | 27299/50000 [17:13:45<12:05:26,  1.92s/it]
Outer training loop:  55%|#####4    | 27300/50000 [17:13:47<11:49:31,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.88s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  55%|#####4    | 27301/50000 [17:13:55<22:42:46,  3.60s/it]
Outer training loop:  55%|#####4    | 27302/50000 [17:13:57<19:21:22,  3.07s/it]
Outer training loop:  55%|#####4    | 27303/50000 [17:13:59<17:33:51,  2.79s/it]
Outer training loop:  55%|#####4    | 27304/50000 [17:14:01<15:45:21,  2.50s/it]
Outer training loop:  55%|#####4    | 27305/50000 [17:14:02<14:22:07,  2.28s/it]
Outer training loop:  55%|#####4    | 27306/50000 [17:14:04<13:48:52,  2.19s/it]
Outer training loop:  55%|#####4    | 27307/50000 [17:14:06<13:10:18,  2.09s/it]
Outer training loop:  55%|#####4    | 27308/50000 [17:14:08<12:51:59,  2.04s/it]
Outer training loop:  55%|#####4    | 27309/50000 [17:14:10<12:33:47,  1.99s/it]
Outer training loop:  55%|#####4    | 27310/50000 [17:14:12<12:26:05,  1.97s/it]
Outer training loop:  55%|#####4    | 27311/50000 [17:14:14<12:24:09,  1.97s/it]
Outer training loop:  55%|#####4    | 27312/50000 [17:14:16<12:10:45,  1.93s/it]
Outer training loop:  55%|#####4    | 27313/50000 [17:14:18<12:21:03,  1.96s/it]
Outer training loop:  55%|#####4    | 27314/50000 [17:14:19<11:55:14,  1.89s/it]
Outer training loop:  55%|#####4    | 27315/50000 [17:14:22<12:21:48,  1.96s/it]
Outer training loop:  55%|#####4    | 27316/50000 [17:14:24<12:37:02,  2.00s/it]
Outer training loop:  55%|#####4    | 27317/50000 [17:14:26<12:38:33,  2.01s/it]
Outer training loop:  55%|#####4    | 27318/50000 [17:14:27<12:16:16,  1.95s/it]
Outer training loop:  55%|#####4    | 27319/50000 [17:14:29<12:02:25,  1.91s/it]
Outer training loop:  55%|#####4    | 27320/50000 [17:14:31<12:14:39,  1.94s/it]
Outer training loop:  55%|#####4    | 27321/50000 [17:14:33<12:08:03,  1.93s/it]
Outer training loop:  55%|#####4    | 27322/50000 [17:14:35<12:01:49,  1.91s/it]
Outer training loop:  55%|#####4    | 27323/50000 [17:14:37<12:13:40,  1.94s/it]
Outer training loop:  55%|#####4    | 27324/50000 [17:14:39<11:49:22,  1.88s/it]
Outer training loop:  55%|#####4    | 27325/50000 [17:14:41<11:42:24,  1.86s/it]
Outer training loop:  55%|#####4    | 27326/50000 [17:14:43<12:05:36,  1.92s/it]
Outer training loop:  55%|#####4    | 27327/50000 [17:14:45<11:54:13,  1.89s/it]
Outer training loop:  55%|#####4    | 27328/50000 [17:14:46<11:53:35,  1.89s/it]
Outer training loop:  55%|#####4    | 27329/50000 [17:14:48<11:55:35,  1.89s/it]
Outer training loop:  55%|#####4    | 27330/50000 [17:14:50<12:01:12,  1.91s/it]
Outer training loop:  55%|#####4    | 27331/50000 [17:14:52<11:40:48,  1.85s/it]
Outer training loop:  55%|#####4    | 27332/50000 [17:14:54<11:29:09,  1.82s/it]
Outer training loop:  55%|#####4    | 27333/50000 [17:14:56<11:58:57,  1.90s/it]
Outer training loop:  55%|#####4    | 27334/50000 [17:14:58<11:49:07,  1.88s/it]
Outer training loop:  55%|#####4    | 27335/50000 [17:14:59<11:45:08,  1.87s/it]
Outer training loop:  55%|#####4    | 27336/50000 [17:15:01<11:54:55,  1.89s/it]
Outer training loop:  55%|#####4    | 27337/50000 [17:15:03<11:51:30,  1.88s/it]
Outer training loop:  55%|#####4    | 27338/50000 [17:15:05<11:55:21,  1.89s/it]
Outer training loop:  55%|#####4    | 27339/50000 [17:15:07<11:37:13,  1.85s/it]
Outer training loop:  55%|#####4    | 27340/50000 [17:15:09<12:07:19,  1.93s/it]
Outer training loop:  55%|#####4    | 27341/50000 [17:15:11<12:09:08,  1.93s/it]
Outer training loop:  55%|#####4    | 27342/50000 [17:15:13<12:09:08,  1.93s/it]
Outer training loop:  55%|#####4    | 27343/50000 [17:15:15<12:13:45,  1.94s/it]
Outer training loop:  55%|#####4    | 27344/50000 [17:15:17<12:02:42,  1.91s/it]
Outer training loop:  55%|#####4    | 27345/50000 [17:15:18<11:41:53,  1.86s/it]
Outer training loop:  55%|#####4    | 27346/50000 [17:15:20<11:56:03,  1.90s/it]
Outer training loop:  55%|#####4    | 27347/50000 [17:15:22<11:44:57,  1.87s/it]
Outer training loop:  55%|#####4    | 27348/50000 [17:15:24<11:31:34,  1.83s/it]
Outer training loop:  55%|#####4    | 27349/50000 [17:15:26<12:10:55,  1.94s/it]
Outer training loop:  55%|#####4    | 27350/50000 [17:15:28<12:20:17,  1.96s/it]
Outer training loop:  55%|#####4    | 27351/50000 [17:15:30<12:13:19,  1.94s/it]
Outer training loop:  55%|#####4    | 27352/50000 [17:15:32<12:26:59,  1.98s/it]
Outer training loop:  55%|#####4    | 27353/50000 [17:15:34<12:15:46,  1.95s/it]
Outer training loop:  55%|#####4    | 27354/50000 [17:15:36<12:23:17,  1.97s/it]
Outer training loop:  55%|#####4    | 27355/50000 [17:15:38<12:08:23,  1.93s/it]
Outer training loop:  55%|#####4    | 27356/50000 [17:15:40<12:28:52,  1.98s/it]
Outer training loop:  55%|#####4    | 27357/50000 [17:15:42<12:06:24,  1.92s/it]
Outer training loop:  55%|#####4    | 27358/50000 [17:15:44<11:48:16,  1.88s/it]
Outer training loop:  55%|#####4    | 27359/50000 [17:15:45<11:53:38,  1.89s/it]
Outer training loop:  55%|#####4    | 27360/50000 [17:15:47<11:42:48,  1.86s/it]
Outer training loop:  55%|#####4    | 27361/50000 [17:15:49<11:36:24,  1.85s/it]
Outer training loop:  55%|#####4    | 27362/50000 [17:15:51<11:32:37,  1.84s/it]
Outer training loop:  55%|#####4    | 27363/50000 [17:15:53<11:45:10,  1.87s/it]
Outer training loop:  55%|#####4    | 27364/50000 [17:15:55<11:29:55,  1.83s/it]
Outer training loop:  55%|#####4    | 27365/50000 [17:15:57<12:01:42,  1.91s/it]
Outer training loop:  55%|#####4    | 27366/50000 [17:15:59<12:22:42,  1.97s/it]
Outer training loop:  55%|#####4    | 27367/50000 [17:16:01<11:57:35,  1.90s/it]
Outer training loop:  55%|#####4    | 27368/50000 [17:16:02<11:40:13,  1.86s/it]
Outer training loop:  55%|#####4    | 27369/50000 [17:16:04<11:37:15,  1.85s/it]
Outer training loop:  55%|#####4    | 27370/50000 [17:16:06<12:01:35,  1.91s/it]
Outer training loop:  55%|#####4    | 27371/50000 [17:16:08<11:41:24,  1.86s/it]
Outer training loop:  55%|#####4    | 27372/50000 [17:16:10<11:44:04,  1.87s/it]
Outer training loop:  55%|#####4    | 27373/50000 [17:16:12<11:51:58,  1.89s/it]
Outer training loop:  55%|#####4    | 27374/50000 [17:16:13<11:33:10,  1.84s/it]
Outer training loop:  55%|#####4    | 27375/50000 [17:16:15<11:32:46,  1.84s/it]
Outer training loop:  55%|#####4    | 27376/50000 [17:16:17<11:22:45,  1.81s/it]
Outer training loop:  55%|#####4    | 27377/50000 [17:16:19<11:47:40,  1.88s/it]
Outer training loop:  55%|#####4    | 27378/50000 [17:16:21<11:45:00,  1.87s/it]
Outer training loop:  55%|#####4    | 27379/50000 [17:16:23<11:36:40,  1.85s/it]
Outer training loop:  55%|#####4    | 27380/50000 [17:16:25<12:00:29,  1.91s/it]
Outer training loop:  55%|#####4    | 27381/50000 [17:16:27<11:45:18,  1.87s/it]
Outer training loop:  55%|#####4    | 27382/50000 [17:16:28<11:36:39,  1.85s/it]
Outer training loop:  55%|#####4    | 27383/50000 [17:16:30<11:50:06,  1.88s/it]
Outer training loop:  55%|#####4    | 27384/50000 [17:16:32<11:38:05,  1.85s/it]
Outer training loop:  55%|#####4    | 27385/50000 [17:16:34<11:45:51,  1.87s/it]
Outer training loop:  55%|#####4    | 27386/50000 [17:16:36<12:01:16,  1.91s/it]
Outer training loop:  55%|#####4    | 27387/50000 [17:16:38<11:37:55,  1.85s/it]
Outer training loop:  55%|#####4    | 27388/50000 [17:16:40<11:41:35,  1.86s/it]
Outer training loop:  55%|#####4    | 27389/50000 [17:16:41<11:26:14,  1.82s/it]
Outer training loop:  55%|#####4    | 27390/50000 [17:16:43<11:48:00,  1.88s/it]
Outer training loop:  55%|#####4    | 27391/50000 [17:16:45<11:41:13,  1.86s/it]
Outer training loop:  55%|#####4    | 27392/50000 [17:16:47<11:28:48,  1.83s/it]
Outer training loop:  55%|#####4    | 27393/50000 [17:16:49<11:38:24,  1.85s/it]
Outer training loop:  55%|#####4    | 27394/50000 [17:16:51<12:03:58,  1.92s/it]
Outer training loop:  55%|#####4    | 27395/50000 [17:16:53<11:46:11,  1.87s/it]
Outer training loop:  55%|#####4    | 27396/50000 [17:16:54<11:34:54,  1.84s/it]
Outer training loop:  55%|#####4    | 27397/50000 [17:16:56<11:56:00,  1.90s/it]
Outer training loop:  55%|#####4    | 27398/50000 [17:16:58<11:50:51,  1.89s/it]
Outer training loop:  55%|#####4    | 27399/50000 [17:17:00<11:40:20,  1.86s/it]
Outer training loop:  55%|#####4    | 27400/50000 [17:17:02<11:44:44,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  55%|#####4    | 27401/50000 [17:17:10<22:33:47,  3.59s/it]
Outer training loop:  55%|#####4    | 27402/50000 [17:17:11<19:07:08,  3.05s/it]
Outer training loop:  55%|#####4    | 27403/50000 [17:17:13<16:54:41,  2.69s/it]
Outer training loop:  55%|#####4    | 27404/50000 [17:17:15<15:15:09,  2.43s/it]
Outer training loop:  55%|#####4    | 27405/50000 [17:17:17<14:28:22,  2.31s/it]
Outer training loop:  55%|#####4    | 27406/50000 [17:17:19<13:34:21,  2.16s/it]
Outer training loop:  55%|#####4    | 27407/50000 [17:17:21<12:54:14,  2.06s/it]
Outer training loop:  55%|#####4    | 27408/50000 [17:17:23<12:42:23,  2.02s/it]
Outer training loop:  55%|#####4    | 27409/50000 [17:17:24<12:14:56,  1.95s/it]
Outer training loop:  55%|#####4    | 27410/50000 [17:17:26<12:09:44,  1.94s/it]
Outer training loop:  55%|#####4    | 27411/50000 [17:17:28<12:02:37,  1.92s/it]
Outer training loop:  55%|#####4    | 27412/50000 [17:17:30<12:25:14,  1.98s/it]
Outer training loop:  55%|#####4    | 27413/50000 [17:17:32<12:15:32,  1.95s/it]
Outer training loop:  55%|#####4    | 27414/50000 [17:17:34<12:11:11,  1.94s/it]
Outer training loop:  55%|#####4    | 27415/50000 [17:17:36<12:26:03,  1.98s/it]
Outer training loop:  55%|#####4    | 27416/50000 [17:17:38<12:08:40,  1.94s/it]
Outer training loop:  55%|#####4    | 27417/50000 [17:17:40<11:53:17,  1.90s/it]
Outer training loop:  55%|#####4    | 27418/50000 [17:17:42<12:15:22,  1.95s/it]
Outer training loop:  55%|#####4    | 27419/50000 [17:17:44<11:57:46,  1.91s/it]
Outer training loop:  55%|#####4    | 27420/50000 [17:17:46<11:54:07,  1.90s/it]
Outer training loop:  55%|#####4    | 27421/50000 [17:17:48<12:09:58,  1.94s/it]
Outer training loop:  55%|#####4    | 27422/50000 [17:17:50<11:58:17,  1.91s/it]
Outer training loop:  55%|#####4    | 27423/50000 [17:17:51<11:43:17,  1.87s/it]
Outer training loop:  55%|#####4    | 27424/50000 [17:17:53<11:42:05,  1.87s/it]
Outer training loop:  55%|#####4    | 27425/50000 [17:17:55<12:03:19,  1.92s/it]
Outer training loop:  55%|#####4    | 27426/50000 [17:17:57<11:57:48,  1.91s/it]
Outer training loop:  55%|#####4    | 27427/50000 [17:17:59<11:38:11,  1.86s/it]
Outer training loop:  55%|#####4    | 27428/50000 [17:18:01<12:08:18,  1.94s/it]
Outer training loop:  55%|#####4    | 27429/50000 [17:18:03<11:50:47,  1.89s/it]
Outer training loop:  55%|#####4    | 27430/50000 [17:18:05<11:53:22,  1.90s/it]
Outer training loop:  55%|#####4    | 27431/50000 [17:18:07<12:10:46,  1.94s/it]
Outer training loop:  55%|#####4    | 27432/50000 [17:18:08<11:52:38,  1.89s/it]
Outer training loop:  55%|#####4    | 27433/50000 [17:18:10<11:56:50,  1.91s/it]
Outer training loop:  55%|#####4    | 27434/50000 [17:18:12<11:44:06,  1.87s/it]
Outer training loop:  55%|#####4    | 27435/50000 [17:18:14<12:03:43,  1.92s/it]
Outer training loop:  55%|#####4    | 27436/50000 [17:18:16<11:50:37,  1.89s/it]
Outer training loop:  55%|#####4    | 27437/50000 [17:18:18<11:38:43,  1.86s/it]
Outer training loop:  55%|#####4    | 27438/50000 [17:18:20<11:53:27,  1.90s/it]
Outer training loop:  55%|#####4    | 27439/50000 [17:18:22<11:41:45,  1.87s/it]
Outer training loop:  55%|#####4    | 27440/50000 [17:18:24<11:41:20,  1.87s/it]
Outer training loop:  55%|#####4    | 27441/50000 [17:18:26<12:17:42,  1.96s/it]
Outer training loop:  55%|#####4    | 27442/50000 [17:18:28<12:06:43,  1.93s/it]
Outer training loop:  55%|#####4    | 27443/50000 [17:18:29<12:00:57,  1.92s/it]
Outer training loop:  55%|#####4    | 27444/50000 [17:18:31<12:11:39,  1.95s/it]
Outer training loop:  55%|#####4    | 27445/50000 [17:18:33<11:42:49,  1.87s/it]
Outer training loop:  55%|#####4    | 27446/50000 [17:18:35<11:36:30,  1.85s/it]
Outer training loop:  55%|#####4    | 27447/50000 [17:18:37<11:37:42,  1.86s/it]
Outer training loop:  55%|#####4    | 27448/50000 [17:18:39<11:53:20,  1.90s/it]
Outer training loop:  55%|#####4    | 27449/50000 [17:18:41<11:53:10,  1.90s/it]
Outer training loop:  55%|#####4    | 27450/50000 [17:18:43<12:05:09,  1.93s/it]
Outer training loop:  55%|#####4    | 27451/50000 [17:18:45<12:17:20,  1.96s/it]
Outer training loop:  55%|#####4    | 27452/50000 [17:18:47<12:00:20,  1.92s/it]
Outer training loop:  55%|#####4    | 27453/50000 [17:18:48<11:48:56,  1.89s/it]
Outer training loop:  55%|#####4    | 27454/50000 [17:18:50<11:50:47,  1.89s/it]
Outer training loop:  55%|#####4    | 27455/50000 [17:18:52<12:07:27,  1.94s/it]
Outer training loop:  55%|#####4    | 27456/50000 [17:18:54<12:06:52,  1.93s/it]
Outer training loop:  55%|#####4    | 27457/50000 [17:18:56<11:51:56,  1.89s/it]
Outer training loop:  55%|#####4    | 27458/50000 [17:18:58<11:59:42,  1.92s/it]
Outer training loop:  55%|#####4    | 27459/50000 [17:19:00<11:43:28,  1.87s/it]
Outer training loop:  55%|#####4    | 27460/50000 [17:19:02<11:44:42,  1.88s/it]
Outer training loop:  55%|#####4    | 27461/50000 [17:19:04<11:52:56,  1.90s/it]
Outer training loop:  55%|#####4    | 27462/50000 [17:19:06<12:05:14,  1.93s/it]
Outer training loop:  55%|#####4    | 27463/50000 [17:19:07<11:41:12,  1.87s/it]
Outer training loop:  55%|#####4    | 27464/50000 [17:19:09<11:41:59,  1.87s/it]
Outer training loop:  55%|#####4    | 27465/50000 [17:19:11<12:02:31,  1.92s/it]
Outer training loop:  55%|#####4    | 27466/50000 [17:19:13<12:02:51,  1.92s/it]
Outer training loop:  55%|#####4    | 27467/50000 [17:19:15<11:47:57,  1.89s/it]
Outer training loop:  55%|#####4    | 27468/50000 [17:19:17<12:13:49,  1.95s/it]
Outer training loop:  55%|#####4    | 27469/50000 [17:19:19<11:55:40,  1.91s/it]
Outer training loop:  55%|#####4    | 27470/50000 [17:19:21<11:50:39,  1.89s/it]
Outer training loop:  55%|#####4    | 27471/50000 [17:19:23<12:02:13,  1.92s/it]
Outer training loop:  55%|#####4    | 27472/50000 [17:19:25<11:49:54,  1.89s/it]
Outer training loop:  55%|#####4    | 27473/50000 [17:19:26<11:45:50,  1.88s/it]
Outer training loop:  55%|#####4    | 27474/50000 [17:19:28<12:04:58,  1.93s/it]
Outer training loop:  55%|#####4    | 27475/50000 [17:19:30<11:53:03,  1.90s/it]
Outer training loop:  55%|#####4    | 27476/50000 [17:19:32<11:46:23,  1.88s/it]
Outer training loop:  55%|#####4    | 27477/50000 [17:19:34<12:01:08,  1.92s/it]
Outer training loop:  55%|#####4    | 27478/50000 [17:19:36<11:45:17,  1.88s/it]
Outer training loop:  55%|#####4    | 27479/50000 [17:19:38<11:41:00,  1.87s/it]
Outer training loop:  55%|#####4    | 27480/50000 [17:19:40<12:10:14,  1.95s/it]
Outer training loop:  55%|#####4    | 27481/50000 [17:19:42<11:48:35,  1.89s/it]
Outer training loop:  55%|#####4    | 27482/50000 [17:19:44<11:46:01,  1.88s/it]
Outer training loop:  55%|#####4    | 27483/50000 [17:19:45<11:39:36,  1.86s/it]
Outer training loop:  55%|#####4    | 27484/50000 [17:19:47<11:33:37,  1.85s/it]
Outer training loop:  55%|#####4    | 27485/50000 [17:19:49<11:26:51,  1.83s/it]
Outer training loop:  55%|#####4    | 27486/50000 [17:19:51<11:27:57,  1.83s/it]
Outer training loop:  55%|#####4    | 27487/50000 [17:19:53<11:45:31,  1.88s/it]
Outer training loop:  55%|#####4    | 27488/50000 [17:19:54<11:25:39,  1.83s/it]
Outer training loop:  55%|#####4    | 27489/50000 [17:19:56<11:26:21,  1.83s/it]
Outer training loop:  55%|#####4    | 27490/50000 [17:19:58<11:33:44,  1.85s/it]
Outer training loop:  55%|#####4    | 27491/50000 [17:20:00<11:51:36,  1.90s/it]
Outer training loop:  55%|#####4    | 27492/50000 [17:20:02<11:38:20,  1.86s/it]
Outer training loop:  55%|#####4    | 27493/50000 [17:20:04<11:43:16,  1.87s/it]
Outer training loop:  55%|#####4    | 27494/50000 [17:20:06<12:09:20,  1.94s/it]
Outer training loop:  55%|#####4    | 27495/50000 [17:20:08<11:42:36,  1.87s/it]
Outer training loop:  55%|#####4    | 27496/50000 [17:20:10<11:50:59,  1.90s/it]
Outer training loop:  55%|#####4    | 27497/50000 [17:20:12<12:15:35,  1.96s/it]
Outer training loop:  55%|#####4    | 27498/50000 [17:20:14<12:21:29,  1.98s/it]
Outer training loop:  55%|#####4    | 27499/50000 [17:20:16<12:05:06,  1.93s/it]
Outer training loop:  55%|#####5    | 27500/50000 [17:20:17<11:48:16,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:41,  1.97s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:36,  1.93s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:30,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.90s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.90s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:21,  1.86s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:29,  1.99s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:26,  1.98s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:21,  1.94s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:17,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:18,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:06,  1.85s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:02,  1.82s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:00,  1.94s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.90s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.88s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:47,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:43,  1.88s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.87s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:39,  1.95s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:27,  1.86s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:29,  1.94s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:22,  1.87s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.88s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:08,  1.84s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:05,  1.81s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:01,  1.82s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.91s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:57,  1.85s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:54,  1.83s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.80s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:49,  1.78s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:48,  1.79s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:46,  1.80s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:49,  2.00s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:43,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.84s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.82s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:34,  1.81s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:32,  1.80s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:33,  1.94s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:30,  1.88s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.86s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.82s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:23,  1.81s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:21,  1.79s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:19,  1.78s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:17,  1.76s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:15,  1.77s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.93s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:25<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:27<00:07,  1.85s/it]

Evaluating policy:  96%|#########6| 80/83 [02:29<00:05,  1.83s/it]

Evaluating policy:  98%|#########7| 81/83 [02:31<00:03,  1.80s/it]

Evaluating policy:  99%|#########8| 82/83 [02:32<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 83/83 [02:34<00:00,  1.73s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:56,  2.15s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:43,  2.02s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:35,  1.95s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.90s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.91s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:16,  1.85s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:13,  1.83s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:13,  1.85s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.88s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:07,  1.82s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:06,  1.84s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:04,  1.83s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:57,  1.81s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:54,  1.79s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:51,  1.76s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:52,  1.82s/it]

Evaluating policy:  27%|##6       | 22/83 [00:40<01:49,  1.79s/it]

Evaluating policy:  28%|##7       | 23/83 [00:42<01:48,  1.80s/it]

Evaluating policy:  29%|##8       | 24/83 [00:43<01:45,  1.78s/it]

Evaluating policy:  30%|###       | 25/83 [00:45<01:47,  1.85s/it]

Evaluating policy:  31%|###1      | 26/83 [00:47<01:48,  1.90s/it]

Evaluating policy:  33%|###2      | 27/83 [00:49<01:43,  1.84s/it]

Evaluating policy:  34%|###3      | 28/83 [00:51<01:40,  1.82s/it]

Evaluating policy:  35%|###4      | 29/83 [00:53<01:37,  1.81s/it]

Evaluating policy:  36%|###6      | 30/83 [00:55<01:35,  1.81s/it]

Evaluating policy:  37%|###7      | 31/83 [00:57<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [00:58<01:35,  1.87s/it]

Evaluating policy:  40%|###9      | 33/83 [01:00<01:32,  1.84s/it]

Evaluating policy:  41%|####      | 34/83 [01:02<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:04<01:27,  1.83s/it]

Evaluating policy:  43%|####3     | 36/83 [01:06<01:27,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:08<01:30,  1.96s/it]

Evaluating policy:  46%|####5     | 38/83 [01:10<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:11<01:21,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:13<01:19,  1.85s/it]

Evaluating policy:  49%|####9     | 41/83 [01:15<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:17<01:14,  1.82s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:19<01:17,  1.93s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:21<01:14,  1.90s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:23<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:25<01:08,  1.85s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:26<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:28<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:30<01:01,  1.82s/it]

Evaluating policy:  60%|######    | 50/83 [01:32<01:02,  1.89s/it]

Evaluating policy:  61%|######1   | 51/83 [01:34<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:36<00:56,  1.83s/it]

Evaluating policy:  64%|######3   | 53/83 [01:37<00:53,  1.80s/it]

Evaluating policy:  65%|######5   | 54/83 [01:39<00:50,  1.75s/it]

Evaluating policy:  66%|######6   | 55/83 [01:41<00:49,  1.78s/it]

Evaluating policy:  67%|######7   | 56/83 [01:43<00:48,  1.81s/it]

Evaluating policy:  69%|######8   | 57/83 [01:44<00:46,  1.78s/it]

Evaluating policy:  70%|######9   | 58/83 [01:47<00:48,  1.94s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:48<00:45,  1.90s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:50<00:43,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:52<00:40,  1.82s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:54<00:37,  1.78s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:55<00:35,  1.76s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:57<00:33,  1.79s/it]

Evaluating policy:  78%|#######8  | 65/83 [01:59<00:31,  1.77s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:01<00:32,  1.91s/it]

Evaluating policy:  81%|########  | 67/83 [02:03<00:30,  1.89s/it]

Evaluating policy:  82%|########1 | 68/83 [02:05<00:28,  1.87s/it]

Evaluating policy:  83%|########3 | 69/83 [02:07<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:09<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:10<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:12<00:19,  1.79s/it]

Evaluating policy:  88%|########7 | 73/83 [02:14<00:17,  1.78s/it]

Evaluating policy:  89%|########9 | 74/83 [02:16<00:17,  1.92s/it]

Evaluating policy:  90%|######### | 75/83 [02:18<00:15,  1.97s/it]

Evaluating policy:  92%|#########1| 76/83 [02:20<00:13,  1.93s/it]

Evaluating policy:  93%|#########2| 77/83 [02:22<00:11,  1.89s/it]

Evaluating policy:  94%|#########3| 78/83 [02:23<00:09,  1.84s/it]

Evaluating policy:  95%|#########5| 79/83 [02:25<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:27<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:29<00:03,  1.84s/it]

Evaluating policy:  99%|#########8| 82/83 [02:31<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:33<00:00,  1.86s/it]

Outer training loop:  55%|#####5    | 27501/50000 [17:25:33<600:06:21, 96.02s/it]
Outer training loop:  55%|#####5    | 27502/50000 [17:25:35<424:33:08, 67.93s/it]
Outer training loop:  55%|#####5    | 27503/50000 [17:25:37<300:40:02, 48.11s/it]
Outer training loop:  55%|#####5    | 27504/50000 [17:25:39<213:53:52, 34.23s/it]
Outer training loop:  55%|#####5    | 27505/50000 [17:25:41<153:18:40, 24.54s/it]
Outer training loop:  55%|#####5    | 27506/50000 [17:25:43<111:12:51, 17.80s/it]
Outer training loop:  55%|#####5    | 27507/50000 [17:25:45<81:12:20, 13.00s/it] 
Outer training loop:  55%|#####5    | 27508/50000 [17:25:47<60:27:24,  9.68s/it]
Outer training loop:  55%|#####5    | 27509/50000 [17:25:49<46:07:28,  7.38s/it]
Outer training loop:  55%|#####5    | 27510/50000 [17:25:51<35:45:16,  5.72s/it]
Outer training loop:  55%|#####5    | 27511/50000 [17:25:53<28:24:56,  4.55s/it]
Outer training loop:  55%|#####5    | 27512/50000 [17:25:54<23:20:01,  3.74s/it]
Outer training loop:  55%|#####5    | 27513/50000 [17:25:56<20:06:08,  3.22s/it]
Outer training loop:  55%|#####5    | 27514/50000 [17:25:58<17:40:59,  2.83s/it]
Outer training loop:  55%|#####5    | 27515/50000 [17:26:00<15:47:37,  2.53s/it]
Outer training loop:  55%|#####5    | 27516/50000 [17:26:02<15:04:37,  2.41s/it]
Outer training loop:  55%|#####5    | 27517/50000 [17:26:04<14:08:36,  2.26s/it]
Outer training loop:  55%|#####5    | 27518/50000 [17:26:06<13:31:35,  2.17s/it]
Outer training loop:  55%|#####5    | 27519/50000 [17:26:08<12:59:14,  2.08s/it]
Outer training loop:  55%|#####5    | 27520/50000 [17:26:10<12:59:13,  2.08s/it]
Outer training loop:  55%|#####5    | 27521/50000 [17:26:12<12:35:28,  2.02s/it]
Outer training loop:  55%|#####5    | 27522/50000 [17:26:14<12:30:55,  2.00s/it]
Outer training loop:  55%|#####5    | 27523/50000 [17:26:16<12:34:59,  2.02s/it]
Outer training loop:  55%|#####5    | 27524/50000 [17:26:18<12:23:23,  1.98s/it]
Outer training loop:  55%|#####5    | 27525/50000 [17:26:20<12:02:48,  1.93s/it]
Outer training loop:  55%|#####5    | 27526/50000 [17:26:22<12:06:52,  1.94s/it]
Outer training loop:  55%|#####5    | 27527/50000 [17:26:24<12:32:39,  2.01s/it]
Outer training loop:  55%|#####5    | 27528/50000 [17:26:26<12:11:24,  1.95s/it]
Outer training loop:  55%|#####5    | 27529/50000 [17:26:27<11:48:03,  1.89s/it]
Outer training loop:  55%|#####5    | 27530/50000 [17:26:29<11:47:29,  1.89s/it]
Outer training loop:  55%|#####5    | 27531/50000 [17:26:31<12:06:34,  1.94s/it]
Outer training loop:  55%|#####5    | 27532/50000 [17:26:33<11:46:51,  1.89s/it]
Outer training loop:  55%|#####5    | 27533/50000 [17:26:35<12:09:49,  1.95s/it]
Outer training loop:  55%|#####5    | 27534/50000 [17:26:37<12:05:11,  1.94s/it]
Outer training loop:  55%|#####5    | 27535/50000 [17:26:39<11:53:39,  1.91s/it]
Outer training loop:  55%|#####5    | 27536/50000 [17:26:41<11:43:32,  1.88s/it]
Outer training loop:  55%|#####5    | 27537/50000 [17:26:43<12:14:10,  1.96s/it]
Outer training loop:  55%|#####5    | 27538/50000 [17:26:45<11:58:30,  1.92s/it]
Outer training loop:  55%|#####5    | 27539/50000 [17:26:47<11:54:53,  1.91s/it]
Outer training loop:  55%|#####5    | 27540/50000 [17:26:49<12:12:36,  1.96s/it]
Outer training loop:  55%|#####5    | 27541/50000 [17:26:51<12:07:12,  1.94s/it]
Outer training loop:  55%|#####5    | 27542/50000 [17:26:52<11:43:47,  1.88s/it]
Outer training loop:  55%|#####5    | 27543/50000 [17:26:55<12:16:35,  1.97s/it]
Outer training loop:  55%|#####5    | 27544/50000 [17:26:56<11:59:54,  1.92s/it]
Outer training loop:  55%|#####5    | 27545/50000 [17:26:58<11:48:54,  1.89s/it]
Outer training loop:  55%|#####5    | 27546/50000 [17:27:00<11:36:16,  1.86s/it]
Outer training loop:  55%|#####5    | 27547/50000 [17:27:02<11:54:16,  1.91s/it]
Outer training loop:  55%|#####5    | 27548/50000 [17:27:04<11:44:53,  1.88s/it]
Outer training loop:  55%|#####5    | 27549/50000 [17:27:06<11:48:26,  1.89s/it]
Outer training loop:  55%|#####5    | 27550/50000 [17:27:08<12:20:52,  1.98s/it]
Outer training loop:  55%|#####5    | 27551/50000 [17:27:10<12:12:26,  1.96s/it]
Outer training loop:  55%|#####5    | 27552/50000 [17:27:12<12:08:36,  1.95s/it]
Outer training loop:  55%|#####5    | 27553/50000 [17:27:14<12:31:41,  2.01s/it]
Outer training loop:  55%|#####5    | 27554/50000 [17:27:16<12:21:10,  1.98s/it]
Outer training loop:  55%|#####5    | 27555/50000 [17:27:18<12:02:47,  1.93s/it]
Outer training loop:  55%|#####5    | 27556/50000 [17:27:20<12:12:05,  1.96s/it]
Outer training loop:  55%|#####5    | 27557/50000 [17:27:22<12:26:59,  2.00s/it]
Outer training loop:  55%|#####5    | 27558/50000 [17:27:24<12:07:28,  1.94s/it]
Outer training loop:  55%|#####5    | 27559/50000 [17:27:26<12:11:46,  1.96s/it]
Outer training loop:  55%|#####5    | 27560/50000 [17:27:27<11:47:58,  1.89s/it]
Outer training loop:  55%|#####5    | 27561/50000 [17:27:29<12:09:25,  1.95s/it]
Outer training loop:  55%|#####5    | 27562/50000 [17:27:31<12:00:15,  1.93s/it]
Outer training loop:  55%|#####5    | 27563/50000 [17:27:33<11:50:59,  1.90s/it]
Outer training loop:  55%|#####5    | 27564/50000 [17:27:35<12:12:19,  1.96s/it]
Outer training loop:  55%|#####5    | 27565/50000 [17:27:37<12:02:37,  1.93s/it]
Outer training loop:  55%|#####5    | 27566/50000 [17:27:39<11:53:40,  1.91s/it]
Outer training loop:  55%|#####5    | 27567/50000 [17:27:41<12:21:07,  1.98s/it]
Outer training loop:  55%|#####5    | 27568/50000 [17:27:43<12:09:26,  1.95s/it]
Outer training loop:  55%|#####5    | 27569/50000 [17:27:45<11:50:03,  1.90s/it]
Outer training loop:  55%|#####5    | 27570/50000 [17:27:47<11:58:13,  1.92s/it]
Outer training loop:  55%|#####5    | 27571/50000 [17:27:49<12:08:24,  1.95s/it]
Outer training loop:  55%|#####5    | 27572/50000 [17:27:51<11:58:22,  1.92s/it]
Outer training loop:  55%|#####5    | 27573/50000 [17:27:52<11:47:20,  1.89s/it]
Outer training loop:  55%|#####5    | 27574/50000 [17:27:54<12:04:42,  1.94s/it]
Outer training loop:  55%|#####5    | 27575/50000 [17:27:56<12:01:49,  1.93s/it]
Outer training loop:  55%|#####5    | 27576/50000 [17:27:58<11:58:41,  1.92s/it]
Outer training loop:  55%|#####5    | 27577/50000 [17:28:00<12:04:20,  1.94s/it]
Outer training loop:  55%|#####5    | 27578/50000 [17:28:02<11:55:27,  1.91s/it]
Outer training loop:  55%|#####5    | 27579/50000 [17:28:04<12:02:46,  1.93s/it]
Outer training loop:  55%|#####5    | 27580/50000 [17:28:06<12:09:19,  1.95s/it]
Outer training loop:  55%|#####5    | 27581/50000 [17:28:08<12:28:41,  2.00s/it]
Outer training loop:  55%|#####5    | 27582/50000 [17:28:10<12:03:13,  1.94s/it]
Outer training loop:  55%|#####5    | 27583/50000 [17:28:12<11:47:23,  1.89s/it]
Outer training loop:  55%|#####5    | 27584/50000 [17:28:14<11:47:31,  1.89s/it]
Outer training loop:  55%|#####5    | 27585/50000 [17:28:16<11:46:27,  1.89s/it]
Outer training loop:  55%|#####5    | 27586/50000 [17:28:17<11:41:58,  1.88s/it]
Outer training loop:  55%|#####5    | 27587/50000 [17:28:19<11:39:19,  1.87s/it]
Outer training loop:  55%|#####5    | 27588/50000 [17:28:21<12:09:05,  1.95s/it]
Outer training loop:  55%|#####5    | 27589/50000 [17:28:23<11:48:44,  1.90s/it]
Outer training loop:  55%|#####5    | 27590/50000 [17:28:25<11:57:25,  1.92s/it]
Outer training loop:  55%|#####5    | 27591/50000 [17:28:27<11:47:54,  1.90s/it]
Outer training loop:  55%|#####5    | 27592/50000 [17:28:29<12:02:39,  1.94s/it]
Outer training loop:  55%|#####5    | 27593/50000 [17:28:31<12:11:51,  1.96s/it]
Outer training loop:  55%|#####5    | 27594/50000 [17:28:33<11:53:49,  1.91s/it]
Outer training loop:  55%|#####5    | 27595/50000 [17:28:35<12:19:31,  1.98s/it]
Outer training loop:  55%|#####5    | 27596/50000 [17:28:37<12:03:17,  1.94s/it]
Outer training loop:  55%|#####5    | 27597/50000 [17:28:39<11:42:10,  1.88s/it]
Outer training loop:  55%|#####5    | 27598/50000 [17:28:40<11:41:53,  1.88s/it]
Outer training loop:  55%|#####5    | 27599/50000 [17:28:43<12:05:38,  1.94s/it]
Outer training loop:  55%|#####5    | 27600/50000 [17:28:44<12:04:48,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  55%|#####5    | 27601/50000 [17:28:52<22:16:17,  3.58s/it]
Outer training loop:  55%|#####5    | 27602/50000 [17:28:54<19:08:45,  3.08s/it]
Outer training loop:  55%|#####5    | 27603/50000 [17:28:56<17:47:01,  2.86s/it]
Outer training loop:  55%|#####5    | 27604/50000 [17:28:58<15:46:31,  2.54s/it]
Outer training loop:  55%|#####5    | 27605/50000 [17:29:00<14:33:27,  2.34s/it]
Outer training loop:  55%|#####5    | 27606/50000 [17:29:02<13:49:24,  2.22s/it]
Outer training loop:  55%|#####5    | 27607/50000 [17:29:04<13:02:29,  2.10s/it]
Outer training loop:  55%|#####5    | 27608/50000 [17:29:05<12:36:25,  2.03s/it]
Outer training loop:  55%|#####5    | 27609/50000 [17:29:07<12:33:16,  2.02s/it]
Outer training loop:  55%|#####5    | 27610/50000 [17:29:09<12:01:40,  1.93s/it]
Outer training loop:  55%|#####5    | 27611/50000 [17:29:11<11:44:50,  1.89s/it]
Outer training loop:  55%|#####5    | 27612/50000 [17:29:13<11:29:21,  1.85s/it]
Outer training loop:  55%|#####5    | 27613/50000 [17:29:15<11:45:28,  1.89s/it]
Outer training loop:  55%|#####5    | 27614/50000 [17:29:16<11:21:16,  1.83s/it]
Outer training loop:  55%|#####5    | 27615/50000 [17:29:18<11:06:23,  1.79s/it]
Outer training loop:  55%|#####5    | 27616/50000 [17:29:20<11:33:10,  1.86s/it]
Outer training loop:  55%|#####5    | 27617/50000 [17:29:22<11:25:04,  1.84s/it]
Outer training loop:  55%|#####5    | 27618/50000 [17:29:24<11:17:43,  1.82s/it]
Outer training loop:  55%|#####5    | 27619/50000 [17:29:26<11:53:56,  1.91s/it]
Outer training loop:  55%|#####5    | 27620/50000 [17:29:28<11:43:43,  1.89s/it]
Outer training loop:  55%|#####5    | 27621/50000 [17:29:29<11:43:16,  1.89s/it]
Outer training loop:  55%|#####5    | 27622/50000 [17:29:31<11:43:07,  1.89s/it]
Outer training loop:  55%|#####5    | 27623/50000 [17:29:33<12:04:56,  1.94s/it]
Outer training loop:  55%|#####5    | 27624/50000 [17:29:35<11:48:00,  1.90s/it]
Outer training loop:  55%|#####5    | 27625/50000 [17:29:37<11:46:09,  1.89s/it]
Outer training loop:  55%|#####5    | 27626/50000 [17:29:39<12:12:12,  1.96s/it]
Outer training loop:  55%|#####5    | 27627/50000 [17:29:41<11:53:05,  1.91s/it]
Outer training loop:  55%|#####5    | 27628/50000 [17:29:43<11:43:25,  1.89s/it]
Outer training loop:  55%|#####5    | 27629/50000 [17:29:45<11:30:12,  1.85s/it]
Outer training loop:  55%|#####5    | 27630/50000 [17:29:47<11:37:04,  1.87s/it]
Outer training loop:  55%|#####5    | 27631/50000 [17:29:48<11:14:52,  1.81s/it]
Outer training loop:  55%|#####5    | 27632/50000 [17:29:50<11:13:48,  1.81s/it]
Outer training loop:  55%|#####5    | 27633/50000 [17:29:52<11:12:09,  1.80s/it]
Outer training loop:  55%|#####5    | 27634/50000 [17:29:54<11:18:47,  1.82s/it]
Outer training loop:  55%|#####5    | 27635/50000 [17:29:56<11:28:51,  1.85s/it]
Outer training loop:  55%|#####5    | 27636/50000 [17:29:57<11:35:43,  1.87s/it]
Outer training loop:  55%|#####5    | 27637/50000 [17:29:59<11:49:32,  1.90s/it]
Outer training loop:  55%|#####5    | 27638/50000 [17:30:01<11:55:47,  1.92s/it]
Outer training loop:  55%|#####5    | 27639/50000 [17:30:03<11:45:28,  1.89s/it]
Outer training loop:  55%|#####5    | 27640/50000 [17:30:06<12:35:14,  2.03s/it]
Outer training loop:  55%|#####5    | 27641/50000 [17:30:07<12:10:22,  1.96s/it]
Outer training loop:  55%|#####5    | 27642/50000 [17:30:09<12:12:34,  1.97s/it]
Outer training loop:  55%|#####5    | 27643/50000 [17:30:11<12:17:41,  1.98s/it]
Outer training loop:  55%|#####5    | 27644/50000 [17:30:13<12:18:56,  1.98s/it]
Outer training loop:  55%|#####5    | 27645/50000 [17:30:15<11:56:38,  1.92s/it]
Outer training loop:  55%|#####5    | 27646/50000 [17:30:17<11:57:17,  1.93s/it]
Outer training loop:  55%|#####5    | 27647/50000 [17:30:19<12:13:43,  1.97s/it]
Outer training loop:  55%|#####5    | 27648/50000 [17:30:21<12:14:28,  1.97s/it]
Outer training loop:  55%|#####5    | 27649/50000 [17:30:23<11:49:13,  1.90s/it]
Outer training loop:  55%|#####5    | 27650/50000 [17:30:25<12:12:03,  1.97s/it]
Outer training loop:  55%|#####5    | 27651/50000 [17:30:27<11:53:02,  1.91s/it]
Outer training loop:  55%|#####5    | 27652/50000 [17:30:29<11:43:43,  1.89s/it]
Outer training loop:  55%|#####5    | 27653/50000 [17:30:31<12:07:53,  1.95s/it]
Outer training loop:  55%|#####5    | 27654/50000 [17:30:33<11:56:12,  1.92s/it]
Outer training loop:  55%|#####5    | 27655/50000 [17:30:34<11:42:11,  1.89s/it]
Outer training loop:  55%|#####5    | 27656/50000 [17:30:36<11:51:08,  1.91s/it]
Outer training loop:  55%|#####5    | 27657/50000 [17:30:38<12:05:17,  1.95s/it]
Outer training loop:  55%|#####5    | 27658/50000 [17:30:40<11:50:20,  1.91s/it]
Outer training loop:  55%|#####5    | 27659/50000 [17:30:42<11:30:14,  1.85s/it]
Outer training loop:  55%|#####5    | 27660/50000 [17:30:44<11:28:58,  1.85s/it]
Outer training loop:  55%|#####5    | 27661/50000 [17:30:46<11:30:42,  1.86s/it]
Outer training loop:  55%|#####5    | 27662/50000 [17:30:47<11:27:23,  1.85s/it]
Outer training loop:  55%|#####5    | 27663/50000 [17:30:49<11:23:59,  1.84s/it]
Outer training loop:  55%|#####5    | 27664/50000 [17:30:51<11:55:53,  1.92s/it]
Outer training loop:  55%|#####5    | 27665/50000 [17:30:53<11:45:32,  1.90s/it]
Outer training loop:  55%|#####5    | 27666/50000 [17:30:55<11:32:13,  1.86s/it]
Outer training loop:  55%|#####5    | 27667/50000 [17:30:57<11:53:46,  1.92s/it]
Outer training loop:  55%|#####5    | 27668/50000 [17:30:59<12:00:33,  1.94s/it]
Outer training loop:  55%|#####5    | 27669/50000 [17:31:01<11:39:17,  1.88s/it]
Outer training loop:  55%|#####5    | 27670/50000 [17:31:02<11:14:01,  1.81s/it]
Outer training loop:  55%|#####5    | 27671/50000 [17:31:04<11:37:33,  1.87s/it]
Outer training loop:  55%|#####5    | 27672/50000 [17:31:06<11:34:05,  1.87s/it]
Outer training loop:  55%|#####5    | 27673/50000 [17:31:08<11:36:30,  1.87s/it]
Outer training loop:  55%|#####5    | 27674/50000 [17:31:10<11:57:50,  1.93s/it]
Outer training loop:  55%|#####5    | 27675/50000 [17:31:12<11:49:53,  1.91s/it]
Outer training loop:  55%|#####5    | 27676/50000 [17:31:14<11:45:42,  1.90s/it]
Outer training loop:  55%|#####5    | 27677/50000 [17:31:16<11:35:12,  1.87s/it]
Outer training loop:  55%|#####5    | 27678/50000 [17:31:18<11:39:04,  1.88s/it]
Outer training loop:  55%|#####5    | 27679/50000 [17:31:20<11:41:11,  1.88s/it]
Outer training loop:  55%|#####5    | 27680/50000 [17:31:21<11:45:23,  1.90s/it]
Outer training loop:  55%|#####5    | 27681/50000 [17:31:23<11:43:14,  1.89s/it]
Outer training loop:  55%|#####5    | 27682/50000 [17:31:25<11:38:20,  1.88s/it]
Outer training loop:  55%|#####5    | 27683/50000 [17:31:27<11:34:41,  1.87s/it]
Outer training loop:  55%|#####5    | 27684/50000 [17:31:29<11:25:30,  1.84s/it]
Outer training loop:  55%|#####5    | 27685/50000 [17:31:31<11:45:37,  1.90s/it]
Outer training loop:  55%|#####5    | 27686/50000 [17:31:33<11:37:55,  1.88s/it]
Outer training loop:  55%|#####5    | 27687/50000 [17:31:35<11:34:05,  1.87s/it]
Outer training loop:  55%|#####5    | 27688/50000 [17:31:36<11:40:38,  1.88s/it]
Outer training loop:  55%|#####5    | 27689/50000 [17:31:38<11:28:52,  1.85s/it]
Outer training loop:  55%|#####5    | 27690/50000 [17:31:40<11:50:59,  1.91s/it]
Outer training loop:  55%|#####5    | 27691/50000 [17:31:42<11:55:00,  1.92s/it]
Outer training loop:  55%|#####5    | 27692/50000 [17:31:44<11:34:57,  1.87s/it]
Outer training loop:  55%|#####5    | 27693/50000 [17:31:46<11:24:35,  1.84s/it]
Outer training loop:  55%|#####5    | 27694/50000 [17:31:48<11:29:02,  1.85s/it]
Outer training loop:  55%|#####5    | 27695/50000 [17:31:50<11:55:44,  1.93s/it]
Outer training loop:  55%|#####5    | 27696/50000 [17:31:52<11:38:43,  1.88s/it]
Outer training loop:  55%|#####5    | 27697/50000 [17:31:53<11:25:50,  1.85s/it]
Outer training loop:  55%|#####5    | 27698/50000 [17:31:55<12:00:03,  1.94s/it]
Outer training loop:  55%|#####5    | 27699/50000 [17:31:57<11:41:18,  1.89s/it]
Outer training loop:  55%|#####5    | 27700/50000 [17:31:59<11:25:19,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  55%|#####5    | 27701/50000 [17:32:06<21:55:56,  3.54s/it]
Outer training loop:  55%|#####5    | 27702/50000 [17:32:08<19:01:08,  3.07s/it]
Outer training loop:  55%|#####5    | 27703/50000 [17:32:10<16:39:08,  2.69s/it]
Outer training loop:  55%|#####5    | 27704/50000 [17:32:12<14:51:12,  2.40s/it]
Outer training loop:  55%|#####5    | 27705/50000 [17:32:14<14:05:46,  2.28s/it]
Outer training loop:  55%|#####5    | 27706/50000 [17:32:16<13:10:26,  2.13s/it]
Outer training loop:  55%|#####5    | 27707/50000 [17:32:17<12:28:06,  2.01s/it]
Outer training loop:  55%|#####5    | 27708/50000 [17:32:19<12:14:58,  1.98s/it]
Outer training loop:  55%|#####5    | 27709/50000 [17:32:21<12:24:17,  2.00s/it]
Outer training loop:  55%|#####5    | 27710/50000 [17:32:23<12:06:50,  1.96s/it]
Outer training loop:  55%|#####5    | 27711/50000 [17:32:25<12:13:00,  1.97s/it]
Outer training loop:  55%|#####5    | 27712/50000 [17:32:27<12:16:22,  1.98s/it]
Outer training loop:  55%|#####5    | 27713/50000 [17:32:29<11:51:01,  1.91s/it]
Outer training loop:  55%|#####5    | 27714/50000 [17:32:31<11:29:56,  1.86s/it]
Outer training loop:  55%|#####5    | 27715/50000 [17:32:33<11:24:53,  1.84s/it]
Outer training loop:  55%|#####5    | 27716/50000 [17:32:35<11:46:05,  1.90s/it]
Outer training loop:  55%|#####5    | 27717/50000 [17:32:36<11:44:05,  1.90s/it]
Outer training loop:  55%|#####5    | 27718/50000 [17:32:38<11:32:20,  1.86s/it]
Outer training loop:  55%|#####5    | 27719/50000 [17:32:40<11:55:59,  1.93s/it]
Outer training loop:  55%|#####5    | 27720/50000 [17:32:42<11:48:26,  1.91s/it]
Outer training loop:  55%|#####5    | 27721/50000 [17:32:44<11:51:42,  1.92s/it]
Outer training loop:  55%|#####5    | 27722/50000 [17:32:46<12:17:20,  1.99s/it]
Outer training loop:  55%|#####5    | 27723/50000 [17:32:48<11:44:40,  1.90s/it]
Outer training loop:  55%|#####5    | 27724/50000 [17:32:50<11:43:49,  1.90s/it]
Outer training loop:  55%|#####5    | 27725/50000 [17:32:52<11:28:16,  1.85s/it]
Outer training loop:  55%|#####5    | 27726/50000 [17:32:54<11:52:35,  1.92s/it]
Outer training loop:  55%|#####5    | 27727/50000 [17:32:56<11:51:57,  1.92s/it]
Outer training loop:  55%|#####5    | 27728/50000 [17:32:58<11:49:46,  1.91s/it]
Outer training loop:  55%|#####5    | 27729/50000 [17:32:59<11:49:55,  1.91s/it]
Outer training loop:  55%|#####5    | 27730/50000 [17:33:01<11:44:05,  1.90s/it]
Outer training loop:  55%|#####5    | 27731/50000 [17:33:03<11:38:18,  1.88s/it]
Outer training loop:  55%|#####5    | 27732/50000 [17:33:05<11:23:59,  1.84s/it]
Outer training loop:  55%|#####5    | 27733/50000 [17:33:07<11:37:15,  1.88s/it]
Outer training loop:  55%|#####5    | 27734/50000 [17:33:09<11:34:49,  1.87s/it]
Outer training loop:  55%|#####5    | 27735/50000 [17:33:11<11:29:48,  1.86s/it]
Outer training loop:  55%|#####5    | 27736/50000 [17:33:13<12:07:34,  1.96s/it]
Outer training loop:  55%|#####5    | 27737/50000 [17:33:15<12:09:07,  1.97s/it]
Outer training loop:  55%|#####5    | 27738/50000 [17:33:17<11:55:26,  1.93s/it]
Outer training loop:  55%|#####5    | 27739/50000 [17:33:19<12:23:04,  2.00s/it]
Outer training loop:  55%|#####5    | 27740/50000 [17:33:21<12:24:02,  2.01s/it]
Outer training loop:  55%|#####5    | 27741/50000 [17:33:23<12:12:39,  1.97s/it]
Outer training loop:  55%|#####5    | 27742/50000 [17:33:25<12:23:33,  2.00s/it]
Outer training loop:  55%|#####5    | 27743/50000 [17:33:27<12:28:21,  2.02s/it]
Outer training loop:  55%|#####5    | 27744/50000 [17:33:29<12:12:05,  1.97s/it]
Outer training loop:  55%|#####5    | 27745/50000 [17:33:30<11:59:22,  1.94s/it]
Outer training loop:  55%|#####5    | 27746/50000 [17:33:33<12:09:58,  1.97s/it]
Outer training loop:  55%|#####5    | 27747/50000 [17:33:34<11:58:30,  1.94s/it]
Outer training loop:  55%|#####5    | 27748/50000 [17:33:36<12:11:59,  1.97s/it]
Outer training loop:  55%|#####5    | 27749/50000 [17:33:38<11:59:27,  1.94s/it]
Outer training loop:  56%|#####5    | 27750/50000 [17:33:40<12:13:20,  1.98s/it]
Outer training loop:  56%|#####5    | 27751/50000 [17:33:42<12:22:23,  2.00s/it]
Outer training loop:  56%|#####5    | 27752/50000 [17:33:44<12:08:11,  1.96s/it]
Outer training loop:  56%|#####5    | 27753/50000 [17:33:46<12:12:35,  1.98s/it]
Outer training loop:  56%|#####5    | 27754/50000 [17:33:48<12:11:53,  1.97s/it]
Outer training loop:  56%|#####5    | 27755/50000 [17:33:50<12:08:05,  1.96s/it]
Outer training loop:  56%|#####5    | 27756/50000 [17:33:52<12:28:54,  2.02s/it]
Outer training loop:  56%|#####5    | 27757/50000 [17:33:54<12:07:01,  1.96s/it]
Outer training loop:  56%|#####5    | 27758/50000 [17:33:56<11:52:25,  1.92s/it]
Outer training loop:  56%|#####5    | 27759/50000 [17:33:58<12:20:48,  2.00s/it]
Outer training loop:  56%|#####5    | 27760/50000 [17:34:00<12:29:11,  2.02s/it]
Outer training loop:  56%|#####5    | 27761/50000 [17:34:02<12:15:35,  1.98s/it]
Outer training loop:  56%|#####5    | 27762/50000 [17:34:04<12:02:00,  1.95s/it]
Outer training loop:  56%|#####5    | 27763/50000 [17:34:06<12:12:25,  1.98s/it]
Outer training loop:  56%|#####5    | 27764/50000 [17:34:08<11:53:21,  1.92s/it]
Outer training loop:  56%|#####5    | 27765/50000 [17:34:10<11:41:38,  1.89s/it]
Outer training loop:  56%|#####5    | 27766/50000 [17:34:12<11:30:14,  1.86s/it]
Outer training loop:  56%|#####5    | 27767/50000 [17:34:13<11:44:22,  1.90s/it]
Outer training loop:  56%|#####5    | 27768/50000 [17:34:15<11:33:25,  1.87s/it]
Outer training loop:  56%|#####5    | 27769/50000 [17:34:17<11:23:40,  1.85s/it]
Outer training loop:  56%|#####5    | 27770/50000 [17:34:19<11:29:40,  1.86s/it]
Outer training loop:  56%|#####5    | 27771/50000 [17:34:21<12:03:33,  1.95s/it]
Outer training loop:  56%|#####5    | 27772/50000 [17:34:23<12:05:10,  1.96s/it]
Outer training loop:  56%|#####5    | 27773/50000 [17:34:25<12:02:44,  1.95s/it]
Outer training loop:  56%|#####5    | 27774/50000 [17:34:27<12:32:08,  2.03s/it]
Outer training loop:  56%|#####5    | 27775/50000 [17:34:29<12:02:36,  1.95s/it]
Outer training loop:  56%|#####5    | 27776/50000 [17:34:31<11:58:16,  1.94s/it]
Outer training loop:  56%|#####5    | 27777/50000 [17:34:33<12:04:47,  1.96s/it]
Outer training loop:  56%|#####5    | 27778/50000 [17:34:35<11:42:52,  1.90s/it]
Outer training loop:  56%|#####5    | 27779/50000 [17:34:37<11:57:47,  1.94s/it]
Outer training loop:  56%|#####5    | 27780/50000 [17:34:38<11:36:14,  1.88s/it]
Outer training loop:  56%|#####5    | 27781/50000 [17:34:41<11:52:34,  1.92s/it]
Outer training loop:  56%|#####5    | 27782/50000 [17:34:42<11:50:42,  1.92s/it]
Outer training loop:  56%|#####5    | 27783/50000 [17:34:44<11:41:46,  1.90s/it]
Outer training loop:  56%|#####5    | 27784/50000 [17:34:46<12:01:52,  1.95s/it]
Outer training loop:  56%|#####5    | 27785/50000 [17:34:48<11:49:34,  1.92s/it]
Outer training loop:  56%|#####5    | 27786/50000 [17:34:50<11:48:05,  1.91s/it]
Outer training loop:  56%|#####5    | 27787/50000 [17:34:52<12:10:58,  1.97s/it]
Outer training loop:  56%|#####5    | 27788/50000 [17:34:54<11:53:43,  1.93s/it]
Outer training loop:  56%|#####5    | 27789/50000 [17:34:56<12:04:53,  1.96s/it]
Outer training loop:  56%|#####5    | 27790/50000 [17:34:58<11:53:35,  1.93s/it]
Outer training loop:  56%|#####5    | 27791/50000 [17:35:00<12:06:59,  1.96s/it]
Outer training loop:  56%|#####5    | 27792/50000 [17:35:02<11:47:33,  1.91s/it]
Outer training loop:  56%|#####5    | 27793/50000 [17:35:04<11:42:39,  1.90s/it]
Outer training loop:  56%|#####5    | 27794/50000 [17:35:06<11:55:53,  1.93s/it]
Outer training loop:  56%|#####5    | 27795/50000 [17:35:07<11:37:54,  1.89s/it]
Outer training loop:  56%|#####5    | 27796/50000 [17:35:09<11:32:41,  1.87s/it]
Outer training loop:  56%|#####5    | 27797/50000 [17:35:11<11:33:12,  1.87s/it]
Outer training loop:  56%|#####5    | 27798/50000 [17:35:13<11:51:33,  1.92s/it]
Outer training loop:  56%|#####5    | 27799/50000 [17:35:15<11:38:29,  1.89s/it]
Outer training loop:  56%|#####5    | 27800/50000 [17:35:17<11:37:26,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  56%|#####5    | 27801/50000 [17:35:24<21:56:00,  3.56s/it]
Outer training loop:  56%|#####5    | 27802/50000 [17:35:26<19:00:49,  3.08s/it]
Outer training loop:  56%|#####5    | 27803/50000 [17:35:28<16:36:35,  2.69s/it]
Outer training loop:  56%|#####5    | 27804/50000 [17:35:30<15:07:10,  2.45s/it]
Outer training loop:  56%|#####5    | 27805/50000 [17:35:32<14:11:32,  2.30s/it]
Outer training loop:  56%|#####5    | 27806/50000 [17:35:34<13:16:13,  2.15s/it]
Outer training loop:  56%|#####5    | 27807/50000 [17:35:36<12:43:39,  2.06s/it]
Outer training loop:  56%|#####5    | 27808/50000 [17:35:38<12:49:43,  2.08s/it]
Outer training loop:  56%|#####5    | 27809/50000 [17:35:40<12:27:09,  2.02s/it]
Outer training loop:  56%|#####5    | 27810/50000 [17:35:41<11:49:57,  1.92s/it]
Outer training loop:  56%|#####5    | 27811/50000 [17:35:43<11:37:33,  1.89s/it]
Outer training loop:  56%|#####5    | 27812/50000 [17:35:45<11:54:15,  1.93s/it]
Outer training loop:  56%|#####5    | 27813/50000 [17:35:47<11:37:15,  1.89s/it]
Outer training loop:  56%|#####5    | 27814/50000 [17:35:49<11:28:14,  1.86s/it]
Outer training loop:  56%|#####5    | 27815/50000 [17:35:51<11:55:11,  1.93s/it]
Outer training loop:  56%|#####5    | 27816/50000 [17:35:53<11:34:34,  1.88s/it]
Outer training loop:  56%|#####5    | 27817/50000 [17:35:54<11:34:44,  1.88s/it]
Outer training loop:  56%|#####5    | 27818/50000 [17:35:56<11:36:53,  1.89s/it]
Outer training loop:  56%|#####5    | 27819/50000 [17:35:59<12:13:12,  1.98s/it]
Outer training loop:  56%|#####5    | 27820/50000 [17:36:00<12:02:39,  1.95s/it]
Outer training loop:  56%|#####5    | 27821/50000 [17:36:02<11:36:21,  1.88s/it]
Outer training loop:  56%|#####5    | 27822/50000 [17:36:04<11:44:05,  1.90s/it]
Outer training loop:  56%|#####5    | 27823/50000 [17:36:06<11:48:21,  1.92s/it]
Outer training loop:  56%|#####5    | 27824/50000 [17:36:08<11:36:19,  1.88s/it]
Outer training loop:  56%|#####5    | 27825/50000 [17:36:10<11:53:52,  1.93s/it]
Outer training loop:  56%|#####5    | 27826/50000 [17:36:12<11:37:45,  1.89s/it]
Outer training loop:  56%|#####5    | 27827/50000 [17:36:13<11:29:19,  1.87s/it]
Outer training loop:  56%|#####5    | 27828/50000 [17:36:15<11:22:16,  1.85s/it]
Outer training loop:  56%|#####5    | 27829/50000 [17:36:17<11:40:39,  1.90s/it]
Outer training loop:  56%|#####5    | 27830/50000 [17:36:19<11:29:57,  1.87s/it]
Outer training loop:  56%|#####5    | 27831/50000 [17:36:21<11:25:38,  1.86s/it]
Outer training loop:  56%|#####5    | 27832/50000 [17:36:23<11:44:50,  1.91s/it]
Outer training loop:  56%|#####5    | 27833/50000 [17:36:25<11:40:32,  1.90s/it]
Outer training loop:  56%|#####5    | 27834/50000 [17:36:27<11:55:37,  1.94s/it]
Outer training loop:  56%|#####5    | 27835/50000 [17:36:29<12:10:06,  1.98s/it]
Outer training loop:  56%|#####5    | 27836/50000 [17:36:31<11:38:27,  1.89s/it]
Outer training loop:  56%|#####5    | 27837/50000 [17:36:32<11:21:24,  1.84s/it]
Outer training loop:  56%|#####5    | 27838/50000 [17:36:34<11:28:56,  1.87s/it]
Outer training loop:  56%|#####5    | 27839/50000 [17:36:36<11:48:26,  1.92s/it]
Outer training loop:  56%|#####5    | 27840/50000 [17:36:38<11:38:29,  1.89s/it]
Outer training loop:  56%|#####5    | 27841/50000 [17:36:40<11:37:41,  1.89s/it]
Outer training loop:  56%|#####5    | 27842/50000 [17:36:42<11:59:12,  1.95s/it]
Outer training loop:  56%|#####5    | 27843/50000 [17:36:44<11:55:41,  1.94s/it]
Outer training loop:  56%|#####5    | 27844/50000 [17:36:46<11:37:40,  1.89s/it]
Outer training loop:  56%|#####5    | 27845/50000 [17:36:48<11:49:30,  1.92s/it]
Outer training loop:  56%|#####5    | 27846/50000 [17:36:50<11:51:07,  1.93s/it]
Outer training loop:  56%|#####5    | 27847/50000 [17:36:52<11:40:58,  1.90s/it]
Outer training loop:  56%|#####5    | 27848/50000 [17:36:53<11:31:35,  1.87s/it]
Outer training loop:  56%|#####5    | 27849/50000 [17:36:55<11:44:39,  1.91s/it]
Outer training loop:  56%|#####5    | 27850/50000 [17:36:57<11:26:51,  1.86s/it]
Outer training loop:  56%|#####5    | 27851/50000 [17:36:59<11:21:56,  1.85s/it]
Outer training loop:  56%|#####5    | 27852/50000 [17:37:01<11:36:08,  1.89s/it]
Outer training loop:  56%|#####5    | 27853/50000 [17:37:03<11:33:55,  1.88s/it]
Outer training loop:  56%|#####5    | 27854/50000 [17:37:05<11:18:50,  1.84s/it]
Outer training loop:  56%|#####5    | 27855/50000 [17:37:06<11:18:15,  1.84s/it]
Outer training loop:  56%|#####5    | 27856/50000 [17:37:08<11:39:48,  1.90s/it]
Outer training loop:  56%|#####5    | 27857/50000 [17:37:10<11:30:41,  1.87s/it]
Outer training loop:  56%|#####5    | 27858/50000 [17:37:12<11:19:04,  1.84s/it]
Outer training loop:  56%|#####5    | 27859/50000 [17:37:14<11:30:08,  1.87s/it]
Outer training loop:  56%|#####5    | 27860/50000 [17:37:16<11:24:49,  1.86s/it]
Outer training loop:  56%|#####5    | 27861/50000 [17:37:18<11:26:12,  1.86s/it]
Outer training loop:  56%|#####5    | 27862/50000 [17:37:20<11:44:34,  1.91s/it]
Outer training loop:  56%|#####5    | 27863/50000 [17:37:21<11:38:41,  1.89s/it]
Outer training loop:  56%|#####5    | 27864/50000 [17:37:23<11:25:10,  1.86s/it]
Outer training loop:  56%|#####5    | 27865/50000 [17:37:25<11:58:20,  1.95s/it]
Outer training loop:  56%|#####5    | 27866/50000 [17:37:27<11:42:04,  1.90s/it]
Outer training loop:  56%|#####5    | 27867/50000 [17:37:29<11:41:13,  1.90s/it]
Outer training loop:  56%|#####5    | 27868/50000 [17:37:31<11:27:00,  1.86s/it]
Outer training loop:  56%|#####5    | 27869/50000 [17:37:33<11:52:02,  1.93s/it]
Outer training loop:  56%|#####5    | 27870/50000 [17:37:35<11:44:26,  1.91s/it]
Outer training loop:  56%|#####5    | 27871/50000 [17:37:37<11:33:22,  1.88s/it]
Outer training loop:  56%|#####5    | 27872/50000 [17:37:39<11:49:58,  1.93s/it]
Outer training loop:  56%|#####5    | 27873/50000 [17:37:41<11:41:44,  1.90s/it]
Outer training loop:  56%|#####5    | 27874/50000 [17:37:42<11:31:20,  1.87s/it]
Outer training loop:  56%|#####5    | 27875/50000 [17:37:44<11:16:37,  1.83s/it]
Outer training loop:  56%|#####5    | 27876/50000 [17:37:46<11:29:42,  1.87s/it]
Outer training loop:  56%|#####5    | 27877/50000 [17:37:48<11:20:50,  1.85s/it]
Outer training loop:  56%|#####5    | 27878/50000 [17:37:50<11:23:41,  1.85s/it]
Outer training loop:  56%|#####5    | 27879/50000 [17:37:52<11:35:59,  1.89s/it]
Outer training loop:  56%|#####5    | 27880/50000 [17:37:53<11:25:17,  1.86s/it]
Outer training loop:  56%|#####5    | 27881/50000 [17:37:55<11:23:56,  1.86s/it]
Outer training loop:  56%|#####5    | 27882/50000 [17:37:57<11:20:07,  1.85s/it]
Outer training loop:  56%|#####5    | 27883/50000 [17:37:59<11:42:45,  1.91s/it]
Outer training loop:  56%|#####5    | 27884/50000 [17:38:01<11:54:36,  1.94s/it]
Outer training loop:  56%|#####5    | 27885/50000 [17:38:03<11:52:56,  1.93s/it]
Outer training loop:  56%|#####5    | 27886/50000 [17:38:05<12:03:36,  1.96s/it]
Outer training loop:  56%|#####5    | 27887/50000 [17:38:07<11:48:57,  1.92s/it]
Outer training loop:  56%|#####5    | 27888/50000 [17:38:09<11:42:06,  1.91s/it]
Outer training loop:  56%|#####5    | 27889/50000 [17:38:11<11:55:01,  1.94s/it]
Outer training loop:  56%|#####5    | 27890/50000 [17:38:13<11:48:38,  1.92s/it]
Outer training loop:  56%|#####5    | 27891/50000 [17:38:15<11:45:10,  1.91s/it]
Outer training loop:  56%|#####5    | 27892/50000 [17:38:16<11:33:41,  1.88s/it]
Outer training loop:  56%|#####5    | 27893/50000 [17:38:18<11:42:15,  1.91s/it]
Outer training loop:  56%|#####5    | 27894/50000 [17:38:20<11:38:25,  1.90s/it]
Outer training loop:  56%|#####5    | 27895/50000 [17:38:22<11:26:40,  1.86s/it]
Outer training loop:  56%|#####5    | 27896/50000 [17:38:24<11:30:12,  1.87s/it]
Outer training loop:  56%|#####5    | 27897/50000 [17:38:26<11:27:51,  1.87s/it]
Outer training loop:  56%|#####5    | 27898/50000 [17:38:28<11:35:57,  1.89s/it]
Outer training loop:  56%|#####5    | 27899/50000 [17:38:30<11:49:29,  1.93s/it]
Outer training loop:  56%|#####5    | 27900/50000 [17:38:32<11:31:32,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  56%|#####5    | 27901/50000 [17:38:39<21:39:49,  3.53s/it]
Outer training loop:  56%|#####5    | 27902/50000 [17:38:41<18:32:10,  3.02s/it]
Outer training loop:  56%|#####5    | 27903/50000 [17:38:43<16:37:04,  2.71s/it]
Outer training loop:  56%|#####5    | 27904/50000 [17:38:44<14:46:11,  2.41s/it]
Outer training loop:  56%|#####5    | 27905/50000 [17:38:46<13:37:14,  2.22s/it]
Outer training loop:  56%|#####5    | 27906/50000 [17:38:48<12:45:08,  2.08s/it]
Outer training loop:  56%|#####5    | 27907/50000 [17:38:50<12:51:18,  2.09s/it]
Outer training loop:  56%|#####5    | 27908/50000 [17:38:52<12:15:50,  2.00s/it]
Outer training loop:  56%|#####5    | 27909/50000 [17:38:54<12:01:38,  1.96s/it]
Outer training loop:  56%|#####5    | 27910/50000 [17:38:56<12:09:10,  1.98s/it]
Outer training loop:  56%|#####5    | 27911/50000 [17:38:58<11:55:21,  1.94s/it]
Outer training loop:  56%|#####5    | 27912/50000 [17:38:59<11:36:40,  1.89s/it]
Outer training loop:  56%|#####5    | 27913/50000 [17:39:01<11:45:14,  1.92s/it]
Outer training loop:  56%|#####5    | 27914/50000 [17:39:03<11:21:03,  1.85s/it]
Outer training loop:  56%|#####5    | 27915/50000 [17:39:05<11:21:38,  1.85s/it]
Outer training loop:  56%|#####5    | 27916/50000 [17:39:07<11:19:45,  1.85s/it]
Outer training loop:  56%|#####5    | 27917/50000 [17:39:09<11:28:54,  1.87s/it]
Outer training loop:  56%|#####5    | 27918/50000 [17:39:10<11:18:47,  1.84s/it]
Outer training loop:  56%|#####5    | 27919/50000 [17:39:12<10:58:41,  1.79s/it]
Outer training loop:  56%|#####5    | 27920/50000 [17:39:14<11:24:29,  1.86s/it]
Outer training loop:  56%|#####5    | 27921/50000 [17:39:16<11:18:46,  1.84s/it]
Outer training loop:  56%|#####5    | 27922/50000 [17:39:18<11:09:48,  1.82s/it]
Outer training loop:  56%|#####5    | 27923/50000 [17:39:20<11:32:01,  1.88s/it]
Outer training loop:  56%|#####5    | 27924/50000 [17:39:22<11:22:37,  1.86s/it]
Outer training loop:  56%|#####5    | 27925/50000 [17:39:23<11:04:13,  1.81s/it]
Outer training loop:  56%|#####5    | 27926/50000 [17:39:25<11:02:49,  1.80s/it]
Outer training loop:  56%|#####5    | 27927/50000 [17:39:27<11:35:45,  1.89s/it]
Outer training loop:  56%|#####5    | 27928/50000 [17:39:29<11:36:34,  1.89s/it]
Outer training loop:  56%|#####5    | 27929/50000 [17:39:31<11:26:43,  1.87s/it]
Outer training loop:  56%|#####5    | 27930/50000 [17:39:33<11:35:44,  1.89s/it]
Outer training loop:  56%|#####5    | 27931/50000 [17:39:35<11:57:53,  1.95s/it]
Outer training loop:  56%|#####5    | 27932/50000 [17:39:37<11:35:10,  1.89s/it]
Outer training loop:  56%|#####5    | 27933/50000 [17:39:38<11:12:09,  1.83s/it]
Outer training loop:  56%|#####5    | 27934/50000 [17:39:40<11:31:46,  1.88s/it]
Outer training loop:  56%|#####5    | 27935/50000 [17:39:42<11:17:41,  1.84s/it]
Outer training loop:  56%|#####5    | 27936/50000 [17:39:44<11:23:29,  1.86s/it]
Outer training loop:  56%|#####5    | 27937/50000 [17:39:46<11:27:08,  1.87s/it]
Outer training loop:  56%|#####5    | 27938/50000 [17:39:48<11:21:31,  1.85s/it]
Outer training loop:  56%|#####5    | 27939/50000 [17:39:49<11:21:30,  1.85s/it]
Outer training loop:  56%|#####5    | 27940/50000 [17:39:51<11:17:41,  1.84s/it]
Outer training loop:  56%|#####5    | 27941/50000 [17:39:53<11:35:57,  1.89s/it]
Outer training loop:  56%|#####5    | 27942/50000 [17:39:55<11:24:28,  1.86s/it]
Outer training loop:  56%|#####5    | 27943/50000 [17:39:57<11:23:14,  1.86s/it]
Outer training loop:  56%|#####5    | 27944/50000 [17:39:59<11:45:45,  1.92s/it]
Outer training loop:  56%|#####5    | 27945/50000 [17:40:01<11:28:11,  1.87s/it]
Outer training loop:  56%|#####5    | 27946/50000 [17:40:03<11:20:01,  1.85s/it]
Outer training loop:  56%|#####5    | 27947/50000 [17:40:05<11:46:29,  1.92s/it]
Outer training loop:  56%|#####5    | 27948/50000 [17:40:07<11:40:27,  1.91s/it]
Outer training loop:  56%|#####5    | 27949/50000 [17:40:08<11:31:57,  1.88s/it]
Outer training loop:  56%|#####5    | 27950/50000 [17:40:10<11:25:09,  1.86s/it]
Outer training loop:  56%|#####5    | 27951/50000 [17:40:12<11:43:49,  1.92s/it]
Outer training loop:  56%|#####5    | 27952/50000 [17:40:14<11:38:44,  1.90s/it]
Outer training loop:  56%|#####5    | 27953/50000 [17:40:16<11:34:23,  1.89s/it]
Outer training loop:  56%|#####5    | 27954/50000 [17:40:18<11:50:44,  1.93s/it]
Outer training loop:  56%|#####5    | 27955/50000 [17:40:20<11:40:57,  1.91s/it]
Outer training loop:  56%|#####5    | 27956/50000 [17:40:22<11:35:03,  1.89s/it]
Outer training loop:  56%|#####5    | 27957/50000 [17:40:24<11:46:27,  1.92s/it]
Outer training loop:  56%|#####5    | 27958/50000 [17:40:25<11:30:17,  1.88s/it]
Outer training loop:  56%|#####5    | 27959/50000 [17:40:27<11:18:46,  1.85s/it]
Outer training loop:  56%|#####5    | 27960/50000 [17:40:29<11:38:00,  1.90s/it]
Outer training loop:  56%|#####5    | 27961/50000 [17:40:31<11:37:33,  1.90s/it]
Outer training loop:  56%|#####5    | 27962/50000 [17:40:33<11:15:48,  1.84s/it]
Outer training loop:  56%|#####5    | 27963/50000 [17:40:35<11:06:12,  1.81s/it]
Outer training loop:  56%|#####5    | 27964/50000 [17:40:37<11:32:54,  1.89s/it]
Outer training loop:  56%|#####5    | 27965/50000 [17:40:39<11:28:23,  1.87s/it]
Outer training loop:  56%|#####5    | 27966/50000 [17:40:40<11:28:05,  1.87s/it]
Outer training loop:  56%|#####5    | 27967/50000 [17:40:42<11:46:24,  1.92s/it]
Outer training loop:  56%|#####5    | 27968/50000 [17:40:44<11:33:21,  1.89s/it]
Outer training loop:  56%|#####5    | 27969/50000 [17:40:46<11:16:24,  1.84s/it]
Outer training loop:  56%|#####5    | 27970/50000 [17:40:48<11:32:02,  1.88s/it]
Outer training loop:  56%|#####5    | 27971/50000 [17:40:50<11:11:47,  1.83s/it]
Outer training loop:  56%|#####5    | 27972/50000 [17:40:51<11:07:19,  1.82s/it]
Outer training loop:  56%|#####5    | 27973/50000 [17:40:53<11:04:49,  1.81s/it]
Outer training loop:  56%|#####5    | 27974/50000 [17:40:55<11:23:03,  1.86s/it]
Outer training loop:  56%|#####5    | 27975/50000 [17:40:57<11:19:56,  1.85s/it]
Outer training loop:  56%|#####5    | 27976/50000 [17:40:59<11:24:32,  1.86s/it]
Outer training loop:  56%|#####5    | 27977/50000 [17:41:01<11:42:11,  1.91s/it]
Outer training loop:  56%|#####5    | 27978/50000 [17:41:03<11:22:45,  1.86s/it]
Outer training loop:  56%|#####5    | 27979/50000 [17:41:05<11:24:14,  1.86s/it]
Outer training loop:  56%|#####5    | 27980/50000 [17:41:07<11:36:19,  1.90s/it]
Outer training loop:  56%|#####5    | 27981/50000 [17:41:09<11:48:16,  1.93s/it]
Outer training loop:  56%|#####5    | 27982/50000 [17:41:10<11:26:07,  1.87s/it]
Outer training loop:  56%|#####5    | 27983/50000 [17:41:12<11:16:00,  1.84s/it]
Outer training loop:  56%|#####5    | 27984/50000 [17:41:14<11:48:15,  1.93s/it]
Outer training loop:  56%|#####5    | 27985/50000 [17:41:16<11:47:24,  1.93s/it]
Outer training loop:  56%|#####5    | 27986/50000 [17:41:18<11:40:20,  1.91s/it]
Outer training loop:  56%|#####5    | 27987/50000 [17:41:20<11:52:11,  1.94s/it]
Outer training loop:  56%|#####5    | 27988/50000 [17:41:22<11:44:24,  1.92s/it]
Outer training loop:  56%|#####5    | 27989/50000 [17:41:24<11:39:19,  1.91s/it]
Outer training loop:  56%|#####5    | 27990/50000 [17:41:26<11:40:05,  1.91s/it]
Outer training loop:  56%|#####5    | 27991/50000 [17:41:28<11:59:46,  1.96s/it]
Outer training loop:  56%|#####5    | 27992/50000 [17:41:30<11:50:37,  1.94s/it]
Outer training loop:  56%|#####5    | 27993/50000 [17:41:31<11:35:04,  1.90s/it]
Outer training loop:  56%|#####5    | 27994/50000 [17:41:33<11:41:44,  1.91s/it]
Outer training loop:  56%|#####5    | 27995/50000 [17:41:35<11:36:07,  1.90s/it]
Outer training loop:  56%|#####5    | 27996/50000 [17:41:37<11:27:57,  1.88s/it]
Outer training loop:  56%|#####5    | 27997/50000 [17:41:39<11:35:17,  1.90s/it]
Outer training loop:  56%|#####5    | 27998/50000 [17:41:41<11:24:44,  1.87s/it]
Outer training loop:  56%|#####5    | 27999/50000 [17:41:43<11:32:00,  1.89s/it]
Outer training loop:  56%|#####6    | 28000/50000 [17:41:45<11:28:37,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.77s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:25,  1.82s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:26,  1.86s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:24,  1.85s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:21,  1.83s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:21,  1.86s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:22,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:20,  1.90s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:14,  1.85s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:15,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:07,  1.86s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:05,  1.85s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:05,  1.87s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:02,  1.85s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:59,  1.85s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:01,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.84s/it]

Evaluating policy:  28%|##7       | 23/83 [00:42<01:50,  1.84s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:45,  1.85s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:42,  1.86s/it]

Evaluating policy:  35%|###4      | 29/83 [00:53<01:37,  1.81s/it]

Evaluating policy:  36%|###6      | 30/83 [00:55<01:35,  1.81s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:41,  1.96s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:27,  1.81s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:25,  1.82s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:10<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:12<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:14<01:18,  1.82s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:17,  1.95s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:23<01:10,  1.85s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:25<01:07,  1.82s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:27<01:05,  1.83s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:29<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:02,  1.84s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<01:04,  1.97s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:02,  1.95s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<01:00,  1.94s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.88s/it]

Evaluating policy:  65%|######5   | 54/83 [01:40<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:42<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:49,  1.98s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:44,  1.92s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:55<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:57<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:33,  1.98s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:31,  1.94s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:28,  1.90s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.84s/it]

Evaluating policy:  84%|########4 | 70/83 [02:10<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:19,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:17,  1.99s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.97s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.95s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.96s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.93s/it]

Evaluating policy:  98%|#########7| 81/83 [02:31<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<03:07,  2.28s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:53,  2.14s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:40,  2.01s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:37,  1.99s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:34,  1.98s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:30,  1.95s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:29,  1.99s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.95s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:15,  1.91s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:15,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:09,  1.96s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:05,  1.93s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:59,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:56,  1.85s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:52,  1.81s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:45,  1.83s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:43,  1.82s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.88s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:42,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:36,  1.82s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:34,  1.82s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:36,  1.93s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:33,  1.90s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.86s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:26,  1.97s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:24,  1.97s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:21,  1.93s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:19,  1.93s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:17,  2.04s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:13,  1.99s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:10,  1.96s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.89s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<01:01,  1.97s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:58,  1.94s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:56,  1.93s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:54,  1.93s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:52,  1.93s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.90s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.90s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:48,  2.03s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:45,  1.97s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:41,  1.91s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:40,  1.91s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:38,  1.90s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.88s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.86s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:29,  1.99s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:27,  1.96s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.91s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.85s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.87s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:12,  1.84s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.98s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.89s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.82s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.86s/it]

Outer training loop:  56%|#####6    | 28001/50000 [17:47:06<596:19:34, 97.59s/it]
Outer training loop:  56%|#####6    | 28002/50000 [17:47:07<420:39:30, 68.84s/it]
Outer training loop:  56%|#####6    | 28003/50000 [17:47:09<297:41:36, 48.72s/it]
Outer training loop:  56%|#####6    | 28004/50000 [17:47:12<212:58:23, 34.86s/it]
Outer training loop:  56%|#####6    | 28005/50000 [17:47:13<152:24:16, 24.94s/it]
Outer training loop:  56%|#####6    | 28006/50000 [17:47:15<109:53:57, 17.99s/it]
Outer training loop:  56%|#####6    | 28007/50000 [17:47:17<80:08:30, 13.12s/it] 
Outer training loop:  56%|#####6    | 28008/50000 [17:47:19<59:55:14,  9.81s/it]
Outer training loop:  56%|#####6    | 28009/50000 [17:47:21<45:29:01,  7.45s/it]
Outer training loop:  56%|#####6    | 28010/50000 [17:47:23<35:06:32,  5.75s/it]
Outer training loop:  56%|#####6    | 28011/50000 [17:47:25<28:10:40,  4.61s/it]
Outer training loop:  56%|#####6    | 28012/50000 [17:47:27<23:28:28,  3.84s/it]
Outer training loop:  56%|#####6    | 28013/50000 [17:47:28<19:39:04,  3.22s/it]
Outer training loop:  56%|#####6    | 28014/50000 [17:47:30<17:08:05,  2.81s/it]
Outer training loop:  56%|#####6    | 28015/50000 [17:47:32<15:34:08,  2.55s/it]
Outer training loop:  56%|#####6    | 28016/50000 [17:47:34<14:13:53,  2.33s/it]
Outer training loop:  56%|#####6    | 28017/50000 [17:47:36<13:23:03,  2.19s/it]
Outer training loop:  56%|#####6    | 28018/50000 [17:47:38<13:12:06,  2.16s/it]
Outer training loop:  56%|#####6    | 28019/50000 [17:47:40<12:31:33,  2.05s/it]
Outer training loop:  56%|#####6    | 28020/50000 [17:47:42<11:55:28,  1.95s/it]
Outer training loop:  56%|#####6    | 28021/50000 [17:47:44<12:11:45,  2.00s/it]
Outer training loop:  56%|#####6    | 28022/50000 [17:47:46<11:59:46,  1.96s/it]
Outer training loop:  56%|#####6    | 28023/50000 [17:47:48<11:59:26,  1.96s/it]
Outer training loop:  56%|#####6    | 28024/50000 [17:47:49<11:50:00,  1.94s/it]
Outer training loop:  56%|#####6    | 28025/50000 [17:47:51<12:04:54,  1.98s/it]
Outer training loop:  56%|#####6    | 28026/50000 [17:47:53<11:55:49,  1.95s/it]
Outer training loop:  56%|#####6    | 28027/50000 [17:47:55<11:46:54,  1.93s/it]
Outer training loop:  56%|#####6    | 28028/50000 [17:47:57<11:51:24,  1.94s/it]
Outer training loop:  56%|#####6    | 28029/50000 [17:47:59<11:39:18,  1.91s/it]
Outer training loop:  56%|#####6    | 28030/50000 [17:48:01<11:23:57,  1.87s/it]
Outer training loop:  56%|#####6    | 28031/50000 [17:48:03<11:29:33,  1.88s/it]
Outer training loop:  56%|#####6    | 28032/50000 [17:48:05<11:20:58,  1.86s/it]
Outer training loop:  56%|#####6    | 28033/50000 [17:48:06<11:14:47,  1.84s/it]
Outer training loop:  56%|#####6    | 28034/50000 [17:48:08<11:40:16,  1.91s/it]
Outer training loop:  56%|#####6    | 28035/50000 [17:48:10<11:45:19,  1.93s/it]
Outer training loop:  56%|#####6    | 28036/50000 [17:48:12<11:46:10,  1.93s/it]
Outer training loop:  56%|#####6    | 28037/50000 [17:48:14<11:52:01,  1.95s/it]
Outer training loop:  56%|#####6    | 28038/50000 [17:48:16<11:32:13,  1.89s/it]
Outer training loop:  56%|#####6    | 28039/50000 [17:48:18<11:12:19,  1.84s/it]
Outer training loop:  56%|#####6    | 28040/50000 [17:48:20<11:15:19,  1.85s/it]
Outer training loop:  56%|#####6    | 28041/50000 [17:48:22<11:32:54,  1.89s/it]
Outer training loop:  56%|#####6    | 28042/50000 [17:48:23<11:20:35,  1.86s/it]
Outer training loop:  56%|#####6    | 28043/50000 [17:48:25<11:25:45,  1.87s/it]
Outer training loop:  56%|#####6    | 28044/50000 [17:48:27<11:42:52,  1.92s/it]
Outer training loop:  56%|#####6    | 28045/50000 [17:48:29<11:34:12,  1.90s/it]
Outer training loop:  56%|#####6    | 28046/50000 [17:48:31<11:26:08,  1.88s/it]
Outer training loop:  56%|#####6    | 28047/50000 [17:48:33<11:32:26,  1.89s/it]
Outer training loop:  56%|#####6    | 28048/50000 [17:48:35<11:56:54,  1.96s/it]
Outer training loop:  56%|#####6    | 28049/50000 [17:48:37<11:47:33,  1.93s/it]
Outer training loop:  56%|#####6    | 28050/50000 [17:48:39<11:41:20,  1.92s/it]
Outer training loop:  56%|#####6    | 28051/50000 [17:48:41<12:08:02,  1.99s/it]
Outer training loop:  56%|#####6    | 28052/50000 [17:48:43<11:50:53,  1.94s/it]
Outer training loop:  56%|#####6    | 28053/50000 [17:48:45<11:46:52,  1.93s/it]
Outer training loop:  56%|#####6    | 28054/50000 [17:48:47<11:53:40,  1.95s/it]
Outer training loop:  56%|#####6    | 28055/50000 [17:48:49<11:37:48,  1.91s/it]
Outer training loop:  56%|#####6    | 28056/50000 [17:48:50<11:18:43,  1.86s/it]
Outer training loop:  56%|#####6    | 28057/50000 [17:48:52<11:08:54,  1.83s/it]
Outer training loop:  56%|#####6    | 28058/50000 [17:48:54<11:20:51,  1.86s/it]
Outer training loop:  56%|#####6    | 28059/50000 [17:48:56<11:14:56,  1.85s/it]
Outer training loop:  56%|#####6    | 28060/50000 [17:48:58<11:20:30,  1.86s/it]
Outer training loop:  56%|#####6    | 28061/50000 [17:49:00<11:48:51,  1.94s/it]
Outer training loop:  56%|#####6    | 28062/50000 [17:49:02<11:50:29,  1.94s/it]
Outer training loop:  56%|#####6    | 28063/50000 [17:49:03<11:24:13,  1.87s/it]
Outer training loop:  56%|#####6    | 28064/50000 [17:49:05<11:31:55,  1.89s/it]
Outer training loop:  56%|#####6    | 28065/50000 [17:49:08<11:57:16,  1.96s/it]
Outer training loop:  56%|#####6    | 28066/50000 [17:49:09<11:31:25,  1.89s/it]
Outer training loop:  56%|#####6    | 28067/50000 [17:49:11<11:35:44,  1.90s/it]
Outer training loop:  56%|#####6    | 28068/50000 [17:49:13<12:06:30,  1.99s/it]
Outer training loop:  56%|#####6    | 28069/50000 [17:49:15<11:45:57,  1.93s/it]
Outer training loop:  56%|#####6    | 28070/50000 [17:49:17<11:38:55,  1.91s/it]
Outer training loop:  56%|#####6    | 28071/50000 [17:49:19<11:44:55,  1.93s/it]
Outer training loop:  56%|#####6    | 28072/50000 [17:49:21<11:39:41,  1.91s/it]
Outer training loop:  56%|#####6    | 28073/50000 [17:49:23<11:25:43,  1.88s/it]
Outer training loop:  56%|#####6    | 28074/50000 [17:49:25<11:51:05,  1.95s/it]
Outer training loop:  56%|#####6    | 28075/50000 [17:49:27<11:44:22,  1.93s/it]
Outer training loop:  56%|#####6    | 28076/50000 [17:49:29<11:41:42,  1.92s/it]
Outer training loop:  56%|#####6    | 28077/50000 [17:49:31<12:02:03,  1.98s/it]
Outer training loop:  56%|#####6    | 28078/50000 [17:49:32<11:38:32,  1.91s/it]
Outer training loop:  56%|#####6    | 28079/50000 [17:49:34<11:21:37,  1.87s/it]
Outer training loop:  56%|#####6    | 28080/50000 [17:49:36<11:16:56,  1.85s/it]
Outer training loop:  56%|#####6    | 28081/50000 [17:49:38<11:34:24,  1.90s/it]
Outer training loop:  56%|#####6    | 28082/50000 [17:49:40<11:34:36,  1.90s/it]
Outer training loop:  56%|#####6    | 28083/50000 [17:49:42<11:15:57,  1.85s/it]
Outer training loop:  56%|#####6    | 28084/50000 [17:49:44<11:33:24,  1.90s/it]
Outer training loop:  56%|#####6    | 28085/50000 [17:49:45<11:21:46,  1.87s/it]
Outer training loop:  56%|#####6    | 28086/50000 [17:49:47<11:17:38,  1.86s/it]
Outer training loop:  56%|#####6    | 28087/50000 [17:49:49<11:27:40,  1.88s/it]
Outer training loop:  56%|#####6    | 28088/50000 [17:49:51<11:35:46,  1.91s/it]
Outer training loop:  56%|#####6    | 28089/50000 [17:49:53<11:24:19,  1.87s/it]
Outer training loop:  56%|#####6    | 28090/50000 [17:49:55<11:25:49,  1.88s/it]
Outer training loop:  56%|#####6    | 28091/50000 [17:49:57<11:37:05,  1.91s/it]
Outer training loop:  56%|#####6    | 28092/50000 [17:49:59<11:33:13,  1.90s/it]
Outer training loop:  56%|#####6    | 28093/50000 [17:50:01<11:26:35,  1.88s/it]
Outer training loop:  56%|#####6    | 28094/50000 [17:50:03<11:53:42,  1.95s/it]
Outer training loop:  56%|#####6    | 28095/50000 [17:50:05<11:43:53,  1.93s/it]
Outer training loop:  56%|#####6    | 28096/50000 [17:50:06<11:43:12,  1.93s/it]
Outer training loop:  56%|#####6    | 28097/50000 [17:50:08<11:38:21,  1.91s/it]
Outer training loop:  56%|#####6    | 28098/50000 [17:50:10<11:48:21,  1.94s/it]
Outer training loop:  56%|#####6    | 28099/50000 [17:50:12<11:36:00,  1.91s/it]
Outer training loop:  56%|#####6    | 28100/50000 [17:50:14<11:38:42,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  56%|#####6    | 28101/50000 [17:50:22<21:37:59,  3.56s/it]
Outer training loop:  56%|#####6    | 28102/50000 [17:50:23<18:20:59,  3.02s/it]
Outer training loop:  56%|#####6    | 28103/50000 [17:50:25<16:12:28,  2.66s/it]
Outer training loop:  56%|#####6    | 28104/50000 [17:50:27<14:30:26,  2.39s/it]
Outer training loop:  56%|#####6    | 28105/50000 [17:50:29<14:04:45,  2.31s/it]
Outer training loop:  56%|#####6    | 28106/50000 [17:50:31<12:58:37,  2.13s/it]
Outer training loop:  56%|#####6    | 28107/50000 [17:50:32<12:19:42,  2.03s/it]
Outer training loop:  56%|#####6    | 28108/50000 [17:50:35<12:21:53,  2.03s/it]
Outer training loop:  56%|#####6    | 28109/50000 [17:50:37<12:24:51,  2.04s/it]
Outer training loop:  56%|#####6    | 28110/50000 [17:50:38<11:54:17,  1.96s/it]
Outer training loop:  56%|#####6    | 28111/50000 [17:50:40<11:52:43,  1.95s/it]
Outer training loop:  56%|#####6    | 28112/50000 [17:50:42<11:55:14,  1.96s/it]
Outer training loop:  56%|#####6    | 28113/50000 [17:50:44<11:42:48,  1.93s/it]
Outer training loop:  56%|#####6    | 28114/50000 [17:50:46<11:31:24,  1.90s/it]
Outer training loop:  56%|#####6    | 28115/50000 [17:50:48<11:32:46,  1.90s/it]
Outer training loop:  56%|#####6    | 28116/50000 [17:50:50<11:30:36,  1.89s/it]
Outer training loop:  56%|#####6    | 28117/50000 [17:50:52<11:22:48,  1.87s/it]
Outer training loop:  56%|#####6    | 28118/50000 [17:50:53<11:23:00,  1.87s/it]
Outer training loop:  56%|#####6    | 28119/50000 [17:50:55<11:39:55,  1.92s/it]
Outer training loop:  56%|#####6    | 28120/50000 [17:50:57<11:37:00,  1.91s/it]
Outer training loop:  56%|#####6    | 28121/50000 [17:50:59<11:24:56,  1.88s/it]
Outer training loop:  56%|#####6    | 28122/50000 [17:51:01<11:37:42,  1.91s/it]
Outer training loop:  56%|#####6    | 28123/50000 [17:51:03<11:26:26,  1.88s/it]
Outer training loop:  56%|#####6    | 28124/50000 [17:51:05<11:30:54,  1.89s/it]
Outer training loop:  56%|#####6    | 28125/50000 [17:51:07<11:44:33,  1.93s/it]
Outer training loop:  56%|#####6    | 28126/50000 [17:51:09<11:22:52,  1.87s/it]
Outer training loop:  56%|#####6    | 28127/50000 [17:51:11<11:31:51,  1.90s/it]
Outer training loop:  56%|#####6    | 28128/50000 [17:51:12<11:29:09,  1.89s/it]
Outer training loop:  56%|#####6    | 28129/50000 [17:51:14<11:37:03,  1.91s/it]
Outer training loop:  56%|#####6    | 28130/50000 [17:51:16<11:13:03,  1.85s/it]
Outer training loop:  56%|#####6    | 28131/50000 [17:51:18<11:07:43,  1.83s/it]
Outer training loop:  56%|#####6    | 28132/50000 [17:51:20<11:35:17,  1.91s/it]
Outer training loop:  56%|#####6    | 28133/50000 [17:51:22<11:29:52,  1.89s/it]
Outer training loop:  56%|#####6    | 28134/50000 [17:51:24<11:15:08,  1.85s/it]
Outer training loop:  56%|#####6    | 28135/50000 [17:51:26<11:35:35,  1.91s/it]
Outer training loop:  56%|#####6    | 28136/50000 [17:51:27<11:26:32,  1.88s/it]
Outer training loop:  56%|#####6    | 28137/50000 [17:51:29<11:19:01,  1.86s/it]
Outer training loop:  56%|#####6    | 28138/50000 [17:51:31<11:04:39,  1.82s/it]
Outer training loop:  56%|#####6    | 28139/50000 [17:51:33<11:31:43,  1.90s/it]
Outer training loop:  56%|#####6    | 28140/50000 [17:51:35<11:25:33,  1.88s/it]
Outer training loop:  56%|#####6    | 28141/50000 [17:51:37<11:19:27,  1.86s/it]
Outer training loop:  56%|#####6    | 28142/50000 [17:51:39<11:28:43,  1.89s/it]
Outer training loop:  56%|#####6    | 28143/50000 [17:51:41<11:26:55,  1.89s/it]
Outer training loop:  56%|#####6    | 28144/50000 [17:51:43<11:29:32,  1.89s/it]
Outer training loop:  56%|#####6    | 28145/50000 [17:51:44<11:20:41,  1.87s/it]
Outer training loop:  56%|#####6    | 28146/50000 [17:51:46<11:44:15,  1.93s/it]
Outer training loop:  56%|#####6    | 28147/50000 [17:51:48<11:31:12,  1.90s/it]
Outer training loop:  56%|#####6    | 28148/50000 [17:51:50<11:17:08,  1.86s/it]
Outer training loop:  56%|#####6    | 28149/50000 [17:51:52<11:34:04,  1.91s/it]
Outer training loop:  56%|#####6    | 28150/50000 [17:51:54<11:25:36,  1.88s/it]
Outer training loop:  56%|#####6    | 28151/50000 [17:51:56<11:28:26,  1.89s/it]
Outer training loop:  56%|#####6    | 28152/50000 [17:51:58<11:48:07,  1.94s/it]
Outer training loop:  56%|#####6    | 28153/50000 [17:52:00<11:29:13,  1.89s/it]
Outer training loop:  56%|#####6    | 28154/50000 [17:52:01<11:28:48,  1.89s/it]
Outer training loop:  56%|#####6    | 28155/50000 [17:52:03<11:18:47,  1.86s/it]
Outer training loop:  56%|#####6    | 28156/50000 [17:52:05<11:41:36,  1.93s/it]
Outer training loop:  56%|#####6    | 28157/50000 [17:52:07<11:26:54,  1.89s/it]
Outer training loop:  56%|#####6    | 28158/50000 [17:52:09<11:18:18,  1.86s/it]
Outer training loop:  56%|#####6    | 28159/50000 [17:52:11<11:52:07,  1.96s/it]
Outer training loop:  56%|#####6    | 28160/50000 [17:52:13<11:49:02,  1.95s/it]
Outer training loop:  56%|#####6    | 28161/50000 [17:52:15<11:35:10,  1.91s/it]
Outer training loop:  56%|#####6    | 28162/50000 [17:52:17<11:53:16,  1.96s/it]
Outer training loop:  56%|#####6    | 28163/50000 [17:52:19<11:45:02,  1.94s/it]
Outer training loop:  56%|#####6    | 28164/50000 [17:52:21<11:35:27,  1.91s/it]
Outer training loop:  56%|#####6    | 28165/50000 [17:52:23<11:56:42,  1.97s/it]
Outer training loop:  56%|#####6    | 28166/50000 [17:52:25<11:32:53,  1.90s/it]
Outer training loop:  56%|#####6    | 28167/50000 [17:52:26<11:28:45,  1.89s/it]
Outer training loop:  56%|#####6    | 28168/50000 [17:52:28<11:24:34,  1.88s/it]
Outer training loop:  56%|#####6    | 28169/50000 [17:52:30<11:38:09,  1.92s/it]
Outer training loop:  56%|#####6    | 28170/50000 [17:52:32<11:20:20,  1.87s/it]
Outer training loop:  56%|#####6    | 28171/50000 [17:52:34<11:20:31,  1.87s/it]
Outer training loop:  56%|#####6    | 28172/50000 [17:52:36<11:39:21,  1.92s/it]
Outer training loop:  56%|#####6    | 28173/50000 [17:52:38<11:27:10,  1.89s/it]
Outer training loop:  56%|#####6    | 28174/50000 [17:52:40<11:22:11,  1.88s/it]
Outer training loop:  56%|#####6    | 28175/50000 [17:52:42<11:34:45,  1.91s/it]
Outer training loop:  56%|#####6    | 28176/50000 [17:52:43<11:23:00,  1.88s/it]
Outer training loop:  56%|#####6    | 28177/50000 [17:52:45<11:08:27,  1.84s/it]
Outer training loop:  56%|#####6    | 28178/50000 [17:52:47<10:59:00,  1.81s/it]
Outer training loop:  56%|#####6    | 28179/50000 [17:52:49<11:22:22,  1.88s/it]
Outer training loop:  56%|#####6    | 28180/50000 [17:52:51<11:09:59,  1.84s/it]
Outer training loop:  56%|#####6    | 28181/50000 [17:52:52<11:01:41,  1.82s/it]
Outer training loop:  56%|#####6    | 28182/50000 [17:52:55<11:33:46,  1.91s/it]
Outer training loop:  56%|#####6    | 28183/50000 [17:52:56<11:12:58,  1.85s/it]
Outer training loop:  56%|#####6    | 28184/50000 [17:52:58<11:18:06,  1.86s/it]
Outer training loop:  56%|#####6    | 28185/50000 [17:53:00<11:04:43,  1.83s/it]
Outer training loop:  56%|#####6    | 28186/50000 [17:53:02<11:17:58,  1.86s/it]
Outer training loop:  56%|#####6    | 28187/50000 [17:53:04<11:28:32,  1.89s/it]
Outer training loop:  56%|#####6    | 28188/50000 [17:53:06<11:19:27,  1.87s/it]
Outer training loop:  56%|#####6    | 28189/50000 [17:53:08<11:23:13,  1.88s/it]
Outer training loop:  56%|#####6    | 28190/50000 [17:53:09<11:19:01,  1.87s/it]
Outer training loop:  56%|#####6    | 28191/50000 [17:53:11<10:59:23,  1.81s/it]
Outer training loop:  56%|#####6    | 28192/50000 [17:53:13<11:07:54,  1.84s/it]
Outer training loop:  56%|#####6    | 28193/50000 [17:53:15<11:19:29,  1.87s/it]
Outer training loop:  56%|#####6    | 28194/50000 [17:53:17<11:16:12,  1.86s/it]
Outer training loop:  56%|#####6    | 28195/50000 [17:53:19<11:13:48,  1.85s/it]
Outer training loop:  56%|#####6    | 28196/50000 [17:53:21<11:30:58,  1.90s/it]
Outer training loop:  56%|#####6    | 28197/50000 [17:53:22<11:15:26,  1.86s/it]
Outer training loop:  56%|#####6    | 28198/50000 [17:53:24<11:11:02,  1.85s/it]
Outer training loop:  56%|#####6    | 28199/50000 [17:53:26<11:10:10,  1.84s/it]
Outer training loop:  56%|#####6    | 28200/50000 [17:53:28<11:38:25,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  56%|#####6    | 28201/50000 [17:53:35<21:18:42,  3.52s/it]
Outer training loop:  56%|#####6    | 28202/50000 [17:53:37<18:09:57,  3.00s/it]
Outer training loop:  56%|#####6    | 28203/50000 [17:53:39<15:51:38,  2.62s/it]
Outer training loop:  56%|#####6    | 28204/50000 [17:53:41<14:43:33,  2.43s/it]
Outer training loop:  56%|#####6    | 28205/50000 [17:53:43<13:45:37,  2.27s/it]
Outer training loop:  56%|#####6    | 28206/50000 [17:53:45<13:13:50,  2.19s/it]
Outer training loop:  56%|#####6    | 28207/50000 [17:53:47<12:52:58,  2.13s/it]
Outer training loop:  56%|#####6    | 28208/50000 [17:53:49<12:10:41,  2.01s/it]
Outer training loop:  56%|#####6    | 28209/50000 [17:53:50<11:46:58,  1.95s/it]
Outer training loop:  56%|#####6    | 28210/50000 [17:53:52<11:27:13,  1.89s/it]
Outer training loop:  56%|#####6    | 28211/50000 [17:53:54<11:50:58,  1.96s/it]
Outer training loop:  56%|#####6    | 28212/50000 [17:53:56<11:37:53,  1.92s/it]
Outer training loop:  56%|#####6    | 28213/50000 [17:53:58<11:19:14,  1.87s/it]
Outer training loop:  56%|#####6    | 28214/50000 [17:54:00<11:34:55,  1.91s/it]
Outer training loop:  56%|#####6    | 28215/50000 [17:54:01<11:10:03,  1.85s/it]
Outer training loop:  56%|#####6    | 28216/50000 [17:54:03<11:07:03,  1.84s/it]
Outer training loop:  56%|#####6    | 28217/50000 [17:54:05<11:27:09,  1.89s/it]
Outer training loop:  56%|#####6    | 28218/50000 [17:54:07<11:17:39,  1.87s/it]
Outer training loop:  56%|#####6    | 28219/50000 [17:54:09<11:18:58,  1.87s/it]
Outer training loop:  56%|#####6    | 28220/50000 [17:54:11<11:18:29,  1.87s/it]
Outer training loop:  56%|#####6    | 28221/50000 [17:54:13<11:36:37,  1.92s/it]
Outer training loop:  56%|#####6    | 28222/50000 [17:54:15<11:24:22,  1.89s/it]
Outer training loop:  56%|#####6    | 28223/50000 [17:54:17<11:21:02,  1.88s/it]
Outer training loop:  56%|#####6    | 28224/50000 [17:54:18<11:25:35,  1.89s/it]
Outer training loop:  56%|#####6    | 28225/50000 [17:54:20<11:31:09,  1.90s/it]
Outer training loop:  56%|#####6    | 28226/50000 [17:54:22<11:28:04,  1.90s/it]
Outer training loop:  56%|#####6    | 28227/50000 [17:54:24<11:36:11,  1.92s/it]
Outer training loop:  56%|#####6    | 28228/50000 [17:54:26<11:36:58,  1.92s/it]
Outer training loop:  56%|#####6    | 28229/50000 [17:54:28<11:30:51,  1.90s/it]
Outer training loop:  56%|#####6    | 28230/50000 [17:54:30<11:24:15,  1.89s/it]
Outer training loop:  56%|#####6    | 28231/50000 [17:54:32<11:33:42,  1.91s/it]
Outer training loop:  56%|#####6    | 28232/50000 [17:54:34<11:27:36,  1.90s/it]
Outer training loop:  56%|#####6    | 28233/50000 [17:54:36<11:27:48,  1.90s/it]
Outer training loop:  56%|#####6    | 28234/50000 [17:54:38<11:37:41,  1.92s/it]
Outer training loop:  56%|#####6    | 28235/50000 [17:54:40<11:35:37,  1.92s/it]
Outer training loop:  56%|#####6    | 28236/50000 [17:54:41<11:28:57,  1.90s/it]
Outer training loop:  56%|#####6    | 28237/50000 [17:54:44<12:03:53,  2.00s/it]
Outer training loop:  56%|#####6    | 28238/50000 [17:54:45<11:34:25,  1.91s/it]
Outer training loop:  56%|#####6    | 28239/50000 [17:54:47<11:24:19,  1.89s/it]
Outer training loop:  56%|#####6    | 28240/50000 [17:54:49<11:31:30,  1.91s/it]
Outer training loop:  56%|#####6    | 28241/50000 [17:54:51<11:39:40,  1.93s/it]
Outer training loop:  56%|#####6    | 28242/50000 [17:54:53<11:15:09,  1.86s/it]
Outer training loop:  56%|#####6    | 28243/50000 [17:54:55<11:07:38,  1.84s/it]
Outer training loop:  56%|#####6    | 28244/50000 [17:54:56<11:12:06,  1.85s/it]
Outer training loop:  56%|#####6    | 28245/50000 [17:54:58<11:04:33,  1.83s/it]
Outer training loop:  56%|#####6    | 28246/50000 [17:55:00<11:04:23,  1.83s/it]
Outer training loop:  56%|#####6    | 28247/50000 [17:55:02<10:58:13,  1.82s/it]
Outer training loop:  56%|#####6    | 28248/50000 [17:55:04<11:17:23,  1.87s/it]
Outer training loop:  56%|#####6    | 28249/50000 [17:55:06<11:08:07,  1.84s/it]
Outer training loop:  56%|#####6    | 28250/50000 [17:55:07<10:51:29,  1.80s/it]
Outer training loop:  57%|#####6    | 28251/50000 [17:55:09<11:12:07,  1.85s/it]
Outer training loop:  57%|#####6    | 28252/50000 [17:55:11<11:10:53,  1.85s/it]
Outer training loop:  57%|#####6    | 28253/50000 [17:55:13<11:03:40,  1.83s/it]
Outer training loop:  57%|#####6    | 28254/50000 [17:55:15<11:14:30,  1.86s/it]
Outer training loop:  57%|#####6    | 28255/50000 [17:55:17<11:02:56,  1.83s/it]
Outer training loop:  57%|#####6    | 28256/50000 [17:55:19<11:24:15,  1.89s/it]
Outer training loop:  57%|#####6    | 28257/50000 [17:55:21<11:33:41,  1.91s/it]
Outer training loop:  57%|#####6    | 28258/50000 [17:55:23<11:41:32,  1.94s/it]
Outer training loop:  57%|#####6    | 28259/50000 [17:55:24<11:28:50,  1.90s/it]
Outer training loop:  57%|#####6    | 28260/50000 [17:55:26<11:21:11,  1.88s/it]
Outer training loop:  57%|#####6    | 28261/50000 [17:55:28<11:57:17,  1.98s/it]
Outer training loop:  57%|#####6    | 28262/50000 [17:55:30<11:34:49,  1.92s/it]
Outer training loop:  57%|#####6    | 28263/50000 [17:55:32<11:27:03,  1.90s/it]
Outer training loop:  57%|#####6    | 28264/50000 [17:55:34<11:36:36,  1.92s/it]
Outer training loop:  57%|#####6    | 28265/50000 [17:55:36<11:25:02,  1.89s/it]
Outer training loop:  57%|#####6    | 28266/50000 [17:55:38<11:32:22,  1.91s/it]
Outer training loop:  57%|#####6    | 28267/50000 [17:55:40<11:17:39,  1.87s/it]
Outer training loop:  57%|#####6    | 28268/50000 [17:55:42<11:31:28,  1.91s/it]
Outer training loop:  57%|#####6    | 28269/50000 [17:55:44<11:31:03,  1.91s/it]
Outer training loop:  57%|#####6    | 28270/50000 [17:55:45<11:17:40,  1.87s/it]
Outer training loop:  57%|#####6    | 28271/50000 [17:55:47<11:22:41,  1.89s/it]
Outer training loop:  57%|#####6    | 28272/50000 [17:55:49<11:27:08,  1.90s/it]
Outer training loop:  57%|#####6    | 28273/50000 [17:55:51<11:24:05,  1.89s/it]
Outer training loop:  57%|#####6    | 28274/50000 [17:55:53<11:31:20,  1.91s/it]
Outer training loop:  57%|#####6    | 28275/50000 [17:55:55<11:24:50,  1.89s/it]
Outer training loop:  57%|#####6    | 28276/50000 [17:55:57<11:16:32,  1.87s/it]
Outer training loop:  57%|#####6    | 28277/50000 [17:55:59<11:23:33,  1.89s/it]
Outer training loop:  57%|#####6    | 28278/50000 [17:56:01<11:32:27,  1.91s/it]
Outer training loop:  57%|#####6    | 28279/50000 [17:56:02<11:30:55,  1.91s/it]
Outer training loop:  57%|#####6    | 28280/50000 [17:56:04<11:14:04,  1.86s/it]
Outer training loop:  57%|#####6    | 28281/50000 [17:56:06<11:30:44,  1.91s/it]
Outer training loop:  57%|#####6    | 28282/50000 [17:56:08<11:16:14,  1.87s/it]
Outer training loop:  57%|#####6    | 28283/50000 [17:56:10<10:55:11,  1.81s/it]
Outer training loop:  57%|#####6    | 28284/50000 [17:56:11<10:48:50,  1.79s/it]
Outer training loop:  57%|#####6    | 28285/50000 [17:56:13<11:15:18,  1.87s/it]
Outer training loop:  57%|#####6    | 28286/50000 [17:56:15<11:06:40,  1.84s/it]
Outer training loop:  57%|#####6    | 28287/50000 [17:56:17<11:08:20,  1.85s/it]
Outer training loop:  57%|#####6    | 28288/50000 [17:56:19<11:17:37,  1.87s/it]
Outer training loop:  57%|#####6    | 28289/50000 [17:56:21<11:05:27,  1.84s/it]
Outer training loop:  57%|#####6    | 28290/50000 [17:56:23<11:06:40,  1.84s/it]
Outer training loop:  57%|#####6    | 28291/50000 [17:56:25<11:36:15,  1.92s/it]
Outer training loop:  57%|#####6    | 28292/50000 [17:56:26<11:14:16,  1.86s/it]
Outer training loop:  57%|#####6    | 28293/50000 [17:56:28<11:04:29,  1.84s/it]
Outer training loop:  57%|#####6    | 28294/50000 [17:56:30<11:26:23,  1.90s/it]
Outer training loop:  57%|#####6    | 28295/50000 [17:56:32<11:09:28,  1.85s/it]
Outer training loop:  57%|#####6    | 28296/50000 [17:56:34<11:09:42,  1.85s/it]
Outer training loop:  57%|#####6    | 28297/50000 [17:56:36<11:17:28,  1.87s/it]
Outer training loop:  57%|#####6    | 28298/50000 [17:56:38<11:23:49,  1.89s/it]
Outer training loop:  57%|#####6    | 28299/50000 [17:56:40<11:12:05,  1.86s/it]
Outer training loop:  57%|#####6    | 28300/50000 [17:56:41<11:00:49,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  57%|#####6    | 28301/50000 [17:56:49<21:03:00,  3.49s/it]
Outer training loop:  57%|#####6    | 28302/50000 [17:56:51<18:32:27,  3.08s/it]
Outer training loop:  57%|#####6    | 28303/50000 [17:56:53<16:36:59,  2.76s/it]
Outer training loop:  57%|#####6    | 28304/50000 [17:56:55<15:20:56,  2.55s/it]
Outer training loop:  57%|#####6    | 28305/50000 [17:56:57<14:10:27,  2.35s/it]
Outer training loop:  57%|#####6    | 28306/50000 [17:56:59<13:13:54,  2.20s/it]
Outer training loop:  57%|#####6    | 28307/50000 [17:57:00<12:34:06,  2.09s/it]
Outer training loop:  57%|#####6    | 28308/50000 [17:57:02<12:22:46,  2.05s/it]
Outer training loop:  57%|#####6    | 28309/50000 [17:57:04<12:09:20,  2.02s/it]
Outer training loop:  57%|#####6    | 28310/50000 [17:57:06<11:49:39,  1.96s/it]
Outer training loop:  57%|#####6    | 28311/50000 [17:57:08<11:29:32,  1.91s/it]
Outer training loop:  57%|#####6    | 28312/50000 [17:57:10<11:40:54,  1.94s/it]
Outer training loop:  57%|#####6    | 28313/50000 [17:57:12<11:16:30,  1.87s/it]
Outer training loop:  57%|#####6    | 28314/50000 [17:57:13<11:09:32,  1.85s/it]
Outer training loop:  57%|#####6    | 28315/50000 [17:57:15<11:01:46,  1.83s/it]
Outer training loop:  57%|#####6    | 28316/50000 [17:57:17<11:20:02,  1.88s/it]
Outer training loop:  57%|#####6    | 28317/50000 [17:57:19<11:20:58,  1.88s/it]
Outer training loop:  57%|#####6    | 28318/50000 [17:57:21<11:07:45,  1.85s/it]
Outer training loop:  57%|#####6    | 28319/50000 [17:57:23<11:00:51,  1.83s/it]
Outer training loop:  57%|#####6    | 28320/50000 [17:57:25<11:33:09,  1.92s/it]
Outer training loop:  57%|#####6    | 28321/50000 [17:57:27<11:20:14,  1.88s/it]
Outer training loop:  57%|#####6    | 28322/50000 [17:57:28<11:08:25,  1.85s/it]
Outer training loop:  57%|#####6    | 28323/50000 [17:57:30<11:16:18,  1.87s/it]
Outer training loop:  57%|#####6    | 28324/50000 [17:57:32<11:04:26,  1.84s/it]
Outer training loop:  57%|#####6    | 28325/50000 [17:57:34<10:54:39,  1.81s/it]
Outer training loop:  57%|#####6    | 28326/50000 [17:57:36<10:51:06,  1.80s/it]
Outer training loop:  57%|#####6    | 28327/50000 [17:57:38<11:05:59,  1.84s/it]
Outer training loop:  57%|#####6    | 28328/50000 [17:57:39<11:15:15,  1.87s/it]
Outer training loop:  57%|#####6    | 28329/50000 [17:57:41<11:04:38,  1.84s/it]
Outer training loop:  57%|#####6    | 28330/50000 [17:57:43<11:16:45,  1.87s/it]
Outer training loop:  57%|#####6    | 28331/50000 [17:57:45<11:12:12,  1.86s/it]
Outer training loop:  57%|#####6    | 28332/50000 [17:57:47<11:07:59,  1.85s/it]
Outer training loop:  57%|#####6    | 28333/50000 [17:57:49<11:36:18,  1.93s/it]
Outer training loop:  57%|#####6    | 28334/50000 [17:57:51<11:20:37,  1.88s/it]
Outer training loop:  57%|#####6    | 28335/50000 [17:57:52<11:08:27,  1.85s/it]
Outer training loop:  57%|#####6    | 28336/50000 [17:57:54<11:11:15,  1.86s/it]
Outer training loop:  57%|#####6    | 28337/50000 [17:57:56<11:36:42,  1.93s/it]
Outer training loop:  57%|#####6    | 28338/50000 [17:57:58<11:25:56,  1.90s/it]
Outer training loop:  57%|#####6    | 28339/50000 [17:58:00<11:42:44,  1.95s/it]
Outer training loop:  57%|#####6    | 28340/50000 [17:58:02<11:27:09,  1.90s/it]
Outer training loop:  57%|#####6    | 28341/50000 [17:58:04<11:25:31,  1.90s/it]
Outer training loop:  57%|#####6    | 28342/50000 [17:58:06<11:25:55,  1.90s/it]
Outer training loop:  57%|#####6    | 28343/50000 [17:58:08<11:31:18,  1.92s/it]
Outer training loop:  57%|#####6    | 28344/50000 [17:58:10<11:21:35,  1.89s/it]
Outer training loop:  57%|#####6    | 28345/50000 [17:58:12<11:14:48,  1.87s/it]
Outer training loop:  57%|#####6    | 28346/50000 [17:58:14<11:30:54,  1.91s/it]
Outer training loop:  57%|#####6    | 28347/50000 [17:58:15<11:17:22,  1.88s/it]
Outer training loop:  57%|#####6    | 28348/50000 [17:58:17<11:04:38,  1.84s/it]
Outer training loop:  57%|#####6    | 28349/50000 [17:58:19<11:23:45,  1.89s/it]
Outer training loop:  57%|#####6    | 28350/50000 [17:58:21<11:23:00,  1.89s/it]
Outer training loop:  57%|#####6    | 28351/50000 [17:58:23<11:11:28,  1.86s/it]
Outer training loop:  57%|#####6    | 28352/50000 [17:58:25<10:53:10,  1.81s/it]
Outer training loop:  57%|#####6    | 28353/50000 [17:58:27<11:34:57,  1.93s/it]
Outer training loop:  57%|#####6    | 28354/50000 [17:58:29<11:33:55,  1.92s/it]
Outer training loop:  57%|#####6    | 28355/50000 [17:58:30<11:17:33,  1.88s/it]
Outer training loop:  57%|#####6    | 28356/50000 [17:58:32<11:30:26,  1.91s/it]
Outer training loop:  57%|#####6    | 28357/50000 [17:58:34<11:17:48,  1.88s/it]
Outer training loop:  57%|#####6    | 28358/50000 [17:58:36<11:07:13,  1.85s/it]
Outer training loop:  57%|#####6    | 28359/50000 [17:58:38<11:27:07,  1.91s/it]
Outer training loop:  57%|#####6    | 28360/50000 [17:58:40<11:16:18,  1.88s/it]
Outer training loop:  57%|#####6    | 28361/50000 [17:58:42<11:20:23,  1.89s/it]
Outer training loop:  57%|#####6    | 28362/50000 [17:58:44<11:22:08,  1.89s/it]
Outer training loop:  57%|#####6    | 28363/50000 [17:58:46<11:29:56,  1.91s/it]
Outer training loop:  57%|#####6    | 28364/50000 [17:58:47<11:27:35,  1.91s/it]
Outer training loop:  57%|#####6    | 28365/50000 [17:58:49<11:20:49,  1.89s/it]
Outer training loop:  57%|#####6    | 28366/50000 [17:58:52<11:54:45,  1.98s/it]
Outer training loop:  57%|#####6    | 28367/50000 [17:58:53<11:43:32,  1.95s/it]
Outer training loop:  57%|#####6    | 28368/50000 [17:58:55<11:17:47,  1.88s/it]
Outer training loop:  57%|#####6    | 28369/50000 [17:58:57<11:05:40,  1.85s/it]
Outer training loop:  57%|#####6    | 28370/50000 [17:58:59<11:29:42,  1.91s/it]
Outer training loop:  57%|#####6    | 28371/50000 [17:59:01<11:24:48,  1.90s/it]
Outer training loop:  57%|#####6    | 28372/50000 [17:59:03<11:14:25,  1.87s/it]
Outer training loop:  57%|#####6    | 28373/50000 [17:59:05<11:40:45,  1.94s/it]
Outer training loop:  57%|#####6    | 28374/50000 [17:59:07<11:30:05,  1.91s/it]
Outer training loop:  57%|#####6    | 28375/50000 [17:59:08<11:10:09,  1.86s/it]
Outer training loop:  57%|#####6    | 28376/50000 [17:59:11<11:45:42,  1.96s/it]
Outer training loop:  57%|#####6    | 28377/50000 [17:59:12<11:26:51,  1.91s/it]
Outer training loop:  57%|#####6    | 28378/50000 [17:59:14<11:19:40,  1.89s/it]
Outer training loop:  57%|#####6    | 28379/50000 [17:59:16<11:17:53,  1.88s/it]
Outer training loop:  57%|#####6    | 28380/50000 [17:59:18<11:22:59,  1.90s/it]
Outer training loop:  57%|#####6    | 28381/50000 [17:59:20<11:22:10,  1.89s/it]
Outer training loop:  57%|#####6    | 28382/50000 [17:59:22<11:08:07,  1.85s/it]
Outer training loop:  57%|#####6    | 28383/50000 [17:59:24<11:29:04,  1.91s/it]
Outer training loop:  57%|#####6    | 28384/50000 [17:59:25<11:20:34,  1.89s/it]
Outer training loop:  57%|#####6    | 28385/50000 [17:59:27<11:12:27,  1.87s/it]
Outer training loop:  57%|#####6    | 28386/50000 [17:59:29<11:08:27,  1.86s/it]
Outer training loop:  57%|#####6    | 28387/50000 [17:59:31<11:22:39,  1.90s/it]
Outer training loop:  57%|#####6    | 28388/50000 [17:59:33<11:11:54,  1.87s/it]
Outer training loop:  57%|#####6    | 28389/50000 [17:59:35<11:18:32,  1.88s/it]
Outer training loop:  57%|#####6    | 28390/50000 [17:59:37<11:31:08,  1.92s/it]
Outer training loop:  57%|#####6    | 28391/50000 [17:59:39<11:21:00,  1.89s/it]
Outer training loop:  57%|#####6    | 28392/50000 [17:59:41<11:17:36,  1.88s/it]
Outer training loop:  57%|#####6    | 28393/50000 [17:59:42<11:11:11,  1.86s/it]
Outer training loop:  57%|#####6    | 28394/50000 [17:59:44<11:16:36,  1.88s/it]
Outer training loop:  57%|#####6    | 28395/50000 [17:59:46<11:13:35,  1.87s/it]
Outer training loop:  57%|#####6    | 28396/50000 [17:59:48<11:15:22,  1.88s/it]
Outer training loop:  57%|#####6    | 28397/50000 [17:59:50<11:29:12,  1.91s/it]
Outer training loop:  57%|#####6    | 28398/50000 [17:59:52<11:19:10,  1.89s/it]
Outer training loop:  57%|#####6    | 28399/50000 [17:59:54<11:29:15,  1.91s/it]
Outer training loop:  57%|#####6    | 28400/50000 [17:59:56<11:57:46,  1.99s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  57%|#####6    | 28401/50000 [18:00:04<21:59:11,  3.66s/it]
Outer training loop:  57%|#####6    | 28402/50000 [18:00:05<18:46:07,  3.13s/it]
Outer training loop:  57%|#####6    | 28403/50000 [18:00:07<16:15:17,  2.71s/it]
Outer training loop:  57%|#####6    | 28404/50000 [18:00:09<15:13:58,  2.54s/it]
Outer training loop:  57%|#####6    | 28405/50000 [18:00:11<13:57:37,  2.33s/it]
Outer training loop:  57%|#####6    | 28406/50000 [18:00:13<13:18:56,  2.22s/it]
Outer training loop:  57%|#####6    | 28407/50000 [18:00:15<12:38:57,  2.11s/it]
Outer training loop:  57%|#####6    | 28408/50000 [18:00:17<12:19:30,  2.05s/it]
Outer training loop:  57%|#####6    | 28409/50000 [18:00:19<11:58:08,  2.00s/it]
Outer training loop:  57%|#####6    | 28410/50000 [18:00:21<11:38:59,  1.94s/it]
Outer training loop:  57%|#####6    | 28411/50000 [18:00:22<11:35:12,  1.93s/it]
Outer training loop:  57%|#####6    | 28412/50000 [18:00:24<11:48:04,  1.97s/it]
Outer training loop:  57%|#####6    | 28413/50000 [18:00:26<11:47:50,  1.97s/it]
Outer training loop:  57%|#####6    | 28414/50000 [18:00:28<11:37:43,  1.94s/it]
Outer training loop:  57%|#####6    | 28415/50000 [18:00:30<11:50:39,  1.98s/it]
Outer training loop:  57%|#####6    | 28416/50000 [18:00:32<11:33:26,  1.93s/it]
Outer training loop:  57%|#####6    | 28417/50000 [18:00:34<11:26:47,  1.91s/it]
Outer training loop:  57%|#####6    | 28418/50000 [18:00:36<11:13:57,  1.87s/it]
Outer training loop:  57%|#####6    | 28419/50000 [18:00:38<11:32:57,  1.93s/it]
Outer training loop:  57%|#####6    | 28420/50000 [18:00:40<11:30:39,  1.92s/it]
Outer training loop:  57%|#####6    | 28421/50000 [18:00:42<11:16:30,  1.88s/it]
Outer training loop:  57%|#####6    | 28422/50000 [18:00:44<11:27:09,  1.91s/it]
Outer training loop:  57%|#####6    | 28423/50000 [18:00:45<11:14:19,  1.88s/it]
Outer training loop:  57%|#####6    | 28424/50000 [18:00:47<11:08:46,  1.86s/it]
Outer training loop:  57%|#####6    | 28425/50000 [18:00:49<11:40:09,  1.95s/it]
Outer training loop:  57%|#####6    | 28426/50000 [18:00:51<11:46:07,  1.96s/it]
Outer training loop:  57%|#####6    | 28427/50000 [18:00:53<11:29:37,  1.92s/it]
Outer training loop:  57%|#####6    | 28428/50000 [18:00:55<11:35:09,  1.93s/it]
Outer training loop:  57%|#####6    | 28429/50000 [18:00:57<11:31:06,  1.92s/it]
Outer training loop:  57%|#####6    | 28430/50000 [18:00:59<11:26:22,  1.91s/it]
Outer training loop:  57%|#####6    | 28431/50000 [18:01:01<11:12:14,  1.87s/it]
Outer training loop:  57%|#####6    | 28432/50000 [18:01:03<11:34:41,  1.93s/it]
Outer training loop:  57%|#####6    | 28433/50000 [18:01:05<11:23:46,  1.90s/it]
Outer training loop:  57%|#####6    | 28434/50000 [18:01:07<11:24:16,  1.90s/it]
Outer training loop:  57%|#####6    | 28435/50000 [18:01:08<11:16:23,  1.88s/it]
Outer training loop:  57%|#####6    | 28436/50000 [18:01:10<11:37:16,  1.94s/it]
Outer training loop:  57%|#####6    | 28437/50000 [18:01:12<11:20:40,  1.89s/it]
Outer training loop:  57%|#####6    | 28438/50000 [18:01:14<11:13:02,  1.87s/it]
Outer training loop:  57%|#####6    | 28439/50000 [18:01:16<11:34:15,  1.93s/it]
Outer training loop:  57%|#####6    | 28440/50000 [18:01:18<11:36:10,  1.94s/it]
Outer training loop:  57%|#####6    | 28441/50000 [18:01:20<11:26:58,  1.91s/it]
Outer training loop:  57%|#####6    | 28442/50000 [18:01:22<11:40:51,  1.95s/it]
Outer training loop:  57%|#####6    | 28443/50000 [18:01:24<11:15:54,  1.88s/it]
Outer training loop:  57%|#####6    | 28444/50000 [18:01:26<11:18:58,  1.89s/it]
Outer training loop:  57%|#####6    | 28445/50000 [18:01:28<11:33:12,  1.93s/it]
Outer training loop:  57%|#####6    | 28446/50000 [18:01:30<11:34:29,  1.93s/it]
Outer training loop:  57%|#####6    | 28447/50000 [18:01:31<11:08:57,  1.86s/it]
Outer training loop:  57%|#####6    | 28448/50000 [18:01:33<11:11:06,  1.87s/it]
Outer training loop:  57%|#####6    | 28449/50000 [18:01:35<11:13:09,  1.87s/it]
Outer training loop:  57%|#####6    | 28450/50000 [18:01:37<11:24:23,  1.91s/it]
Outer training loop:  57%|#####6    | 28451/50000 [18:01:39<11:13:18,  1.87s/it]
Outer training loop:  57%|#####6    | 28452/50000 [18:01:41<11:24:09,  1.91s/it]
Outer training loop:  57%|#####6    | 28453/50000 [18:01:43<11:18:30,  1.89s/it]
Outer training loop:  57%|#####6    | 28454/50000 [18:01:45<11:22:00,  1.90s/it]
Outer training loop:  57%|#####6    | 28455/50000 [18:01:46<11:26:34,  1.91s/it]
Outer training loop:  57%|#####6    | 28456/50000 [18:01:48<11:23:10,  1.90s/it]
Outer training loop:  57%|#####6    | 28457/50000 [18:01:50<11:22:59,  1.90s/it]
Outer training loop:  57%|#####6    | 28458/50000 [18:01:52<11:04:21,  1.85s/it]
Outer training loop:  57%|#####6    | 28459/50000 [18:01:54<11:30:40,  1.92s/it]
Outer training loop:  57%|#####6    | 28460/50000 [18:01:56<11:17:38,  1.89s/it]
Outer training loop:  57%|#####6    | 28461/50000 [18:01:58<11:16:02,  1.88s/it]
Outer training loop:  57%|#####6    | 28462/50000 [18:02:00<11:31:58,  1.93s/it]
Outer training loop:  57%|#####6    | 28463/50000 [18:02:02<11:15:23,  1.88s/it]
Outer training loop:  57%|#####6    | 28464/50000 [18:02:03<11:02:07,  1.84s/it]
Outer training loop:  57%|#####6    | 28465/50000 [18:02:05<11:21:09,  1.90s/it]
Outer training loop:  57%|#####6    | 28466/50000 [18:02:07<11:12:55,  1.87s/it]
Outer training loop:  57%|#####6    | 28467/50000 [18:02:09<10:57:16,  1.83s/it]
Outer training loop:  57%|#####6    | 28468/50000 [18:02:11<10:57:24,  1.83s/it]
Outer training loop:  57%|#####6    | 28469/50000 [18:02:13<11:23:00,  1.90s/it]
Outer training loop:  57%|#####6    | 28470/50000 [18:02:15<11:05:40,  1.86s/it]
Outer training loop:  57%|#####6    | 28471/50000 [18:02:16<10:51:25,  1.82s/it]
Outer training loop:  57%|#####6    | 28472/50000 [18:02:18<11:24:20,  1.91s/it]
Outer training loop:  57%|#####6    | 28473/50000 [18:02:20<11:30:42,  1.93s/it]
Outer training loop:  57%|#####6    | 28474/50000 [18:02:22<11:15:36,  1.88s/it]
Outer training loop:  57%|#####6    | 28475/50000 [18:02:24<11:18:42,  1.89s/it]
Outer training loop:  57%|#####6    | 28476/50000 [18:02:26<11:24:45,  1.91s/it]
Outer training loop:  57%|#####6    | 28477/50000 [18:02:28<11:05:54,  1.86s/it]
Outer training loop:  57%|#####6    | 28478/50000 [18:02:30<11:09:47,  1.87s/it]
Outer training loop:  57%|#####6    | 28479/50000 [18:02:31<11:00:46,  1.84s/it]
Outer training loop:  57%|#####6    | 28480/50000 [18:02:33<10:49:25,  1.81s/it]
Outer training loop:  57%|#####6    | 28481/50000 [18:02:35<10:47:42,  1.81s/it]
Outer training loop:  57%|#####6    | 28482/50000 [18:02:37<11:09:19,  1.87s/it]
Outer training loop:  57%|#####6    | 28483/50000 [18:02:39<10:57:53,  1.83s/it]
Outer training loop:  57%|#####6    | 28484/50000 [18:02:41<11:03:23,  1.85s/it]
Outer training loop:  57%|#####6    | 28485/50000 [18:02:43<11:20:40,  1.90s/it]
Outer training loop:  57%|#####6    | 28486/50000 [18:02:44<11:10:11,  1.87s/it]
Outer training loop:  57%|#####6    | 28487/50000 [18:02:46<11:07:01,  1.86s/it]
Outer training loop:  57%|#####6    | 28488/50000 [18:02:48<11:09:17,  1.87s/it]
Outer training loop:  57%|#####6    | 28489/50000 [18:02:50<11:14:04,  1.88s/it]
Outer training loop:  57%|#####6    | 28490/50000 [18:02:52<10:50:30,  1.81s/it]
Outer training loop:  57%|#####6    | 28491/50000 [18:02:53<10:46:26,  1.80s/it]
Outer training loop:  57%|#####6    | 28492/50000 [18:02:56<11:11:37,  1.87s/it]
Outer training loop:  57%|#####6    | 28493/50000 [18:02:57<11:21:43,  1.90s/it]
Outer training loop:  57%|#####6    | 28494/50000 [18:02:59<11:20:31,  1.90s/it]
Outer training loop:  57%|#####6    | 28495/50000 [18:03:01<11:23:01,  1.91s/it]
Outer training loop:  57%|#####6    | 28496/50000 [18:03:03<11:06:57,  1.86s/it]
Outer training loop:  57%|#####6    | 28497/50000 [18:03:05<11:07:43,  1.86s/it]
Outer training loop:  57%|#####6    | 28498/50000 [18:03:07<11:34:38,  1.94s/it]
Outer training loop:  57%|#####6    | 28499/50000 [18:03:09<11:38:25,  1.95s/it]
Outer training loop:  57%|#####6    | 28500/50000 [18:03:11<11:43:00,  1.96s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.93s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:41,  1.97s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:35,  1.92s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:32,  1.90s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:32,  1.93s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:29,  1.92s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.93s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:17,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:16,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:08,  1.86s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:04,  1.83s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.87s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:02,  1.85s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:59,  1.83s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:56,  1.82s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:55,  1.84s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.85s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:46,  1.84s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:46,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:43,  1.89s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:36,  1.82s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:40,  1.93s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.89s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:33,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:26,  1.80s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:25,  1.82s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:12<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:14<01:17,  1.81s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:11,  1.87s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:25<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:27<01:06,  1.86s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:29<01:03,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:01,  1.81s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:02,  1.94s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:38<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:40<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:42<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:49,  1.90s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:47,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:47,  1.98s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:44,  1.94s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.91s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:55<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:57<00:37,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:30,  1.79s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:30,  1.92s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:28,  1.93s/it]

Evaluating policy:  83%|########3 | 69/83 [02:08<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:10<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:19,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:19<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:14,  2.04s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.99s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.95s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.96s/it]

Evaluating policy:  96%|#########6| 80/83 [02:29<00:05,  1.93s/it]

Evaluating policy:  98%|#########7| 81/83 [02:31<00:03,  1.90s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.82s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.74s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:33,  1.92s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.92s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.86s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.92s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:19,  1.86s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:16,  1.85s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:20,  1.93s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.88s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:16,  1.98s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:12,  1.95s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.92s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.91s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:05,  1.96s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:57,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:57,  1.95s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:54,  1.93s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.90s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:44,  1.86s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:47,  1.96s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:44,  1.93s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:35,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:37,  1.96s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:32,  1.92s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.93s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:22,  1.83s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:24,  1.93s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:23,  1.93s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:20,  1.92s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:16,  1.91s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:14,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:13,  1.99s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:11,  1.99s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:08,  1.95s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:06,  1.96s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:03,  1.92s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:01,  1.91s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<01:00,  2.01s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:56,  1.94s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:53,  1.90s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:51,  1.91s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:45,  2.00s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:42,  1.94s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:39,  1.89s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:38,  1.93s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:36,  1.91s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:33,  1.88s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:06<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:29,  1.84s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:29,  1.97s/it]

Evaluating policy:  83%|########3 | 69/83 [02:12<00:27,  1.97s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:24,  1.92s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.89s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.83s/it]

Evaluating policy:  88%|########7 | 73/83 [02:19<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:14,  1.86s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:13,  1.88s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:12,  2.03s/it]

Evaluating policy:  94%|#########3| 78/83 [02:29<00:09,  1.95s/it]

Evaluating policy:  95%|#########5| 79/83 [02:31<00:07,  1.94s/it]

Evaluating policy:  96%|#########6| 80/83 [02:33<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.90s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.91s/it]

Outer training loop:  57%|#####7    | 28501/50000 [18:08:33<584:25:43, 97.86s/it]
Outer training loop:  57%|#####7    | 28502/50000 [18:08:34<412:25:03, 69.06s/it]
Outer training loop:  57%|#####7    | 28503/50000 [18:08:36<291:52:59, 48.88s/it]
Outer training loop:  57%|#####7    | 28504/50000 [18:08:38<207:33:56, 34.76s/it]
Outer training loop:  57%|#####7    | 28505/50000 [18:08:41<149:44:02, 25.08s/it]
Outer training loop:  57%|#####7    | 28506/50000 [18:08:43<108:16:34, 18.14s/it]
Outer training loop:  57%|#####7    | 28507/50000 [18:08:44<78:46:35, 13.19s/it] 
Outer training loop:  57%|#####7    | 28508/50000 [18:08:46<58:47:24,  9.85s/it]
Outer training loop:  57%|#####7    | 28509/50000 [18:08:48<44:29:54,  7.45s/it]
Outer training loop:  57%|#####7    | 28510/50000 [18:08:50<34:40:02,  5.81s/it]
Outer training loop:  57%|#####7    | 28511/50000 [18:08:52<27:14:08,  4.56s/it]
Outer training loop:  57%|#####7    | 28512/50000 [18:08:54<22:35:27,  3.78s/it]
Outer training loop:  57%|#####7    | 28513/50000 [18:08:56<19:14:18,  3.22s/it]
Outer training loop:  57%|#####7    | 28514/50000 [18:08:57<16:48:20,  2.82s/it]
Outer training loop:  57%|#####7    | 28515/50000 [18:09:00<15:28:20,  2.59s/it]
Outer training loop:  57%|#####7    | 28516/50000 [18:09:01<14:08:17,  2.37s/it]
Outer training loop:  57%|#####7    | 28517/50000 [18:09:03<13:00:45,  2.18s/it]
Outer training loop:  57%|#####7    | 28518/50000 [18:09:05<12:56:47,  2.17s/it]
Outer training loop:  57%|#####7    | 28519/50000 [18:09:07<12:13:35,  2.05s/it]
Outer training loop:  57%|#####7    | 28520/50000 [18:09:09<11:58:09,  2.01s/it]
Outer training loop:  57%|#####7    | 28521/50000 [18:09:11<12:04:40,  2.02s/it]
Outer training loop:  57%|#####7    | 28522/50000 [18:09:13<11:42:12,  1.96s/it]
Outer training loop:  57%|#####7    | 28523/50000 [18:09:15<11:31:53,  1.93s/it]
Outer training loop:  57%|#####7    | 28524/50000 [18:09:17<11:19:35,  1.90s/it]
Outer training loop:  57%|#####7    | 28525/50000 [18:09:18<11:23:29,  1.91s/it]
Outer training loop:  57%|#####7    | 28526/50000 [18:09:20<11:19:33,  1.90s/it]
Outer training loop:  57%|#####7    | 28527/50000 [18:09:22<11:13:01,  1.88s/it]
Outer training loop:  57%|#####7    | 28528/50000 [18:09:24<11:19:53,  1.90s/it]
Outer training loop:  57%|#####7    | 28529/50000 [18:09:26<11:20:55,  1.90s/it]
Outer training loop:  57%|#####7    | 28530/50000 [18:09:28<11:10:55,  1.87s/it]
Outer training loop:  57%|#####7    | 28531/50000 [18:09:30<12:00:41,  2.01s/it]
Outer training loop:  57%|#####7    | 28532/50000 [18:09:32<11:58:14,  2.01s/it]
Outer training loop:  57%|#####7    | 28533/50000 [18:09:34<11:33:04,  1.94s/it]
Outer training loop:  57%|#####7    | 28534/50000 [18:09:36<11:25:24,  1.92s/it]
Outer training loop:  57%|#####7    | 28535/50000 [18:09:38<11:40:43,  1.96s/it]
Outer training loop:  57%|#####7    | 28536/50000 [18:09:40<11:28:53,  1.93s/it]
Outer training loop:  57%|#####7    | 28537/50000 [18:09:42<11:16:15,  1.89s/it]
Outer training loop:  57%|#####7    | 28538/50000 [18:09:44<11:33:47,  1.94s/it]
Outer training loop:  57%|#####7    | 28539/50000 [18:09:45<11:25:16,  1.92s/it]
Outer training loop:  57%|#####7    | 28540/50000 [18:09:47<11:13:29,  1.88s/it]
Outer training loop:  57%|#####7    | 28541/50000 [18:09:49<11:40:10,  1.96s/it]
Outer training loop:  57%|#####7    | 28542/50000 [18:09:51<11:34:59,  1.94s/it]
Outer training loop:  57%|#####7    | 28543/50000 [18:09:53<11:26:30,  1.92s/it]
Outer training loop:  57%|#####7    | 28544/50000 [18:09:55<11:34:27,  1.94s/it]
Outer training loop:  57%|#####7    | 28545/50000 [18:09:57<11:25:35,  1.92s/it]
Outer training loop:  57%|#####7    | 28546/50000 [18:09:59<11:22:08,  1.91s/it]
Outer training loop:  57%|#####7    | 28547/50000 [18:10:01<10:54:03,  1.83s/it]
Outer training loop:  57%|#####7    | 28548/50000 [18:10:02<11:07:50,  1.87s/it]
Outer training loop:  57%|#####7    | 28549/50000 [18:10:04<11:13:54,  1.88s/it]
Outer training loop:  57%|#####7    | 28550/50000 [18:10:06<11:02:25,  1.85s/it]
Outer training loop:  57%|#####7    | 28551/50000 [18:10:08<11:13:16,  1.88s/it]
Outer training loop:  57%|#####7    | 28552/50000 [18:10:10<11:05:28,  1.86s/it]
Outer training loop:  57%|#####7    | 28553/50000 [18:10:12<11:00:23,  1.85s/it]
Outer training loop:  57%|#####7    | 28554/50000 [18:10:14<11:31:04,  1.93s/it]
Outer training loop:  57%|#####7    | 28555/50000 [18:10:16<11:12:04,  1.88s/it]
Outer training loop:  57%|#####7    | 28556/50000 [18:10:17<11:06:09,  1.86s/it]
Outer training loop:  57%|#####7    | 28557/50000 [18:10:20<11:27:28,  1.92s/it]
Outer training loop:  57%|#####7    | 28558/50000 [18:10:21<11:11:15,  1.88s/it]
Outer training loop:  57%|#####7    | 28559/50000 [18:10:23<10:59:14,  1.84s/it]
Outer training loop:  57%|#####7    | 28560/50000 [18:10:25<10:58:13,  1.84s/it]
Outer training loop:  57%|#####7    | 28561/50000 [18:10:27<11:15:26,  1.89s/it]
Outer training loop:  57%|#####7    | 28562/50000 [18:10:29<10:52:42,  1.83s/it]
Outer training loop:  57%|#####7    | 28563/50000 [18:10:30<10:44:26,  1.80s/it]
Outer training loop:  57%|#####7    | 28564/50000 [18:10:32<11:07:55,  1.87s/it]
Outer training loop:  57%|#####7    | 28565/50000 [18:10:34<11:12:32,  1.88s/it]
Outer training loop:  57%|#####7    | 28566/50000 [18:10:36<11:23:01,  1.91s/it]
Outer training loop:  57%|#####7    | 28567/50000 [18:10:38<11:11:24,  1.88s/it]
Outer training loop:  57%|#####7    | 28568/50000 [18:10:40<11:26:06,  1.92s/it]
Outer training loop:  57%|#####7    | 28569/50000 [18:10:42<11:14:39,  1.89s/it]
Outer training loop:  57%|#####7    | 28570/50000 [18:10:44<11:07:21,  1.87s/it]
Outer training loop:  57%|#####7    | 28571/50000 [18:10:46<11:22:46,  1.91s/it]
Outer training loop:  57%|#####7    | 28572/50000 [18:10:48<11:15:10,  1.89s/it]
Outer training loop:  57%|#####7    | 28573/50000 [18:10:49<11:08:43,  1.87s/it]
Outer training loop:  57%|#####7    | 28574/50000 [18:10:51<11:19:17,  1.90s/it]
Outer training loop:  57%|#####7    | 28575/50000 [18:10:53<11:09:31,  1.88s/it]
Outer training loop:  57%|#####7    | 28576/50000 [18:10:55<11:07:47,  1.87s/it]
Outer training loop:  57%|#####7    | 28577/50000 [18:10:57<10:49:01,  1.82s/it]
Outer training loop:  57%|#####7    | 28578/50000 [18:10:59<11:23:58,  1.92s/it]
Outer training loop:  57%|#####7    | 28579/50000 [18:11:01<11:06:30,  1.87s/it]
Outer training loop:  57%|#####7    | 28580/50000 [18:11:02<10:47:05,  1.81s/it]
Outer training loop:  57%|#####7    | 28581/50000 [18:11:05<11:33:33,  1.94s/it]
Outer training loop:  57%|#####7    | 28582/50000 [18:11:06<11:28:57,  1.93s/it]
Outer training loop:  57%|#####7    | 28583/50000 [18:11:08<11:37:43,  1.95s/it]
Outer training loop:  57%|#####7    | 28584/50000 [18:11:11<11:44:19,  1.97s/it]
Outer training loop:  57%|#####7    | 28585/50000 [18:11:12<11:19:53,  1.90s/it]
Outer training loop:  57%|#####7    | 28586/50000 [18:11:14<11:17:24,  1.90s/it]
Outer training loop:  57%|#####7    | 28587/50000 [18:11:16<11:19:13,  1.90s/it]
Outer training loop:  57%|#####7    | 28588/50000 [18:11:18<11:39:37,  1.96s/it]
Outer training loop:  57%|#####7    | 28589/50000 [18:11:20<11:31:18,  1.94s/it]
Outer training loop:  57%|#####7    | 28590/50000 [18:11:22<11:11:54,  1.88s/it]
Outer training loop:  57%|#####7    | 28591/50000 [18:11:24<11:38:26,  1.96s/it]
Outer training loop:  57%|#####7    | 28592/50000 [18:11:26<11:32:05,  1.94s/it]
Outer training loop:  57%|#####7    | 28593/50000 [18:11:28<11:23:40,  1.92s/it]
Outer training loop:  57%|#####7    | 28594/50000 [18:11:29<11:12:44,  1.89s/it]
Outer training loop:  57%|#####7    | 28595/50000 [18:11:32<11:27:40,  1.93s/it]
Outer training loop:  57%|#####7    | 28596/50000 [18:11:33<11:14:28,  1.89s/it]
Outer training loop:  57%|#####7    | 28597/50000 [18:11:35<11:09:47,  1.88s/it]
Outer training loop:  57%|#####7    | 28598/50000 [18:11:37<11:29:18,  1.93s/it]
Outer training loop:  57%|#####7    | 28599/50000 [18:11:39<11:19:35,  1.91s/it]
Outer training loop:  57%|#####7    | 28600/50000 [18:11:41<11:10:03,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.00s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]

Outer training loop:  57%|#####7    | 28601/50000 [18:11:48<21:02:16,  3.54s/it]
Outer training loop:  57%|#####7    | 28602/50000 [18:11:51<18:39:39,  3.14s/it]
Outer training loop:  57%|#####7    | 28603/50000 [18:11:52<16:15:48,  2.74s/it]
Outer training loop:  57%|#####7    | 28604/50000 [18:11:54<14:30:52,  2.44s/it]
Outer training loop:  57%|#####7    | 28605/50000 [18:11:56<13:17:43,  2.24s/it]
Outer training loop:  57%|#####7    | 28606/50000 [18:11:58<12:51:55,  2.16s/it]
Outer training loop:  57%|#####7    | 28607/50000 [18:12:00<12:17:10,  2.07s/it]
Outer training loop:  57%|#####7    | 28608/50000 [18:12:01<11:49:01,  1.99s/it]
Outer training loop:  57%|#####7    | 28609/50000 [18:12:04<11:58:34,  2.02s/it]
Outer training loop:  57%|#####7    | 28610/50000 [18:12:05<11:50:52,  1.99s/it]
Outer training loop:  57%|#####7    | 28611/50000 [18:12:07<11:28:06,  1.93s/it]
Outer training loop:  57%|#####7    | 28612/50000 [18:12:09<11:37:44,  1.96s/it]
Outer training loop:  57%|#####7    | 28613/50000 [18:12:11<11:19:04,  1.91s/it]
Outer training loop:  57%|#####7    | 28614/50000 [18:12:13<11:22:56,  1.92s/it]
Outer training loop:  57%|#####7    | 28615/50000 [18:12:15<11:25:37,  1.92s/it]
Outer training loop:  57%|#####7    | 28616/50000 [18:12:17<11:22:15,  1.91s/it]
Outer training loop:  57%|#####7    | 28617/50000 [18:12:19<11:10:44,  1.88s/it]
Outer training loop:  57%|#####7    | 28618/50000 [18:12:20<10:59:50,  1.85s/it]
Outer training loop:  57%|#####7    | 28619/50000 [18:12:22<11:10:14,  1.88s/it]
Outer training loop:  57%|#####7    | 28620/50000 [18:12:24<10:58:16,  1.85s/it]
Outer training loop:  57%|#####7    | 28621/50000 [18:12:26<10:47:31,  1.82s/it]
Outer training loop:  57%|#####7    | 28622/50000 [18:12:28<11:17:37,  1.90s/it]
Outer training loop:  57%|#####7    | 28623/50000 [18:12:30<11:13:54,  1.89s/it]
Outer training loop:  57%|#####7    | 28624/50000 [18:12:32<10:59:41,  1.85s/it]
Outer training loop:  57%|#####7    | 28625/50000 [18:12:34<11:22:09,  1.91s/it]
Outer training loop:  57%|#####7    | 28626/50000 [18:12:35<11:09:34,  1.88s/it]
Outer training loop:  57%|#####7    | 28627/50000 [18:12:37<11:06:24,  1.87s/it]
Outer training loop:  57%|#####7    | 28628/50000 [18:12:39<11:21:55,  1.91s/it]
Outer training loop:  57%|#####7    | 28629/50000 [18:12:41<11:37:28,  1.96s/it]
Outer training loop:  57%|#####7    | 28630/50000 [18:12:43<11:20:08,  1.91s/it]
Outer training loop:  57%|#####7    | 28631/50000 [18:12:45<11:03:51,  1.86s/it]
Outer training loop:  57%|#####7    | 28632/50000 [18:12:47<11:26:15,  1.93s/it]
Outer training loop:  57%|#####7    | 28633/50000 [18:12:49<11:18:14,  1.90s/it]
Outer training loop:  57%|#####7    | 28634/50000 [18:12:51<11:23:51,  1.92s/it]
Outer training loop:  57%|#####7    | 28635/50000 [18:12:53<11:25:56,  1.93s/it]
Outer training loop:  57%|#####7    | 28636/50000 [18:12:55<11:26:09,  1.93s/it]
Outer training loop:  57%|#####7    | 28637/50000 [18:12:56<11:09:04,  1.88s/it]
Outer training loop:  57%|#####7    | 28638/50000 [18:12:58<11:08:15,  1.88s/it]
Outer training loop:  57%|#####7    | 28639/50000 [18:13:01<11:38:38,  1.96s/it]
Outer training loop:  57%|#####7    | 28640/50000 [18:13:02<11:20:44,  1.91s/it]
Outer training loop:  57%|#####7    | 28641/50000 [18:13:04<11:15:32,  1.90s/it]
Outer training loop:  57%|#####7    | 28642/50000 [18:13:06<11:32:41,  1.95s/it]
Outer training loop:  57%|#####7    | 28643/50000 [18:13:08<11:12:36,  1.89s/it]
Outer training loop:  57%|#####7    | 28644/50000 [18:13:10<10:59:01,  1.85s/it]
Outer training loop:  57%|#####7    | 28645/50000 [18:13:12<10:55:21,  1.84s/it]
Outer training loop:  57%|#####7    | 28646/50000 [18:13:14<11:14:02,  1.89s/it]
Outer training loop:  57%|#####7    | 28647/50000 [18:13:15<11:08:14,  1.88s/it]
Outer training loop:  57%|#####7    | 28648/50000 [18:13:17<11:07:57,  1.88s/it]
Outer training loop:  57%|#####7    | 28649/50000 [18:13:19<11:20:41,  1.91s/it]
Outer training loop:  57%|#####7    | 28650/50000 [18:13:21<11:12:03,  1.89s/it]
Outer training loop:  57%|#####7    | 28651/50000 [18:13:23<11:02:08,  1.86s/it]
Outer training loop:  57%|#####7    | 28652/50000 [18:13:25<11:03:25,  1.86s/it]
Outer training loop:  57%|#####7    | 28653/50000 [18:13:27<11:36:07,  1.96s/it]
Outer training loop:  57%|#####7    | 28654/50000 [18:13:29<11:25:04,  1.93s/it]
Outer training loop:  57%|#####7    | 28655/50000 [18:13:31<11:04:16,  1.87s/it]
Outer training loop:  57%|#####7    | 28656/50000 [18:13:33<11:27:26,  1.93s/it]
Outer training loop:  57%|#####7    | 28657/50000 [18:13:35<11:22:11,  1.92s/it]
Outer training loop:  57%|#####7    | 28658/50000 [18:13:36<11:19:37,  1.91s/it]
Outer training loop:  57%|#####7    | 28659/50000 [18:13:39<11:43:48,  1.98s/it]
Outer training loop:  57%|#####7    | 28660/50000 [18:13:40<11:30:05,  1.94s/it]
Outer training loop:  57%|#####7    | 28661/50000 [18:13:42<11:16:16,  1.90s/it]
Outer training loop:  57%|#####7    | 28662/50000 [18:13:44<11:27:09,  1.93s/it]
Outer training loop:  57%|#####7    | 28663/50000 [18:13:46<11:11:43,  1.89s/it]
Outer training loop:  57%|#####7    | 28664/50000 [18:13:48<10:58:05,  1.85s/it]
Outer training loop:  57%|#####7    | 28665/50000 [18:13:50<10:55:18,  1.84s/it]
Outer training loop:  57%|#####7    | 28666/50000 [18:13:52<11:20:35,  1.91s/it]
Outer training loop:  57%|#####7    | 28667/50000 [18:13:53<11:09:19,  1.88s/it]
Outer training loop:  57%|#####7    | 28668/50000 [18:13:55<10:57:47,  1.85s/it]
Outer training loop:  57%|#####7    | 28669/50000 [18:13:57<11:07:04,  1.88s/it]
Outer training loop:  57%|#####7    | 28670/50000 [18:13:59<10:58:09,  1.85s/it]
Outer training loop:  57%|#####7    | 28671/50000 [18:14:01<11:00:24,  1.86s/it]
Outer training loop:  57%|#####7    | 28672/50000 [18:14:03<11:16:26,  1.90s/it]
Outer training loop:  57%|#####7    | 28673/50000 [18:14:05<11:08:32,  1.88s/it]
Outer training loop:  57%|#####7    | 28674/50000 [18:14:07<11:05:28,  1.87s/it]
Outer training loop:  57%|#####7    | 28675/50000 [18:14:08<10:51:32,  1.83s/it]
Outer training loop:  57%|#####7    | 28676/50000 [18:14:10<11:05:20,  1.87s/it]
Outer training loop:  57%|#####7    | 28677/50000 [18:14:12<10:52:44,  1.84s/it]
Outer training loop:  57%|#####7    | 28678/50000 [18:14:14<11:23:49,  1.92s/it]
Outer training loop:  57%|#####7    | 28679/50000 [18:14:16<11:23:44,  1.92s/it]
Outer training loop:  57%|#####7    | 28680/50000 [18:14:18<11:13:06,  1.89s/it]
Outer training loop:  57%|#####7    | 28681/50000 [18:14:20<11:22:15,  1.92s/it]
Outer training loop:  57%|#####7    | 28682/50000 [18:14:22<11:33:55,  1.95s/it]
Outer training loop:  57%|#####7    | 28683/50000 [18:14:24<11:12:48,  1.89s/it]
Outer training loop:  57%|#####7    | 28684/50000 [18:14:25<11:04:47,  1.87s/it]
Outer training loop:  57%|#####7    | 28685/50000 [18:14:27<10:59:02,  1.86s/it]
Outer training loop:  57%|#####7    | 28686/50000 [18:14:29<11:11:52,  1.89s/it]
Outer training loop:  57%|#####7    | 28687/50000 [18:14:31<11:02:59,  1.87s/it]
Outer training loop:  57%|#####7    | 28688/50000 [18:14:33<10:57:07,  1.85s/it]
Outer training loop:  57%|#####7    | 28689/50000 [18:14:35<11:14:41,  1.90s/it]
Outer training loop:  57%|#####7    | 28690/50000 [18:14:37<10:56:58,  1.85s/it]
Outer training loop:  57%|#####7    | 28691/50000 [18:14:38<10:55:10,  1.84s/it]
Outer training loop:  57%|#####7    | 28692/50000 [18:14:41<11:27:15,  1.94s/it]
Outer training loop:  57%|#####7    | 28693/50000 [18:14:42<11:21:39,  1.92s/it]
Outer training loop:  57%|#####7    | 28694/50000 [18:14:44<10:59:30,  1.86s/it]
Outer training loop:  57%|#####7    | 28695/50000 [18:14:46<11:04:58,  1.87s/it]
Outer training loop:  57%|#####7    | 28696/50000 [18:14:48<11:27:52,  1.94s/it]
Outer training loop:  57%|#####7    | 28697/50000 [18:14:50<11:23:39,  1.93s/it]
Outer training loop:  57%|#####7    | 28698/50000 [18:14:52<11:09:28,  1.89s/it]
Outer training loop:  57%|#####7    | 28699/50000 [18:14:54<11:09:10,  1.88s/it]
Outer training loop:  57%|#####7    | 28700/50000 [18:14:56<10:58:56,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  57%|#####7    | 28701/50000 [18:15:03<20:36:30,  3.48s/it]
Outer training loop:  57%|#####7    | 28702/50000 [18:15:05<17:43:59,  3.00s/it]
Outer training loop:  57%|#####7    | 28703/50000 [18:15:07<15:39:43,  2.65s/it]
Outer training loop:  57%|#####7    | 28704/50000 [18:15:08<14:18:58,  2.42s/it]
Outer training loop:  57%|#####7    | 28705/50000 [18:15:10<13:30:07,  2.28s/it]
Outer training loop:  57%|#####7    | 28706/50000 [18:15:12<12:46:46,  2.16s/it]
Outer training loop:  57%|#####7    | 28707/50000 [18:15:14<12:25:42,  2.10s/it]
Outer training loop:  57%|#####7    | 28708/50000 [18:15:16<11:56:07,  2.02s/it]
Outer training loop:  57%|#####7    | 28709/50000 [18:15:18<11:40:09,  1.97s/it]
Outer training loop:  57%|#####7    | 28710/50000 [18:15:20<11:29:09,  1.94s/it]
Outer training loop:  57%|#####7    | 28711/50000 [18:15:22<11:37:38,  1.97s/it]
Outer training loop:  57%|#####7    | 28712/50000 [18:15:24<11:25:00,  1.93s/it]
Outer training loop:  57%|#####7    | 28713/50000 [18:15:26<11:32:44,  1.95s/it]
Outer training loop:  57%|#####7    | 28714/50000 [18:15:28<11:45:44,  1.99s/it]
Outer training loop:  57%|#####7    | 28715/50000 [18:15:30<11:27:06,  1.94s/it]
Outer training loop:  57%|#####7    | 28716/50000 [18:15:31<11:09:35,  1.89s/it]
Outer training loop:  57%|#####7    | 28717/50000 [18:15:33<11:17:41,  1.91s/it]
Outer training loop:  57%|#####7    | 28718/50000 [18:15:35<11:03:46,  1.87s/it]
Outer training loop:  57%|#####7    | 28719/50000 [18:15:37<10:54:50,  1.85s/it]
Outer training loop:  57%|#####7    | 28720/50000 [18:15:39<11:19:44,  1.92s/it]
Outer training loop:  57%|#####7    | 28721/50000 [18:15:41<11:14:44,  1.90s/it]
Outer training loop:  57%|#####7    | 28722/50000 [18:15:43<11:05:33,  1.88s/it]
Outer training loop:  57%|#####7    | 28723/50000 [18:15:44<11:02:37,  1.87s/it]
Outer training loop:  57%|#####7    | 28724/50000 [18:15:47<11:31:44,  1.95s/it]
Outer training loop:  57%|#####7    | 28725/50000 [18:15:49<11:38:00,  1.97s/it]
Outer training loop:  57%|#####7    | 28726/50000 [18:15:51<11:38:45,  1.97s/it]
Outer training loop:  57%|#####7    | 28727/50000 [18:15:52<11:28:20,  1.94s/it]
Outer training loop:  57%|#####7    | 28728/50000 [18:15:54<11:22:28,  1.92s/it]
Outer training loop:  57%|#####7    | 28729/50000 [18:15:56<11:13:36,  1.90s/it]
Outer training loop:  57%|#####7    | 28730/50000 [18:15:58<11:42:11,  1.98s/it]
Outer training loop:  57%|#####7    | 28731/50000 [18:16:00<11:26:13,  1.94s/it]
Outer training loop:  57%|#####7    | 28732/50000 [18:16:02<11:06:18,  1.88s/it]
Outer training loop:  57%|#####7    | 28733/50000 [18:16:04<11:24:04,  1.93s/it]
Outer training loop:  57%|#####7    | 28734/50000 [18:16:06<11:15:17,  1.91s/it]
Outer training loop:  57%|#####7    | 28735/50000 [18:16:08<11:07:08,  1.88s/it]
Outer training loop:  57%|#####7    | 28736/50000 [18:16:09<10:59:55,  1.86s/it]
Outer training loop:  57%|#####7    | 28737/50000 [18:16:12<11:16:35,  1.91s/it]
Outer training loop:  57%|#####7    | 28738/50000 [18:16:13<11:11:09,  1.89s/it]
Outer training loop:  57%|#####7    | 28739/50000 [18:16:15<11:11:37,  1.90s/it]
Outer training loop:  57%|#####7    | 28740/50000 [18:16:17<11:17:21,  1.91s/it]
Outer training loop:  57%|#####7    | 28741/50000 [18:16:19<11:08:11,  1.89s/it]
Outer training loop:  57%|#####7    | 28742/50000 [18:16:21<10:48:30,  1.83s/it]
Outer training loop:  57%|#####7    | 28743/50000 [18:16:23<10:59:23,  1.86s/it]
Outer training loop:  57%|#####7    | 28744/50000 [18:16:25<11:09:09,  1.89s/it]
Outer training loop:  57%|#####7    | 28745/50000 [18:16:27<11:10:16,  1.89s/it]
Outer training loop:  57%|#####7    | 28746/50000 [18:16:28<10:57:16,  1.86s/it]
Outer training loop:  57%|#####7    | 28747/50000 [18:16:30<11:18:44,  1.92s/it]
Outer training loop:  57%|#####7    | 28748/50000 [18:16:32<11:03:05,  1.87s/it]
Outer training loop:  57%|#####7    | 28749/50000 [18:16:34<10:56:00,  1.85s/it]
Outer training loop:  57%|#####7    | 28750/50000 [18:16:36<11:23:53,  1.93s/it]
Outer training loop:  58%|#####7    | 28751/50000 [18:16:38<11:03:09,  1.87s/it]
Outer training loop:  58%|#####7    | 28752/50000 [18:16:40<10:50:24,  1.84s/it]
Outer training loop:  58%|#####7    | 28753/50000 [18:16:41<10:43:26,  1.82s/it]
Outer training loop:  58%|#####7    | 28754/50000 [18:16:43<11:11:34,  1.90s/it]
Outer training loop:  58%|#####7    | 28755/50000 [18:16:45<11:09:05,  1.89s/it]
Outer training loop:  58%|#####7    | 28756/50000 [18:16:47<10:53:06,  1.84s/it]
Outer training loop:  58%|#####7    | 28757/50000 [18:16:49<11:00:07,  1.86s/it]
Outer training loop:  58%|#####7    | 28758/50000 [18:16:51<10:54:50,  1.85s/it]
Outer training loop:  58%|#####7    | 28759/50000 [18:16:52<10:37:45,  1.80s/it]
Outer training loop:  58%|#####7    | 28760/50000 [18:16:55<11:17:46,  1.91s/it]
Outer training loop:  58%|#####7    | 28761/50000 [18:16:56<11:00:55,  1.87s/it]
Outer training loop:  58%|#####7    | 28762/50000 [18:16:58<10:50:13,  1.84s/it]
Outer training loop:  58%|#####7    | 28763/50000 [18:17:00<10:38:37,  1.80s/it]
Outer training loop:  58%|#####7    | 28764/50000 [18:17:02<10:54:06,  1.85s/it]
Outer training loop:  58%|#####7    | 28765/50000 [18:17:04<10:50:10,  1.84s/it]
Outer training loop:  58%|#####7    | 28766/50000 [18:17:05<10:49:49,  1.84s/it]
Outer training loop:  58%|#####7    | 28767/50000 [18:17:07<11:04:37,  1.88s/it]
Outer training loop:  58%|#####7    | 28768/50000 [18:17:09<11:06:48,  1.88s/it]
Outer training loop:  58%|#####7    | 28769/50000 [18:17:11<11:01:50,  1.87s/it]
Outer training loop:  58%|#####7    | 28770/50000 [18:17:13<10:59:12,  1.86s/it]
Outer training loop:  58%|#####7    | 28771/50000 [18:17:15<11:13:14,  1.90s/it]
Outer training loop:  58%|#####7    | 28772/50000 [18:17:17<11:15:24,  1.91s/it]
Outer training loop:  58%|#####7    | 28773/50000 [18:17:19<11:08:21,  1.89s/it]
Outer training loop:  58%|#####7    | 28774/50000 [18:17:21<11:31:57,  1.96s/it]
Outer training loop:  58%|#####7    | 28775/50000 [18:17:23<11:50:14,  2.01s/it]
Outer training loop:  58%|#####7    | 28776/50000 [18:17:25<11:27:30,  1.94s/it]
Outer training loop:  58%|#####7    | 28777/50000 [18:17:27<11:14:28,  1.91s/it]
Outer training loop:  58%|#####7    | 28778/50000 [18:17:29<11:35:58,  1.97s/it]
Outer training loop:  58%|#####7    | 28779/50000 [18:17:30<11:13:31,  1.90s/it]
Outer training loop:  58%|#####7    | 28780/50000 [18:17:32<11:08:38,  1.89s/it]
Outer training loop:  58%|#####7    | 28781/50000 [18:17:34<11:33:22,  1.96s/it]
Outer training loop:  58%|#####7    | 28782/50000 [18:17:36<11:21:24,  1.93s/it]
Outer training loop:  58%|#####7    | 28783/50000 [18:17:38<11:09:07,  1.89s/it]
Outer training loop:  58%|#####7    | 28784/50000 [18:17:40<11:32:08,  1.96s/it]
Outer training loop:  58%|#####7    | 28785/50000 [18:17:42<11:17:00,  1.91s/it]
Outer training loop:  58%|#####7    | 28786/50000 [18:17:44<11:09:39,  1.89s/it]
Outer training loop:  58%|#####7    | 28787/50000 [18:17:46<11:07:17,  1.89s/it]
Outer training loop:  58%|#####7    | 28788/50000 [18:17:48<11:17:36,  1.92s/it]
Outer training loop:  58%|#####7    | 28789/50000 [18:17:50<11:16:16,  1.91s/it]
Outer training loop:  58%|#####7    | 28790/50000 [18:17:51<11:05:02,  1.88s/it]
Outer training loop:  58%|#####7    | 28791/50000 [18:17:53<11:13:11,  1.90s/it]
Outer training loop:  58%|#####7    | 28792/50000 [18:17:55<11:14:26,  1.91s/it]
Outer training loop:  58%|#####7    | 28793/50000 [18:17:57<10:58:14,  1.86s/it]
Outer training loop:  58%|#####7    | 28794/50000 [18:17:59<11:20:03,  1.92s/it]
Outer training loop:  58%|#####7    | 28795/50000 [18:18:01<11:08:00,  1.89s/it]
Outer training loop:  58%|#####7    | 28796/50000 [18:18:03<11:00:05,  1.87s/it]
Outer training loop:  58%|#####7    | 28797/50000 [18:18:05<11:08:03,  1.89s/it]
Outer training loop:  58%|#####7    | 28798/50000 [18:18:07<11:31:39,  1.96s/it]
Outer training loop:  58%|#####7    | 28799/50000 [18:18:09<11:12:00,  1.90s/it]
Outer training loop:  58%|#####7    | 28800/50000 [18:18:10<10:55:04,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.74s/it]

Outer training loop:  58%|#####7    | 28801/50000 [18:18:18<20:39:32,  3.51s/it]
Outer training loop:  58%|#####7    | 28802/50000 [18:18:20<18:12:03,  3.09s/it]
Outer training loop:  58%|#####7    | 28803/50000 [18:18:22<15:50:05,  2.69s/it]
Outer training loop:  58%|#####7    | 28804/50000 [18:18:23<14:03:14,  2.39s/it]
Outer training loop:  58%|#####7    | 28805/50000 [18:18:25<13:42:30,  2.33s/it]
Outer training loop:  58%|#####7    | 28806/50000 [18:18:27<12:47:44,  2.17s/it]
Outer training loop:  58%|#####7    | 28807/50000 [18:18:29<12:25:23,  2.11s/it]
Outer training loop:  58%|#####7    | 28808/50000 [18:18:31<12:17:42,  2.09s/it]
Outer training loop:  58%|#####7    | 28809/50000 [18:18:33<12:00:10,  2.04s/it]
Outer training loop:  58%|#####7    | 28810/50000 [18:18:35<11:48:52,  2.01s/it]
Outer training loop:  58%|#####7    | 28811/50000 [18:18:37<11:33:42,  1.96s/it]
Outer training loop:  58%|#####7    | 28812/50000 [18:18:39<11:31:16,  1.96s/it]
Outer training loop:  58%|#####7    | 28813/50000 [18:18:41<11:17:04,  1.92s/it]
Outer training loop:  58%|#####7    | 28814/50000 [18:18:43<11:21:25,  1.93s/it]
Outer training loop:  58%|#####7    | 28815/50000 [18:18:45<11:33:31,  1.96s/it]
Outer training loop:  58%|#####7    | 28816/50000 [18:18:47<11:10:39,  1.90s/it]
Outer training loop:  58%|#####7    | 28817/50000 [18:18:48<11:00:21,  1.87s/it]
Outer training loop:  58%|#####7    | 28818/50000 [18:18:50<10:57:05,  1.86s/it]
Outer training loop:  58%|#####7    | 28819/50000 [18:18:52<10:57:48,  1.86s/it]
Outer training loop:  58%|#####7    | 28820/50000 [18:18:54<10:50:12,  1.84s/it]
Outer training loop:  58%|#####7    | 28821/50000 [18:18:56<10:59:20,  1.87s/it]
Outer training loop:  58%|#####7    | 28822/50000 [18:18:58<11:42:04,  1.99s/it]
Outer training loop:  58%|#####7    | 28823/50000 [18:19:00<11:18:50,  1.92s/it]
Outer training loop:  58%|#####7    | 28824/50000 [18:19:02<11:03:15,  1.88s/it]
Outer training loop:  58%|#####7    | 28825/50000 [18:19:03<10:56:53,  1.86s/it]
Outer training loop:  58%|#####7    | 28826/50000 [18:19:06<11:24:43,  1.94s/it]
Outer training loop:  58%|#####7    | 28827/50000 [18:19:07<11:13:14,  1.91s/it]
Outer training loop:  58%|#####7    | 28828/50000 [18:19:09<11:22:37,  1.93s/it]
Outer training loop:  58%|#####7    | 28829/50000 [18:19:11<11:33:01,  1.96s/it]
Outer training loop:  58%|#####7    | 28830/50000 [18:19:13<11:19:56,  1.93s/it]
Outer training loop:  58%|#####7    | 28831/50000 [18:19:15<11:08:52,  1.90s/it]
Outer training loop:  58%|#####7    | 28832/50000 [18:19:17<11:14:20,  1.91s/it]
Outer training loop:  58%|#####7    | 28833/50000 [18:19:19<11:11:37,  1.90s/it]
Outer training loop:  58%|#####7    | 28834/50000 [18:19:21<11:07:43,  1.89s/it]
Outer training loop:  58%|#####7    | 28835/50000 [18:19:23<10:58:56,  1.87s/it]
Outer training loop:  58%|#####7    | 28836/50000 [18:19:25<11:35:41,  1.97s/it]
Outer training loop:  58%|#####7    | 28837/50000 [18:19:27<11:18:36,  1.92s/it]
Outer training loop:  58%|#####7    | 28838/50000 [18:19:28<11:17:13,  1.92s/it]
Outer training loop:  58%|#####7    | 28839/50000 [18:19:31<11:32:07,  1.96s/it]
Outer training loop:  58%|#####7    | 28840/50000 [18:19:32<11:24:14,  1.94s/it]
Outer training loop:  58%|#####7    | 28841/50000 [18:19:34<11:18:34,  1.92s/it]
Outer training loop:  58%|#####7    | 28842/50000 [18:19:36<11:31:46,  1.96s/it]
Outer training loop:  58%|#####7    | 28843/50000 [18:19:38<11:18:54,  1.93s/it]
Outer training loop:  58%|#####7    | 28844/50000 [18:19:40<11:14:57,  1.91s/it]
Outer training loop:  58%|#####7    | 28845/50000 [18:19:42<11:02:34,  1.88s/it]
Outer training loop:  58%|#####7    | 28846/50000 [18:19:44<11:27:35,  1.95s/it]
Outer training loop:  58%|#####7    | 28847/50000 [18:19:46<11:29:57,  1.96s/it]
Outer training loop:  58%|#####7    | 28848/50000 [18:19:48<11:15:22,  1.92s/it]
Outer training loop:  58%|#####7    | 28849/50000 [18:19:50<11:33:12,  1.97s/it]
Outer training loop:  58%|#####7    | 28850/50000 [18:19:52<11:27:39,  1.95s/it]
Outer training loop:  58%|#####7    | 28851/50000 [18:19:54<11:06:58,  1.89s/it]
Outer training loop:  58%|#####7    | 28852/50000 [18:19:56<11:31:03,  1.96s/it]
Outer training loop:  58%|#####7    | 28853/50000 [18:19:58<11:28:39,  1.95s/it]
Outer training loop:  58%|#####7    | 28854/50000 [18:19:59<11:17:04,  1.92s/it]
Outer training loop:  58%|#####7    | 28855/50000 [18:20:02<11:39:57,  1.99s/it]
Outer training loop:  58%|#####7    | 28856/50000 [18:20:04<11:42:25,  1.99s/it]
Outer training loop:  58%|#####7    | 28857/50000 [18:20:05<11:30:21,  1.96s/it]
Outer training loop:  58%|#####7    | 28858/50000 [18:20:08<11:41:00,  1.99s/it]
Outer training loop:  58%|#####7    | 28859/50000 [18:20:09<11:31:59,  1.96s/it]
Outer training loop:  58%|#####7    | 28860/50000 [18:20:11<11:29:59,  1.96s/it]
Outer training loop:  58%|#####7    | 28861/50000 [18:20:13<11:19:20,  1.93s/it]
Outer training loop:  58%|#####7    | 28862/50000 [18:20:15<11:25:48,  1.95s/it]
Outer training loop:  58%|#####7    | 28863/50000 [18:20:17<11:03:01,  1.88s/it]
Outer training loop:  58%|#####7    | 28864/50000 [18:20:19<10:48:27,  1.84s/it]
Outer training loop:  58%|#####7    | 28865/50000 [18:20:21<11:08:26,  1.90s/it]
Outer training loop:  58%|#####7    | 28866/50000 [18:20:23<10:57:09,  1.87s/it]
Outer training loop:  58%|#####7    | 28867/50000 [18:20:25<11:10:06,  1.90s/it]
Outer training loop:  58%|#####7    | 28868/50000 [18:20:26<11:06:11,  1.89s/it]
Outer training loop:  58%|#####7    | 28869/50000 [18:20:28<11:11:58,  1.91s/it]
Outer training loop:  58%|#####7    | 28870/50000 [18:20:30<10:56:43,  1.86s/it]
Outer training loop:  58%|#####7    | 28871/50000 [18:20:32<10:42:09,  1.82s/it]
Outer training loop:  58%|#####7    | 28872/50000 [18:20:34<11:23:44,  1.94s/it]
Outer training loop:  58%|#####7    | 28873/50000 [18:20:36<10:59:23,  1.87s/it]
Outer training loop:  58%|#####7    | 28874/50000 [18:20:38<10:59:01,  1.87s/it]
Outer training loop:  58%|#####7    | 28875/50000 [18:20:39<10:56:49,  1.87s/it]
Outer training loop:  58%|#####7    | 28876/50000 [18:20:42<11:23:07,  1.94s/it]
Outer training loop:  58%|#####7    | 28877/50000 [18:20:44<11:38:29,  1.98s/it]
Outer training loop:  58%|#####7    | 28878/50000 [18:20:45<11:13:13,  1.91s/it]
Outer training loop:  58%|#####7    | 28879/50000 [18:20:47<11:11:27,  1.91s/it]
Outer training loop:  58%|#####7    | 28880/50000 [18:20:49<11:06:53,  1.89s/it]
Outer training loop:  58%|#####7    | 28881/50000 [18:20:51<11:04:12,  1.89s/it]
Outer training loop:  58%|#####7    | 28882/50000 [18:20:53<11:22:05,  1.94s/it]
Outer training loop:  58%|#####7    | 28883/50000 [18:20:55<11:24:01,  1.94s/it]
Outer training loop:  58%|#####7    | 28884/50000 [18:20:57<11:16:32,  1.92s/it]
Outer training loop:  58%|#####7    | 28885/50000 [18:20:59<11:25:38,  1.95s/it]
Outer training loop:  58%|#####7    | 28886/50000 [18:21:01<11:10:12,  1.90s/it]
Outer training loop:  58%|#####7    | 28887/50000 [18:21:03<11:03:16,  1.88s/it]
Outer training loop:  58%|#####7    | 28888/50000 [18:21:04<10:56:51,  1.87s/it]
Outer training loop:  58%|#####7    | 28889/50000 [18:21:07<11:18:21,  1.93s/it]
Outer training loop:  58%|#####7    | 28890/50000 [18:21:08<11:01:44,  1.88s/it]
Outer training loop:  58%|#####7    | 28891/50000 [18:21:10<10:59:01,  1.87s/it]
Outer training loop:  58%|#####7    | 28892/50000 [18:21:12<11:12:28,  1.91s/it]
Outer training loop:  58%|#####7    | 28893/50000 [18:21:14<11:20:23,  1.93s/it]
Outer training loop:  58%|#####7    | 28894/50000 [18:21:16<11:06:01,  1.89s/it]
Outer training loop:  58%|#####7    | 28895/50000 [18:21:18<10:55:34,  1.86s/it]
Outer training loop:  58%|#####7    | 28896/50000 [18:21:20<11:27:13,  1.95s/it]
Outer training loop:  58%|#####7    | 28897/50000 [18:21:22<11:09:36,  1.90s/it]
Outer training loop:  58%|#####7    | 28898/50000 [18:21:23<10:56:01,  1.87s/it]
Outer training loop:  58%|#####7    | 28899/50000 [18:21:26<11:25:07,  1.95s/it]
Outer training loop:  58%|#####7    | 28900/50000 [18:21:27<11:14:47,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  58%|#####7    | 28901/50000 [18:21:35<20:41:59,  3.53s/it]
Outer training loop:  58%|#####7    | 28902/50000 [18:21:37<17:43:06,  3.02s/it]
Outer training loop:  58%|#####7    | 28903/50000 [18:21:39<15:52:46,  2.71s/it]
Outer training loop:  58%|#####7    | 28904/50000 [18:21:40<14:23:32,  2.46s/it]
Outer training loop:  58%|#####7    | 28905/50000 [18:21:42<13:25:10,  2.29s/it]
Outer training loop:  58%|#####7    | 28906/50000 [18:21:44<12:27:29,  2.13s/it]
Outer training loop:  58%|#####7    | 28907/50000 [18:21:46<11:58:57,  2.05s/it]
Outer training loop:  58%|#####7    | 28908/50000 [18:21:48<11:33:24,  1.97s/it]
Outer training loop:  58%|#####7    | 28909/50000 [18:21:49<11:14:12,  1.92s/it]
Outer training loop:  58%|#####7    | 28910/50000 [18:21:51<11:18:25,  1.93s/it]
Outer training loop:  58%|#####7    | 28911/50000 [18:21:53<11:13:25,  1.92s/it]
Outer training loop:  58%|#####7    | 28912/50000 [18:21:55<11:09:39,  1.91s/it]
Outer training loop:  58%|#####7    | 28913/50000 [18:21:57<11:17:57,  1.93s/it]
Outer training loop:  58%|#####7    | 28914/50000 [18:21:59<11:05:29,  1.89s/it]
Outer training loop:  58%|#####7    | 28915/50000 [18:22:01<10:51:44,  1.85s/it]
Outer training loop:  58%|#####7    | 28916/50000 [18:22:03<10:40:52,  1.82s/it]
Outer training loop:  58%|#####7    | 28917/50000 [18:22:05<11:09:23,  1.91s/it]
Outer training loop:  58%|#####7    | 28918/50000 [18:22:06<10:58:23,  1.87s/it]
Outer training loop:  58%|#####7    | 28919/50000 [18:22:08<11:11:07,  1.91s/it]
Outer training loop:  58%|#####7    | 28920/50000 [18:22:10<11:24:36,  1.95s/it]
Outer training loop:  58%|#####7    | 28921/50000 [18:22:12<11:04:24,  1.89s/it]
Outer training loop:  58%|#####7    | 28922/50000 [18:22:14<10:54:32,  1.86s/it]
Outer training loop:  58%|#####7    | 28923/50000 [18:22:16<10:44:51,  1.84s/it]
Outer training loop:  58%|#####7    | 28924/50000 [18:22:18<11:10:55,  1.91s/it]
Outer training loop:  58%|#####7    | 28925/50000 [18:22:20<11:07:19,  1.90s/it]
Outer training loop:  58%|#####7    | 28926/50000 [18:22:22<10:59:12,  1.88s/it]
Outer training loop:  58%|#####7    | 28927/50000 [18:22:24<11:17:26,  1.93s/it]
Outer training loop:  58%|#####7    | 28928/50000 [18:22:25<11:10:42,  1.91s/it]
Outer training loop:  58%|#####7    | 28929/50000 [18:22:27<10:57:59,  1.87s/it]
Outer training loop:  58%|#####7    | 28930/50000 [18:22:29<10:52:19,  1.86s/it]
Outer training loop:  58%|#####7    | 28931/50000 [18:22:31<10:54:07,  1.86s/it]
Outer training loop:  58%|#####7    | 28932/50000 [18:22:33<10:47:57,  1.85s/it]
Outer training loop:  58%|#####7    | 28933/50000 [18:22:35<10:45:57,  1.84s/it]
Outer training loop:  58%|#####7    | 28934/50000 [18:22:37<11:06:17,  1.90s/it]
Outer training loop:  58%|#####7    | 28935/50000 [18:22:38<10:50:20,  1.85s/it]
Outer training loop:  58%|#####7    | 28936/50000 [18:22:40<10:52:20,  1.86s/it]
Outer training loop:  58%|#####7    | 28937/50000 [18:22:42<10:59:06,  1.88s/it]
Outer training loop:  58%|#####7    | 28938/50000 [18:22:44<10:59:50,  1.88s/it]
Outer training loop:  58%|#####7    | 28939/50000 [18:22:46<10:45:36,  1.84s/it]
Outer training loop:  58%|#####7    | 28940/50000 [18:22:47<10:24:04,  1.78s/it]
Outer training loop:  58%|#####7    | 28941/50000 [18:22:49<10:45:28,  1.84s/it]
Outer training loop:  58%|#####7    | 28942/50000 [18:22:51<11:00:30,  1.88s/it]
Outer training loop:  58%|#####7    | 28943/50000 [18:22:53<10:52:19,  1.86s/it]
Outer training loop:  58%|#####7    | 28944/50000 [18:22:55<10:58:56,  1.88s/it]
Outer training loop:  58%|#####7    | 28945/50000 [18:22:57<10:47:39,  1.85s/it]
Outer training loop:  58%|#####7    | 28946/50000 [18:22:59<10:47:40,  1.85s/it]
Outer training loop:  58%|#####7    | 28947/50000 [18:23:01<10:41:37,  1.83s/it]
Outer training loop:  58%|#####7    | 28948/50000 [18:23:02<10:53:15,  1.86s/it]
Outer training loop:  58%|#####7    | 28949/50000 [18:23:04<10:54:26,  1.87s/it]
Outer training loop:  58%|#####7    | 28950/50000 [18:23:06<10:54:12,  1.86s/it]
Outer training loop:  58%|#####7    | 28951/50000 [18:23:08<11:08:42,  1.91s/it]
Outer training loop:  58%|#####7    | 28952/50000 [18:23:10<11:00:15,  1.88s/it]
Outer training loop:  58%|#####7    | 28953/50000 [18:23:12<10:44:47,  1.84s/it]
Outer training loop:  58%|#####7    | 28954/50000 [18:23:14<10:40:01,  1.82s/it]
Outer training loop:  58%|#####7    | 28955/50000 [18:23:16<11:01:33,  1.89s/it]
Outer training loop:  58%|#####7    | 28956/50000 [18:23:17<10:48:58,  1.85s/it]
Outer training loop:  58%|#####7    | 28957/50000 [18:23:19<10:43:57,  1.84s/it]
Outer training loop:  58%|#####7    | 28958/50000 [18:23:21<11:01:41,  1.89s/it]
Outer training loop:  58%|#####7    | 28959/50000 [18:23:23<10:57:21,  1.87s/it]
Outer training loop:  58%|#####7    | 28960/50000 [18:23:25<10:43:43,  1.84s/it]
Outer training loop:  58%|#####7    | 28961/50000 [18:23:26<10:32:39,  1.80s/it]
Outer training loop:  58%|#####7    | 28962/50000 [18:23:29<10:54:06,  1.87s/it]
Outer training loop:  58%|#####7    | 28963/50000 [18:23:30<10:38:25,  1.82s/it]
Outer training loop:  58%|#####7    | 28964/50000 [18:23:32<10:29:26,  1.80s/it]
Outer training loop:  58%|#####7    | 28965/50000 [18:23:34<10:53:22,  1.86s/it]
Outer training loop:  58%|#####7    | 28966/50000 [18:23:36<10:55:19,  1.87s/it]
Outer training loop:  58%|#####7    | 28967/50000 [18:23:38<10:32:27,  1.80s/it]
Outer training loop:  58%|#####7    | 28968/50000 [18:23:39<10:36:52,  1.82s/it]
Outer training loop:  58%|#####7    | 28969/50000 [18:23:41<10:56:43,  1.87s/it]
Outer training loop:  58%|#####7    | 28970/50000 [18:23:43<10:46:50,  1.85s/it]
Outer training loop:  58%|#####7    | 28971/50000 [18:23:45<11:00:56,  1.89s/it]
Outer training loop:  58%|#####7    | 28972/50000 [18:23:47<10:40:44,  1.83s/it]
Outer training loop:  58%|#####7    | 28973/50000 [18:23:49<10:35:15,  1.81s/it]
Outer training loop:  58%|#####7    | 28974/50000 [18:23:50<10:37:21,  1.82s/it]
Outer training loop:  58%|#####7    | 28975/50000 [18:23:52<10:52:30,  1.86s/it]
Outer training loop:  58%|#####7    | 28976/50000 [18:23:54<10:39:35,  1.83s/it]
Outer training loop:  58%|#####7    | 28977/50000 [18:23:56<10:30:21,  1.80s/it]
Outer training loop:  58%|#####7    | 28978/50000 [18:23:58<10:44:34,  1.84s/it]
Outer training loop:  58%|#####7    | 28979/50000 [18:24:00<10:30:02,  1.80s/it]
Outer training loop:  58%|#####7    | 28980/50000 [18:24:01<10:28:40,  1.79s/it]
Outer training loop:  58%|#####7    | 28981/50000 [18:24:03<10:49:00,  1.85s/it]
Outer training loop:  58%|#####7    | 28982/50000 [18:24:05<10:52:49,  1.86s/it]
Outer training loop:  58%|#####7    | 28983/50000 [18:24:07<10:41:14,  1.83s/it]
Outer training loop:  58%|#####7    | 28984/50000 [18:24:09<10:42:39,  1.83s/it]
Outer training loop:  58%|#####7    | 28985/50000 [18:24:11<10:58:35,  1.88s/it]
Outer training loop:  58%|#####7    | 28986/50000 [18:24:13<10:55:06,  1.87s/it]
Outer training loop:  58%|#####7    | 28987/50000 [18:24:14<10:43:23,  1.84s/it]
Outer training loop:  58%|#####7    | 28988/50000 [18:24:16<10:54:08,  1.87s/it]
Outer training loop:  58%|#####7    | 28989/50000 [18:24:18<10:39:55,  1.83s/it]
Outer training loop:  58%|#####7    | 28990/50000 [18:24:20<10:40:35,  1.83s/it]
Outer training loop:  58%|#####7    | 28991/50000 [18:24:22<10:22:06,  1.78s/it]
Outer training loop:  58%|#####7    | 28992/50000 [18:24:23<10:39:41,  1.83s/it]
Outer training loop:  58%|#####7    | 28993/50000 [18:24:25<10:28:23,  1.79s/it]
Outer training loop:  58%|#####7    | 28994/50000 [18:24:27<10:32:54,  1.81s/it]
Outer training loop:  58%|#####7    | 28995/50000 [18:24:29<10:50:28,  1.86s/it]
Outer training loop:  58%|#####7    | 28996/50000 [18:24:31<10:39:07,  1.83s/it]
Outer training loop:  58%|#####7    | 28997/50000 [18:24:32<10:28:45,  1.80s/it]
Outer training loop:  58%|#####7    | 28998/50000 [18:24:34<10:28:07,  1.79s/it]
Outer training loop:  58%|#####7    | 28999/50000 [18:24:36<10:51:42,  1.86s/it]
Outer training loop:  58%|#####8    | 29000/50000 [18:24:38<10:46:14,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.99s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:42,  1.98s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:35,  1.92s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:24,  1.83s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.89s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:21,  1.84s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.83s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:19,  1.86s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:14,  1.84s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:11,  1.83s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:09,  1.85s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:06,  1.83s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:07,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:02,  1.82s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:02,  1.85s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:58,  1.82s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:57,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:36<01:56,  1.85s/it]

Evaluating policy:  25%|##5       | 21/83 [00:38<01:53,  1.83s/it]

Evaluating policy:  27%|##6       | 22/83 [00:40<01:57,  1.92s/it]

Evaluating policy:  28%|##7       | 23/83 [00:42<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:47,  1.85s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:44,  1.83s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:50,  1.97s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:53<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:55<01:38,  1.86s/it]

Evaluating policy:  37%|###7      | 31/83 [00:57<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:39,  1.95s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:36,  1.92s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:32,  1.89s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:26,  1.84s/it]

Evaluating policy:  45%|####4     | 37/83 [01:08<01:23,  1.81s/it]

Evaluating policy:  46%|####5     | 38/83 [01:10<01:25,  1.89s/it]

Evaluating policy:  47%|####6     | 39/83 [01:12<01:22,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:14<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:15,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:19<01:13,  1.84s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:21<01:11,  1.84s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:23<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:25<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:27<01:06,  1.86s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:29<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:01,  1.81s/it]

Evaluating policy:  60%|######    | 50/83 [01:32<00:59,  1.81s/it]

Evaluating policy:  61%|######1   | 51/83 [01:34<00:58,  1.83s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<01:00,  1.96s/it]

Evaluating policy:  64%|######3   | 53/83 [01:38<00:57,  1.90s/it]

Evaluating policy:  65%|######5   | 54/83 [01:40<00:54,  1.87s/it]

Evaluating policy:  66%|######6   | 55/83 [01:42<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:50,  1.86s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:47<00:46,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:46,  1.96s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:44,  1.94s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:53<00:42,  1.93s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:55<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:57<00:37,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:31,  1.97s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:29,  1.95s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.90s/it]

Evaluating policy:  84%|########4 | 70/83 [02:10<00:24,  1.89s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:22,  1.91s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:20,  1.88s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.87s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.89s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:14,  2.03s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.99s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.95s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.97s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.94s/it]

Evaluating policy:  98%|#########7| 81/83 [02:31<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.84s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:31,  1.85s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:29,  1.85s/it]

Evaluating policy:   4%|3         | 3/83 [00:06<02:42,  2.04s/it]

Evaluating policy:   5%|4         | 4/83 [00:08<02:37,  1.99s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:30,  1.93s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:30,  1.95s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:30,  1.98s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:26,  1.95s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:23,  1.97s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.97s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:11,  1.88s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:16,  1.97s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:14,  1.97s/it]

Evaluating policy:  19%|#9        | 16/83 [00:31<02:09,  1.93s/it]

Evaluating policy:  20%|##        | 17/83 [00:33<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:35<02:06,  1.95s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:02,  1.92s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:58,  1.97s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:53,  1.93s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:46,  1.87s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:37,  1.87s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:37,  1.92s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:36,  1.98s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:34,  1.96s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:31,  1.95s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:24,  1.87s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:17<01:24,  1.97s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:21,  1.93s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:16,  1.91s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:13,  1.89s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:13,  2.00s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:10,  1.96s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:07,  1.92s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:04,  1.91s/it]

Evaluating policy:  60%|######    | 50/83 [01:36<01:02,  1.88s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:55,  1.84s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:56,  1.96s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:53,  1.93s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:52,  1.94s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:50,  1.93s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:48,  1.93s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:43,  1.88s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:44,  2.03s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:41,  1.99s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:01<00:38,  1.95s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:03<00:37,  1.96s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:05<00:34,  1.92s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:07<00:32,  1.90s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:30,  1.89s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:28,  1.87s/it]

Evaluating policy:  83%|########3 | 69/83 [02:13<00:28,  2.02s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:25,  1.96s/it]

Evaluating policy:  86%|########5 | 71/83 [02:16<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:18<00:20,  1.85s/it]

Evaluating policy:  88%|########7 | 73/83 [02:20<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:22<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:14,  1.84s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:12,  1.83s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:11,  1.85s/it]

Evaluating policy:  94%|#########3| 78/83 [02:29<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:31<00:08,  2.01s/it]

Evaluating policy:  96%|#########6| 80/83 [02:33<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:35<00:03,  1.91s/it]

Evaluating policy:  99%|#########8| 82/83 [02:37<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 83/83 [02:39<00:00,  1.95s/it]

Outer training loop:  58%|#####8    | 29001/50000 [18:30:01<572:19:29, 98.12s/it]
Outer training loop:  58%|#####8    | 29002/50000 [18:30:03<403:51:07, 69.24s/it]
Outer training loop:  58%|#####8    | 29003/50000 [18:30:05<286:20:28, 49.09s/it]
Outer training loop:  58%|#####8    | 29004/50000 [18:30:07<203:38:48, 34.92s/it]
Outer training loop:  58%|#####8    | 29005/50000 [18:30:09<145:49:47, 25.01s/it]
Outer training loop:  58%|#####8    | 29006/50000 [18:30:10<105:27:52, 18.08s/it]
Outer training loop:  58%|#####8    | 29007/50000 [18:30:13<78:08:51, 13.40s/it] 
Outer training loop:  58%|#####8    | 29008/50000 [18:30:15<58:04:01,  9.96s/it]
Outer training loop:  58%|#####8    | 29009/50000 [18:30:17<43:46:52,  7.51s/it]
Outer training loop:  58%|#####8    | 29010/50000 [18:30:19<34:13:06,  5.87s/it]
Outer training loop:  58%|#####8    | 29011/50000 [18:30:21<27:22:29,  4.70s/it]
Outer training loop:  58%|#####8    | 29012/50000 [18:30:23<22:31:40,  3.86s/it]
Outer training loop:  58%|#####8    | 29013/50000 [18:30:25<19:33:06,  3.35s/it]
Outer training loop:  58%|#####8    | 29014/50000 [18:30:27<16:50:12,  2.89s/it]
Outer training loop:  58%|#####8    | 29015/50000 [18:30:28<14:53:58,  2.56s/it]
Outer training loop:  58%|#####8    | 29016/50000 [18:30:30<13:27:49,  2.31s/it]
Outer training loop:  58%|#####8    | 29017/50000 [18:30:32<12:52:16,  2.21s/it]
Outer training loop:  58%|#####8    | 29018/50000 [18:30:34<12:21:16,  2.12s/it]
Outer training loop:  58%|#####8    | 29019/50000 [18:30:36<12:05:35,  2.07s/it]
Outer training loop:  58%|#####8    | 29020/50000 [18:30:38<11:49:16,  2.03s/it]
Outer training loop:  58%|#####8    | 29021/50000 [18:30:40<11:40:23,  2.00s/it]
Outer training loop:  58%|#####8    | 29022/50000 [18:30:42<11:22:02,  1.95s/it]
Outer training loop:  58%|#####8    | 29023/50000 [18:30:43<11:06:59,  1.91s/it]
Outer training loop:  58%|#####8    | 29024/50000 [18:30:45<11:11:01,  1.92s/it]
Outer training loop:  58%|#####8    | 29025/50000 [18:30:47<11:04:26,  1.90s/it]
Outer training loop:  58%|#####8    | 29026/50000 [18:30:49<10:59:21,  1.89s/it]
Outer training loop:  58%|#####8    | 29027/50000 [18:30:51<11:17:24,  1.94s/it]
Outer training loop:  58%|#####8    | 29028/50000 [18:30:53<11:08:11,  1.91s/it]
Outer training loop:  58%|#####8    | 29029/50000 [18:30:55<10:57:56,  1.88s/it]
Outer training loop:  58%|#####8    | 29030/50000 [18:30:57<10:41:49,  1.84s/it]
Outer training loop:  58%|#####8    | 29031/50000 [18:30:59<11:10:12,  1.92s/it]
Outer training loop:  58%|#####8    | 29032/50000 [18:31:00<10:53:31,  1.87s/it]
Outer training loop:  58%|#####8    | 29033/50000 [18:31:02<10:49:51,  1.86s/it]
Outer training loop:  58%|#####8    | 29034/50000 [18:31:04<10:42:22,  1.84s/it]
Outer training loop:  58%|#####8    | 29035/50000 [18:31:06<10:42:02,  1.84s/it]
Outer training loop:  58%|#####8    | 29036/50000 [18:31:08<10:41:12,  1.84s/it]
Outer training loop:  58%|#####8    | 29037/50000 [18:31:10<10:53:00,  1.87s/it]
Outer training loop:  58%|#####8    | 29038/50000 [18:31:11<10:37:32,  1.82s/it]
Outer training loop:  58%|#####8    | 29039/50000 [18:31:13<10:41:45,  1.84s/it]
Outer training loop:  58%|#####8    | 29040/50000 [18:31:15<10:52:37,  1.87s/it]
Outer training loop:  58%|#####8    | 29041/50000 [18:31:17<10:39:01,  1.83s/it]
Outer training loop:  58%|#####8    | 29042/50000 [18:31:19<10:30:33,  1.81s/it]
Outer training loop:  58%|#####8    | 29043/50000 [18:31:20<10:24:49,  1.79s/it]
Outer training loop:  58%|#####8    | 29044/50000 [18:31:22<10:38:13,  1.83s/it]
Outer training loop:  58%|#####8    | 29045/50000 [18:31:24<10:27:24,  1.80s/it]
Outer training loop:  58%|#####8    | 29046/50000 [18:31:26<10:41:28,  1.84s/it]
Outer training loop:  58%|#####8    | 29047/50000 [18:31:28<10:56:19,  1.88s/it]
Outer training loop:  58%|#####8    | 29048/50000 [18:31:30<10:38:14,  1.83s/it]
Outer training loop:  58%|#####8    | 29049/50000 [18:31:31<10:33:45,  1.81s/it]
Outer training loop:  58%|#####8    | 29050/50000 [18:31:33<10:54:05,  1.87s/it]
Outer training loop:  58%|#####8    | 29051/50000 [18:31:35<11:00:59,  1.89s/it]
Outer training loop:  58%|#####8    | 29052/50000 [18:31:37<10:59:39,  1.89s/it]
Outer training loop:  58%|#####8    | 29053/50000 [18:31:39<10:53:52,  1.87s/it]
Outer training loop:  58%|#####8    | 29054/50000 [18:31:41<11:10:21,  1.92s/it]
Outer training loop:  58%|#####8    | 29055/50000 [18:31:43<11:07:09,  1.91s/it]
Outer training loop:  58%|#####8    | 29056/50000 [18:31:45<10:49:25,  1.86s/it]
Outer training loop:  58%|#####8    | 29057/50000 [18:31:47<11:05:48,  1.91s/it]
Outer training loop:  58%|#####8    | 29058/50000 [18:31:49<11:00:39,  1.89s/it]
Outer training loop:  58%|#####8    | 29059/50000 [18:31:50<10:54:55,  1.88s/it]
Outer training loop:  58%|#####8    | 29060/50000 [18:31:52<11:03:57,  1.90s/it]
Outer training loop:  58%|#####8    | 29061/50000 [18:31:54<11:04:51,  1.91s/it]
Outer training loop:  58%|#####8    | 29062/50000 [18:31:56<10:48:36,  1.86s/it]
Outer training loop:  58%|#####8    | 29063/50000 [18:31:58<10:49:11,  1.86s/it]
Outer training loop:  58%|#####8    | 29064/50000 [18:32:00<11:07:15,  1.91s/it]
Outer training loop:  58%|#####8    | 29065/50000 [18:32:02<10:48:45,  1.86s/it]
Outer training loop:  58%|#####8    | 29066/50000 [18:32:03<10:21:34,  1.78s/it]
Outer training loop:  58%|#####8    | 29067/50000 [18:32:05<10:32:08,  1.81s/it]
Outer training loop:  58%|#####8    | 29068/50000 [18:32:07<10:41:50,  1.84s/it]
Outer training loop:  58%|#####8    | 29069/50000 [18:32:09<10:32:34,  1.81s/it]
Outer training loop:  58%|#####8    | 29070/50000 [18:32:11<10:26:33,  1.80s/it]
Outer training loop:  58%|#####8    | 29071/50000 [18:32:13<10:47:22,  1.86s/it]
Outer training loop:  58%|#####8    | 29072/50000 [18:32:14<10:51:25,  1.87s/it]
Outer training loop:  58%|#####8    | 29073/50000 [18:32:16<10:44:47,  1.85s/it]
Outer training loop:  58%|#####8    | 29074/50000 [18:32:18<10:43:26,  1.84s/it]
Outer training loop:  58%|#####8    | 29075/50000 [18:32:20<10:52:52,  1.87s/it]
Outer training loop:  58%|#####8    | 29076/50000 [18:32:22<10:39:45,  1.83s/it]
Outer training loop:  58%|#####8    | 29077/50000 [18:32:24<10:29:22,  1.80s/it]
Outer training loop:  58%|#####8    | 29078/50000 [18:32:26<10:55:14,  1.88s/it]
Outer training loop:  58%|#####8    | 29079/50000 [18:32:28<10:59:46,  1.89s/it]
Outer training loop:  58%|#####8    | 29080/50000 [18:32:29<10:59:11,  1.89s/it]
Outer training loop:  58%|#####8    | 29081/50000 [18:32:31<11:16:10,  1.94s/it]
Outer training loop:  58%|#####8    | 29082/50000 [18:32:33<10:56:48,  1.88s/it]
Outer training loop:  58%|#####8    | 29083/50000 [18:32:35<10:50:18,  1.87s/it]
Outer training loop:  58%|#####8    | 29084/50000 [18:32:37<10:27:54,  1.80s/it]
Outer training loop:  58%|#####8    | 29085/50000 [18:32:39<10:50:33,  1.87s/it]
Outer training loop:  58%|#####8    | 29086/50000 [18:32:40<10:32:38,  1.81s/it]
Outer training loop:  58%|#####8    | 29087/50000 [18:32:42<10:20:46,  1.78s/it]
Outer training loop:  58%|#####8    | 29088/50000 [18:32:44<10:21:30,  1.78s/it]
Outer training loop:  58%|#####8    | 29089/50000 [18:32:46<10:32:34,  1.82s/it]
Outer training loop:  58%|#####8    | 29090/50000 [18:32:47<10:16:09,  1.77s/it]
Outer training loop:  58%|#####8    | 29091/50000 [18:32:49<10:11:28,  1.75s/it]
Outer training loop:  58%|#####8    | 29092/50000 [18:32:51<10:27:06,  1.80s/it]
Outer training loop:  58%|#####8    | 29093/50000 [18:32:53<10:45:51,  1.85s/it]
Outer training loop:  58%|#####8    | 29094/50000 [18:32:55<10:41:07,  1.84s/it]
Outer training loop:  58%|#####8    | 29095/50000 [18:32:57<10:34:16,  1.82s/it]
Outer training loop:  58%|#####8    | 29096/50000 [18:32:59<10:43:02,  1.85s/it]
Outer training loop:  58%|#####8    | 29097/50000 [18:33:00<10:35:45,  1.82s/it]
Outer training loop:  58%|#####8    | 29098/50000 [18:33:02<10:23:22,  1.79s/it]
Outer training loop:  58%|#####8    | 29099/50000 [18:33:04<10:46:31,  1.86s/it]
Outer training loop:  58%|#####8    | 29100/50000 [18:33:06<10:58:56,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  58%|#####8    | 29101/50000 [18:33:13<20:08:03,  3.47s/it]
Outer training loop:  58%|#####8    | 29102/50000 [18:33:15<17:14:10,  2.97s/it]
Outer training loop:  58%|#####8    | 29103/50000 [18:33:17<15:29:56,  2.67s/it]
Outer training loop:  58%|#####8    | 29104/50000 [18:33:19<14:06:43,  2.43s/it]
Outer training loop:  58%|#####8    | 29105/50000 [18:33:21<12:56:33,  2.23s/it]
Outer training loop:  58%|#####8    | 29106/50000 [18:33:23<12:30:27,  2.16s/it]
Outer training loop:  58%|#####8    | 29107/50000 [18:33:24<11:59:12,  2.07s/it]
Outer training loop:  58%|#####8    | 29108/50000 [18:33:26<11:31:33,  1.99s/it]
Outer training loop:  58%|#####8    | 29109/50000 [18:33:28<11:32:21,  1.99s/it]
Outer training loop:  58%|#####8    | 29110/50000 [18:33:30<11:07:04,  1.92s/it]
Outer training loop:  58%|#####8    | 29111/50000 [18:33:32<10:49:32,  1.87s/it]
Outer training loop:  58%|#####8    | 29112/50000 [18:33:33<10:40:15,  1.84s/it]
Outer training loop:  58%|#####8    | 29113/50000 [18:33:35<10:50:09,  1.87s/it]
Outer training loop:  58%|#####8    | 29114/50000 [18:33:37<10:48:19,  1.86s/it]
Outer training loop:  58%|#####8    | 29115/50000 [18:33:39<10:28:19,  1.81s/it]
Outer training loop:  58%|#####8    | 29116/50000 [18:33:41<10:52:40,  1.88s/it]
Outer training loop:  58%|#####8    | 29117/50000 [18:33:43<10:35:42,  1.83s/it]
Outer training loop:  58%|#####8    | 29118/50000 [18:33:44<10:30:28,  1.81s/it]
Outer training loop:  58%|#####8    | 29119/50000 [18:33:46<10:26:40,  1.80s/it]
Outer training loop:  58%|#####8    | 29120/50000 [18:33:48<10:54:52,  1.88s/it]
Outer training loop:  58%|#####8    | 29121/50000 [18:33:50<10:51:54,  1.87s/it]
Outer training loop:  58%|#####8    | 29122/50000 [18:33:52<10:49:45,  1.87s/it]
Outer training loop:  58%|#####8    | 29123/50000 [18:33:54<10:57:15,  1.89s/it]
Outer training loop:  58%|#####8    | 29124/50000 [18:33:56<10:33:45,  1.82s/it]
Outer training loop:  58%|#####8    | 29125/50000 [18:33:58<10:59:16,  1.89s/it]
Outer training loop:  58%|#####8    | 29126/50000 [18:33:59<10:37:02,  1.83s/it]
Outer training loop:  58%|#####8    | 29127/50000 [18:34:01<10:20:41,  1.78s/it]
Outer training loop:  58%|#####8    | 29128/50000 [18:34:03<10:11:25,  1.76s/it]
Outer training loop:  58%|#####8    | 29129/50000 [18:34:05<10:34:38,  1.82s/it]
Outer training loop:  58%|#####8    | 29130/50000 [18:34:06<10:24:21,  1.79s/it]
Outer training loop:  58%|#####8    | 29131/50000 [18:34:08<10:21:49,  1.79s/it]
Outer training loop:  58%|#####8    | 29132/50000 [18:34:10<10:45:31,  1.86s/it]
Outer training loop:  58%|#####8    | 29133/50000 [18:34:12<10:28:46,  1.81s/it]
Outer training loop:  58%|#####8    | 29134/50000 [18:34:14<10:24:58,  1.80s/it]
Outer training loop:  58%|#####8    | 29135/50000 [18:34:16<10:45:51,  1.86s/it]
Outer training loop:  58%|#####8    | 29136/50000 [18:34:18<10:41:14,  1.84s/it]
Outer training loop:  58%|#####8    | 29137/50000 [18:34:19<10:41:01,  1.84s/it]
Outer training loop:  58%|#####8    | 29138/50000 [18:34:21<10:26:59,  1.80s/it]
Outer training loop:  58%|#####8    | 29139/50000 [18:34:23<10:47:23,  1.86s/it]
Outer training loop:  58%|#####8    | 29140/50000 [18:34:25<10:39:31,  1.84s/it]
Outer training loop:  58%|#####8    | 29141/50000 [18:34:27<10:40:35,  1.84s/it]
Outer training loop:  58%|#####8    | 29142/50000 [18:34:29<10:44:52,  1.86s/it]
Outer training loop:  58%|#####8    | 29143/50000 [18:34:30<10:39:45,  1.84s/it]
Outer training loop:  58%|#####8    | 29144/50000 [18:34:32<10:34:10,  1.82s/it]
Outer training loop:  58%|#####8    | 29145/50000 [18:34:34<10:59:26,  1.90s/it]
Outer training loop:  58%|#####8    | 29146/50000 [18:34:36<10:51:07,  1.87s/it]
Outer training loop:  58%|#####8    | 29147/50000 [18:34:38<10:46:10,  1.86s/it]
Outer training loop:  58%|#####8    | 29148/50000 [18:34:40<10:54:00,  1.88s/it]
Outer training loop:  58%|#####8    | 29149/50000 [18:34:42<10:40:09,  1.84s/it]
Outer training loop:  58%|#####8    | 29150/50000 [18:34:43<10:24:55,  1.80s/it]
Outer training loop:  58%|#####8    | 29151/50000 [18:34:45<10:29:02,  1.81s/it]
Outer training loop:  58%|#####8    | 29152/50000 [18:34:47<10:43:22,  1.85s/it]
Outer training loop:  58%|#####8    | 29153/50000 [18:34:49<10:41:16,  1.85s/it]
Outer training loop:  58%|#####8    | 29154/50000 [18:34:51<10:31:21,  1.82s/it]
Outer training loop:  58%|#####8    | 29155/50000 [18:34:53<10:39:54,  1.84s/it]
Outer training loop:  58%|#####8    | 29156/50000 [18:34:54<10:27:02,  1.80s/it]
Outer training loop:  58%|#####8    | 29157/50000 [18:34:56<10:18:40,  1.78s/it]
Outer training loop:  58%|#####8    | 29158/50000 [18:34:58<10:14:08,  1.77s/it]
Outer training loop:  58%|#####8    | 29159/50000 [18:35:00<10:40:52,  1.85s/it]
Outer training loop:  58%|#####8    | 29160/50000 [18:35:01<10:30:44,  1.82s/it]
Outer training loop:  58%|#####8    | 29161/50000 [18:35:03<10:24:53,  1.80s/it]
Outer training loop:  58%|#####8    | 29162/50000 [18:35:05<10:51:28,  1.88s/it]
Outer training loop:  58%|#####8    | 29163/50000 [18:35:07<10:44:50,  1.86s/it]
Outer training loop:  58%|#####8    | 29164/50000 [18:35:09<10:38:03,  1.84s/it]
Outer training loop:  58%|#####8    | 29165/50000 [18:35:11<10:53:32,  1.88s/it]
Outer training loop:  58%|#####8    | 29166/50000 [18:35:13<10:30:52,  1.82s/it]
Outer training loop:  58%|#####8    | 29167/50000 [18:35:14<10:23:22,  1.80s/it]
Outer training loop:  58%|#####8    | 29168/50000 [18:35:16<10:45:34,  1.86s/it]
Outer training loop:  58%|#####8    | 29169/50000 [18:35:18<10:27:40,  1.81s/it]
Outer training loop:  58%|#####8    | 29170/50000 [18:35:20<10:27:04,  1.81s/it]
Outer training loop:  58%|#####8    | 29171/50000 [18:35:21<10:08:05,  1.75s/it]
Outer training loop:  58%|#####8    | 29172/50000 [18:35:23<10:37:03,  1.84s/it]
Outer training loop:  58%|#####8    | 29173/50000 [18:35:25<10:39:06,  1.84s/it]
Outer training loop:  58%|#####8    | 29174/50000 [18:35:27<10:28:19,  1.81s/it]
Outer training loop:  58%|#####8    | 29175/50000 [18:35:29<10:44:04,  1.86s/it]
Outer training loop:  58%|#####8    | 29176/50000 [18:35:31<10:40:03,  1.84s/it]
Outer training loop:  58%|#####8    | 29177/50000 [18:35:33<10:43:43,  1.85s/it]
Outer training loop:  58%|#####8    | 29178/50000 [18:35:35<10:49:12,  1.87s/it]
Outer training loop:  58%|#####8    | 29179/50000 [18:35:37<11:00:29,  1.90s/it]
Outer training loop:  58%|#####8    | 29180/50000 [18:35:38<10:51:05,  1.88s/it]
Outer training loop:  58%|#####8    | 29181/50000 [18:35:40<11:10:59,  1.93s/it]
Outer training loop:  58%|#####8    | 29182/50000 [18:35:43<11:30:15,  1.99s/it]
Outer training loop:  58%|#####8    | 29183/50000 [18:35:44<11:08:29,  1.93s/it]
Outer training loop:  58%|#####8    | 29184/50000 [18:35:46<10:50:38,  1.88s/it]
Outer training loop:  58%|#####8    | 29185/50000 [18:35:48<11:03:04,  1.91s/it]
Outer training loop:  58%|#####8    | 29186/50000 [18:35:50<10:52:23,  1.88s/it]
Outer training loop:  58%|#####8    | 29187/50000 [18:35:52<10:51:35,  1.88s/it]
Outer training loop:  58%|#####8    | 29188/50000 [18:35:54<10:56:23,  1.89s/it]
Outer training loop:  58%|#####8    | 29189/50000 [18:35:56<11:06:06,  1.92s/it]
Outer training loop:  58%|#####8    | 29190/50000 [18:35:58<10:54:17,  1.89s/it]
Outer training loop:  58%|#####8    | 29191/50000 [18:35:59<10:50:34,  1.88s/it]
Outer training loop:  58%|#####8    | 29192/50000 [18:36:01<10:55:18,  1.89s/it]
Outer training loop:  58%|#####8    | 29193/50000 [18:36:03<10:47:06,  1.87s/it]
Outer training loop:  58%|#####8    | 29194/50000 [18:36:05<10:45:01,  1.86s/it]
Outer training loop:  58%|#####8    | 29195/50000 [18:36:07<10:47:54,  1.87s/it]
Outer training loop:  58%|#####8    | 29196/50000 [18:36:09<10:45:03,  1.86s/it]
Outer training loop:  58%|#####8    | 29197/50000 [18:36:11<10:48:20,  1.87s/it]
Outer training loop:  58%|#####8    | 29198/50000 [18:36:13<10:56:50,  1.89s/it]
Outer training loop:  58%|#####8    | 29199/50000 [18:36:14<10:39:38,  1.85s/it]
Outer training loop:  58%|#####8    | 29200/50000 [18:36:16<10:36:28,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.94s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  58%|#####8    | 29201/50000 [18:36:23<20:00:54,  3.46s/it]
Outer training loop:  58%|#####8    | 29202/50000 [18:36:25<17:07:50,  2.97s/it]
Outer training loop:  58%|#####8    | 29203/50000 [18:36:27<15:25:10,  2.67s/it]
Outer training loop:  58%|#####8    | 29204/50000 [18:36:29<13:55:27,  2.41s/it]
Outer training loop:  58%|#####8    | 29205/50000 [18:36:31<12:52:19,  2.23s/it]
Outer training loop:  58%|#####8    | 29206/50000 [18:36:33<12:23:27,  2.15s/it]
Outer training loop:  58%|#####8    | 29207/50000 [18:36:35<11:54:43,  2.06s/it]
Outer training loop:  58%|#####8    | 29208/50000 [18:36:36<11:30:26,  1.99s/it]
Outer training loop:  58%|#####8    | 29209/50000 [18:36:38<11:14:11,  1.95s/it]
Outer training loop:  58%|#####8    | 29210/50000 [18:36:40<11:22:48,  1.97s/it]
Outer training loop:  58%|#####8    | 29211/50000 [18:36:42<11:01:33,  1.91s/it]
Outer training loop:  58%|#####8    | 29212/50000 [18:36:44<10:46:24,  1.87s/it]
Outer training loop:  58%|#####8    | 29213/50000 [18:36:46<10:54:57,  1.89s/it]
Outer training loop:  58%|#####8    | 29214/50000 [18:36:47<10:38:37,  1.84s/it]
Outer training loop:  58%|#####8    | 29215/50000 [18:36:49<10:35:38,  1.83s/it]
Outer training loop:  58%|#####8    | 29216/50000 [18:36:51<10:41:32,  1.85s/it]
Outer training loop:  58%|#####8    | 29217/50000 [18:36:53<10:59:42,  1.90s/it]
Outer training loop:  58%|#####8    | 29218/50000 [18:36:55<10:51:10,  1.88s/it]
Outer training loop:  58%|#####8    | 29219/50000 [18:36:57<10:50:37,  1.88s/it]
Outer training loop:  58%|#####8    | 29220/50000 [18:36:59<11:02:13,  1.91s/it]
Outer training loop:  58%|#####8    | 29221/50000 [18:37:01<10:54:35,  1.89s/it]
Outer training loop:  58%|#####8    | 29222/50000 [18:37:02<10:40:02,  1.85s/it]
Outer training loop:  58%|#####8    | 29223/50000 [18:37:04<10:38:12,  1.84s/it]
Outer training loop:  58%|#####8    | 29224/50000 [18:37:06<10:53:21,  1.89s/it]
Outer training loop:  58%|#####8    | 29225/50000 [18:37:08<10:32:47,  1.83s/it]
Outer training loop:  58%|#####8    | 29226/50000 [18:37:10<10:24:42,  1.80s/it]
Outer training loop:  58%|#####8    | 29227/50000 [18:37:12<10:33:49,  1.83s/it]
Outer training loop:  58%|#####8    | 29228/50000 [18:37:14<10:43:25,  1.86s/it]
Outer training loop:  58%|#####8    | 29229/50000 [18:37:15<10:31:53,  1.83s/it]
Outer training loop:  58%|#####8    | 29230/50000 [18:37:17<10:55:30,  1.89s/it]
Outer training loop:  58%|#####8    | 29231/50000 [18:37:19<10:49:35,  1.88s/it]
Outer training loop:  58%|#####8    | 29232/50000 [18:37:21<11:04:16,  1.92s/it]
Outer training loop:  58%|#####8    | 29233/50000 [18:37:23<11:01:52,  1.91s/it]
Outer training loop:  58%|#####8    | 29234/50000 [18:37:25<11:22:15,  1.97s/it]
Outer training loop:  58%|#####8    | 29235/50000 [18:37:27<11:06:09,  1.92s/it]
Outer training loop:  58%|#####8    | 29236/50000 [18:37:29<10:51:24,  1.88s/it]
Outer training loop:  58%|#####8    | 29237/50000 [18:37:31<10:51:05,  1.88s/it]
Outer training loop:  58%|#####8    | 29238/50000 [18:37:33<10:50:58,  1.88s/it]
Outer training loop:  58%|#####8    | 29239/50000 [18:37:34<10:46:04,  1.87s/it]
Outer training loop:  58%|#####8    | 29240/50000 [18:37:36<10:56:57,  1.90s/it]
Outer training loop:  58%|#####8    | 29241/50000 [18:37:38<11:06:20,  1.93s/it]
Outer training loop:  58%|#####8    | 29242/50000 [18:37:40<10:58:23,  1.90s/it]
Outer training loop:  58%|#####8    | 29243/50000 [18:37:42<11:18:03,  1.96s/it]
Outer training loop:  58%|#####8    | 29244/50000 [18:37:44<11:22:00,  1.97s/it]
Outer training loop:  58%|#####8    | 29245/50000 [18:37:46<11:06:47,  1.93s/it]
Outer training loop:  58%|#####8    | 29246/50000 [18:37:48<10:57:53,  1.90s/it]
Outer training loop:  58%|#####8    | 29247/50000 [18:37:50<11:24:01,  1.98s/it]
Outer training loop:  58%|#####8    | 29248/50000 [18:37:52<11:21:33,  1.97s/it]
Outer training loop:  58%|#####8    | 29249/50000 [18:37:54<11:13:55,  1.95s/it]
Outer training loop:  58%|#####8    | 29250/50000 [18:37:56<11:19:23,  1.96s/it]
Outer training loop:  59%|#####8    | 29251/50000 [18:37:58<11:01:00,  1.91s/it]
Outer training loop:  59%|#####8    | 29252/50000 [18:38:00<10:47:25,  1.87s/it]
Outer training loop:  59%|#####8    | 29253/50000 [18:38:01<10:54:52,  1.89s/it]
Outer training loop:  59%|#####8    | 29254/50000 [18:38:03<10:45:09,  1.87s/it]
Outer training loop:  59%|#####8    | 29255/50000 [18:38:05<10:53:03,  1.89s/it]
Outer training loop:  59%|#####8    | 29256/50000 [18:38:07<10:48:53,  1.88s/it]
Outer training loop:  59%|#####8    | 29257/50000 [18:38:09<10:58:39,  1.91s/it]
Outer training loop:  59%|#####8    | 29258/50000 [18:38:11<10:58:56,  1.91s/it]
Outer training loop:  59%|#####8    | 29259/50000 [18:38:13<10:52:23,  1.89s/it]
Outer training loop:  59%|#####8    | 29260/50000 [18:38:15<10:54:30,  1.89s/it]
Outer training loop:  59%|#####8    | 29261/50000 [18:38:16<10:36:33,  1.84s/it]
Outer training loop:  59%|#####8    | 29262/50000 [18:38:18<10:36:01,  1.84s/it]
Outer training loop:  59%|#####8    | 29263/50000 [18:38:20<10:58:47,  1.91s/it]
Outer training loop:  59%|#####8    | 29264/50000 [18:38:22<10:52:20,  1.89s/it]
Outer training loop:  59%|#####8    | 29265/50000 [18:38:24<10:53:23,  1.89s/it]
Outer training loop:  59%|#####8    | 29266/50000 [18:38:26<10:50:04,  1.88s/it]
Outer training loop:  59%|#####8    | 29267/50000 [18:38:28<10:58:04,  1.90s/it]
Outer training loop:  59%|#####8    | 29268/50000 [18:38:30<10:50:08,  1.88s/it]
Outer training loop:  59%|#####8    | 29269/50000 [18:38:31<10:38:56,  1.85s/it]
Outer training loop:  59%|#####8    | 29270/50000 [18:38:34<10:55:40,  1.90s/it]
Outer training loop:  59%|#####8    | 29271/50000 [18:38:35<10:49:40,  1.88s/it]
Outer training loop:  59%|#####8    | 29272/50000 [18:38:37<10:37:16,  1.84s/it]
Outer training loop:  59%|#####8    | 29273/50000 [18:38:39<10:40:32,  1.85s/it]
Outer training loop:  59%|#####8    | 29274/50000 [18:38:41<11:04:30,  1.92s/it]
Outer training loop:  59%|#####8    | 29275/50000 [18:38:43<11:04:17,  1.92s/it]
Outer training loop:  59%|#####8    | 29276/50000 [18:38:45<11:06:45,  1.93s/it]
Outer training loop:  59%|#####8    | 29277/50000 [18:38:47<11:26:45,  1.99s/it]
Outer training loop:  59%|#####8    | 29278/50000 [18:38:49<11:14:54,  1.95s/it]
Outer training loop:  59%|#####8    | 29279/50000 [18:38:51<11:03:43,  1.92s/it]
Outer training loop:  59%|#####8    | 29280/50000 [18:38:53<11:15:55,  1.96s/it]
Outer training loop:  59%|#####8    | 29281/50000 [18:38:55<11:06:43,  1.93s/it]
Outer training loop:  59%|#####8    | 29282/50000 [18:38:57<10:55:31,  1.90s/it]
Outer training loop:  59%|#####8    | 29283/50000 [18:38:58<10:54:23,  1.90s/it]
Outer training loop:  59%|#####8    | 29284/50000 [18:39:00<11:04:30,  1.92s/it]
Outer training loop:  59%|#####8    | 29285/50000 [18:39:02<10:53:44,  1.89s/it]
Outer training loop:  59%|#####8    | 29286/50000 [18:39:04<11:03:45,  1.92s/it]
Outer training loop:  59%|#####8    | 29287/50000 [18:39:06<11:18:21,  1.97s/it]
Outer training loop:  59%|#####8    | 29288/50000 [18:39:08<11:04:15,  1.92s/it]
Outer training loop:  59%|#####8    | 29289/50000 [18:39:10<10:57:08,  1.90s/it]
Outer training loop:  59%|#####8    | 29290/50000 [18:39:12<10:45:16,  1.87s/it]
Outer training loop:  59%|#####8    | 29291/50000 [18:39:14<11:19:04,  1.97s/it]
Outer training loop:  59%|#####8    | 29292/50000 [18:39:16<11:03:51,  1.92s/it]
Outer training loop:  59%|#####8    | 29293/50000 [18:39:18<10:54:11,  1.90s/it]
Outer training loop:  59%|#####8    | 29294/50000 [18:39:20<11:35:30,  2.02s/it]
Outer training loop:  59%|#####8    | 29295/50000 [18:39:22<11:27:00,  1.99s/it]
Outer training loop:  59%|#####8    | 29296/50000 [18:39:24<11:22:38,  1.98s/it]
Outer training loop:  59%|#####8    | 29297/50000 [18:39:26<11:37:46,  2.02s/it]
Outer training loop:  59%|#####8    | 29298/50000 [18:39:28<11:32:45,  2.01s/it]
Outer training loop:  59%|#####8    | 29299/50000 [18:39:30<11:16:38,  1.96s/it]
Outer training loop:  59%|#####8    | 29300/50000 [18:39:32<11:34:32,  2.01s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  59%|#####8    | 29301/50000 [18:39:40<21:19:22,  3.71s/it]
Outer training loop:  59%|#####8    | 29302/50000 [18:39:41<18:06:43,  3.15s/it]
Outer training loop:  59%|#####8    | 29303/50000 [18:39:43<16:13:01,  2.82s/it]
Outer training loop:  59%|#####8    | 29304/50000 [18:39:45<14:53:18,  2.59s/it]
Outer training loop:  59%|#####8    | 29305/50000 [18:39:47<13:34:45,  2.36s/it]
Outer training loop:  59%|#####8    | 29306/50000 [18:39:49<12:49:11,  2.23s/it]
Outer training loop:  59%|#####8    | 29307/50000 [18:39:51<12:22:52,  2.15s/it]
Outer training loop:  59%|#####8    | 29308/50000 [18:39:53<11:45:08,  2.04s/it]
Outer training loop:  59%|#####8    | 29309/50000 [18:39:55<11:20:08,  1.97s/it]
Outer training loop:  59%|#####8    | 29310/50000 [18:39:57<11:00:10,  1.91s/it]
Outer training loop:  59%|#####8    | 29311/50000 [18:39:59<11:12:47,  1.95s/it]
Outer training loop:  59%|#####8    | 29312/50000 [18:40:00<10:54:29,  1.90s/it]
Outer training loop:  59%|#####8    | 29313/50000 [18:40:02<10:40:32,  1.86s/it]
Outer training loop:  59%|#####8    | 29314/50000 [18:40:04<11:08:31,  1.94s/it]
Outer training loop:  59%|#####8    | 29315/50000 [18:40:06<11:02:41,  1.92s/it]
Outer training loop:  59%|#####8    | 29316/50000 [18:40:08<10:55:19,  1.90s/it]
Outer training loop:  59%|#####8    | 29317/50000 [18:40:10<11:14:34,  1.96s/it]
Outer training loop:  59%|#####8    | 29318/50000 [18:40:12<10:55:16,  1.90s/it]
Outer training loop:  59%|#####8    | 29319/50000 [18:40:14<10:57:14,  1.91s/it]
Outer training loop:  59%|#####8    | 29320/50000 [18:40:16<10:54:06,  1.90s/it]
Outer training loop:  59%|#####8    | 29321/50000 [18:40:18<11:07:41,  1.94s/it]
Outer training loop:  59%|#####8    | 29322/50000 [18:40:20<10:54:20,  1.90s/it]
Outer training loop:  59%|#####8    | 29323/50000 [18:40:21<10:49:14,  1.88s/it]
Outer training loop:  59%|#####8    | 29324/50000 [18:40:23<10:44:04,  1.87s/it]
Outer training loop:  59%|#####8    | 29325/50000 [18:40:25<11:14:39,  1.96s/it]
Outer training loop:  59%|#####8    | 29326/50000 [18:40:27<10:55:09,  1.90s/it]
Outer training loop:  59%|#####8    | 29327/50000 [18:40:29<10:52:59,  1.90s/it]
Outer training loop:  59%|#####8    | 29328/50000 [18:40:31<11:05:55,  1.93s/it]
Outer training loop:  59%|#####8    | 29329/50000 [18:40:33<10:51:39,  1.89s/it]
Outer training loop:  59%|#####8    | 29330/50000 [18:40:35<11:02:14,  1.92s/it]
Outer training loop:  59%|#####8    | 29331/50000 [18:40:37<11:10:12,  1.95s/it]
Outer training loop:  59%|#####8    | 29332/50000 [18:40:39<11:05:04,  1.93s/it]
Outer training loop:  59%|#####8    | 29333/50000 [18:40:41<11:01:31,  1.92s/it]
Outer training loop:  59%|#####8    | 29334/50000 [18:40:42<10:49:20,  1.89s/it]
Outer training loop:  59%|#####8    | 29335/50000 [18:40:44<11:02:49,  1.92s/it]
Outer training loop:  59%|#####8    | 29336/50000 [18:40:46<10:55:04,  1.90s/it]
Outer training loop:  59%|#####8    | 29337/50000 [18:40:48<10:38:29,  1.85s/it]
Outer training loop:  59%|#####8    | 29338/50000 [18:40:50<10:58:46,  1.91s/it]
Outer training loop:  59%|#####8    | 29339/50000 [18:40:52<10:49:47,  1.89s/it]
Outer training loop:  59%|#####8    | 29340/50000 [18:40:54<10:47:08,  1.88s/it]
Outer training loop:  59%|#####8    | 29341/50000 [18:40:56<11:19:54,  1.97s/it]
Outer training loop:  59%|#####8    | 29342/50000 [18:40:58<11:14:15,  1.96s/it]
Outer training loop:  59%|#####8    | 29343/50000 [18:41:00<10:58:25,  1.91s/it]
Outer training loop:  59%|#####8    | 29344/50000 [18:41:02<10:59:38,  1.92s/it]
Outer training loop:  59%|#####8    | 29345/50000 [18:41:04<11:07:00,  1.94s/it]
Outer training loop:  59%|#####8    | 29346/50000 [18:41:06<11:14:50,  1.96s/it]
Outer training loop:  59%|#####8    | 29347/50000 [18:41:07<11:03:33,  1.93s/it]
Outer training loop:  59%|#####8    | 29348/50000 [18:41:09<11:08:35,  1.94s/it]
Outer training loop:  59%|#####8    | 29349/50000 [18:41:11<10:58:05,  1.91s/it]
Outer training loop:  59%|#####8    | 29350/50000 [18:41:13<10:40:20,  1.86s/it]
Outer training loop:  59%|#####8    | 29351/50000 [18:41:15<11:00:30,  1.92s/it]
Outer training loop:  59%|#####8    | 29352/50000 [18:41:17<10:45:08,  1.87s/it]
Outer training loop:  59%|#####8    | 29353/50000 [18:41:19<10:47:54,  1.88s/it]
Outer training loop:  59%|#####8    | 29354/50000 [18:41:21<10:52:23,  1.90s/it]
Outer training loop:  59%|#####8    | 29355/50000 [18:41:23<11:14:21,  1.96s/it]
Outer training loop:  59%|#####8    | 29356/50000 [18:41:25<11:03:43,  1.93s/it]
Outer training loop:  59%|#####8    | 29357/50000 [18:41:27<10:58:42,  1.91s/it]
Outer training loop:  59%|#####8    | 29358/50000 [18:41:29<11:10:17,  1.95s/it]
Outer training loop:  59%|#####8    | 29359/50000 [18:41:30<11:07:36,  1.94s/it]
Outer training loop:  59%|#####8    | 29360/50000 [18:41:32<10:55:13,  1.90s/it]
Outer training loop:  59%|#####8    | 29361/50000 [18:41:34<10:48:19,  1.88s/it]
Outer training loop:  59%|#####8    | 29362/50000 [18:41:36<10:55:39,  1.91s/it]
Outer training loop:  59%|#####8    | 29363/50000 [18:41:38<10:41:05,  1.86s/it]
Outer training loop:  59%|#####8    | 29364/50000 [18:41:40<10:43:21,  1.87s/it]
Outer training loop:  59%|#####8    | 29365/50000 [18:41:42<11:00:03,  1.92s/it]
Outer training loop:  59%|#####8    | 29366/50000 [18:41:44<10:46:56,  1.88s/it]
Outer training loop:  59%|#####8    | 29367/50000 [18:41:45<10:41:37,  1.87s/it]
Outer training loop:  59%|#####8    | 29368/50000 [18:41:47<10:47:34,  1.88s/it]
Outer training loop:  59%|#####8    | 29369/50000 [18:41:49<10:41:44,  1.87s/it]
Outer training loop:  59%|#####8    | 29370/50000 [18:41:51<10:38:15,  1.86s/it]
Outer training loop:  59%|#####8    | 29371/50000 [18:41:53<10:20:28,  1.80s/it]
Outer training loop:  59%|#####8    | 29372/50000 [18:41:55<10:43:03,  1.87s/it]
Outer training loop:  59%|#####8    | 29373/50000 [18:41:57<10:42:58,  1.87s/it]
Outer training loop:  59%|#####8    | 29374/50000 [18:41:58<10:40:32,  1.86s/it]
Outer training loop:  59%|#####8    | 29375/50000 [18:42:00<10:47:32,  1.88s/it]
Outer training loop:  59%|#####8    | 29376/50000 [18:42:02<10:42:18,  1.87s/it]
Outer training loop:  59%|#####8    | 29377/50000 [18:42:04<10:29:27,  1.83s/it]
Outer training loop:  59%|#####8    | 29378/50000 [18:42:06<10:45:42,  1.88s/it]
Outer training loop:  59%|#####8    | 29379/50000 [18:42:08<10:45:33,  1.88s/it]
Outer training loop:  59%|#####8    | 29380/50000 [18:42:10<10:46:27,  1.88s/it]
Outer training loop:  59%|#####8    | 29381/50000 [18:42:11<10:37:53,  1.86s/it]
Outer training loop:  59%|#####8    | 29382/50000 [18:42:13<10:46:31,  1.88s/it]
Outer training loop:  59%|#####8    | 29383/50000 [18:42:15<10:30:20,  1.83s/it]
Outer training loop:  59%|#####8    | 29384/50000 [18:42:17<10:29:04,  1.83s/it]
Outer training loop:  59%|#####8    | 29385/50000 [18:42:19<10:40:45,  1.86s/it]
Outer training loop:  59%|#####8    | 29386/50000 [18:42:21<10:39:58,  1.86s/it]
Outer training loop:  59%|#####8    | 29387/50000 [18:42:23<10:40:11,  1.86s/it]
Outer training loop:  59%|#####8    | 29388/50000 [18:42:25<10:47:42,  1.89s/it]
Outer training loop:  59%|#####8    | 29389/50000 [18:42:26<10:46:40,  1.88s/it]
Outer training loop:  59%|#####8    | 29390/50000 [18:42:28<10:39:46,  1.86s/it]
Outer training loop:  59%|#####8    | 29391/50000 [18:42:30<11:02:15,  1.93s/it]
Outer training loop:  59%|#####8    | 29392/50000 [18:42:32<11:01:15,  1.93s/it]
Outer training loop:  59%|#####8    | 29393/50000 [18:42:34<10:54:56,  1.91s/it]
Outer training loop:  59%|#####8    | 29394/50000 [18:42:36<10:40:42,  1.87s/it]
Outer training loop:  59%|#####8    | 29395/50000 [18:42:38<11:09:55,  1.95s/it]
Outer training loop:  59%|#####8    | 29396/50000 [18:42:40<10:57:41,  1.92s/it]
Outer training loop:  59%|#####8    | 29397/50000 [18:42:42<10:50:26,  1.89s/it]
Outer training loop:  59%|#####8    | 29398/50000 [18:42:44<11:14:53,  1.97s/it]
Outer training loop:  59%|#####8    | 29399/50000 [18:42:46<11:05:34,  1.94s/it]
Outer training loop:  59%|#####8    | 29400/50000 [18:42:48<10:59:34,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.96s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  59%|#####8    | 29401/50000 [18:42:55<20:30:45,  3.58s/it]
Outer training loop:  59%|#####8    | 29402/50000 [18:42:57<17:41:05,  3.09s/it]
Outer training loop:  59%|#####8    | 29403/50000 [18:42:59<15:29:36,  2.71s/it]
Outer training loop:  59%|#####8    | 29404/50000 [18:43:01<13:56:30,  2.44s/it]
Outer training loop:  59%|#####8    | 29405/50000 [18:43:03<13:07:48,  2.30s/it]
Outer training loop:  59%|#####8    | 29406/50000 [18:43:05<12:49:13,  2.24s/it]
Outer training loop:  59%|#####8    | 29407/50000 [18:43:07<12:05:36,  2.11s/it]
Outer training loop:  59%|#####8    | 29408/50000 [18:43:08<11:49:42,  2.07s/it]
Outer training loop:  59%|#####8    | 29409/50000 [18:43:10<11:42:27,  2.05s/it]
Outer training loop:  59%|#####8    | 29410/50000 [18:43:12<11:20:31,  1.98s/it]
Outer training loop:  59%|#####8    | 29411/50000 [18:43:14<11:02:10,  1.93s/it]
Outer training loop:  59%|#####8    | 29412/50000 [18:43:16<10:42:08,  1.87s/it]
Outer training loop:  59%|#####8    | 29413/50000 [18:43:18<10:48:20,  1.89s/it]
Outer training loop:  59%|#####8    | 29414/50000 [18:43:20<10:32:55,  1.84s/it]
Outer training loop:  59%|#####8    | 29415/50000 [18:43:21<10:38:06,  1.86s/it]
Outer training loop:  59%|#####8    | 29416/50000 [18:43:23<10:54:59,  1.91s/it]
Outer training loop:  59%|#####8    | 29417/50000 [18:43:25<10:59:58,  1.92s/it]
Outer training loop:  59%|#####8    | 29418/50000 [18:43:27<10:34:47,  1.85s/it]
Outer training loop:  59%|#####8    | 29419/50000 [18:43:29<10:29:50,  1.84s/it]
Outer training loop:  59%|#####8    | 29420/50000 [18:43:31<10:41:39,  1.87s/it]
Outer training loop:  59%|#####8    | 29421/50000 [18:43:33<10:37:42,  1.86s/it]
Outer training loop:  59%|#####8    | 29422/50000 [18:43:35<10:38:53,  1.86s/it]
Outer training loop:  59%|#####8    | 29423/50000 [18:43:37<10:56:34,  1.91s/it]
Outer training loop:  59%|#####8    | 29424/50000 [18:43:38<10:39:37,  1.87s/it]
Outer training loop:  59%|#####8    | 29425/50000 [18:43:40<10:34:41,  1.85s/it]
Outer training loop:  59%|#####8    | 29426/50000 [18:43:42<10:44:16,  1.88s/it]
Outer training loop:  59%|#####8    | 29427/50000 [18:43:44<10:50:40,  1.90s/it]
Outer training loop:  59%|#####8    | 29428/50000 [18:43:46<10:38:22,  1.86s/it]
Outer training loop:  59%|#####8    | 29429/50000 [18:43:48<10:29:01,  1.83s/it]
Outer training loop:  59%|#####8    | 29430/50000 [18:43:50<10:59:03,  1.92s/it]
Outer training loop:  59%|#####8    | 29431/50000 [18:43:52<10:59:22,  1.92s/it]
Outer training loop:  59%|#####8    | 29432/50000 [18:43:53<10:43:41,  1.88s/it]
Outer training loop:  59%|#####8    | 29433/50000 [18:43:56<11:14:38,  1.97s/it]
Outer training loop:  59%|#####8    | 29434/50000 [18:43:57<11:00:12,  1.93s/it]
Outer training loop:  59%|#####8    | 29435/50000 [18:43:59<10:55:35,  1.91s/it]
Outer training loop:  59%|#####8    | 29436/50000 [18:44:01<10:55:21,  1.91s/it]
Outer training loop:  59%|#####8    | 29437/50000 [18:44:03<10:51:22,  1.90s/it]
Outer training loop:  59%|#####8    | 29438/50000 [18:44:05<11:07:34,  1.95s/it]
Outer training loop:  59%|#####8    | 29439/50000 [18:44:07<10:50:12,  1.90s/it]
Outer training loop:  59%|#####8    | 29440/50000 [18:44:09<11:03:13,  1.94s/it]
Outer training loop:  59%|#####8    | 29441/50000 [18:44:11<10:48:37,  1.89s/it]
Outer training loop:  59%|#####8    | 29442/50000 [18:44:13<10:47:52,  1.89s/it]
Outer training loop:  59%|#####8    | 29443/50000 [18:44:15<10:54:54,  1.91s/it]
Outer training loop:  59%|#####8    | 29444/50000 [18:44:16<10:55:46,  1.91s/it]
Outer training loop:  59%|#####8    | 29445/50000 [18:44:18<10:55:07,  1.91s/it]
Outer training loop:  59%|#####8    | 29446/50000 [18:44:20<11:06:28,  1.95s/it]
Outer training loop:  59%|#####8    | 29447/50000 [18:44:22<10:38:42,  1.86s/it]
Outer training loop:  59%|#####8    | 29448/50000 [18:44:24<10:38:14,  1.86s/it]
Outer training loop:  59%|#####8    | 29449/50000 [18:44:26<10:56:40,  1.92s/it]
Outer training loop:  59%|#####8    | 29450/50000 [18:44:28<10:45:01,  1.88s/it]
Outer training loop:  59%|#####8    | 29451/50000 [18:44:30<10:37:28,  1.86s/it]
Outer training loop:  59%|#####8    | 29452/50000 [18:44:31<10:24:26,  1.82s/it]
Outer training loop:  59%|#####8    | 29453/50000 [18:44:33<10:42:13,  1.88s/it]
Outer training loop:  59%|#####8    | 29454/50000 [18:44:35<10:42:06,  1.88s/it]
Outer training loop:  59%|#####8    | 29455/50000 [18:44:37<10:45:54,  1.89s/it]
Outer training loop:  59%|#####8    | 29456/50000 [18:44:39<11:02:10,  1.93s/it]
Outer training loop:  59%|#####8    | 29457/50000 [18:44:41<10:45:49,  1.89s/it]
Outer training loop:  59%|#####8    | 29458/50000 [18:44:43<10:39:22,  1.87s/it]
Outer training loop:  59%|#####8    | 29459/50000 [18:44:45<10:59:34,  1.93s/it]
Outer training loop:  59%|#####8    | 29460/50000 [18:44:47<10:44:13,  1.88s/it]
Outer training loop:  59%|#####8    | 29461/50000 [18:44:48<10:39:06,  1.87s/it]
Outer training loop:  59%|#####8    | 29462/50000 [18:44:50<10:41:16,  1.87s/it]
Outer training loop:  59%|#####8    | 29463/50000 [18:44:52<10:57:51,  1.92s/it]
Outer training loop:  59%|#####8    | 29464/50000 [18:44:54<10:49:39,  1.90s/it]
Outer training loop:  59%|#####8    | 29465/50000 [18:44:56<10:39:16,  1.87s/it]
Outer training loop:  59%|#####8    | 29466/50000 [18:44:58<10:44:45,  1.88s/it]
Outer training loop:  59%|#####8    | 29467/50000 [18:45:00<10:41:01,  1.87s/it]
Outer training loop:  59%|#####8    | 29468/50000 [18:45:02<10:26:45,  1.83s/it]
Outer training loop:  59%|#####8    | 29469/50000 [18:45:03<10:29:07,  1.84s/it]
Outer training loop:  59%|#####8    | 29470/50000 [18:45:05<10:44:39,  1.88s/it]
Outer training loop:  59%|#####8    | 29471/50000 [18:45:07<10:27:03,  1.83s/it]
Outer training loop:  59%|#####8    | 29472/50000 [18:45:09<10:40:25,  1.87s/it]
Outer training loop:  59%|#####8    | 29473/50000 [18:45:11<10:50:31,  1.90s/it]
Outer training loop:  59%|#####8    | 29474/50000 [18:45:13<10:48:53,  1.90s/it]
Outer training loop:  59%|#####8    | 29475/50000 [18:45:15<10:42:04,  1.88s/it]
Outer training loop:  59%|#####8    | 29476/50000 [18:45:17<11:02:07,  1.94s/it]
Outer training loop:  59%|#####8    | 29477/50000 [18:45:19<11:06:38,  1.95s/it]
Outer training loop:  59%|#####8    | 29478/50000 [18:45:21<10:51:20,  1.90s/it]
Outer training loop:  59%|#####8    | 29479/50000 [18:45:22<10:36:43,  1.86s/it]
Outer training loop:  59%|#####8    | 29480/50000 [18:45:24<10:39:15,  1.87s/it]
Outer training loop:  59%|#####8    | 29481/50000 [18:45:26<10:31:45,  1.85s/it]
Outer training loop:  59%|#####8    | 29482/50000 [18:45:28<10:23:33,  1.82s/it]
Outer training loop:  59%|#####8    | 29483/50000 [18:45:30<10:15:12,  1.80s/it]
Outer training loop:  59%|#####8    | 29484/50000 [18:45:31<10:31:38,  1.85s/it]
Outer training loop:  59%|#####8    | 29485/50000 [18:45:33<10:26:20,  1.83s/it]
Outer training loop:  59%|#####8    | 29486/50000 [18:45:35<10:35:07,  1.86s/it]
Outer training loop:  59%|#####8    | 29487/50000 [18:45:37<10:45:31,  1.89s/it]
Outer training loop:  59%|#####8    | 29488/50000 [18:45:39<10:48:43,  1.90s/it]
Outer training loop:  59%|#####8    | 29489/50000 [18:45:41<10:47:36,  1.89s/it]
Outer training loop:  59%|#####8    | 29490/50000 [18:45:43<10:49:54,  1.90s/it]
Outer training loop:  59%|#####8    | 29491/50000 [18:45:45<10:32:50,  1.85s/it]
Outer training loop:  59%|#####8    | 29492/50000 [18:45:47<10:39:41,  1.87s/it]
Outer training loop:  59%|#####8    | 29493/50000 [18:45:48<10:34:23,  1.86s/it]
Outer training loop:  59%|#####8    | 29494/50000 [18:45:50<10:53:59,  1.91s/it]
Outer training loop:  59%|#####8    | 29495/50000 [18:45:52<10:47:03,  1.89s/it]
Outer training loop:  59%|#####8    | 29496/50000 [18:45:54<10:27:57,  1.84s/it]
Outer training loop:  59%|#####8    | 29497/50000 [18:45:56<10:57:39,  1.92s/it]
Outer training loop:  59%|#####8    | 29498/50000 [18:45:58<10:41:01,  1.88s/it]
Outer training loop:  59%|#####8    | 29499/50000 [18:46:00<10:45:11,  1.89s/it]
Outer training loop:  59%|#####8    | 29500/50000 [18:46:02<10:30:09,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:24,  1.76s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.77s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:24,  1.81s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.88s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.86s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:20,  1.87s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:19,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.88s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.92s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:04,  1.86s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:59,  1.84s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:57,  1.83s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:01,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.85s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:47,  1.83s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:51,  1.93s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:44,  1.86s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:41,  1.84s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:47,  2.02s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:43,  1.99s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:43,  2.03s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:40,  2.00s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:37,  2.00s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:33,  2.00s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:28,  1.93s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:25,  1.91s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:24,  1.91s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.85s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.86s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:18,  1.96s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:09,  1.87s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:05,  1.87s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:01,  1.81s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:04,  1.94s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.91s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:52,  2.02s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:49,  1.98s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:46,  1.92s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.90s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:34,  1.94s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:32,  1.89s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.84s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.80s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:21,  1.81s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:17,  1.80s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:17,  1.97s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.91s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.87s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.89s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.83s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.95s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:35,  1.90s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:31,  1.87s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:29,  1.86s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:24,  1.85s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:21,  1.84s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.84s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:16,  1.82s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:13,  1.80s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:11,  1.80s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:12,  1.84s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:24,  2.03s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:17,  1.96s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:10,  1.95s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:06,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:03,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.86s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<02:00,  1.95s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:56,  1.91s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.88s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:53,  1.99s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.92s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:44,  1.89s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:39,  1.94s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:32,  1.88s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:27,  1.86s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:27,  1.95s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:22,  1.88s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:20,  1.88s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.91s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:15,  1.99s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:11,  1.94s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:08,  1.90s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:05,  1.87s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.85s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<01:01,  2.00s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:58,  1.95s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:47,  1.99s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:44,  1.93s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.84s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:29,  2.00s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:27,  1.97s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.93s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.89s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.83s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.84s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:12,  2.01s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.93s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.88s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.83s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.87s/it]

Outer training loop:  59%|#####9    | 29501/50000 [18:51:23<555:59:14, 97.64s/it]
Outer training loop:  59%|#####9    | 29502/50000 [18:51:24<392:13:46, 68.89s/it]
Outer training loop:  59%|#####9    | 29503/50000 [18:51:26<277:52:02, 48.80s/it]
Outer training loop:  59%|#####9    | 29504/50000 [18:51:29<198:39:53, 34.89s/it]
Outer training loop:  59%|#####9    | 29505/50000 [18:51:31<141:57:27, 24.94s/it]
Outer training loop:  59%|#####9    | 29506/50000 [18:51:32<102:35:17, 18.02s/it]
Outer training loop:  59%|#####9    | 29507/50000 [18:51:34<74:49:02, 13.14s/it] 
Outer training loop:  59%|#####9    | 29508/50000 [18:51:36<55:40:53,  9.78s/it]
Outer training loop:  59%|#####9    | 29509/50000 [18:51:38<42:06:03,  7.40s/it]
Outer training loop:  59%|#####9    | 29510/50000 [18:51:40<32:30:27,  5.71s/it]
Outer training loop:  59%|#####9    | 29511/50000 [18:51:42<26:05:34,  4.58s/it]
Outer training loop:  59%|#####9    | 29512/50000 [18:51:44<21:25:18,  3.76s/it]
Outer training loop:  59%|#####9    | 29513/50000 [18:51:45<17:56:47,  3.15s/it]
Outer training loop:  59%|#####9    | 29514/50000 [18:51:47<15:50:09,  2.78s/it]
Outer training loop:  59%|#####9    | 29515/50000 [18:51:49<14:27:39,  2.54s/it]
Outer training loop:  59%|#####9    | 29516/50000 [18:51:51<13:24:20,  2.36s/it]
Outer training loop:  59%|#####9    | 29517/50000 [18:51:53<12:25:11,  2.18s/it]
Outer training loop:  59%|#####9    | 29518/50000 [18:51:55<12:11:52,  2.14s/it]
Outer training loop:  59%|#####9    | 29519/50000 [18:51:57<12:05:02,  2.12s/it]
Outer training loop:  59%|#####9    | 29520/50000 [18:51:59<11:41:20,  2.05s/it]
Outer training loop:  59%|#####9    | 29521/50000 [18:52:01<11:15:01,  1.98s/it]
Outer training loop:  59%|#####9    | 29522/50000 [18:52:03<11:13:12,  1.97s/it]
Outer training loop:  59%|#####9    | 29523/50000 [18:52:05<11:05:06,  1.95s/it]
Outer training loop:  59%|#####9    | 29524/50000 [18:52:06<10:46:01,  1.89s/it]
Outer training loop:  59%|#####9    | 29525/50000 [18:52:08<10:45:01,  1.89s/it]
Outer training loop:  59%|#####9    | 29526/50000 [18:52:10<10:40:47,  1.88s/it]
Outer training loop:  59%|#####9    | 29527/50000 [18:52:12<10:32:47,  1.85s/it]
Outer training loop:  59%|#####9    | 29528/50000 [18:52:14<10:51:23,  1.91s/it]
Outer training loop:  59%|#####9    | 29529/50000 [18:52:16<10:44:05,  1.89s/it]
Outer training loop:  59%|#####9    | 29530/50000 [18:52:18<10:50:02,  1.91s/it]
Outer training loop:  59%|#####9    | 29531/50000 [18:52:20<10:42:09,  1.88s/it]
Outer training loop:  59%|#####9    | 29532/50000 [18:52:22<10:58:49,  1.93s/it]
Outer training loop:  59%|#####9    | 29533/50000 [18:52:23<10:42:07,  1.88s/it]
Outer training loop:  59%|#####9    | 29534/50000 [18:52:25<10:51:52,  1.91s/it]
Outer training loop:  59%|#####9    | 29535/50000 [18:52:27<11:16:25,  1.98s/it]
Outer training loop:  59%|#####9    | 29536/50000 [18:52:29<11:00:33,  1.94s/it]
Outer training loop:  59%|#####9    | 29537/50000 [18:52:31<10:50:03,  1.91s/it]
Outer training loop:  59%|#####9    | 29538/50000 [18:52:33<11:04:53,  1.95s/it]
Outer training loop:  59%|#####9    | 29539/50000 [18:52:35<10:53:04,  1.92s/it]
Outer training loop:  59%|#####9    | 29540/50000 [18:52:37<10:39:51,  1.88s/it]
Outer training loop:  59%|#####9    | 29541/50000 [18:52:39<10:37:12,  1.87s/it]
Outer training loop:  59%|#####9    | 29542/50000 [18:52:41<10:58:17,  1.93s/it]
Outer training loop:  59%|#####9    | 29543/50000 [18:52:43<10:55:55,  1.92s/it]
Outer training loop:  59%|#####9    | 29544/50000 [18:52:44<10:44:13,  1.89s/it]
Outer training loop:  59%|#####9    | 29545/50000 [18:52:46<10:54:36,  1.92s/it]
Outer training loop:  59%|#####9    | 29546/50000 [18:52:48<10:45:38,  1.89s/it]
Outer training loop:  59%|#####9    | 29547/50000 [18:52:50<10:39:26,  1.88s/it]
Outer training loop:  59%|#####9    | 29548/50000 [18:52:52<10:51:39,  1.91s/it]
Outer training loop:  59%|#####9    | 29549/50000 [18:52:54<10:37:21,  1.87s/it]
Outer training loop:  59%|#####9    | 29550/50000 [18:52:56<10:38:20,  1.87s/it]
Outer training loop:  59%|#####9    | 29551/50000 [18:52:58<10:39:24,  1.88s/it]
Outer training loop:  59%|#####9    | 29552/50000 [18:53:00<10:51:54,  1.91s/it]
Outer training loop:  59%|#####9    | 29553/50000 [18:53:01<10:40:29,  1.88s/it]
Outer training loop:  59%|#####9    | 29554/50000 [18:53:03<10:22:21,  1.83s/it]
Outer training loop:  59%|#####9    | 29555/50000 [18:53:05<10:54:23,  1.92s/it]
Outer training loop:  59%|#####9    | 29556/50000 [18:53:07<10:39:41,  1.88s/it]
Outer training loop:  59%|#####9    | 29557/50000 [18:53:09<10:29:07,  1.85s/it]
Outer training loop:  59%|#####9    | 29558/50000 [18:53:11<10:45:33,  1.89s/it]
Outer training loop:  59%|#####9    | 29559/50000 [18:53:13<10:38:29,  1.87s/it]
Outer training loop:  59%|#####9    | 29560/50000 [18:53:14<10:28:08,  1.84s/it]
Outer training loop:  59%|#####9    | 29561/50000 [18:53:16<10:18:45,  1.82s/it]
Outer training loop:  59%|#####9    | 29562/50000 [18:53:18<10:44:51,  1.89s/it]
Outer training loop:  59%|#####9    | 29563/50000 [18:53:20<10:49:54,  1.91s/it]
Outer training loop:  59%|#####9    | 29564/50000 [18:53:22<10:43:12,  1.89s/it]
Outer training loop:  59%|#####9    | 29565/50000 [18:53:24<10:38:35,  1.87s/it]
Outer training loop:  59%|#####9    | 29566/50000 [18:53:26<10:55:16,  1.92s/it]
Outer training loop:  59%|#####9    | 29567/50000 [18:53:28<10:54:01,  1.92s/it]
Outer training loop:  59%|#####9    | 29568/50000 [18:53:30<10:49:21,  1.91s/it]
Outer training loop:  59%|#####9    | 29569/50000 [18:53:32<11:20:30,  2.00s/it]
Outer training loop:  59%|#####9    | 29570/50000 [18:53:34<11:02:25,  1.95s/it]
Outer training loop:  59%|#####9    | 29571/50000 [18:53:36<10:50:26,  1.91s/it]
Outer training loop:  59%|#####9    | 29572/50000 [18:53:37<10:51:49,  1.91s/it]
Outer training loop:  59%|#####9    | 29573/50000 [18:53:39<10:37:20,  1.87s/it]
Outer training loop:  59%|#####9    | 29574/50000 [18:53:41<10:36:41,  1.87s/it]
Outer training loop:  59%|#####9    | 29575/50000 [18:53:43<10:41:53,  1.89s/it]
Outer training loop:  59%|#####9    | 29576/50000 [18:53:45<10:52:34,  1.92s/it]
Outer training loop:  59%|#####9    | 29577/50000 [18:53:47<10:39:03,  1.88s/it]
Outer training loop:  59%|#####9    | 29578/50000 [18:53:49<10:44:19,  1.89s/it]
Outer training loop:  59%|#####9    | 29579/50000 [18:53:51<10:58:45,  1.94s/it]
Outer training loop:  59%|#####9    | 29580/50000 [18:53:53<10:45:38,  1.90s/it]
Outer training loop:  59%|#####9    | 29581/50000 [18:53:54<10:41:52,  1.89s/it]
Outer training loop:  59%|#####9    | 29582/50000 [18:53:56<10:31:31,  1.86s/it]
Outer training loop:  59%|#####9    | 29583/50000 [18:53:58<10:47:25,  1.90s/it]
Outer training loop:  59%|#####9    | 29584/50000 [18:54:00<10:38:59,  1.88s/it]
Outer training loop:  59%|#####9    | 29585/50000 [18:54:02<10:31:09,  1.85s/it]
Outer training loop:  59%|#####9    | 29586/50000 [18:54:04<10:25:49,  1.84s/it]
Outer training loop:  59%|#####9    | 29587/50000 [18:54:06<10:50:34,  1.91s/it]
Outer training loop:  59%|#####9    | 29588/50000 [18:54:08<10:42:44,  1.89s/it]
Outer training loop:  59%|#####9    | 29589/50000 [18:54:09<10:40:33,  1.88s/it]
Outer training loop:  59%|#####9    | 29590/50000 [18:54:12<11:03:49,  1.95s/it]
Outer training loop:  59%|#####9    | 29591/50000 [18:54:13<10:51:24,  1.92s/it]
Outer training loop:  59%|#####9    | 29592/50000 [18:54:15<10:46:32,  1.90s/it]
Outer training loop:  59%|#####9    | 29593/50000 [18:54:17<10:44:04,  1.89s/it]
Outer training loop:  59%|#####9    | 29594/50000 [18:54:19<10:57:44,  1.93s/it]
Outer training loop:  59%|#####9    | 29595/50000 [18:54:21<10:37:57,  1.88s/it]
Outer training loop:  59%|#####9    | 29596/50000 [18:54:23<10:17:11,  1.81s/it]
Outer training loop:  59%|#####9    | 29597/50000 [18:54:25<10:28:58,  1.85s/it]
Outer training loop:  59%|#####9    | 29598/50000 [18:54:26<10:36:08,  1.87s/it]
Outer training loop:  59%|#####9    | 29599/50000 [18:54:28<10:24:42,  1.84s/it]
Outer training loop:  59%|#####9    | 29600/50000 [18:54:30<10:39:32,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.86s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  59%|#####9    | 29601/50000 [18:54:38<20:01:49,  3.53s/it]
Outer training loop:  59%|#####9    | 29602/50000 [18:54:39<17:04:19,  3.01s/it]
Outer training loop:  59%|#####9    | 29603/50000 [18:54:41<15:02:49,  2.66s/it]
Outer training loop:  59%|#####9    | 29604/50000 [18:54:43<13:43:08,  2.42s/it]
Outer training loop:  59%|#####9    | 29605/50000 [18:54:45<12:54:39,  2.28s/it]
Outer training loop:  59%|#####9    | 29606/50000 [18:54:47<12:00:31,  2.12s/it]
Outer training loop:  59%|#####9    | 29607/50000 [18:54:49<11:35:44,  2.05s/it]
Outer training loop:  59%|#####9    | 29608/50000 [18:54:51<11:30:49,  2.03s/it]
Outer training loop:  59%|#####9    | 29609/50000 [18:54:52<11:00:36,  1.94s/it]
Outer training loop:  59%|#####9    | 29610/50000 [18:54:54<10:55:46,  1.93s/it]
Outer training loop:  59%|#####9    | 29611/50000 [18:54:56<11:06:48,  1.96s/it]
Outer training loop:  59%|#####9    | 29612/50000 [18:54:58<11:00:37,  1.94s/it]
Outer training loop:  59%|#####9    | 29613/50000 [18:55:00<10:46:37,  1.90s/it]
Outer training loop:  59%|#####9    | 29614/50000 [18:55:02<10:31:59,  1.86s/it]
Outer training loop:  59%|#####9    | 29615/50000 [18:55:04<10:53:22,  1.92s/it]
Outer training loop:  59%|#####9    | 29616/50000 [18:55:06<11:02:13,  1.95s/it]
Outer training loop:  59%|#####9    | 29617/50000 [18:55:08<10:47:00,  1.90s/it]
Outer training loop:  59%|#####9    | 29618/50000 [18:55:10<10:55:19,  1.93s/it]
Outer training loop:  59%|#####9    | 29619/50000 [18:55:11<10:41:15,  1.89s/it]
Outer training loop:  59%|#####9    | 29620/50000 [18:55:13<10:35:47,  1.87s/it]
Outer training loop:  59%|#####9    | 29621/50000 [18:55:15<10:24:14,  1.84s/it]
Outer training loop:  59%|#####9    | 29622/50000 [18:55:17<10:38:43,  1.88s/it]
Outer training loop:  59%|#####9    | 29623/50000 [18:55:19<10:22:29,  1.83s/it]
Outer training loop:  59%|#####9    | 29624/50000 [18:55:21<10:34:07,  1.87s/it]
Outer training loop:  59%|#####9    | 29625/50000 [18:55:23<10:44:35,  1.90s/it]
Outer training loop:  59%|#####9    | 29626/50000 [18:55:24<10:39:32,  1.88s/it]
Outer training loop:  59%|#####9    | 29627/50000 [18:55:26<10:38:08,  1.88s/it]
Outer training loop:  59%|#####9    | 29628/50000 [18:55:28<10:29:39,  1.85s/it]
Outer training loop:  59%|#####9    | 29629/50000 [18:55:30<10:51:38,  1.92s/it]
Outer training loop:  59%|#####9    | 29630/50000 [18:55:32<10:49:37,  1.91s/it]
Outer training loop:  59%|#####9    | 29631/50000 [18:55:34<10:37:09,  1.88s/it]
Outer training loop:  59%|#####9    | 29632/50000 [18:55:36<10:54:58,  1.93s/it]
Outer training loop:  59%|#####9    | 29633/50000 [18:55:38<10:38:10,  1.88s/it]
Outer training loop:  59%|#####9    | 29634/50000 [18:55:40<10:39:14,  1.88s/it]
Outer training loop:  59%|#####9    | 29635/50000 [18:55:42<10:53:32,  1.93s/it]
Outer training loop:  59%|#####9    | 29636/50000 [18:55:43<10:45:19,  1.90s/it]
Outer training loop:  59%|#####9    | 29637/50000 [18:55:45<10:37:04,  1.88s/it]
Outer training loop:  59%|#####9    | 29638/50000 [18:55:47<10:29:16,  1.85s/it]
Outer training loop:  59%|#####9    | 29639/50000 [18:55:49<10:50:05,  1.92s/it]
Outer training loop:  59%|#####9    | 29640/50000 [18:55:51<10:50:11,  1.92s/it]
Outer training loop:  59%|#####9    | 29641/50000 [18:55:53<10:47:15,  1.91s/it]
Outer training loop:  59%|#####9    | 29642/50000 [18:55:55<10:57:07,  1.94s/it]
Outer training loop:  59%|#####9    | 29643/50000 [18:55:57<11:02:27,  1.95s/it]
Outer training loop:  59%|#####9    | 29644/50000 [18:55:59<10:45:05,  1.90s/it]
Outer training loop:  59%|#####9    | 29645/50000 [18:56:01<10:32:56,  1.87s/it]
Outer training loop:  59%|#####9    | 29646/50000 [18:56:03<10:43:04,  1.90s/it]
Outer training loop:  59%|#####9    | 29647/50000 [18:56:04<10:44:10,  1.90s/it]
Outer training loop:  59%|#####9    | 29648/50000 [18:56:06<10:46:27,  1.91s/it]
Outer training loop:  59%|#####9    | 29649/50000 [18:56:08<10:56:41,  1.94s/it]
Outer training loop:  59%|#####9    | 29650/50000 [18:56:10<10:43:48,  1.90s/it]
Outer training loop:  59%|#####9    | 29651/50000 [18:56:12<10:35:58,  1.88s/it]
Outer training loop:  59%|#####9    | 29652/50000 [18:56:14<10:59:18,  1.94s/it]
Outer training loop:  59%|#####9    | 29653/50000 [18:56:16<10:44:01,  1.90s/it]
Outer training loop:  59%|#####9    | 29654/50000 [18:56:18<10:38:43,  1.88s/it]
Outer training loop:  59%|#####9    | 29655/50000 [18:56:20<11:05:41,  1.96s/it]
Outer training loop:  59%|#####9    | 29656/50000 [18:56:22<10:45:47,  1.90s/it]
Outer training loop:  59%|#####9    | 29657/50000 [18:56:23<10:36:13,  1.88s/it]
Outer training loop:  59%|#####9    | 29658/50000 [18:56:25<10:33:45,  1.87s/it]
Outer training loop:  59%|#####9    | 29659/50000 [18:56:27<10:48:26,  1.91s/it]
Outer training loop:  59%|#####9    | 29660/50000 [18:56:29<10:38:47,  1.88s/it]
Outer training loop:  59%|#####9    | 29661/50000 [18:56:31<10:37:20,  1.88s/it]
Outer training loop:  59%|#####9    | 29662/50000 [18:56:33<10:16:25,  1.82s/it]
Outer training loop:  59%|#####9    | 29663/50000 [18:56:35<10:41:09,  1.89s/it]
Outer training loop:  59%|#####9    | 29664/50000 [18:56:37<10:41:54,  1.89s/it]
Outer training loop:  59%|#####9    | 29665/50000 [18:56:38<10:34:21,  1.87s/it]
Outer training loop:  59%|#####9    | 29666/50000 [18:56:41<11:19:42,  2.01s/it]
Outer training loop:  59%|#####9    | 29667/50000 [18:56:43<11:11:04,  1.98s/it]
Outer training loop:  59%|#####9    | 29668/50000 [18:56:44<10:47:04,  1.91s/it]
Outer training loop:  59%|#####9    | 29669/50000 [18:56:46<10:44:55,  1.90s/it]
Outer training loop:  59%|#####9    | 29670/50000 [18:56:48<10:51:23,  1.92s/it]
Outer training loop:  59%|#####9    | 29671/50000 [18:56:50<10:51:15,  1.92s/it]
Outer training loop:  59%|#####9    | 29672/50000 [18:56:52<10:42:38,  1.90s/it]
Outer training loop:  59%|#####9    | 29673/50000 [18:56:54<10:53:11,  1.93s/it]
Outer training loop:  59%|#####9    | 29674/50000 [18:56:56<10:39:15,  1.89s/it]
Outer training loop:  59%|#####9    | 29675/50000 [18:56:58<10:50:20,  1.92s/it]
Outer training loop:  59%|#####9    | 29676/50000 [18:57:00<10:41:14,  1.89s/it]
Outer training loop:  59%|#####9    | 29677/50000 [18:57:02<11:04:10,  1.96s/it]
Outer training loop:  59%|#####9    | 29678/50000 [18:57:04<10:57:41,  1.94s/it]
Outer training loop:  59%|#####9    | 29679/50000 [18:57:06<10:51:39,  1.92s/it]
Outer training loop:  59%|#####9    | 29680/50000 [18:57:08<11:04:21,  1.96s/it]
Outer training loop:  59%|#####9    | 29681/50000 [18:57:09<10:46:34,  1.91s/it]
Outer training loop:  59%|#####9    | 29682/50000 [18:57:11<10:22:17,  1.84s/it]
Outer training loop:  59%|#####9    | 29683/50000 [18:57:13<10:29:38,  1.86s/it]
Outer training loop:  59%|#####9    | 29684/50000 [18:57:15<10:50:47,  1.92s/it]
Outer training loop:  59%|#####9    | 29685/50000 [18:57:17<10:36:38,  1.88s/it]
Outer training loop:  59%|#####9    | 29686/50000 [18:57:19<10:24:57,  1.85s/it]
Outer training loop:  59%|#####9    | 29687/50000 [18:57:21<10:44:23,  1.90s/it]
Outer training loop:  59%|#####9    | 29688/50000 [18:57:23<10:45:22,  1.91s/it]
Outer training loop:  59%|#####9    | 29689/50000 [18:57:24<10:45:44,  1.91s/it]
Outer training loop:  59%|#####9    | 29690/50000 [18:57:26<10:37:44,  1.88s/it]
Outer training loop:  59%|#####9    | 29691/50000 [18:57:28<10:52:41,  1.93s/it]
Outer training loop:  59%|#####9    | 29692/50000 [18:57:30<10:36:13,  1.88s/it]
Outer training loop:  59%|#####9    | 29693/50000 [18:57:32<10:27:51,  1.86s/it]
Outer training loop:  59%|#####9    | 29694/50000 [18:57:34<10:45:59,  1.91s/it]
Outer training loop:  59%|#####9    | 29695/50000 [18:57:36<10:33:09,  1.87s/it]
Outer training loop:  59%|#####9    | 29696/50000 [18:57:37<10:17:37,  1.83s/it]
Outer training loop:  59%|#####9    | 29697/50000 [18:57:39<10:41:57,  1.90s/it]
Outer training loop:  59%|#####9    | 29698/50000 [18:57:41<10:43:28,  1.90s/it]
Outer training loop:  59%|#####9    | 29699/50000 [18:57:43<10:37:32,  1.88s/it]
Outer training loop:  59%|#####9    | 29700/50000 [18:57:45<10:23:30,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  59%|#####9    | 29701/50000 [18:57:53<20:05:56,  3.56s/it]
Outer training loop:  59%|#####9    | 29702/50000 [18:57:54<17:13:34,  3.06s/it]
Outer training loop:  59%|#####9    | 29703/50000 [18:57:56<14:54:46,  2.65s/it]
Outer training loop:  59%|#####9    | 29704/50000 [18:57:58<13:40:57,  2.43s/it]
Outer training loop:  59%|#####9    | 29705/50000 [18:58:00<12:58:48,  2.30s/it]
Outer training loop:  59%|#####9    | 29706/50000 [18:58:02<12:08:45,  2.15s/it]
Outer training loop:  59%|#####9    | 29707/50000 [18:58:04<11:24:06,  2.02s/it]
Outer training loop:  59%|#####9    | 29708/50000 [18:58:06<11:28:47,  2.04s/it]
Outer training loop:  59%|#####9    | 29709/50000 [18:58:07<11:07:42,  1.97s/it]
Outer training loop:  59%|#####9    | 29710/50000 [18:58:09<11:01:48,  1.96s/it]
Outer training loop:  59%|#####9    | 29711/50000 [18:58:11<10:47:31,  1.91s/it]
Outer training loop:  59%|#####9    | 29712/50000 [18:58:13<11:06:41,  1.97s/it]
Outer training loop:  59%|#####9    | 29713/50000 [18:58:15<11:14:12,  1.99s/it]
Outer training loop:  59%|#####9    | 29714/50000 [18:58:17<11:06:58,  1.97s/it]
Outer training loop:  59%|#####9    | 29715/50000 [18:58:19<11:10:03,  1.98s/it]
Outer training loop:  59%|#####9    | 29716/50000 [18:58:21<10:54:46,  1.94s/it]
Outer training loop:  59%|#####9    | 29717/50000 [18:58:23<10:44:47,  1.91s/it]
Outer training loop:  59%|#####9    | 29718/50000 [18:58:25<10:49:45,  1.92s/it]
Outer training loop:  59%|#####9    | 29719/50000 [18:58:27<11:06:12,  1.97s/it]
Outer training loop:  59%|#####9    | 29720/50000 [18:58:29<11:00:17,  1.95s/it]
Outer training loop:  59%|#####9    | 29721/50000 [18:58:31<10:42:21,  1.90s/it]
Outer training loop:  59%|#####9    | 29722/50000 [18:58:33<10:45:22,  1.91s/it]
Outer training loop:  59%|#####9    | 29723/50000 [18:58:34<10:35:44,  1.88s/it]
Outer training loop:  59%|#####9    | 29724/50000 [18:58:36<10:24:00,  1.85s/it]
Outer training loop:  59%|#####9    | 29725/50000 [18:58:38<10:09:55,  1.80s/it]
Outer training loop:  59%|#####9    | 29726/50000 [18:58:40<10:33:11,  1.87s/it]
Outer training loop:  59%|#####9    | 29727/50000 [18:58:42<10:33:52,  1.88s/it]
Outer training loop:  59%|#####9    | 29728/50000 [18:58:44<10:37:23,  1.89s/it]
Outer training loop:  59%|#####9    | 29729/50000 [18:58:46<10:41:58,  1.90s/it]
Outer training loop:  59%|#####9    | 29730/50000 [18:58:47<10:26:55,  1.86s/it]
Outer training loop:  59%|#####9    | 29731/50000 [18:58:49<10:20:56,  1.84s/it]
Outer training loop:  59%|#####9    | 29732/50000 [18:58:51<10:30:35,  1.87s/it]
Outer training loop:  59%|#####9    | 29733/50000 [18:58:53<10:19:39,  1.83s/it]
Outer training loop:  59%|#####9    | 29734/50000 [18:58:55<10:30:28,  1.87s/it]
Outer training loop:  59%|#####9    | 29735/50000 [18:58:57<10:43:30,  1.91s/it]
Outer training loop:  59%|#####9    | 29736/50000 [18:58:59<10:27:45,  1.86s/it]
Outer training loop:  59%|#####9    | 29737/50000 [18:59:00<10:24:50,  1.85s/it]
Outer training loop:  59%|#####9    | 29738/50000 [18:59:02<10:15:06,  1.82s/it]
Outer training loop:  59%|#####9    | 29739/50000 [18:59:04<10:36:17,  1.88s/it]
Outer training loop:  59%|#####9    | 29740/50000 [18:59:06<10:35:00,  1.88s/it]
Outer training loop:  59%|#####9    | 29741/50000 [18:59:08<10:37:37,  1.89s/it]
Outer training loop:  59%|#####9    | 29742/50000 [18:59:10<11:10:54,  1.99s/it]
Outer training loop:  59%|#####9    | 29743/50000 [18:59:12<10:54:37,  1.94s/it]
Outer training loop:  59%|#####9    | 29744/50000 [18:59:14<10:48:21,  1.92s/it]
Outer training loop:  59%|#####9    | 29745/50000 [18:59:16<10:52:39,  1.93s/it]
Outer training loop:  59%|#####9    | 29746/50000 [18:59:18<10:30:58,  1.87s/it]
Outer training loop:  59%|#####9    | 29747/50000 [18:59:19<10:28:28,  1.86s/it]
Outer training loop:  59%|#####9    | 29748/50000 [18:59:21<10:29:14,  1.86s/it]
Outer training loop:  59%|#####9    | 29749/50000 [18:59:23<10:39:35,  1.89s/it]
Outer training loop:  60%|#####9    | 29750/50000 [18:59:25<10:43:24,  1.91s/it]
Outer training loop:  60%|#####9    | 29751/50000 [18:59:27<10:31:43,  1.87s/it]
Outer training loop:  60%|#####9    | 29752/50000 [18:59:29<10:57:16,  1.95s/it]
Outer training loop:  60%|#####9    | 29753/50000 [18:59:31<10:43:08,  1.91s/it]
Outer training loop:  60%|#####9    | 29754/50000 [18:59:33<10:30:12,  1.87s/it]
Outer training loop:  60%|#####9    | 29755/50000 [18:59:35<10:42:10,  1.90s/it]
Outer training loop:  60%|#####9    | 29756/50000 [18:59:37<10:48:04,  1.92s/it]
Outer training loop:  60%|#####9    | 29757/50000 [18:59:39<10:39:06,  1.89s/it]
Outer training loop:  60%|#####9    | 29758/50000 [18:59:40<10:48:47,  1.92s/it]
Outer training loop:  60%|#####9    | 29759/50000 [18:59:42<10:57:06,  1.95s/it]
Outer training loop:  60%|#####9    | 29760/50000 [18:59:44<10:49:37,  1.93s/it]
Outer training loop:  60%|#####9    | 29761/50000 [18:59:46<10:36:00,  1.89s/it]
Outer training loop:  60%|#####9    | 29762/50000 [18:59:48<10:30:29,  1.87s/it]
Outer training loop:  60%|#####9    | 29763/50000 [18:59:50<11:15:48,  2.00s/it]
Outer training loop:  60%|#####9    | 29764/50000 [18:59:52<11:00:53,  1.96s/it]
Outer training loop:  60%|#####9    | 29765/50000 [18:59:54<10:50:01,  1.93s/it]
Outer training loop:  60%|#####9    | 29766/50000 [18:59:56<11:03:43,  1.97s/it]
Outer training loop:  60%|#####9    | 29767/50000 [18:59:58<10:39:26,  1.90s/it]
Outer training loop:  60%|#####9    | 29768/50000 [19:00:00<10:27:36,  1.86s/it]
Outer training loop:  60%|#####9    | 29769/50000 [19:00:02<10:41:21,  1.90s/it]
Outer training loop:  60%|#####9    | 29770/50000 [19:00:03<10:25:40,  1.86s/it]
Outer training loop:  60%|#####9    | 29771/50000 [19:00:05<10:36:33,  1.89s/it]
Outer training loop:  60%|#####9    | 29772/50000 [19:00:07<10:27:52,  1.86s/it]
Outer training loop:  60%|#####9    | 29773/50000 [19:00:09<10:36:46,  1.89s/it]
Outer training loop:  60%|#####9    | 29774/50000 [19:00:11<10:31:45,  1.87s/it]
Outer training loop:  60%|#####9    | 29775/50000 [19:00:13<10:26:33,  1.86s/it]
Outer training loop:  60%|#####9    | 29776/50000 [19:00:15<10:44:58,  1.91s/it]
Outer training loop:  60%|#####9    | 29777/50000 [19:00:17<10:38:04,  1.89s/it]
Outer training loop:  60%|#####9    | 29778/50000 [19:00:18<10:30:00,  1.87s/it]
Outer training loop:  60%|#####9    | 29779/50000 [19:00:20<10:37:33,  1.89s/it]
Outer training loop:  60%|#####9    | 29780/50000 [19:00:22<10:56:23,  1.95s/it]
Outer training loop:  60%|#####9    | 29781/50000 [19:00:24<10:29:47,  1.87s/it]
Outer training loop:  60%|#####9    | 29782/50000 [19:00:26<10:30:04,  1.87s/it]
Outer training loop:  60%|#####9    | 29783/50000 [19:00:28<11:03:45,  1.97s/it]
Outer training loop:  60%|#####9    | 29784/50000 [19:00:30<10:37:45,  1.89s/it]
Outer training loop:  60%|#####9    | 29785/50000 [19:00:32<10:15:36,  1.83s/it]
Outer training loop:  60%|#####9    | 29786/50000 [19:00:33<10:12:52,  1.82s/it]
Outer training loop:  60%|#####9    | 29787/50000 [19:00:36<10:53:11,  1.94s/it]
Outer training loop:  60%|#####9    | 29788/50000 [19:00:38<10:50:34,  1.93s/it]
Outer training loop:  60%|#####9    | 29789/50000 [19:00:39<10:39:58,  1.90s/it]
Outer training loop:  60%|#####9    | 29790/50000 [19:00:41<10:48:06,  1.92s/it]
Outer training loop:  60%|#####9    | 29791/50000 [19:00:43<10:39:33,  1.90s/it]
Outer training loop:  60%|#####9    | 29792/50000 [19:00:45<10:32:18,  1.88s/it]
Outer training loop:  60%|#####9    | 29793/50000 [19:00:47<10:13:45,  1.82s/it]
Outer training loop:  60%|#####9    | 29794/50000 [19:00:49<10:50:00,  1.93s/it]
Outer training loop:  60%|#####9    | 29795/50000 [19:00:51<10:36:49,  1.89s/it]
Outer training loop:  60%|#####9    | 29796/50000 [19:00:52<10:23:59,  1.85s/it]
Outer training loop:  60%|#####9    | 29797/50000 [19:00:54<10:41:33,  1.91s/it]
Outer training loop:  60%|#####9    | 29798/50000 [19:00:56<10:30:59,  1.87s/it]
Outer training loop:  60%|#####9    | 29799/50000 [19:00:58<10:25:30,  1.86s/it]
Outer training loop:  60%|#####9    | 29800/50000 [19:01:00<10:30:50,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  60%|#####9    | 29801/50000 [19:01:07<19:41:01,  3.51s/it]
Outer training loop:  60%|#####9    | 29802/50000 [19:01:09<16:54:28,  3.01s/it]
Outer training loop:  60%|#####9    | 29803/50000 [19:01:11<14:59:07,  2.67s/it]
Outer training loop:  60%|#####9    | 29804/50000 [19:01:13<13:57:55,  2.49s/it]
Outer training loop:  60%|#####9    | 29805/50000 [19:01:15<12:46:20,  2.28s/it]
Outer training loop:  60%|#####9    | 29806/50000 [19:01:17<11:57:58,  2.13s/it]
Outer training loop:  60%|#####9    | 29807/50000 [19:01:18<11:18:46,  2.02s/it]
Outer training loop:  60%|#####9    | 29808/50000 [19:01:20<11:17:12,  2.01s/it]
Outer training loop:  60%|#####9    | 29809/50000 [19:01:22<10:51:08,  1.93s/it]
Outer training loop:  60%|#####9    | 29810/50000 [19:01:24<11:04:42,  1.98s/it]
Outer training loop:  60%|#####9    | 29811/50000 [19:01:26<11:08:33,  1.99s/it]
Outer training loop:  60%|#####9    | 29812/50000 [19:01:28<10:56:47,  1.95s/it]
Outer training loop:  60%|#####9    | 29813/50000 [19:01:30<10:40:20,  1.90s/it]
Outer training loop:  60%|#####9    | 29814/50000 [19:01:32<11:00:06,  1.96s/it]
Outer training loop:  60%|#####9    | 29815/50000 [19:01:34<10:55:59,  1.95s/it]
Outer training loop:  60%|#####9    | 29816/50000 [19:01:36<11:00:33,  1.96s/it]
Outer training loop:  60%|#####9    | 29817/50000 [19:01:38<11:12:52,  2.00s/it]
Outer training loop:  60%|#####9    | 29818/50000 [19:01:40<10:53:51,  1.94s/it]
Outer training loop:  60%|#####9    | 29819/50000 [19:01:42<10:46:58,  1.92s/it]
Outer training loop:  60%|#####9    | 29820/50000 [19:01:44<10:35:56,  1.89s/it]
Outer training loop:  60%|#####9    | 29821/50000 [19:01:46<10:49:33,  1.93s/it]
Outer training loop:  60%|#####9    | 29822/50000 [19:01:47<10:48:26,  1.93s/it]
Outer training loop:  60%|#####9    | 29823/50000 [19:01:49<10:41:11,  1.91s/it]
Outer training loop:  60%|#####9    | 29824/50000 [19:01:51<11:00:16,  1.96s/it]
Outer training loop:  60%|#####9    | 29825/50000 [19:01:53<10:32:43,  1.88s/it]
Outer training loop:  60%|#####9    | 29826/50000 [19:01:55<10:38:50,  1.90s/it]
Outer training loop:  60%|#####9    | 29827/50000 [19:01:57<10:40:05,  1.90s/it]
Outer training loop:  60%|#####9    | 29828/50000 [19:01:59<10:53:53,  1.94s/it]
Outer training loop:  60%|#####9    | 29829/50000 [19:02:01<10:34:09,  1.89s/it]
Outer training loop:  60%|#####9    | 29830/50000 [19:02:03<10:34:09,  1.89s/it]
Outer training loop:  60%|#####9    | 29831/50000 [19:02:05<10:44:36,  1.92s/it]
Outer training loop:  60%|#####9    | 29832/50000 [19:02:07<10:39:48,  1.90s/it]
Outer training loop:  60%|#####9    | 29833/50000 [19:02:08<10:32:29,  1.88s/it]
Outer training loop:  60%|#####9    | 29834/50000 [19:02:10<10:57:35,  1.96s/it]
Outer training loop:  60%|#####9    | 29835/50000 [19:02:12<10:44:03,  1.92s/it]
Outer training loop:  60%|#####9    | 29836/50000 [19:02:14<10:26:00,  1.86s/it]
Outer training loop:  60%|#####9    | 29837/50000 [19:02:16<10:30:00,  1.87s/it]
Outer training loop:  60%|#####9    | 29838/50000 [19:02:18<10:50:01,  1.93s/it]
Outer training loop:  60%|#####9    | 29839/50000 [19:02:20<10:41:24,  1.91s/it]
Outer training loop:  60%|#####9    | 29840/50000 [19:02:22<10:19:03,  1.84s/it]
Outer training loop:  60%|#####9    | 29841/50000 [19:02:23<10:10:04,  1.82s/it]
Outer training loop:  60%|#####9    | 29842/50000 [19:02:25<10:39:17,  1.90s/it]
Outer training loop:  60%|#####9    | 29843/50000 [19:02:27<10:31:13,  1.88s/it]
Outer training loop:  60%|#####9    | 29844/50000 [19:02:29<10:25:32,  1.86s/it]
Outer training loop:  60%|#####9    | 29845/50000 [19:02:31<10:39:28,  1.90s/it]
Outer training loop:  60%|#####9    | 29846/50000 [19:02:33<10:31:33,  1.88s/it]
Outer training loop:  60%|#####9    | 29847/50000 [19:02:35<10:29:07,  1.87s/it]
Outer training loop:  60%|#####9    | 29848/50000 [19:02:37<10:42:03,  1.91s/it]
Outer training loop:  60%|#####9    | 29849/50000 [19:02:39<10:34:18,  1.89s/it]
Outer training loop:  60%|#####9    | 29850/50000 [19:02:41<10:46:32,  1.93s/it]
Outer training loop:  60%|#####9    | 29851/50000 [19:02:43<10:48:53,  1.93s/it]
Outer training loop:  60%|#####9    | 29852/50000 [19:02:45<10:55:45,  1.95s/it]
Outer training loop:  60%|#####9    | 29853/50000 [19:02:46<10:36:17,  1.89s/it]
Outer training loop:  60%|#####9    | 29854/50000 [19:02:48<10:35:34,  1.89s/it]
Outer training loop:  60%|#####9    | 29855/50000 [19:02:50<10:58:45,  1.96s/it]
Outer training loop:  60%|#####9    | 29856/50000 [19:02:52<10:45:25,  1.92s/it]
Outer training loop:  60%|#####9    | 29857/50000 [19:02:54<10:40:21,  1.91s/it]
Outer training loop:  60%|#####9    | 29858/50000 [19:02:56<10:35:57,  1.89s/it]
Outer training loop:  60%|#####9    | 29859/50000 [19:02:58<10:33:31,  1.89s/it]
Outer training loop:  60%|#####9    | 29860/50000 [19:03:00<10:42:33,  1.91s/it]
Outer training loop:  60%|#####9    | 29861/50000 [19:03:02<10:29:35,  1.88s/it]
Outer training loop:  60%|#####9    | 29862/50000 [19:03:03<10:29:01,  1.87s/it]
Outer training loop:  60%|#####9    | 29863/50000 [19:03:05<10:29:20,  1.88s/it]
Outer training loop:  60%|#####9    | 29864/50000 [19:03:07<10:25:10,  1.86s/it]
Outer training loop:  60%|#####9    | 29865/50000 [19:03:09<11:01:27,  1.97s/it]
Outer training loop:  60%|#####9    | 29866/50000 [19:03:11<10:49:19,  1.94s/it]
Outer training loop:  60%|#####9    | 29867/50000 [19:03:13<10:40:00,  1.91s/it]
Outer training loop:  60%|#####9    | 29868/50000 [19:03:15<10:26:51,  1.87s/it]
Outer training loop:  60%|#####9    | 29869/50000 [19:03:17<10:34:27,  1.89s/it]
Outer training loop:  60%|#####9    | 29870/50000 [19:03:19<10:24:55,  1.86s/it]
Outer training loop:  60%|#####9    | 29871/50000 [19:03:20<10:28:20,  1.87s/it]
Outer training loop:  60%|#####9    | 29872/50000 [19:03:22<10:45:56,  1.93s/it]
Outer training loop:  60%|#####9    | 29873/50000 [19:03:24<10:41:45,  1.91s/it]
Outer training loop:  60%|#####9    | 29874/50000 [19:03:26<10:47:02,  1.93s/it]
Outer training loop:  60%|#####9    | 29875/50000 [19:03:28<10:46:01,  1.93s/it]
Outer training loop:  60%|#####9    | 29876/50000 [19:03:30<10:59:32,  1.97s/it]
Outer training loop:  60%|#####9    | 29877/50000 [19:03:32<10:48:58,  1.94s/it]
Outer training loop:  60%|#####9    | 29878/50000 [19:03:34<10:44:50,  1.92s/it]
Outer training loop:  60%|#####9    | 29879/50000 [19:03:36<10:57:27,  1.96s/it]
Outer training loop:  60%|#####9    | 29880/50000 [19:03:38<10:37:15,  1.90s/it]
Outer training loop:  60%|#####9    | 29881/50000 [19:03:40<10:21:29,  1.85s/it]
Outer training loop:  60%|#####9    | 29882/50000 [19:03:41<10:13:31,  1.83s/it]
Outer training loop:  60%|#####9    | 29883/50000 [19:03:44<10:49:57,  1.94s/it]
Outer training loop:  60%|#####9    | 29884/50000 [19:03:45<10:26:50,  1.87s/it]
Outer training loop:  60%|#####9    | 29885/50000 [19:03:47<10:13:05,  1.83s/it]
Outer training loop:  60%|#####9    | 29886/50000 [19:03:49<10:36:00,  1.90s/it]
Outer training loop:  60%|#####9    | 29887/50000 [19:03:51<10:20:45,  1.85s/it]
Outer training loop:  60%|#####9    | 29888/50000 [19:03:53<10:13:17,  1.83s/it]
Outer training loop:  60%|#####9    | 29889/50000 [19:03:54<10:11:21,  1.82s/it]
Outer training loop:  60%|#####9    | 29890/50000 [19:03:56<10:14:24,  1.83s/it]
Outer training loop:  60%|#####9    | 29891/50000 [19:03:58<10:15:44,  1.84s/it]
Outer training loop:  60%|#####9    | 29892/50000 [19:04:00<10:03:44,  1.80s/it]
Outer training loop:  60%|#####9    | 29893/50000 [19:04:02<10:24:35,  1.86s/it]
Outer training loop:  60%|#####9    | 29894/50000 [19:04:04<10:13:09,  1.83s/it]
Outer training loop:  60%|#####9    | 29895/50000 [19:04:05<10:14:30,  1.83s/it]
Outer training loop:  60%|#####9    | 29896/50000 [19:04:07<10:10:48,  1.82s/it]
Outer training loop:  60%|#####9    | 29897/50000 [19:04:09<10:27:38,  1.87s/it]
Outer training loop:  60%|#####9    | 29898/50000 [19:04:11<10:18:02,  1.84s/it]
Outer training loop:  60%|#####9    | 29899/50000 [19:04:13<10:22:58,  1.86s/it]
Outer training loop:  60%|#####9    | 29900/50000 [19:04:15<10:38:26,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  60%|#####9    | 29901/50000 [19:04:22<19:32:39,  3.50s/it]
Outer training loop:  60%|#####9    | 29902/50000 [19:04:24<16:45:16,  3.00s/it]
Outer training loop:  60%|#####9    | 29903/50000 [19:04:26<14:54:22,  2.67s/it]
Outer training loop:  60%|#####9    | 29904/50000 [19:04:28<13:47:53,  2.47s/it]
Outer training loop:  60%|#####9    | 29905/50000 [19:04:30<12:42:00,  2.28s/it]
Outer training loop:  60%|#####9    | 29906/50000 [19:04:32<12:05:53,  2.17s/it]
Outer training loop:  60%|#####9    | 29907/50000 [19:04:34<12:20:08,  2.21s/it]
Outer training loop:  60%|#####9    | 29908/50000 [19:04:36<11:33:38,  2.07s/it]
Outer training loop:  60%|#####9    | 29909/50000 [19:04:37<11:04:56,  1.99s/it]
Outer training loop:  60%|#####9    | 29910/50000 [19:04:39<10:52:01,  1.95s/it]
Outer training loop:  60%|#####9    | 29911/50000 [19:04:41<10:59:55,  1.97s/it]
Outer training loop:  60%|#####9    | 29912/50000 [19:04:43<10:56:46,  1.96s/it]
Outer training loop:  60%|#####9    | 29913/50000 [19:04:45<10:44:09,  1.92s/it]
Outer training loop:  60%|#####9    | 29914/50000 [19:04:47<10:27:01,  1.87s/it]
Outer training loop:  60%|#####9    | 29915/50000 [19:04:49<10:53:43,  1.95s/it]
Outer training loop:  60%|#####9    | 29916/50000 [19:04:51<10:50:55,  1.94s/it]
Outer training loop:  60%|#####9    | 29917/50000 [19:04:53<10:43:58,  1.92s/it]
Outer training loop:  60%|#####9    | 29918/50000 [19:04:55<10:49:53,  1.94s/it]
Outer training loop:  60%|#####9    | 29919/50000 [19:04:57<10:35:03,  1.90s/it]
Outer training loop:  60%|#####9    | 29920/50000 [19:04:58<10:31:51,  1.89s/it]
Outer training loop:  60%|#####9    | 29921/50000 [19:05:00<10:26:36,  1.87s/it]
Outer training loop:  60%|#####9    | 29922/50000 [19:05:02<10:28:47,  1.88s/it]
Outer training loop:  60%|#####9    | 29923/50000 [19:05:04<10:23:34,  1.86s/it]
Outer training loop:  60%|#####9    | 29924/50000 [19:05:06<10:22:17,  1.86s/it]
Outer training loop:  60%|#####9    | 29925/50000 [19:05:08<10:37:07,  1.90s/it]
Outer training loop:  60%|#####9    | 29926/50000 [19:05:10<10:31:50,  1.89s/it]
Outer training loop:  60%|#####9    | 29927/50000 [19:05:11<10:15:55,  1.84s/it]
Outer training loop:  60%|#####9    | 29928/50000 [19:05:14<10:49:18,  1.94s/it]
Outer training loop:  60%|#####9    | 29929/50000 [19:05:15<10:37:24,  1.91s/it]
Outer training loop:  60%|#####9    | 29930/50000 [19:05:17<10:33:20,  1.89s/it]
Outer training loop:  60%|#####9    | 29931/50000 [19:05:19<10:29:16,  1.88s/it]
Outer training loop:  60%|#####9    | 29932/50000 [19:05:21<10:42:09,  1.92s/it]
Outer training loop:  60%|#####9    | 29933/50000 [19:05:23<10:21:22,  1.86s/it]
Outer training loop:  60%|#####9    | 29934/50000 [19:05:25<10:23:35,  1.86s/it]
Outer training loop:  60%|#####9    | 29935/50000 [19:05:27<10:37:11,  1.91s/it]
Outer training loop:  60%|#####9    | 29936/50000 [19:05:28<10:18:48,  1.85s/it]
Outer training loop:  60%|#####9    | 29937/50000 [19:05:30<10:10:58,  1.83s/it]
Outer training loop:  60%|#####9    | 29938/50000 [19:05:32<10:04:52,  1.81s/it]
Outer training loop:  60%|#####9    | 29939/50000 [19:05:34<10:27:14,  1.88s/it]
Outer training loop:  60%|#####9    | 29940/50000 [19:05:36<10:14:35,  1.84s/it]
Outer training loop:  60%|#####9    | 29941/50000 [19:05:38<10:20:14,  1.86s/it]
Outer training loop:  60%|#####9    | 29942/50000 [19:05:40<10:31:43,  1.89s/it]
Outer training loop:  60%|#####9    | 29943/50000 [19:05:41<10:14:21,  1.84s/it]
Outer training loop:  60%|#####9    | 29944/50000 [19:05:43<10:19:42,  1.85s/it]
Outer training loop:  60%|#####9    | 29945/50000 [19:05:45<10:13:19,  1.83s/it]
Outer training loop:  60%|#####9    | 29946/50000 [19:05:47<10:55:05,  1.96s/it]
Outer training loop:  60%|#####9    | 29947/50000 [19:05:49<10:34:41,  1.90s/it]
Outer training loop:  60%|#####9    | 29948/50000 [19:05:51<10:30:38,  1.89s/it]
Outer training loop:  60%|#####9    | 29949/50000 [19:05:53<10:34:43,  1.90s/it]
Outer training loop:  60%|#####9    | 29950/50000 [19:05:55<10:23:31,  1.87s/it]
Outer training loop:  60%|#####9    | 29951/50000 [19:05:56<10:15:38,  1.84s/it]
Outer training loop:  60%|#####9    | 29952/50000 [19:05:58<10:33:14,  1.90s/it]
Outer training loop:  60%|#####9    | 29953/50000 [19:06:00<10:25:33,  1.87s/it]
Outer training loop:  60%|#####9    | 29954/50000 [19:06:02<10:20:40,  1.86s/it]
Outer training loop:  60%|#####9    | 29955/50000 [19:06:04<10:20:13,  1.86s/it]
Outer training loop:  60%|#####9    | 29956/50000 [19:06:06<10:46:52,  1.94s/it]
Outer training loop:  60%|#####9    | 29957/50000 [19:06:08<11:08:25,  2.00s/it]
Outer training loop:  60%|#####9    | 29958/50000 [19:06:10<11:02:48,  1.98s/it]
Outer training loop:  60%|#####9    | 29959/50000 [19:06:12<11:05:06,  1.99s/it]
Outer training loop:  60%|#####9    | 29960/50000 [19:06:14<10:58:23,  1.97s/it]
Outer training loop:  60%|#####9    | 29961/50000 [19:06:16<10:50:01,  1.95s/it]
Outer training loop:  60%|#####9    | 29962/50000 [19:06:18<10:58:02,  1.97s/it]
Outer training loop:  60%|#####9    | 29963/50000 [19:06:20<10:50:00,  1.95s/it]
Outer training loop:  60%|#####9    | 29964/50000 [19:06:22<10:28:33,  1.88s/it]
Outer training loop:  60%|#####9    | 29965/50000 [19:06:24<10:49:47,  1.95s/it]
Outer training loop:  60%|#####9    | 29966/50000 [19:06:26<10:51:29,  1.95s/it]
Outer training loop:  60%|#####9    | 29967/50000 [19:06:28<10:43:34,  1.93s/it]
Outer training loop:  60%|#####9    | 29968/50000 [19:06:30<11:06:14,  2.00s/it]
Outer training loop:  60%|#####9    | 29969/50000 [19:06:32<10:52:57,  1.96s/it]
Outer training loop:  60%|#####9    | 29970/50000 [19:06:33<10:40:59,  1.92s/it]
Outer training loop:  60%|#####9    | 29971/50000 [19:06:36<11:13:01,  2.02s/it]
Outer training loop:  60%|#####9    | 29972/50000 [19:06:38<11:04:06,  1.99s/it]
Outer training loop:  60%|#####9    | 29973/50000 [19:06:39<10:47:11,  1.94s/it]
Outer training loop:  60%|#####9    | 29974/50000 [19:06:41<10:39:05,  1.91s/it]
Outer training loop:  60%|#####9    | 29975/50000 [19:06:44<11:12:41,  2.02s/it]
Outer training loop:  60%|#####9    | 29976/50000 [19:06:45<10:52:32,  1.96s/it]
Outer training loop:  60%|#####9    | 29977/50000 [19:06:47<10:35:48,  1.91s/it]
Outer training loop:  60%|#####9    | 29978/50000 [19:06:49<10:22:13,  1.86s/it]
Outer training loop:  60%|#####9    | 29979/50000 [19:06:51<10:41:10,  1.92s/it]
Outer training loop:  60%|#####9    | 29980/50000 [19:06:53<10:35:40,  1.91s/it]
Outer training loop:  60%|#####9    | 29981/50000 [19:06:55<10:36:28,  1.91s/it]
Outer training loop:  60%|#####9    | 29982/50000 [19:06:57<10:50:59,  1.95s/it]
Outer training loop:  60%|#####9    | 29983/50000 [19:06:59<10:41:24,  1.92s/it]
Outer training loop:  60%|#####9    | 29984/50000 [19:07:00<10:30:33,  1.89s/it]
Outer training loop:  60%|#####9    | 29985/50000 [19:07:02<10:23:57,  1.87s/it]
Outer training loop:  60%|#####9    | 29986/50000 [19:07:04<10:44:53,  1.93s/it]
Outer training loop:  60%|#####9    | 29987/50000 [19:07:06<10:36:54,  1.91s/it]
Outer training loop:  60%|#####9    | 29988/50000 [19:07:08<10:19:59,  1.86s/it]
Outer training loop:  60%|#####9    | 29989/50000 [19:07:10<10:31:44,  1.89s/it]
Outer training loop:  60%|#####9    | 29990/50000 [19:07:12<10:33:03,  1.90s/it]
Outer training loop:  60%|#####9    | 29991/50000 [19:07:14<10:29:47,  1.89s/it]
Outer training loop:  60%|#####9    | 29992/50000 [19:07:16<10:22:07,  1.87s/it]
Outer training loop:  60%|#####9    | 29993/50000 [19:07:18<10:40:39,  1.92s/it]
Outer training loop:  60%|#####9    | 29994/50000 [19:07:19<10:26:49,  1.88s/it]
Outer training loop:  60%|#####9    | 29995/50000 [19:07:21<10:24:47,  1.87s/it]
Outer training loop:  60%|#####9    | 29996/50000 [19:07:23<10:41:00,  1.92s/it]
Outer training loop:  60%|#####9    | 29997/50000 [19:07:25<10:40:07,  1.92s/it]
Outer training loop:  60%|#####9    | 29998/50000 [19:07:27<10:36:23,  1.91s/it]
Outer training loop:  60%|#####9    | 29999/50000 [19:07:29<10:42:54,  1.93s/it]
Outer training loop:  60%|######    | 30000/50000 [19:07:31<10:21:43,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:27,  1.80s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:30,  1.86s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:34,  1.93s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.90s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.94s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.93s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.89s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:20,  1.87s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:25,  1.97s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:20,  1.93s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:15,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:16,  1.94s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:14,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:04,  1.85s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:05,  1.93s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.91s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<02:00,  1.97s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:57,  1.95s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:54,  1.95s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:53,  1.95s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:49,  1.93s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.92s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:49,  1.99s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.92s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:35,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:37,  1.96s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:32,  1.90s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:28,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.88s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:22,  1.82s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:25,  1.93s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:20,  1.87s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:17,  1.85s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:17,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:11,  1.83s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:08,  1.82s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.91s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:07,  1.87s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.86s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:56,  1.81s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:57,  1.98s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:53,  1.91s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.90s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:50,  1.94s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.91s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.86s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:43,  1.99s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.93s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:28,  1.80s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:26,  1.80s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:24,  1.78s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.95s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.91s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.87s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.87s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.86s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.85s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.85s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:08,  2.09s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:06,  2.02s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.94s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.86s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:20,  1.71s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:20,  1.74s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:20,  1.75s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.79s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:19,  1.79s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:33,  1.99s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:28,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.90s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:18,  1.88s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:20,  1.93s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:12,  1.87s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:08,  1.84s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:03,  1.85s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:00,  1.82s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:58,  1.83s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:01,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.84s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:57,  1.96s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:46,  1.87s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:40,  1.82s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:35,  1.84s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:31,  1.84s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.87s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:24,  1.96s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.93s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.91s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:16,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:12,  1.87s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:10,  1.87s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:07,  1.84s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:10,  1.97s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:06,  1.90s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:57,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.80s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:54,  1.96s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:51,  1.90s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:44,  1.83s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:41,  1.98s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:39,  1.98s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:36,  1.93s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:34,  1.91s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:32,  1.92s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.92s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.90s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.93s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.97s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:25,  2.09s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:21,  1.99s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.95s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.94s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.92s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.91s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.84s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.85s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.84s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.99s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  2.00s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.98s/it]

Outer training loop:  60%|######    | 30001/50000 [19:12:53<544:18:45, 97.98s/it]
Outer training loop:  60%|######    | 30002/50000 [19:12:55<384:03:33, 69.14s/it]
Outer training loop:  60%|######    | 30003/50000 [19:12:57<271:40:10, 48.91s/it]
Outer training loop:  60%|######    | 30004/50000 [19:12:58<193:18:41, 34.80s/it]
Outer training loop:  60%|######    | 30005/50000 [19:13:00<138:27:53, 24.93s/it]
Outer training loop:  60%|######    | 30006/50000 [19:13:02<99:54:48, 17.99s/it] 
Outer training loop:  60%|######    | 30007/50000 [19:13:04<72:52:47, 13.12s/it]
Outer training loop:  60%|######    | 30008/50000 [19:13:06<54:03:48,  9.74s/it]
Outer training loop:  60%|######    | 30009/50000 [19:13:08<41:47:24,  7.53s/it]
Outer training loop:  60%|######    | 30010/50000 [19:13:10<32:17:41,  5.82s/it]
Outer training loop:  60%|######    | 30011/50000 [19:13:12<25:41:06,  4.63s/it]
Outer training loop:  60%|######    | 30012/50000 [19:13:14<21:19:09,  3.84s/it]
Outer training loop:  60%|######    | 30013/50000 [19:13:16<17:55:40,  3.23s/it]
Outer training loop:  60%|######    | 30014/50000 [19:13:17<15:30:56,  2.79s/it]
Outer training loop:  60%|######    | 30015/50000 [19:13:19<13:56:36,  2.51s/it]
Outer training loop:  60%|######    | 30016/50000 [19:13:21<13:01:01,  2.34s/it]
Outer training loop:  60%|######    | 30017/50000 [19:13:23<12:06:15,  2.18s/it]
Outer training loop:  60%|######    | 30018/50000 [19:13:25<11:26:55,  2.06s/it]
Outer training loop:  60%|######    | 30019/50000 [19:13:27<11:19:30,  2.04s/it]
Outer training loop:  60%|######    | 30020/50000 [19:13:29<10:58:56,  1.98s/it]
Outer training loop:  60%|######    | 30021/50000 [19:13:30<10:52:45,  1.96s/it]
Outer training loop:  60%|######    | 30022/50000 [19:13:32<10:34:07,  1.90s/it]
Outer training loop:  60%|######    | 30023/50000 [19:13:34<10:35:16,  1.91s/it]
Outer training loop:  60%|######    | 30024/50000 [19:13:36<10:40:41,  1.92s/it]
Outer training loop:  60%|######    | 30025/50000 [19:13:38<10:17:53,  1.86s/it]
Outer training loop:  60%|######    | 30026/50000 [19:13:40<10:42:46,  1.93s/it]
Outer training loop:  60%|######    | 30027/50000 [19:13:42<10:28:05,  1.89s/it]
Outer training loop:  60%|######    | 30028/50000 [19:13:44<10:21:05,  1.87s/it]
Outer training loop:  60%|######    | 30029/50000 [19:13:46<10:36:13,  1.91s/it]
Outer training loop:  60%|######    | 30030/50000 [19:13:47<10:24:09,  1.88s/it]
Outer training loop:  60%|######    | 30031/50000 [19:13:49<10:08:21,  1.83s/it]
Outer training loop:  60%|######    | 30032/50000 [19:13:51<10:09:41,  1.83s/it]
Outer training loop:  60%|######    | 30033/50000 [19:13:53<10:31:54,  1.90s/it]
Outer training loop:  60%|######    | 30034/50000 [19:13:55<10:21:49,  1.87s/it]
Outer training loop:  60%|######    | 30035/50000 [19:13:57<10:13:22,  1.84s/it]
Outer training loop:  60%|######    | 30036/50000 [19:13:59<10:28:11,  1.89s/it]
Outer training loop:  60%|######    | 30037/50000 [19:14:00<10:13:05,  1.84s/it]
Outer training loop:  60%|######    | 30038/50000 [19:14:02<10:28:06,  1.89s/it]
Outer training loop:  60%|######    | 30039/50000 [19:14:04<10:19:58,  1.86s/it]
Outer training loop:  60%|######    | 30040/50000 [19:14:06<10:50:23,  1.96s/it]
Outer training loop:  60%|######    | 30041/50000 [19:14:08<10:40:38,  1.93s/it]
Outer training loop:  60%|######    | 30042/50000 [19:14:10<10:47:05,  1.95s/it]
Outer training loop:  60%|######    | 30043/50000 [19:14:12<10:45:45,  1.94s/it]
Outer training loop:  60%|######    | 30044/50000 [19:14:14<10:50:57,  1.96s/it]
Outer training loop:  60%|######    | 30045/50000 [19:14:16<10:37:05,  1.92s/it]
Outer training loop:  60%|######    | 30046/50000 [19:14:18<10:24:56,  1.88s/it]
Outer training loop:  60%|######    | 30047/50000 [19:14:20<11:00:09,  1.99s/it]
Outer training loop:  60%|######    | 30048/50000 [19:14:22<10:38:02,  1.92s/it]
Outer training loop:  60%|######    | 30049/50000 [19:14:23<10:31:23,  1.90s/it]
Outer training loop:  60%|######    | 30050/50000 [19:14:25<10:33:28,  1.91s/it]
Outer training loop:  60%|######    | 30051/50000 [19:14:27<10:42:49,  1.93s/it]
Outer training loop:  60%|######    | 30052/50000 [19:14:29<10:37:40,  1.92s/it]
Outer training loop:  60%|######    | 30053/50000 [19:14:31<10:27:35,  1.89s/it]
Outer training loop:  60%|######    | 30054/50000 [19:14:33<10:42:27,  1.93s/it]
Outer training loop:  60%|######    | 30055/50000 [19:14:35<10:27:04,  1.89s/it]
Outer training loop:  60%|######    | 30056/50000 [19:14:37<10:16:01,  1.85s/it]
Outer training loop:  60%|######    | 30057/50000 [19:14:39<10:41:23,  1.93s/it]
Outer training loop:  60%|######    | 30058/50000 [19:14:41<10:37:15,  1.92s/it]
Outer training loop:  60%|######    | 30059/50000 [19:14:43<10:30:56,  1.90s/it]
Outer training loop:  60%|######    | 30060/50000 [19:14:45<10:40:26,  1.93s/it]
Outer training loop:  60%|######    | 30061/50000 [19:14:46<10:29:23,  1.89s/it]
Outer training loop:  60%|######    | 30062/50000 [19:14:48<10:19:23,  1.86s/it]
Outer training loop:  60%|######    | 30063/50000 [19:14:50<10:19:12,  1.86s/it]
Outer training loop:  60%|######    | 30064/50000 [19:14:52<10:24:34,  1.88s/it]
Outer training loop:  60%|######    | 30065/50000 [19:14:54<10:11:26,  1.84s/it]
Outer training loop:  60%|######    | 30066/50000 [19:14:56<10:20:22,  1.87s/it]
Outer training loop:  60%|######    | 30067/50000 [19:14:58<10:29:30,  1.89s/it]
Outer training loop:  60%|######    | 30068/50000 [19:14:59<10:28:25,  1.89s/it]
Outer training loop:  60%|######    | 30069/50000 [19:15:01<10:15:43,  1.85s/it]
Outer training loop:  60%|######    | 30070/50000 [19:15:03<10:07:38,  1.83s/it]
Outer training loop:  60%|######    | 30071/50000 [19:15:05<10:27:18,  1.89s/it]
Outer training loop:  60%|######    | 30072/50000 [19:15:07<10:09:23,  1.83s/it]
Outer training loop:  60%|######    | 30073/50000 [19:15:09<10:06:43,  1.83s/it]
Outer training loop:  60%|######    | 30074/50000 [19:15:10<10:00:43,  1.81s/it]
Outer training loop:  60%|######    | 30075/50000 [19:15:12<10:32:43,  1.91s/it]
Outer training loop:  60%|######    | 30076/50000 [19:15:14<10:31:23,  1.90s/it]
Outer training loop:  60%|######    | 30077/50000 [19:15:16<10:23:15,  1.88s/it]
Outer training loop:  60%|######    | 30078/50000 [19:15:18<10:33:38,  1.91s/it]
Outer training loop:  60%|######    | 30079/50000 [19:15:20<10:38:43,  1.92s/it]
Outer training loop:  60%|######    | 30080/50000 [19:15:22<10:23:56,  1.88s/it]
Outer training loop:  60%|######    | 30081/50000 [19:15:24<10:32:54,  1.91s/it]
Outer training loop:  60%|######    | 30082/50000 [19:15:26<10:41:03,  1.93s/it]
Outer training loop:  60%|######    | 30083/50000 [19:15:28<10:34:16,  1.91s/it]
Outer training loop:  60%|######    | 30084/50000 [19:15:30<10:47:08,  1.95s/it]
Outer training loop:  60%|######    | 30085/50000 [19:15:32<10:41:08,  1.93s/it]
Outer training loop:  60%|######    | 30086/50000 [19:15:33<10:23:41,  1.88s/it]
Outer training loop:  60%|######    | 30087/50000 [19:15:35<10:30:05,  1.90s/it]
Outer training loop:  60%|######    | 30088/50000 [19:15:37<10:49:32,  1.96s/it]
Outer training loop:  60%|######    | 30089/50000 [19:15:39<10:38:11,  1.92s/it]
Outer training loop:  60%|######    | 30090/50000 [19:15:41<10:48:36,  1.95s/it]
Outer training loop:  60%|######    | 30091/50000 [19:15:43<10:38:09,  1.92s/it]
Outer training loop:  60%|######    | 30092/50000 [19:15:45<10:26:53,  1.89s/it]
Outer training loop:  60%|######    | 30093/50000 [19:15:47<10:17:50,  1.86s/it]
Outer training loop:  60%|######    | 30094/50000 [19:15:49<10:44:35,  1.94s/it]
Outer training loop:  60%|######    | 30095/50000 [19:15:51<10:46:35,  1.95s/it]
Outer training loop:  60%|######    | 30096/50000 [19:15:53<10:26:57,  1.89s/it]
Outer training loop:  60%|######    | 30097/50000 [19:15:55<10:43:03,  1.94s/it]
Outer training loop:  60%|######    | 30098/50000 [19:15:57<10:35:45,  1.92s/it]
Outer training loop:  60%|######    | 30099/50000 [19:15:58<10:31:44,  1.90s/it]
Outer training loop:  60%|######    | 30100/50000 [19:16:00<10:21:01,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  60%|######    | 30101/50000 [19:16:08<19:57:46,  3.61s/it]
Outer training loop:  60%|######    | 30102/50000 [19:16:10<17:08:44,  3.10s/it]
Outer training loop:  60%|######    | 30103/50000 [19:16:12<15:07:08,  2.74s/it]
Outer training loop:  60%|######    | 30104/50000 [19:16:13<13:32:42,  2.45s/it]
Outer training loop:  60%|######    | 30105/50000 [19:16:15<12:51:16,  2.33s/it]
Outer training loop:  60%|######    | 30106/50000 [19:16:17<11:59:18,  2.17s/it]
Outer training loop:  60%|######    | 30107/50000 [19:16:19<11:17:52,  2.04s/it]
Outer training loop:  60%|######    | 30108/50000 [19:16:21<11:22:14,  2.06s/it]
Outer training loop:  60%|######    | 30109/50000 [19:16:23<11:00:00,  1.99s/it]
Outer training loop:  60%|######    | 30110/50000 [19:16:25<10:50:26,  1.96s/it]
Outer training loop:  60%|######    | 30111/50000 [19:16:27<10:35:21,  1.92s/it]
Outer training loop:  60%|######    | 30112/50000 [19:16:29<10:44:50,  1.95s/it]
Outer training loop:  60%|######    | 30113/50000 [19:16:30<10:32:57,  1.91s/it]
Outer training loop:  60%|######    | 30114/50000 [19:16:32<10:42:22,  1.94s/it]
Outer training loop:  60%|######    | 30115/50000 [19:16:34<10:46:17,  1.95s/it]
Outer training loop:  60%|######    | 30116/50000 [19:16:36<10:40:47,  1.93s/it]
Outer training loop:  60%|######    | 30117/50000 [19:16:38<10:23:11,  1.88s/it]
Outer training loop:  60%|######    | 30118/50000 [19:16:40<10:58:52,  1.99s/it]
Outer training loop:  60%|######    | 30119/50000 [19:16:42<10:51:02,  1.96s/it]
Outer training loop:  60%|######    | 30120/50000 [19:16:44<10:41:21,  1.94s/it]
Outer training loop:  60%|######    | 30121/50000 [19:16:46<10:25:42,  1.89s/it]
Outer training loop:  60%|######    | 30122/50000 [19:16:48<10:25:30,  1.89s/it]
Outer training loop:  60%|######    | 30123/50000 [19:16:50<10:29:01,  1.90s/it]
Outer training loop:  60%|######    | 30124/50000 [19:16:51<10:13:41,  1.85s/it]
Outer training loop:  60%|######    | 30125/50000 [19:16:53<10:13:37,  1.85s/it]
Outer training loop:  60%|######    | 30126/50000 [19:16:55<10:17:14,  1.86s/it]
Outer training loop:  60%|######    | 30127/50000 [19:16:57<10:25:53,  1.89s/it]
Outer training loop:  60%|######    | 30128/50000 [19:16:59<10:12:33,  1.85s/it]
Outer training loop:  60%|######    | 30129/50000 [19:17:01<10:36:46,  1.92s/it]
Outer training loop:  60%|######    | 30130/50000 [19:17:03<10:23:19,  1.88s/it]
Outer training loop:  60%|######    | 30131/50000 [19:17:05<10:15:44,  1.86s/it]
Outer training loop:  60%|######    | 30132/50000 [19:17:07<10:41:39,  1.94s/it]
Outer training loop:  60%|######    | 30133/50000 [19:17:09<10:28:08,  1.90s/it]
Outer training loop:  60%|######    | 30134/50000 [19:17:10<10:25:27,  1.89s/it]
Outer training loop:  60%|######    | 30135/50000 [19:17:13<11:09:30,  2.02s/it]
Outer training loop:  60%|######    | 30136/50000 [19:17:15<10:49:03,  1.96s/it]
Outer training loop:  60%|######    | 30137/50000 [19:17:16<10:37:18,  1.93s/it]
Outer training loop:  60%|######    | 30138/50000 [19:17:18<10:17:22,  1.86s/it]
Outer training loop:  60%|######    | 30139/50000 [19:17:20<10:50:57,  1.97s/it]
Outer training loop:  60%|######    | 30140/50000 [19:17:22<10:41:12,  1.94s/it]
Outer training loop:  60%|######    | 30141/50000 [19:17:24<10:29:41,  1.90s/it]
Outer training loop:  60%|######    | 30142/50000 [19:17:26<10:49:54,  1.96s/it]
Outer training loop:  60%|######    | 30143/50000 [19:17:28<10:42:09,  1.94s/it]
Outer training loop:  60%|######    | 30144/50000 [19:17:30<10:51:33,  1.97s/it]
Outer training loop:  60%|######    | 30145/50000 [19:17:32<10:58:57,  1.99s/it]
Outer training loop:  60%|######    | 30146/50000 [19:17:34<10:52:49,  1.97s/it]
Outer training loop:  60%|######    | 30147/50000 [19:17:36<10:32:11,  1.91s/it]
Outer training loop:  60%|######    | 30148/50000 [19:17:38<10:19:42,  1.87s/it]
Outer training loop:  60%|######    | 30149/50000 [19:17:40<10:38:53,  1.93s/it]
Outer training loop:  60%|######    | 30150/50000 [19:17:41<10:28:14,  1.90s/it]
Outer training loop:  60%|######    | 30151/50000 [19:17:43<10:26:53,  1.89s/it]
Outer training loop:  60%|######    | 30152/50000 [19:17:45<10:50:07,  1.97s/it]
Outer training loop:  60%|######    | 30153/50000 [19:17:47<10:40:11,  1.94s/it]
Outer training loop:  60%|######    | 30154/50000 [19:17:49<10:30:15,  1.91s/it]
Outer training loop:  60%|######    | 30155/50000 [19:17:51<10:21:40,  1.88s/it]
Outer training loop:  60%|######    | 30156/50000 [19:17:53<10:22:00,  1.88s/it]
Outer training loop:  60%|######    | 30157/50000 [19:17:55<10:18:29,  1.87s/it]
Outer training loop:  60%|######    | 30158/50000 [19:17:57<10:21:46,  1.88s/it]
Outer training loop:  60%|######    | 30159/50000 [19:17:59<10:35:32,  1.92s/it]
Outer training loop:  60%|######    | 30160/50000 [19:18:01<10:29:30,  1.90s/it]
Outer training loop:  60%|######    | 30161/50000 [19:18:02<10:29:12,  1.90s/it]
Outer training loop:  60%|######    | 30162/50000 [19:18:04<10:25:33,  1.89s/it]
Outer training loop:  60%|######    | 30163/50000 [19:18:06<10:43:05,  1.95s/it]
Outer training loop:  60%|######    | 30164/50000 [19:18:08<10:38:44,  1.93s/it]
Outer training loop:  60%|######    | 30165/50000 [19:18:10<10:28:04,  1.90s/it]
Outer training loop:  60%|######    | 30166/50000 [19:18:12<10:33:30,  1.92s/it]
Outer training loop:  60%|######    | 30167/50000 [19:18:14<10:32:19,  1.91s/it]
Outer training loop:  60%|######    | 30168/50000 [19:18:16<10:16:44,  1.87s/it]
Outer training loop:  60%|######    | 30169/50000 [19:18:17<10:02:37,  1.82s/it]
Outer training loop:  60%|######    | 30170/50000 [19:18:19<10:26:18,  1.90s/it]
Outer training loop:  60%|######    | 30171/50000 [19:18:21<10:23:25,  1.89s/it]
Outer training loop:  60%|######    | 30172/50000 [19:18:23<10:06:25,  1.84s/it]
Outer training loop:  60%|######    | 30173/50000 [19:18:25<10:35:19,  1.92s/it]
Outer training loop:  60%|######    | 30174/50000 [19:18:27<10:25:26,  1.89s/it]
Outer training loop:  60%|######    | 30175/50000 [19:18:29<10:26:31,  1.90s/it]
Outer training loop:  60%|######    | 30176/50000 [19:18:31<10:37:14,  1.93s/it]
Outer training loop:  60%|######    | 30177/50000 [19:18:33<10:49:59,  1.97s/it]
Outer training loop:  60%|######    | 30178/50000 [19:18:35<11:03:41,  2.01s/it]
Outer training loop:  60%|######    | 30179/50000 [19:18:37<10:41:33,  1.94s/it]
Outer training loop:  60%|######    | 30180/50000 [19:18:39<11:03:34,  2.01s/it]
Outer training loop:  60%|######    | 30181/50000 [19:18:41<10:56:01,  1.99s/it]
Outer training loop:  60%|######    | 30182/50000 [19:18:43<10:26:38,  1.90s/it]
Outer training loop:  60%|######    | 30183/50000 [19:18:45<10:31:47,  1.91s/it]
Outer training loop:  60%|######    | 30184/50000 [19:18:46<10:12:11,  1.85s/it]
Outer training loop:  60%|######    | 30185/50000 [19:18:48<10:31:29,  1.91s/it]
Outer training loop:  60%|######    | 30186/50000 [19:18:51<10:56:14,  1.99s/it]
Outer training loop:  60%|######    | 30187/50000 [19:18:52<10:37:02,  1.93s/it]
Outer training loop:  60%|######    | 30188/50000 [19:18:54<10:29:15,  1.91s/it]
Outer training loop:  60%|######    | 30189/50000 [19:18:56<10:34:06,  1.92s/it]
Outer training loop:  60%|######    | 30190/50000 [19:18:58<11:04:48,  2.01s/it]
Outer training loop:  60%|######    | 30191/50000 [19:19:00<10:48:15,  1.96s/it]
Outer training loop:  60%|######    | 30192/50000 [19:19:02<10:25:51,  1.90s/it]
Outer training loop:  60%|######    | 30193/50000 [19:19:04<10:38:25,  1.93s/it]
Outer training loop:  60%|######    | 30194/50000 [19:19:06<10:33:53,  1.92s/it]
Outer training loop:  60%|######    | 30195/50000 [19:19:08<10:24:43,  1.89s/it]
Outer training loop:  60%|######    | 30196/50000 [19:19:10<10:39:57,  1.94s/it]
Outer training loop:  60%|######    | 30197/50000 [19:19:12<10:26:55,  1.90s/it]
Outer training loop:  60%|######    | 30198/50000 [19:19:13<10:29:04,  1.91s/it]
Outer training loop:  60%|######    | 30199/50000 [19:19:15<10:24:15,  1.89s/it]
Outer training loop:  60%|######    | 30200/50000 [19:19:17<10:22:12,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  60%|######    | 30201/50000 [19:19:25<19:24:31,  3.53s/it]
Outer training loop:  60%|######    | 30202/50000 [19:19:26<16:46:56,  3.05s/it]
Outer training loop:  60%|######    | 30203/50000 [19:19:29<15:17:37,  2.78s/it]
Outer training loop:  60%|######    | 30204/50000 [19:19:31<13:47:47,  2.51s/it]
Outer training loop:  60%|######    | 30205/50000 [19:19:32<12:44:58,  2.32s/it]
Outer training loop:  60%|######    | 30206/50000 [19:19:34<11:58:40,  2.18s/it]
Outer training loop:  60%|######    | 30207/50000 [19:19:36<11:40:44,  2.12s/it]
Outer training loop:  60%|######    | 30208/50000 [19:19:38<11:09:19,  2.03s/it]
Outer training loop:  60%|######    | 30209/50000 [19:19:40<10:59:01,  2.00s/it]
Outer training loop:  60%|######    | 30210/50000 [19:19:42<11:00:06,  2.00s/it]
Outer training loop:  60%|######    | 30211/50000 [19:19:44<10:34:16,  1.92s/it]
Outer training loop:  60%|######    | 30212/50000 [19:19:46<10:21:19,  1.88s/it]
Outer training loop:  60%|######    | 30213/50000 [19:19:48<10:35:13,  1.93s/it]
Outer training loop:  60%|######    | 30214/50000 [19:19:49<10:30:00,  1.91s/it]
Outer training loop:  60%|######    | 30215/50000 [19:19:51<10:16:15,  1.87s/it]
Outer training loop:  60%|######    | 30216/50000 [19:19:53<10:08:29,  1.85s/it]
Outer training loop:  60%|######    | 30217/50000 [19:19:55<10:36:00,  1.93s/it]
Outer training loop:  60%|######    | 30218/50000 [19:19:57<10:19:26,  1.88s/it]
Outer training loop:  60%|######    | 30219/50000 [19:19:59<10:20:38,  1.88s/it]
Outer training loop:  60%|######    | 30220/50000 [19:20:01<10:30:27,  1.91s/it]
Outer training loop:  60%|######    | 30221/50000 [19:20:03<10:19:26,  1.88s/it]
Outer training loop:  60%|######    | 30222/50000 [19:20:04<10:07:30,  1.84s/it]
Outer training loop:  60%|######    | 30223/50000 [19:20:06<10:08:01,  1.84s/it]
Outer training loop:  60%|######    | 30224/50000 [19:20:08<10:15:13,  1.87s/it]
Outer training loop:  60%|######    | 30225/50000 [19:20:10<10:25:38,  1.90s/it]
Outer training loop:  60%|######    | 30226/50000 [19:20:12<10:28:53,  1.91s/it]
Outer training loop:  60%|######    | 30227/50000 [19:20:14<10:41:03,  1.95s/it]
Outer training loop:  60%|######    | 30228/50000 [19:20:16<10:27:27,  1.90s/it]
Outer training loop:  60%|######    | 30229/50000 [19:20:18<10:27:00,  1.90s/it]
Outer training loop:  60%|######    | 30230/50000 [19:20:20<10:56:30,  1.99s/it]
Outer training loop:  60%|######    | 30231/50000 [19:20:22<10:41:09,  1.95s/it]
Outer training loop:  60%|######    | 30232/50000 [19:20:24<10:56:03,  1.99s/it]
Outer training loop:  60%|######    | 30233/50000 [19:20:26<10:52:42,  1.98s/it]
Outer training loop:  60%|######    | 30234/50000 [19:20:28<10:51:49,  1.98s/it]
Outer training loop:  60%|######    | 30235/50000 [19:20:30<10:37:05,  1.93s/it]
Outer training loop:  60%|######    | 30236/50000 [19:20:31<10:32:27,  1.92s/it]
Outer training loop:  60%|######    | 30237/50000 [19:20:34<10:50:39,  1.98s/it]
Outer training loop:  60%|######    | 30238/50000 [19:20:36<10:45:27,  1.96s/it]
Outer training loop:  60%|######    | 30239/50000 [19:20:37<10:29:57,  1.91s/it]
Outer training loop:  60%|######    | 30240/50000 [19:20:39<10:33:25,  1.92s/it]
Outer training loop:  60%|######    | 30241/50000 [19:20:41<10:27:12,  1.90s/it]
Outer training loop:  60%|######    | 30242/50000 [19:20:43<10:23:35,  1.89s/it]
Outer training loop:  60%|######    | 30243/50000 [19:20:45<10:30:19,  1.91s/it]
Outer training loop:  60%|######    | 30244/50000 [19:20:47<10:21:22,  1.89s/it]
Outer training loop:  60%|######    | 30245/50000 [19:20:49<10:26:16,  1.90s/it]
Outer training loop:  60%|######    | 30246/50000 [19:20:51<10:34:52,  1.93s/it]
Outer training loop:  60%|######    | 30247/50000 [19:20:53<10:47:35,  1.97s/it]
Outer training loop:  60%|######    | 30248/50000 [19:20:55<10:34:56,  1.93s/it]
Outer training loop:  60%|######    | 30249/50000 [19:20:56<10:17:22,  1.88s/it]
Outer training loop:  60%|######    | 30250/50000 [19:20:58<10:29:22,  1.91s/it]
Outer training loop:  61%|######    | 30251/50000 [19:21:00<10:19:03,  1.88s/it]
Outer training loop:  61%|######    | 30252/50000 [19:21:02<10:21:07,  1.89s/it]
Outer training loop:  61%|######    | 30253/50000 [19:21:04<10:45:44,  1.96s/it]
Outer training loop:  61%|######    | 30254/50000 [19:21:06<10:30:58,  1.92s/it]
Outer training loop:  61%|######    | 30255/50000 [19:21:08<10:35:45,  1.93s/it]
Outer training loop:  61%|######    | 30256/50000 [19:21:10<10:26:28,  1.90s/it]
Outer training loop:  61%|######    | 30257/50000 [19:21:12<10:29:06,  1.91s/it]
Outer training loop:  61%|######    | 30258/50000 [19:21:14<10:29:50,  1.91s/it]
Outer training loop:  61%|######    | 30259/50000 [19:21:16<10:29:33,  1.91s/it]
Outer training loop:  61%|######    | 30260/50000 [19:21:18<10:36:19,  1.93s/it]
Outer training loop:  61%|######    | 30261/50000 [19:21:19<10:21:48,  1.89s/it]
Outer training loop:  61%|######    | 30262/50000 [19:21:21<10:17:11,  1.88s/it]
Outer training loop:  61%|######    | 30263/50000 [19:21:23<10:36:32,  1.94s/it]
Outer training loop:  61%|######    | 30264/50000 [19:21:25<10:33:15,  1.93s/it]
Outer training loop:  61%|######    | 30265/50000 [19:21:27<10:18:12,  1.88s/it]
Outer training loop:  61%|######    | 30266/50000 [19:21:29<10:24:20,  1.90s/it]
Outer training loop:  61%|######    | 30267/50000 [19:21:31<10:17:51,  1.88s/it]
Outer training loop:  61%|######    | 30268/50000 [19:21:33<10:23:15,  1.90s/it]
Outer training loop:  61%|######    | 30269/50000 [19:21:35<10:39:09,  1.94s/it]
Outer training loop:  61%|######    | 30270/50000 [19:21:36<10:24:19,  1.90s/it]
Outer training loop:  61%|######    | 30271/50000 [19:21:38<10:08:44,  1.85s/it]
Outer training loop:  61%|######    | 30272/50000 [19:21:40<10:12:58,  1.86s/it]
Outer training loop:  61%|######    | 30273/50000 [19:21:42<10:25:23,  1.90s/it]
Outer training loop:  61%|######    | 30274/50000 [19:21:44<10:24:05,  1.90s/it]
Outer training loop:  61%|######    | 30275/50000 [19:21:46<10:14:12,  1.87s/it]
Outer training loop:  61%|######    | 30276/50000 [19:21:48<10:32:13,  1.92s/it]
Outer training loop:  61%|######    | 30277/50000 [19:21:50<10:30:30,  1.92s/it]
Outer training loop:  61%|######    | 30278/50000 [19:21:52<10:13:32,  1.87s/it]
Outer training loop:  61%|######    | 30279/50000 [19:21:53<10:22:53,  1.90s/it]
Outer training loop:  61%|######    | 30280/50000 [19:21:55<10:17:48,  1.88s/it]
Outer training loop:  61%|######    | 30281/50000 [19:21:57<10:26:51,  1.91s/it]
Outer training loop:  61%|######    | 30282/50000 [19:21:59<10:48:19,  1.97s/it]
Outer training loop:  61%|######    | 30283/50000 [19:22:01<10:16:33,  1.88s/it]
Outer training loop:  61%|######    | 30284/50000 [19:22:03<10:08:53,  1.85s/it]
Outer training loop:  61%|######    | 30285/50000 [19:22:05<9:57:11,  1.82s/it] 
Outer training loop:  61%|######    | 30286/50000 [19:22:07<10:21:10,  1.89s/it]
Outer training loop:  61%|######    | 30287/50000 [19:22:08<10:01:01,  1.83s/it]
Outer training loop:  61%|######    | 30288/50000 [19:22:10<10:05:53,  1.84s/it]
Outer training loop:  61%|######    | 30289/50000 [19:22:12<10:29:34,  1.92s/it]
Outer training loop:  61%|######    | 30290/50000 [19:22:14<10:13:32,  1.87s/it]
Outer training loop:  61%|######    | 30291/50000 [19:22:16<10:02:59,  1.84s/it]
Outer training loop:  61%|######    | 30292/50000 [19:22:18<10:24:40,  1.90s/it]
Outer training loop:  61%|######    | 30293/50000 [19:22:20<10:36:04,  1.94s/it]
Outer training loop:  61%|######    | 30294/50000 [19:22:22<10:21:01,  1.89s/it]
Outer training loop:  61%|######    | 30295/50000 [19:22:23<10:11:43,  1.86s/it]
Outer training loop:  61%|######    | 30296/50000 [19:22:25<10:25:43,  1.91s/it]
Outer training loop:  61%|######    | 30297/50000 [19:22:27<10:15:48,  1.88s/it]
Outer training loop:  61%|######    | 30298/50000 [19:22:29<9:58:31,  1.82s/it] 
Outer training loop:  61%|######    | 30299/50000 [19:22:31<10:27:43,  1.91s/it]
Outer training loop:  61%|######    | 30300/50000 [19:22:33<10:07:47,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  61%|######    | 30301/50000 [19:22:40<18:52:17,  3.45s/it]
Outer training loop:  61%|######    | 30302/50000 [19:22:42<16:20:53,  2.99s/it]
Outer training loop:  61%|######    | 30303/50000 [19:22:44<14:13:40,  2.60s/it]
Outer training loop:  61%|######    | 30304/50000 [19:22:45<12:50:00,  2.35s/it]
Outer training loop:  61%|######    | 30305/50000 [19:22:47<11:47:50,  2.16s/it]
Outer training loop:  61%|######    | 30306/50000 [19:22:49<11:28:02,  2.10s/it]
Outer training loop:  61%|######    | 30307/50000 [19:22:51<10:55:10,  2.00s/it]
Outer training loop:  61%|######    | 30308/50000 [19:22:53<10:33:47,  1.93s/it]
Outer training loop:  61%|######    | 30309/50000 [19:22:54<10:30:16,  1.92s/it]
Outer training loop:  61%|######    | 30310/50000 [19:22:56<10:05:35,  1.85s/it]
Outer training loop:  61%|######    | 30311/50000 [19:22:58<9:57:29,  1.82s/it] 
Outer training loop:  61%|######    | 30312/50000 [19:23:00<9:52:22,  1.81s/it]
Outer training loop:  61%|######    | 30313/50000 [19:23:02<10:00:50,  1.83s/it]
Outer training loop:  61%|######    | 30314/50000 [19:23:03<9:55:30,  1.82s/it] 
Outer training loop:  61%|######    | 30315/50000 [19:23:05<9:45:56,  1.79s/it]
Outer training loop:  61%|######    | 30316/50000 [19:23:07<9:46:42,  1.79s/it]
Outer training loop:  61%|######    | 30317/50000 [19:23:09<10:05:24,  1.85s/it]
Outer training loop:  61%|######    | 30318/50000 [19:23:11<10:05:41,  1.85s/it]
Outer training loop:  61%|######    | 30319/50000 [19:23:12<10:01:07,  1.83s/it]
Outer training loop:  61%|######    | 30320/50000 [19:23:14<10:07:03,  1.85s/it]
Outer training loop:  61%|######    | 30321/50000 [19:23:16<10:04:59,  1.84s/it]
Outer training loop:  61%|######    | 30322/50000 [19:23:18<9:49:06,  1.80s/it] 
Outer training loop:  61%|######    | 30323/50000 [19:23:20<10:11:17,  1.86s/it]
Outer training loop:  61%|######    | 30324/50000 [19:23:22<9:57:41,  1.82s/it] 
Outer training loop:  61%|######    | 30325/50000 [19:23:23<10:00:12,  1.83s/it]
Outer training loop:  61%|######    | 30326/50000 [19:23:25<9:55:30,  1.82s/it] 
Outer training loop:  61%|######    | 30327/50000 [19:23:27<9:59:22,  1.83s/it]
Outer training loop:  61%|######    | 30328/50000 [19:23:29<9:49:50,  1.80s/it]
Outer training loop:  61%|######    | 30329/50000 [19:23:31<10:10:13,  1.86s/it]
Outer training loop:  61%|######    | 30330/50000 [19:23:33<10:16:17,  1.88s/it]
Outer training loop:  61%|######    | 30331/50000 [19:23:35<10:04:59,  1.85s/it]
Outer training loop:  61%|######    | 30332/50000 [19:23:36<9:56:23,  1.82s/it] 
Outer training loop:  61%|######    | 30333/50000 [19:23:38<9:49:53,  1.80s/it]
Outer training loop:  61%|######    | 30334/50000 [19:23:40<10:13:20,  1.87s/it]
Outer training loop:  61%|######    | 30335/50000 [19:23:42<10:00:13,  1.83s/it]
Outer training loop:  61%|######    | 30336/50000 [19:23:44<10:04:21,  1.84s/it]
Outer training loop:  61%|######    | 30337/50000 [19:23:46<10:17:13,  1.88s/it]
Outer training loop:  61%|######    | 30338/50000 [19:23:47<10:05:39,  1.85s/it]
Outer training loop:  61%|######    | 30339/50000 [19:23:49<10:04:45,  1.85s/it]
Outer training loop:  61%|######    | 30340/50000 [19:23:51<10:10:35,  1.86s/it]
Outer training loop:  61%|######    | 30341/50000 [19:23:53<10:12:02,  1.87s/it]
Outer training loop:  61%|######    | 30342/50000 [19:23:55<10:05:52,  1.85s/it]
Outer training loop:  61%|######    | 30343/50000 [19:23:57<9:54:38,  1.82s/it] 
Outer training loop:  61%|######    | 30344/50000 [19:23:59<10:04:34,  1.85s/it]
Outer training loop:  61%|######    | 30345/50000 [19:24:00<10:00:02,  1.83s/it]
Outer training loop:  61%|######    | 30346/50000 [19:24:02<9:53:09,  1.81s/it] 
Outer training loop:  61%|######    | 30347/50000 [19:24:04<10:11:38,  1.87s/it]
Outer training loop:  61%|######    | 30348/50000 [19:24:06<10:04:45,  1.85s/it]
Outer training loop:  61%|######    | 30349/50000 [19:24:08<10:07:01,  1.85s/it]
Outer training loop:  61%|######    | 30350/50000 [19:24:10<10:01:58,  1.84s/it]
Outer training loop:  61%|######    | 30351/50000 [19:24:12<10:17:06,  1.88s/it]
Outer training loop:  61%|######    | 30352/50000 [19:24:13<10:16:22,  1.88s/it]
Outer training loop:  61%|######    | 30353/50000 [19:24:15<10:03:05,  1.84s/it]
Outer training loop:  61%|######    | 30354/50000 [19:24:17<10:19:15,  1.89s/it]
Outer training loop:  61%|######    | 30355/50000 [19:24:19<10:17:32,  1.89s/it]
Outer training loop:  61%|######    | 30356/50000 [19:24:21<10:02:55,  1.84s/it]
Outer training loop:  61%|######    | 30357/50000 [19:24:23<10:14:03,  1.88s/it]
Outer training loop:  61%|######    | 30358/50000 [19:24:25<10:11:31,  1.87s/it]
Outer training loop:  61%|######    | 30359/50000 [19:24:26<10:01:55,  1.84s/it]
Outer training loop:  61%|######    | 30360/50000 [19:24:28<9:56:14,  1.82s/it] 
Outer training loop:  61%|######    | 30361/50000 [19:24:30<10:15:20,  1.88s/it]
Outer training loop:  61%|######    | 30362/50000 [19:24:32<9:58:54,  1.83s/it] 
Outer training loop:  61%|######    | 30363/50000 [19:24:34<9:59:44,  1.83s/it]
Outer training loop:  61%|######    | 30364/50000 [19:24:36<10:03:14,  1.84s/it]
Outer training loop:  61%|######    | 30365/50000 [19:24:38<10:11:23,  1.87s/it]
Outer training loop:  61%|######    | 30366/50000 [19:24:39<9:54:22,  1.82s/it] 
Outer training loop:  61%|######    | 30367/50000 [19:24:41<9:56:01,  1.82s/it]
Outer training loop:  61%|######    | 30368/50000 [19:24:43<10:17:17,  1.89s/it]
Outer training loop:  61%|######    | 30369/50000 [19:24:45<10:17:00,  1.89s/it]
Outer training loop:  61%|######    | 30370/50000 [19:24:47<10:01:37,  1.84s/it]
Outer training loop:  61%|######    | 30371/50000 [19:24:48<9:58:25,  1.83s/it] 
Outer training loop:  61%|######    | 30372/50000 [19:24:51<10:16:36,  1.88s/it]
Outer training loop:  61%|######    | 30373/50000 [19:24:52<10:17:55,  1.89s/it]
Outer training loop:  61%|######    | 30374/50000 [19:24:54<10:07:24,  1.86s/it]
Outer training loop:  61%|######    | 30375/50000 [19:24:56<10:16:27,  1.88s/it]
Outer training loop:  61%|######    | 30376/50000 [19:24:58<10:13:29,  1.88s/it]
Outer training loop:  61%|######    | 30377/50000 [19:25:00<10:13:49,  1.88s/it]
Outer training loop:  61%|######    | 30378/50000 [19:25:02<10:02:25,  1.84s/it]
Outer training loop:  61%|######    | 30379/50000 [19:25:04<10:51:32,  1.99s/it]
Outer training loop:  61%|######    | 30380/50000 [19:25:06<10:39:18,  1.96s/it]
Outer training loop:  61%|######    | 30381/50000 [19:25:08<10:22:02,  1.90s/it]
Outer training loop:  61%|######    | 30382/50000 [19:25:10<10:27:18,  1.92s/it]
Outer training loop:  61%|######    | 30383/50000 [19:25:11<10:21:49,  1.90s/it]
Outer training loop:  61%|######    | 30384/50000 [19:25:13<10:07:24,  1.86s/it]
Outer training loop:  61%|######    | 30385/50000 [19:25:15<9:59:58,  1.84s/it] 
Outer training loop:  61%|######    | 30386/50000 [19:25:17<10:02:50,  1.84s/it]
Outer training loop:  61%|######    | 30387/50000 [19:25:19<9:59:10,  1.83s/it] 
Outer training loop:  61%|######    | 30388/50000 [19:25:20<9:51:30,  1.81s/it]
Outer training loop:  61%|######    | 30389/50000 [19:25:22<10:06:52,  1.86s/it]
Outer training loop:  61%|######    | 30390/50000 [19:25:24<9:54:34,  1.82s/it] 
Outer training loop:  61%|######    | 30391/50000 [19:25:26<10:01:40,  1.84s/it]
Outer training loop:  61%|######    | 30392/50000 [19:25:28<10:14:33,  1.88s/it]
Outer training loop:  61%|######    | 30393/50000 [19:25:30<10:02:44,  1.84s/it]
Outer training loop:  61%|######    | 30394/50000 [19:25:32<9:58:39,  1.83s/it] 
Outer training loop:  61%|######    | 30395/50000 [19:25:34<10:22:40,  1.91s/it]
Outer training loop:  61%|######    | 30396/50000 [19:25:35<10:18:27,  1.89s/it]
Outer training loop:  61%|######    | 30397/50000 [19:25:37<10:18:03,  1.89s/it]
Outer training loop:  61%|######    | 30398/50000 [19:25:39<10:14:48,  1.88s/it]
Outer training loop:  61%|######    | 30399/50000 [19:25:41<10:25:25,  1.91s/it]
Outer training loop:  61%|######    | 30400/50000 [19:25:43<10:22:51,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.90s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  61%|######    | 30401/50000 [19:25:50<19:14:00,  3.53s/it]
Outer training loop:  61%|######    | 30402/50000 [19:25:52<16:28:16,  3.03s/it]
Outer training loop:  61%|######    | 30403/50000 [19:25:54<14:49:32,  2.72s/it]
Outer training loop:  61%|######    | 30404/50000 [19:25:56<13:18:58,  2.45s/it]
Outer training loop:  61%|######    | 30405/50000 [19:25:58<12:18:41,  2.26s/it]
Outer training loop:  61%|######    | 30406/50000 [19:26:00<11:46:29,  2.16s/it]
Outer training loop:  61%|######    | 30407/50000 [19:26:02<11:10:45,  2.05s/it]
Outer training loop:  61%|######    | 30408/50000 [19:26:03<10:45:37,  1.98s/it]
Outer training loop:  61%|######    | 30409/50000 [19:26:05<10:39:25,  1.96s/it]
Outer training loop:  61%|######    | 30410/50000 [19:26:07<10:52:30,  2.00s/it]
Outer training loop:  61%|######    | 30411/50000 [19:26:09<10:43:11,  1.97s/it]
Outer training loop:  61%|######    | 30412/50000 [19:26:11<10:36:12,  1.95s/it]
Outer training loop:  61%|######    | 30413/50000 [19:26:13<10:45:01,  1.98s/it]
Outer training loop:  61%|######    | 30414/50000 [19:26:15<10:25:38,  1.92s/it]
Outer training loop:  61%|######    | 30415/50000 [19:26:17<10:15:10,  1.88s/it]
Outer training loop:  61%|######    | 30416/50000 [19:26:19<10:33:57,  1.94s/it]
Outer training loop:  61%|######    | 30417/50000 [19:26:21<10:21:49,  1.91s/it]
Outer training loop:  61%|######    | 30418/50000 [19:26:23<10:10:22,  1.87s/it]
Outer training loop:  61%|######    | 30419/50000 [19:26:25<10:17:53,  1.89s/it]
Outer training loop:  61%|######    | 30420/50000 [19:26:26<10:00:13,  1.84s/it]
Outer training loop:  61%|######    | 30421/50000 [19:26:28<10:05:02,  1.85s/it]
Outer training loop:  61%|######    | 30422/50000 [19:26:30<10:00:44,  1.84s/it]
Outer training loop:  61%|######    | 30423/50000 [19:26:32<10:26:10,  1.92s/it]
Outer training loop:  61%|######    | 30424/50000 [19:26:34<10:20:26,  1.90s/it]
Outer training loop:  61%|######    | 30425/50000 [19:26:36<10:14:37,  1.88s/it]
Outer training loop:  61%|######    | 30426/50000 [19:26:38<10:52:44,  2.00s/it]
Outer training loop:  61%|######    | 30427/50000 [19:26:40<10:39:42,  1.96s/it]
Outer training loop:  61%|######    | 30428/50000 [19:26:42<10:19:26,  1.90s/it]
Outer training loop:  61%|######    | 30429/50000 [19:26:44<10:22:55,  1.91s/it]
Outer training loop:  61%|######    | 30430/50000 [19:26:46<10:34:53,  1.95s/it]
Outer training loop:  61%|######    | 30431/50000 [19:26:47<10:17:54,  1.89s/it]
Outer training loop:  61%|######    | 30432/50000 [19:26:49<10:05:19,  1.86s/it]
Outer training loop:  61%|######    | 30433/50000 [19:26:51<10:23:44,  1.91s/it]
Outer training loop:  61%|######    | 30434/50000 [19:26:53<10:12:20,  1.88s/it]
Outer training loop:  61%|######    | 30435/50000 [19:26:55<10:22:18,  1.91s/it]
Outer training loop:  61%|######    | 30436/50000 [19:26:57<10:33:00,  1.94s/it]
Outer training loop:  61%|######    | 30437/50000 [19:26:59<10:21:13,  1.91s/it]
Outer training loop:  61%|######    | 30438/50000 [19:27:01<10:15:31,  1.89s/it]
Outer training loop:  61%|######    | 30439/50000 [19:27:03<10:13:53,  1.88s/it]
Outer training loop:  61%|######    | 30440/50000 [19:27:05<10:32:05,  1.94s/it]
Outer training loop:  61%|######    | 30441/50000 [19:27:06<10:15:07,  1.89s/it]
Outer training loop:  61%|######    | 30442/50000 [19:27:08<10:00:31,  1.84s/it]
Outer training loop:  61%|######    | 30443/50000 [19:27:10<10:31:36,  1.94s/it]
Outer training loop:  61%|######    | 30444/50000 [19:27:12<10:17:32,  1.89s/it]
Outer training loop:  61%|######    | 30445/50000 [19:27:14<10:16:30,  1.89s/it]
Outer training loop:  61%|######    | 30446/50000 [19:27:16<10:36:53,  1.95s/it]
Outer training loop:  61%|######    | 30447/50000 [19:27:18<10:18:29,  1.90s/it]
Outer training loop:  61%|######    | 30448/50000 [19:27:20<10:11:41,  1.88s/it]
Outer training loop:  61%|######    | 30449/50000 [19:27:21<10:04:59,  1.86s/it]
Outer training loop:  61%|######    | 30450/50000 [19:27:23<10:11:54,  1.88s/it]
Outer training loop:  61%|######    | 30451/50000 [19:27:25<10:09:19,  1.87s/it]
Outer training loop:  61%|######    | 30452/50000 [19:27:27<10:07:34,  1.86s/it]
Outer training loop:  61%|######    | 30453/50000 [19:27:29<10:42:19,  1.97s/it]
Outer training loop:  61%|######    | 30454/50000 [19:27:31<10:23:52,  1.92s/it]
Outer training loop:  61%|######    | 30455/50000 [19:27:33<10:09:16,  1.87s/it]
Outer training loop:  61%|######    | 30456/50000 [19:27:35<10:29:15,  1.93s/it]
Outer training loop:  61%|######    | 30457/50000 [19:27:37<10:21:55,  1.91s/it]
Outer training loop:  61%|######    | 30458/50000 [19:27:39<10:13:21,  1.88s/it]
Outer training loop:  61%|######    | 30459/50000 [19:27:40<10:15:24,  1.89s/it]
Outer training loop:  61%|######    | 30460/50000 [19:27:42<10:14:16,  1.89s/it]
Outer training loop:  61%|######    | 30461/50000 [19:27:44<10:05:54,  1.86s/it]
Outer training loop:  61%|######    | 30462/50000 [19:27:46<10:05:02,  1.86s/it]
Outer training loop:  61%|######    | 30463/50000 [19:27:48<10:06:28,  1.86s/it]
Outer training loop:  61%|######    | 30464/50000 [19:27:50<10:15:01,  1.89s/it]
Outer training loop:  61%|######    | 30465/50000 [19:27:52<10:06:38,  1.86s/it]
Outer training loop:  61%|######    | 30466/50000 [19:27:54<10:14:40,  1.89s/it]
Outer training loop:  61%|######    | 30467/50000 [19:27:55<9:56:11,  1.83s/it] 
Outer training loop:  61%|######    | 30468/50000 [19:27:57<9:48:40,  1.81s/it]
Outer training loop:  61%|######    | 30469/50000 [19:27:59<10:08:14,  1.87s/it]
Outer training loop:  61%|######    | 30470/50000 [19:28:01<9:50:44,  1.81s/it] 
Outer training loop:  61%|######    | 30471/50000 [19:28:03<9:50:59,  1.82s/it]
Outer training loop:  61%|######    | 30472/50000 [19:28:04<9:51:55,  1.82s/it]
Outer training loop:  61%|######    | 30473/50000 [19:28:07<10:21:08,  1.91s/it]
Outer training loop:  61%|######    | 30474/50000 [19:28:08<10:04:46,  1.86s/it]
Outer training loop:  61%|######    | 30475/50000 [19:28:10<10:05:45,  1.86s/it]
Outer training loop:  61%|######    | 30476/50000 [19:28:12<10:45:37,  1.98s/it]
Outer training loop:  61%|######    | 30477/50000 [19:28:14<10:21:04,  1.91s/it]
Outer training loop:  61%|######    | 30478/50000 [19:28:16<10:03:19,  1.85s/it]
Outer training loop:  61%|######    | 30479/50000 [19:28:18<9:56:17,  1.83s/it] 
Outer training loop:  61%|######    | 30480/50000 [19:28:20<10:16:46,  1.90s/it]
Outer training loop:  61%|######    | 30481/50000 [19:28:21<10:05:46,  1.86s/it]
Outer training loop:  61%|######    | 30482/50000 [19:28:23<10:04:31,  1.86s/it]
Outer training loop:  61%|######    | 30483/50000 [19:28:25<10:03:31,  1.86s/it]
Outer training loop:  61%|######    | 30484/50000 [19:28:27<10:19:05,  1.90s/it]
Outer training loop:  61%|######    | 30485/50000 [19:28:29<10:01:24,  1.85s/it]
Outer training loop:  61%|######    | 30486/50000 [19:28:31<10:06:39,  1.87s/it]
Outer training loop:  61%|######    | 30487/50000 [19:28:33<10:09:21,  1.87s/it]
Outer training loop:  61%|######    | 30488/50000 [19:28:35<10:20:22,  1.91s/it]
Outer training loop:  61%|######    | 30489/50000 [19:28:37<10:19:05,  1.90s/it]
Outer training loop:  61%|######    | 30490/50000 [19:28:38<10:06:39,  1.87s/it]
Outer training loop:  61%|######    | 30491/50000 [19:28:40<10:23:46,  1.92s/it]
Outer training loop:  61%|######    | 30492/50000 [19:28:42<10:23:26,  1.92s/it]
Outer training loop:  61%|######    | 30493/50000 [19:28:44<10:07:37,  1.87s/it]
Outer training loop:  61%|######    | 30494/50000 [19:28:46<10:18:13,  1.90s/it]
Outer training loop:  61%|######    | 30495/50000 [19:28:48<9:57:06,  1.84s/it] 
Outer training loop:  61%|######    | 30496/50000 [19:28:50<10:01:08,  1.85s/it]
Outer training loop:  61%|######    | 30497/50000 [19:28:51<9:49:22,  1.81s/it] 
Outer training loop:  61%|######    | 30498/50000 [19:28:53<10:18:58,  1.90s/it]
Outer training loop:  61%|######    | 30499/50000 [19:28:55<10:19:25,  1.91s/it]
Outer training loop:  61%|######1   | 30500/50000 [19:28:57<10:12:17,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.77s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:24,  1.79s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:24,  1.81s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.86s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.86s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:22,  1.85s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:19,  1.84s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:22,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:21,  1.97s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:00,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:04,  1.97s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:58,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:58,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.90s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.93s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:43,  1.95s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:40,  1.94s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:29,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.93s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:20,  1.83s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.82s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:20,  1.96s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:16,  1.91s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:09,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:07,  1.82s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:04,  1.80s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:04,  1.90s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:58,  1.84s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:56,  1.83s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:54,  1.81s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.80s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:50,  1.80s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:48,  1.81s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.95s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:42,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.82s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.81s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:35,  1.95s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:32,  1.89s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:29,  1.85s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.80s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:23,  1.79s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:21,  1.77s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:20,  1.83s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:19,  1.96s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:17,  1.92s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.93s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.93s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.86s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.87s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:31<00:03,  1.84s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.95s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:28,  1.81s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.82s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.80s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:24,  1.83s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:20,  1.80s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:19,  1.81s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:17,  1.83s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:14,  1.82s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:26,  2.01s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:21,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.93s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:14,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:15,  1.97s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:00,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:02,  1.92s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:56,  1.94s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:46,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:42,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:45,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.86s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:34,  1.82s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:32,  1.82s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:31,  1.84s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:25,  1.98s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:21,  1.93s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:19,  1.93s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:13,  1.87s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:14,  2.01s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:10,  1.96s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:06,  1.90s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:04,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<01:00,  2.01s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:56,  1.94s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:53,  1.91s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.88s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:43,  1.99s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.93s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:38,  1.90s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.84s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.84s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.99s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.93s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.85s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.83s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.84s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.84s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.81s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:08,  2.01s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.96s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.90s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.89s/it]

Outer training loop:  61%|######1   | 30501/50000 [19:34:18<527:37:56, 97.41s/it]
Outer training loop:  61%|######1   | 30502/50000 [19:34:19<372:09:29, 68.71s/it]
Outer training loop:  61%|######1   | 30503/50000 [19:34:21<263:40:12, 48.69s/it]
Outer training loop:  61%|######1   | 30504/50000 [19:34:23<187:27:51, 34.62s/it]
Outer training loop:  61%|######1   | 30505/50000 [19:34:25<134:04:00, 24.76s/it]
Outer training loop:  61%|######1   | 30506/50000 [19:34:27<96:59:38, 17.91s/it] 
Outer training loop:  61%|######1   | 30507/50000 [19:34:29<72:21:02, 13.36s/it]
Outer training loop:  61%|######1   | 30508/50000 [19:34:31<53:42:51,  9.92s/it]
Outer training loop:  61%|######1   | 30509/50000 [19:34:33<40:34:23,  7.49s/it]
Outer training loop:  61%|######1   | 30510/50000 [19:34:35<31:49:29,  5.88s/it]
Outer training loop:  61%|######1   | 30511/50000 [19:34:37<25:13:15,  4.66s/it]
Outer training loop:  61%|######1   | 30512/50000 [19:34:39<20:48:38,  3.84s/it]
Outer training loop:  61%|######1   | 30513/50000 [19:34:41<17:56:55,  3.32s/it]
Outer training loop:  61%|######1   | 30514/50000 [19:34:43<15:24:30,  2.85s/it]
Outer training loop:  61%|######1   | 30515/50000 [19:34:45<13:36:33,  2.51s/it]
Outer training loop:  61%|######1   | 30516/50000 [19:34:46<12:26:24,  2.30s/it]
Outer training loop:  61%|######1   | 30517/50000 [19:34:48<11:57:46,  2.21s/it]
Outer training loop:  61%|######1   | 30518/50000 [19:34:50<11:17:46,  2.09s/it]
Outer training loop:  61%|######1   | 30519/50000 [19:34:52<10:52:37,  2.01s/it]
Outer training loop:  61%|######1   | 30520/50000 [19:34:54<10:55:05,  2.02s/it]
Outer training loop:  61%|######1   | 30521/50000 [19:34:56<10:40:45,  1.97s/it]
Outer training loop:  61%|######1   | 30522/50000 [19:34:58<10:37:55,  1.97s/it]
Outer training loop:  61%|######1   | 30523/50000 [19:35:00<10:39:00,  1.97s/it]
Outer training loop:  61%|######1   | 30524/50000 [19:35:02<10:43:43,  1.98s/it]
Outer training loop:  61%|######1   | 30525/50000 [19:35:04<10:11:36,  1.88s/it]
Outer training loop:  61%|######1   | 30526/50000 [19:35:06<10:21:06,  1.91s/it]
Outer training loop:  61%|######1   | 30527/50000 [19:35:07<10:27:45,  1.93s/it]
Outer training loop:  61%|######1   | 30528/50000 [19:35:09<10:13:37,  1.89s/it]
Outer training loop:  61%|######1   | 30529/50000 [19:35:11<10:05:55,  1.87s/it]
Outer training loop:  61%|######1   | 30530/50000 [19:35:13<10:05:30,  1.87s/it]
Outer training loop:  61%|######1   | 30531/50000 [19:35:15<10:25:27,  1.93s/it]
Outer training loop:  61%|######1   | 30532/50000 [19:35:17<10:11:47,  1.89s/it]
Outer training loop:  61%|######1   | 30533/50000 [19:35:19<10:00:15,  1.85s/it]
Outer training loop:  61%|######1   | 30534/50000 [19:35:21<10:28:41,  1.94s/it]
Outer training loop:  61%|######1   | 30535/50000 [19:35:23<10:19:07,  1.91s/it]
Outer training loop:  61%|######1   | 30536/50000 [19:35:24<10:11:01,  1.88s/it]
Outer training loop:  61%|######1   | 30537/50000 [19:35:27<10:38:46,  1.97s/it]
Outer training loop:  61%|######1   | 30538/50000 [19:35:28<10:19:10,  1.91s/it]
Outer training loop:  61%|######1   | 30539/50000 [19:35:30<10:12:40,  1.89s/it]
Outer training loop:  61%|######1   | 30540/50000 [19:35:32<10:05:58,  1.87s/it]
Outer training loop:  61%|######1   | 30541/50000 [19:35:34<10:30:14,  1.94s/it]
Outer training loop:  61%|######1   | 30542/50000 [19:35:36<10:21:09,  1.92s/it]
Outer training loop:  61%|######1   | 30543/50000 [19:35:38<10:24:21,  1.93s/it]
Outer training loop:  61%|######1   | 30544/50000 [19:35:40<10:21:36,  1.92s/it]
Outer training loop:  61%|######1   | 30545/50000 [19:35:42<10:15:02,  1.90s/it]
Outer training loop:  61%|######1   | 30546/50000 [19:35:44<10:14:18,  1.89s/it]
Outer training loop:  61%|######1   | 30547/50000 [19:35:45<10:08:40,  1.88s/it]
Outer training loop:  61%|######1   | 30548/50000 [19:35:47<10:30:15,  1.94s/it]
Outer training loop:  61%|######1   | 30549/50000 [19:35:49<10:17:42,  1.91s/it]
Outer training loop:  61%|######1   | 30550/50000 [19:35:51<9:57:47,  1.84s/it] 
Outer training loop:  61%|######1   | 30551/50000 [19:35:53<10:07:07,  1.87s/it]
Outer training loop:  61%|######1   | 30552/50000 [19:35:55<9:57:17,  1.84s/it] 
Outer training loop:  61%|######1   | 30553/50000 [19:35:57<9:57:55,  1.84s/it]
Outer training loop:  61%|######1   | 30554/50000 [19:35:58<9:50:51,  1.82s/it]
Outer training loop:  61%|######1   | 30555/50000 [19:36:00<10:02:33,  1.86s/it]
Outer training loop:  61%|######1   | 30556/50000 [19:36:02<9:58:12,  1.85s/it] 
Outer training loop:  61%|######1   | 30557/50000 [19:36:04<10:04:11,  1.86s/it]
Outer training loop:  61%|######1   | 30558/50000 [19:36:06<10:27:29,  1.94s/it]
Outer training loop:  61%|######1   | 30559/50000 [19:36:08<10:13:24,  1.89s/it]
Outer training loop:  61%|######1   | 30560/50000 [19:36:10<10:11:35,  1.89s/it]
Outer training loop:  61%|######1   | 30561/50000 [19:36:12<10:15:20,  1.90s/it]
Outer training loop:  61%|######1   | 30562/50000 [19:36:14<10:10:03,  1.88s/it]
Outer training loop:  61%|######1   | 30563/50000 [19:36:15<10:11:12,  1.89s/it]
Outer training loop:  61%|######1   | 30564/50000 [19:36:17<9:54:19,  1.83s/it] 
Outer training loop:  61%|######1   | 30565/50000 [19:36:19<10:13:59,  1.90s/it]
Outer training loop:  61%|######1   | 30566/50000 [19:36:21<10:14:04,  1.90s/it]
Outer training loop:  61%|######1   | 30567/50000 [19:36:23<10:14:30,  1.90s/it]
Outer training loop:  61%|######1   | 30568/50000 [19:36:25<10:32:50,  1.95s/it]
Outer training loop:  61%|######1   | 30569/50000 [19:36:27<10:31:02,  1.95s/it]
Outer training loop:  61%|######1   | 30570/50000 [19:36:29<10:16:10,  1.90s/it]
Outer training loop:  61%|######1   | 30571/50000 [19:36:31<9:59:45,  1.85s/it] 
Outer training loop:  61%|######1   | 30572/50000 [19:36:33<10:16:26,  1.90s/it]
Outer training loop:  61%|######1   | 30573/50000 [19:36:34<10:13:01,  1.89s/it]
Outer training loop:  61%|######1   | 30574/50000 [19:36:36<10:07:59,  1.88s/it]
Outer training loop:  61%|######1   | 30575/50000 [19:36:38<10:18:42,  1.91s/it]
Outer training loop:  61%|######1   | 30576/50000 [19:36:40<10:11:31,  1.89s/it]
Outer training loop:  61%|######1   | 30577/50000 [19:36:42<9:59:46,  1.85s/it] 
Outer training loop:  61%|######1   | 30578/50000 [19:36:44<10:17:50,  1.91s/it]
Outer training loop:  61%|######1   | 30579/50000 [19:36:46<10:14:58,  1.90s/it]
Outer training loop:  61%|######1   | 30580/50000 [19:36:47<9:54:51,  1.84s/it] 
Outer training loop:  61%|######1   | 30581/50000 [19:36:49<9:46:31,  1.81s/it]
Outer training loop:  61%|######1   | 30582/50000 [19:36:51<10:06:31,  1.87s/it]
Outer training loop:  61%|######1   | 30583/50000 [19:36:53<10:12:39,  1.89s/it]
Outer training loop:  61%|######1   | 30584/50000 [19:36:55<10:13:17,  1.90s/it]
Outer training loop:  61%|######1   | 30585/50000 [19:36:57<10:36:18,  1.97s/it]
Outer training loop:  61%|######1   | 30586/50000 [19:36:59<10:33:49,  1.96s/it]
Outer training loop:  61%|######1   | 30587/50000 [19:37:01<10:17:41,  1.91s/it]
Outer training loop:  61%|######1   | 30588/50000 [19:37:03<10:08:35,  1.88s/it]
Outer training loop:  61%|######1   | 30589/50000 [19:37:05<10:11:06,  1.89s/it]
Outer training loop:  61%|######1   | 30590/50000 [19:37:07<10:15:03,  1.90s/it]
Outer training loop:  61%|######1   | 30591/50000 [19:37:08<10:13:27,  1.90s/it]
Outer training loop:  61%|######1   | 30592/50000 [19:37:10<10:18:27,  1.91s/it]
Outer training loop:  61%|######1   | 30593/50000 [19:37:12<10:30:41,  1.95s/it]
Outer training loop:  61%|######1   | 30594/50000 [19:37:14<10:17:44,  1.91s/it]
Outer training loop:  61%|######1   | 30595/50000 [19:37:16<10:08:34,  1.88s/it]
Outer training loop:  61%|######1   | 30596/50000 [19:37:18<10:17:14,  1.91s/it]
Outer training loop:  61%|######1   | 30597/50000 [19:37:20<10:15:53,  1.90s/it]
Outer training loop:  61%|######1   | 30598/50000 [19:37:22<10:13:38,  1.90s/it]
Outer training loop:  61%|######1   | 30599/50000 [19:37:24<10:25:48,  1.94s/it]
Outer training loop:  61%|######1   | 30600/50000 [19:37:26<10:17:09,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  61%|######1   | 30601/50000 [19:37:33<18:58:37,  3.52s/it]
Outer training loop:  61%|######1   | 30602/50000 [19:37:35<16:11:34,  3.01s/it]
Outer training loop:  61%|######1   | 30603/50000 [19:37:37<14:13:29,  2.64s/it]
Outer training loop:  61%|######1   | 30604/50000 [19:37:39<13:15:58,  2.46s/it]
Outer training loop:  61%|######1   | 30605/50000 [19:37:41<12:23:52,  2.30s/it]
Outer training loop:  61%|######1   | 30606/50000 [19:37:43<11:52:06,  2.20s/it]
Outer training loop:  61%|######1   | 30607/50000 [19:37:44<11:09:52,  2.07s/it]
Outer training loop:  61%|######1   | 30608/50000 [19:37:46<10:58:16,  2.04s/it]
Outer training loop:  61%|######1   | 30609/50000 [19:37:48<10:48:23,  2.01s/it]
Outer training loop:  61%|######1   | 30610/50000 [19:37:50<10:49:47,  2.01s/it]
Outer training loop:  61%|######1   | 30611/50000 [19:37:52<10:30:09,  1.95s/it]
Outer training loop:  61%|######1   | 30612/50000 [19:37:54<10:17:05,  1.91s/it]
Outer training loop:  61%|######1   | 30613/50000 [19:37:56<10:32:13,  1.96s/it]
Outer training loop:  61%|######1   | 30614/50000 [19:37:58<10:22:23,  1.93s/it]
Outer training loop:  61%|######1   | 30615/50000 [19:38:00<10:06:23,  1.88s/it]
Outer training loop:  61%|######1   | 30616/50000 [19:38:02<10:21:18,  1.92s/it]
Outer training loop:  61%|######1   | 30617/50000 [19:38:03<10:08:49,  1.88s/it]
Outer training loop:  61%|######1   | 30618/50000 [19:38:05<10:07:53,  1.88s/it]
Outer training loop:  61%|######1   | 30619/50000 [19:38:07<10:01:15,  1.86s/it]
Outer training loop:  61%|######1   | 30620/50000 [19:38:09<10:02:39,  1.87s/it]
Outer training loop:  61%|######1   | 30621/50000 [19:38:11<9:52:54,  1.84s/it] 
Outer training loop:  61%|######1   | 30622/50000 [19:38:13<9:50:06,  1.83s/it]
Outer training loop:  61%|######1   | 30623/50000 [19:38:14<9:59:01,  1.85s/it]
Outer training loop:  61%|######1   | 30624/50000 [19:38:16<9:47:54,  1.82s/it]
Outer training loop:  61%|######1   | 30625/50000 [19:38:18<9:58:30,  1.85s/it]
Outer training loop:  61%|######1   | 30626/50000 [19:38:20<10:07:39,  1.88s/it]
Outer training loop:  61%|######1   | 30627/50000 [19:38:22<10:17:55,  1.91s/it]
Outer training loop:  61%|######1   | 30628/50000 [19:38:24<10:24:59,  1.94s/it]
Outer training loop:  61%|######1   | 30629/50000 [19:38:26<10:20:19,  1.92s/it]
Outer training loop:  61%|######1   | 30630/50000 [19:38:28<10:27:10,  1.94s/it]
Outer training loop:  61%|######1   | 30631/50000 [19:38:30<10:17:34,  1.91s/it]
Outer training loop:  61%|######1   | 30632/50000 [19:38:31<10:00:44,  1.86s/it]
Outer training loop:  61%|######1   | 30633/50000 [19:38:33<10:10:11,  1.89s/it]
Outer training loop:  61%|######1   | 30634/50000 [19:38:35<10:05:03,  1.87s/it]
Outer training loop:  61%|######1   | 30635/50000 [19:38:37<10:08:48,  1.89s/it]
Outer training loop:  61%|######1   | 30636/50000 [19:38:39<10:12:34,  1.90s/it]
Outer training loop:  61%|######1   | 30637/50000 [19:38:41<10:07:12,  1.88s/it]
Outer training loop:  61%|######1   | 30638/50000 [19:38:43<10:03:13,  1.87s/it]
Outer training loop:  61%|######1   | 30639/50000 [19:38:45<10:10:03,  1.89s/it]
Outer training loop:  61%|######1   | 30640/50000 [19:38:47<10:26:23,  1.94s/it]
Outer training loop:  61%|######1   | 30641/50000 [19:38:49<10:11:44,  1.90s/it]
Outer training loop:  61%|######1   | 30642/50000 [19:38:50<10:10:14,  1.89s/it]
Outer training loop:  61%|######1   | 30643/50000 [19:38:52<10:22:50,  1.93s/it]
Outer training loop:  61%|######1   | 30644/50000 [19:38:54<10:08:50,  1.89s/it]
Outer training loop:  61%|######1   | 30645/50000 [19:38:56<9:58:17,  1.85s/it] 
Outer training loop:  61%|######1   | 30646/50000 [19:38:58<10:16:42,  1.91s/it]
Outer training loop:  61%|######1   | 30647/50000 [19:39:00<9:59:13,  1.86s/it] 
Outer training loop:  61%|######1   | 30648/50000 [19:39:02<9:56:33,  1.85s/it]
Outer training loop:  61%|######1   | 30649/50000 [19:39:04<9:58:57,  1.86s/it]
Outer training loop:  61%|######1   | 30650/50000 [19:39:06<10:12:44,  1.90s/it]
Outer training loop:  61%|######1   | 30651/50000 [19:39:07<10:08:32,  1.89s/it]
Outer training loop:  61%|######1   | 30652/50000 [19:39:09<9:57:23,  1.85s/it] 
Outer training loop:  61%|######1   | 30653/50000 [19:39:11<10:05:46,  1.88s/it]
Outer training loop:  61%|######1   | 30654/50000 [19:39:13<10:10:50,  1.89s/it]
Outer training loop:  61%|######1   | 30655/50000 [19:39:15<9:44:40,  1.81s/it] 
Outer training loop:  61%|######1   | 30656/50000 [19:39:16<9:43:54,  1.81s/it]
Outer training loop:  61%|######1   | 30657/50000 [19:39:19<10:18:02,  1.92s/it]
Outer training loop:  61%|######1   | 30658/50000 [19:39:21<10:13:46,  1.90s/it]
Outer training loop:  61%|######1   | 30659/50000 [19:39:22<10:09:56,  1.89s/it]
Outer training loop:  61%|######1   | 30660/50000 [19:39:25<10:33:47,  1.97s/it]
Outer training loop:  61%|######1   | 30661/50000 [19:39:26<10:11:17,  1.90s/it]
Outer training loop:  61%|######1   | 30662/50000 [19:39:28<9:55:34,  1.85s/it] 
Outer training loop:  61%|######1   | 30663/50000 [19:39:30<10:09:23,  1.89s/it]
Outer training loop:  61%|######1   | 30664/50000 [19:39:32<10:07:12,  1.88s/it]
Outer training loop:  61%|######1   | 30665/50000 [19:39:34<10:05:23,  1.88s/it]
Outer training loop:  61%|######1   | 30666/50000 [19:39:36<9:59:45,  1.86s/it] 
Outer training loop:  61%|######1   | 30667/50000 [19:39:38<10:12:57,  1.90s/it]
Outer training loop:  61%|######1   | 30668/50000 [19:39:39<10:06:51,  1.88s/it]
Outer training loop:  61%|######1   | 30669/50000 [19:39:41<10:02:11,  1.87s/it]
Outer training loop:  61%|######1   | 30670/50000 [19:39:43<9:54:08,  1.84s/it] 
Outer training loop:  61%|######1   | 30671/50000 [19:39:45<10:03:48,  1.87s/it]
Outer training loop:  61%|######1   | 30672/50000 [19:39:47<9:49:39,  1.83s/it] 
Outer training loop:  61%|######1   | 30673/50000 [19:39:48<9:42:23,  1.81s/it]
Outer training loop:  61%|######1   | 30674/50000 [19:39:51<10:14:10,  1.91s/it]
Outer training loop:  61%|######1   | 30675/50000 [19:39:52<10:11:40,  1.90s/it]
Outer training loop:  61%|######1   | 30676/50000 [19:39:54<10:00:29,  1.86s/it]
Outer training loop:  61%|######1   | 30677/50000 [19:39:56<10:04:07,  1.88s/it]
Outer training loop:  61%|######1   | 30678/50000 [19:39:58<10:34:20,  1.97s/it]
Outer training loop:  61%|######1   | 30679/50000 [19:40:00<10:13:51,  1.91s/it]
Outer training loop:  61%|######1   | 30680/50000 [19:40:02<10:12:13,  1.90s/it]
Outer training loop:  61%|######1   | 30681/50000 [19:40:04<10:32:41,  1.96s/it]
Outer training loop:  61%|######1   | 30682/50000 [19:40:06<10:12:12,  1.90s/it]
Outer training loop:  61%|######1   | 30683/50000 [19:40:08<9:58:52,  1.86s/it] 
Outer training loop:  61%|######1   | 30684/50000 [19:40:10<10:04:45,  1.88s/it]
Outer training loop:  61%|######1   | 30685/50000 [19:40:11<9:50:45,  1.84s/it] 
Outer training loop:  61%|######1   | 30686/50000 [19:40:13<9:42:28,  1.81s/it]
Outer training loop:  61%|######1   | 30687/50000 [19:40:15<10:05:05,  1.88s/it]
Outer training loop:  61%|######1   | 30688/50000 [19:40:17<9:57:41,  1.86s/it] 
Outer training loop:  61%|######1   | 30689/50000 [19:40:19<9:56:46,  1.85s/it]
Outer training loop:  61%|######1   | 30690/50000 [19:40:21<9:57:17,  1.86s/it]
Outer training loop:  61%|######1   | 30691/50000 [19:40:23<10:22:47,  1.94s/it]
Outer training loop:  61%|######1   | 30692/50000 [19:40:25<10:14:21,  1.91s/it]
Outer training loop:  61%|######1   | 30693/50000 [19:40:26<10:11:08,  1.90s/it]
Outer training loop:  61%|######1   | 30694/50000 [19:40:28<10:12:40,  1.90s/it]
Outer training loop:  61%|######1   | 30695/50000 [19:40:30<10:06:43,  1.89s/it]
Outer training loop:  61%|######1   | 30696/50000 [19:40:32<10:06:31,  1.89s/it]
Outer training loop:  61%|######1   | 30697/50000 [19:40:34<10:23:56,  1.94s/it]
Outer training loop:  61%|######1   | 30698/50000 [19:40:36<10:13:41,  1.91s/it]
Outer training loop:  61%|######1   | 30699/50000 [19:40:38<9:57:42,  1.86s/it] 
Outer training loop:  61%|######1   | 30700/50000 [19:40:40<10:12:12,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  61%|######1   | 30701/50000 [19:40:47<19:12:36,  3.58s/it]
Outer training loop:  61%|######1   | 30702/50000 [19:40:49<16:20:04,  3.05s/it]
Outer training loop:  61%|######1   | 30703/50000 [19:40:51<14:26:52,  2.70s/it]
Outer training loop:  61%|######1   | 30704/50000 [19:40:53<13:02:20,  2.43s/it]
Outer training loop:  61%|######1   | 30705/50000 [19:40:55<12:17:42,  2.29s/it]
Outer training loop:  61%|######1   | 30706/50000 [19:40:56<11:24:10,  2.13s/it]
Outer training loop:  61%|######1   | 30707/50000 [19:40:58<10:49:28,  2.02s/it]
Outer training loop:  61%|######1   | 30708/50000 [19:41:00<10:49:21,  2.02s/it]
Outer training loop:  61%|######1   | 30709/50000 [19:41:02<10:17:53,  1.92s/it]
Outer training loop:  61%|######1   | 30710/50000 [19:41:04<10:17:54,  1.92s/it]
Outer training loop:  61%|######1   | 30711/50000 [19:41:06<10:24:43,  1.94s/it]
Outer training loop:  61%|######1   | 30712/50000 [19:41:08<10:11:44,  1.90s/it]
Outer training loop:  61%|######1   | 30713/50000 [19:41:09<10:03:02,  1.88s/it]
Outer training loop:  61%|######1   | 30714/50000 [19:41:11<10:17:18,  1.92s/it]
Outer training loop:  61%|######1   | 30715/50000 [19:41:13<10:05:15,  1.88s/it]
Outer training loop:  61%|######1   | 30716/50000 [19:41:15<10:05:18,  1.88s/it]
Outer training loop:  61%|######1   | 30717/50000 [19:41:17<10:12:26,  1.91s/it]
Outer training loop:  61%|######1   | 30718/50000 [19:41:19<10:06:04,  1.89s/it]
Outer training loop:  61%|######1   | 30719/50000 [19:41:21<10:11:36,  1.90s/it]
Outer training loop:  61%|######1   | 30720/50000 [19:41:23<10:14:10,  1.91s/it]
Outer training loop:  61%|######1   | 30721/50000 [19:41:25<10:40:20,  1.99s/it]
Outer training loop:  61%|######1   | 30722/50000 [19:41:27<10:18:14,  1.92s/it]
Outer training loop:  61%|######1   | 30723/50000 [19:41:29<10:28:22,  1.96s/it]
Outer training loop:  61%|######1   | 30724/50000 [19:41:31<10:24:17,  1.94s/it]
Outer training loop:  61%|######1   | 30725/50000 [19:41:32<10:08:25,  1.89s/it]
Outer training loop:  61%|######1   | 30726/50000 [19:41:34<10:02:03,  1.87s/it]
Outer training loop:  61%|######1   | 30727/50000 [19:41:36<10:23:53,  1.94s/it]
Outer training loop:  61%|######1   | 30728/50000 [19:41:38<10:15:16,  1.92s/it]
Outer training loop:  61%|######1   | 30729/50000 [19:41:40<10:23:56,  1.94s/it]
Outer training loop:  61%|######1   | 30730/50000 [19:41:42<10:35:45,  1.98s/it]
Outer training loop:  61%|######1   | 30731/50000 [19:41:44<10:29:22,  1.96s/it]
Outer training loop:  61%|######1   | 30732/50000 [19:41:46<10:10:11,  1.90s/it]
Outer training loop:  61%|######1   | 30733/50000 [19:41:48<10:21:03,  1.93s/it]
Outer training loop:  61%|######1   | 30734/50000 [19:41:50<10:15:31,  1.92s/it]
Outer training loop:  61%|######1   | 30735/50000 [19:41:52<10:02:51,  1.88s/it]
Outer training loop:  61%|######1   | 30736/50000 [19:41:54<9:59:55,  1.87s/it] 
Outer training loop:  61%|######1   | 30737/50000 [19:41:56<10:18:39,  1.93s/it]
Outer training loop:  61%|######1   | 30738/50000 [19:41:57<10:17:28,  1.92s/it]
Outer training loop:  61%|######1   | 30739/50000 [19:41:59<10:08:51,  1.90s/it]
Outer training loop:  61%|######1   | 30740/50000 [19:42:01<10:27:49,  1.96s/it]
Outer training loop:  61%|######1   | 30741/50000 [19:42:03<10:16:39,  1.92s/it]
Outer training loop:  61%|######1   | 30742/50000 [19:42:05<10:12:00,  1.91s/it]
Outer training loop:  61%|######1   | 30743/50000 [19:42:07<10:06:39,  1.89s/it]
Outer training loop:  61%|######1   | 30744/50000 [19:42:09<10:31:31,  1.97s/it]
Outer training loop:  61%|######1   | 30745/50000 [19:42:11<10:26:00,  1.95s/it]
Outer training loop:  61%|######1   | 30746/50000 [19:42:13<10:35:52,  1.98s/it]
Outer training loop:  61%|######1   | 30747/50000 [19:42:15<10:12:44,  1.91s/it]
Outer training loop:  61%|######1   | 30748/50000 [19:42:17<10:20:46,  1.93s/it]
Outer training loop:  61%|######1   | 30749/50000 [19:42:19<10:15:22,  1.92s/it]
Outer training loop:  62%|######1   | 30750/50000 [19:42:21<10:16:51,  1.92s/it]
Outer training loop:  62%|######1   | 30751/50000 [19:42:23<10:47:19,  2.02s/it]
Outer training loop:  62%|######1   | 30752/50000 [19:42:25<10:34:42,  1.98s/it]
Outer training loop:  62%|######1   | 30753/50000 [19:42:27<10:30:22,  1.97s/it]
Outer training loop:  62%|######1   | 30754/50000 [19:42:29<10:31:39,  1.97s/it]
Outer training loop:  62%|######1   | 30755/50000 [19:42:31<10:36:06,  1.98s/it]
Outer training loop:  62%|######1   | 30756/50000 [19:42:33<10:34:33,  1.98s/it]
Outer training loop:  62%|######1   | 30757/50000 [19:42:35<10:26:12,  1.95s/it]
Outer training loop:  62%|######1   | 30758/50000 [19:42:37<10:44:46,  2.01s/it]
Outer training loop:  62%|######1   | 30759/50000 [19:42:38<10:21:57,  1.94s/it]
Outer training loop:  62%|######1   | 30760/50000 [19:42:40<10:13:41,  1.91s/it]
Outer training loop:  62%|######1   | 30761/50000 [19:42:42<10:30:57,  1.97s/it]
Outer training loop:  62%|######1   | 30762/50000 [19:42:44<10:19:22,  1.93s/it]
Outer training loop:  62%|######1   | 30763/50000 [19:42:46<10:26:13,  1.95s/it]
Outer training loop:  62%|######1   | 30764/50000 [19:42:48<10:13:04,  1.91s/it]
Outer training loop:  62%|######1   | 30765/50000 [19:42:50<10:46:09,  2.02s/it]
Outer training loop:  62%|######1   | 30766/50000 [19:42:52<10:22:49,  1.94s/it]
Outer training loop:  62%|######1   | 30767/50000 [19:42:54<10:19:45,  1.93s/it]
Outer training loop:  62%|######1   | 30768/50000 [19:42:56<10:28:03,  1.96s/it]
Outer training loop:  62%|######1   | 30769/50000 [19:42:58<10:18:18,  1.93s/it]
Outer training loop:  62%|######1   | 30770/50000 [19:43:00<10:08:44,  1.90s/it]
Outer training loop:  62%|######1   | 30771/50000 [19:43:02<10:17:44,  1.93s/it]
Outer training loop:  62%|######1   | 30772/50000 [19:43:04<10:14:20,  1.92s/it]
Outer training loop:  62%|######1   | 30773/50000 [19:43:06<10:22:01,  1.94s/it]
Outer training loop:  62%|######1   | 30774/50000 [19:43:07<10:14:52,  1.92s/it]
Outer training loop:  62%|######1   | 30775/50000 [19:43:09<10:11:01,  1.91s/it]
Outer training loop:  62%|######1   | 30776/50000 [19:43:11<10:06:35,  1.89s/it]
Outer training loop:  62%|######1   | 30777/50000 [19:43:13<10:01:58,  1.88s/it]
Outer training loop:  62%|######1   | 30778/50000 [19:43:15<9:45:42,  1.83s/it] 
Outer training loop:  62%|######1   | 30779/50000 [19:43:17<10:15:29,  1.92s/it]
Outer training loop:  62%|######1   | 30780/50000 [19:43:19<10:08:21,  1.90s/it]
Outer training loop:  62%|######1   | 30781/50000 [19:43:21<10:20:43,  1.94s/it]
Outer training loop:  62%|######1   | 30782/50000 [19:43:23<10:30:11,  1.97s/it]
Outer training loop:  62%|######1   | 30783/50000 [19:43:25<10:27:38,  1.96s/it]
Outer training loop:  62%|######1   | 30784/50000 [19:43:27<10:08:26,  1.90s/it]
Outer training loop:  62%|######1   | 30785/50000 [19:43:28<10:11:08,  1.91s/it]
Outer training loop:  62%|######1   | 30786/50000 [19:43:31<10:25:26,  1.95s/it]
Outer training loop:  62%|######1   | 30787/50000 [19:43:32<10:22:46,  1.94s/it]
Outer training loop:  62%|######1   | 30788/50000 [19:43:34<10:16:12,  1.92s/it]
Outer training loop:  62%|######1   | 30789/50000 [19:43:36<10:36:29,  1.99s/it]
Outer training loop:  62%|######1   | 30790/50000 [19:43:38<10:21:50,  1.94s/it]
Outer training loop:  62%|######1   | 30791/50000 [19:43:40<10:30:54,  1.97s/it]
Outer training loop:  62%|######1   | 30792/50000 [19:43:42<10:37:17,  1.99s/it]
Outer training loop:  62%|######1   | 30793/50000 [19:43:44<10:28:02,  1.96s/it]
Outer training loop:  62%|######1   | 30794/50000 [19:43:46<10:14:14,  1.92s/it]
Outer training loop:  62%|######1   | 30795/50000 [19:43:48<10:15:31,  1.92s/it]
Outer training loop:  62%|######1   | 30796/50000 [19:43:50<10:29:42,  1.97s/it]
Outer training loop:  62%|######1   | 30797/50000 [19:43:52<10:28:22,  1.96s/it]
Outer training loop:  62%|######1   | 30798/50000 [19:43:54<10:21:10,  1.94s/it]
Outer training loop:  62%|######1   | 30799/50000 [19:43:56<10:28:32,  1.96s/it]
Outer training loop:  62%|######1   | 30800/50000 [19:43:58<10:19:30,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.93s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]

Outer training loop:  62%|######1   | 30801/50000 [19:44:05<19:27:18,  3.65s/it]
Outer training loop:  62%|######1   | 30802/50000 [19:44:07<16:28:29,  3.09s/it]
Outer training loop:  62%|######1   | 30803/50000 [19:44:10<15:10:04,  2.84s/it]
Outer training loop:  62%|######1   | 30804/50000 [19:44:11<13:37:44,  2.56s/it]
Outer training loop:  62%|######1   | 30805/50000 [19:44:13<12:27:22,  2.34s/it]
Outer training loop:  62%|######1   | 30806/50000 [19:44:15<11:55:32,  2.24s/it]
Outer training loop:  62%|######1   | 30807/50000 [19:44:17<11:10:59,  2.10s/it]
Outer training loop:  62%|######1   | 30808/50000 [19:44:19<10:52:46,  2.04s/it]
Outer training loop:  62%|######1   | 30809/50000 [19:44:21<11:08:47,  2.09s/it]
Outer training loop:  62%|######1   | 30810/50000 [19:44:23<10:50:02,  2.03s/it]
Outer training loop:  62%|######1   | 30811/50000 [19:44:25<10:41:02,  2.00s/it]
Outer training loop:  62%|######1   | 30812/50000 [19:44:27<10:35:21,  1.99s/it]
Outer training loop:  62%|######1   | 30813/50000 [19:44:29<10:43:22,  2.01s/it]
Outer training loop:  62%|######1   | 30814/50000 [19:44:31<10:30:48,  1.97s/it]
Outer training loop:  62%|######1   | 30815/50000 [19:44:33<10:18:14,  1.93s/it]
Outer training loop:  62%|######1   | 30816/50000 [19:44:35<10:41:41,  2.01s/it]
Outer training loop:  62%|######1   | 30817/50000 [19:44:37<10:26:07,  1.96s/it]
Outer training loop:  62%|######1   | 30818/50000 [19:44:38<10:03:39,  1.89s/it]
Outer training loop:  62%|######1   | 30819/50000 [19:44:40<9:54:07,  1.86s/it] 
Outer training loop:  62%|######1   | 30820/50000 [19:44:42<10:18:11,  1.93s/it]
Outer training loop:  62%|######1   | 30821/50000 [19:44:44<10:12:09,  1.92s/it]
Outer training loop:  62%|######1   | 30822/50000 [19:44:46<10:17:31,  1.93s/it]
Outer training loop:  62%|######1   | 30823/50000 [19:44:48<10:26:40,  1.96s/it]
Outer training loop:  62%|######1   | 30824/50000 [19:44:50<10:13:02,  1.92s/it]
Outer training loop:  62%|######1   | 30825/50000 [19:44:52<10:11:53,  1.91s/it]
Outer training loop:  62%|######1   | 30826/50000 [19:44:54<10:22:28,  1.95s/it]
Outer training loop:  62%|######1   | 30827/50000 [19:44:56<10:15:15,  1.93s/it]
Outer training loop:  62%|######1   | 30828/50000 [19:44:58<10:16:39,  1.93s/it]
Outer training loop:  62%|######1   | 30829/50000 [19:45:00<10:30:10,  1.97s/it]
Outer training loop:  62%|######1   | 30830/50000 [19:45:02<10:10:49,  1.91s/it]
Outer training loop:  62%|######1   | 30831/50000 [19:45:03<10:08:14,  1.90s/it]
Outer training loop:  62%|######1   | 30832/50000 [19:45:05<10:04:52,  1.89s/it]
Outer training loop:  62%|######1   | 30833/50000 [19:45:07<10:10:05,  1.91s/it]
Outer training loop:  62%|######1   | 30834/50000 [19:45:09<10:06:18,  1.90s/it]
Outer training loop:  62%|######1   | 30835/50000 [19:45:11<9:54:01,  1.86s/it] 
Outer training loop:  62%|######1   | 30836/50000 [19:45:13<10:12:14,  1.92s/it]
Outer training loop:  62%|######1   | 30837/50000 [19:45:15<10:11:44,  1.92s/it]
Outer training loop:  62%|######1   | 30838/50000 [19:45:17<10:07:12,  1.90s/it]
Outer training loop:  62%|######1   | 30839/50000 [19:45:19<10:16:56,  1.93s/it]
Outer training loop:  62%|######1   | 30840/50000 [19:45:21<10:05:06,  1.89s/it]
Outer training loop:  62%|######1   | 30841/50000 [19:45:22<10:02:17,  1.89s/it]
Outer training loop:  62%|######1   | 30842/50000 [19:45:24<10:03:51,  1.89s/it]
Outer training loop:  62%|######1   | 30843/50000 [19:45:27<10:27:52,  1.97s/it]
Outer training loop:  62%|######1   | 30844/50000 [19:45:28<10:16:17,  1.93s/it]
Outer training loop:  62%|######1   | 30845/50000 [19:45:30<10:03:31,  1.89s/it]
Outer training loop:  62%|######1   | 30846/50000 [19:45:32<10:11:37,  1.92s/it]
Outer training loop:  62%|######1   | 30847/50000 [19:45:34<10:01:39,  1.88s/it]
Outer training loop:  62%|######1   | 30848/50000 [19:45:36<10:24:25,  1.96s/it]
Outer training loop:  62%|######1   | 30849/50000 [19:45:38<10:04:42,  1.89s/it]
Outer training loop:  62%|######1   | 30850/50000 [19:45:40<10:18:45,  1.94s/it]
Outer training loop:  62%|######1   | 30851/50000 [19:45:42<10:15:53,  1.93s/it]
Outer training loop:  62%|######1   | 30852/50000 [19:45:44<10:15:11,  1.93s/it]
Outer training loop:  62%|######1   | 30853/50000 [19:45:46<10:30:42,  1.98s/it]
Outer training loop:  62%|######1   | 30854/50000 [19:45:48<10:09:13,  1.91s/it]
Outer training loop:  62%|######1   | 30855/50000 [19:45:49<10:01:00,  1.88s/it]
Outer training loop:  62%|######1   | 30856/50000 [19:45:51<10:02:07,  1.89s/it]
Outer training loop:  62%|######1   | 30857/50000 [19:45:53<10:16:29,  1.93s/it]
Outer training loop:  62%|######1   | 30858/50000 [19:45:55<10:02:26,  1.89s/it]
Outer training loop:  62%|######1   | 30859/50000 [19:45:57<9:57:15,  1.87s/it] 
Outer training loop:  62%|######1   | 30860/50000 [19:45:59<10:17:49,  1.94s/it]
Outer training loop:  62%|######1   | 30861/50000 [19:46:01<10:08:24,  1.91s/it]
Outer training loop:  62%|######1   | 30862/50000 [19:46:03<10:07:17,  1.90s/it]
Outer training loop:  62%|######1   | 30863/50000 [19:46:05<10:15:46,  1.93s/it]
Outer training loop:  62%|######1   | 30864/50000 [19:46:06<9:59:45,  1.88s/it] 
Outer training loop:  62%|######1   | 30865/50000 [19:46:08<9:59:49,  1.88s/it]
Outer training loop:  62%|######1   | 30866/50000 [19:46:10<10:13:17,  1.92s/it]
Outer training loop:  62%|######1   | 30867/50000 [19:46:12<10:05:46,  1.90s/it]
Outer training loop:  62%|######1   | 30868/50000 [19:46:14<9:54:18,  1.86s/it] 
Outer training loop:  62%|######1   | 30869/50000 [19:46:16<9:58:43,  1.88s/it]
Outer training loop:  62%|######1   | 30870/50000 [19:46:18<10:14:02,  1.93s/it]
Outer training loop:  62%|######1   | 30871/50000 [19:46:20<10:08:45,  1.91s/it]
Outer training loop:  62%|######1   | 30872/50000 [19:46:22<10:08:51,  1.91s/it]
Outer training loop:  62%|######1   | 30873/50000 [19:46:24<10:20:16,  1.95s/it]
Outer training loop:  62%|######1   | 30874/50000 [19:46:26<10:00:54,  1.89s/it]
Outer training loop:  62%|######1   | 30875/50000 [19:46:27<9:52:33,  1.86s/it] 
Outer training loop:  62%|######1   | 30876/50000 [19:46:29<9:47:51,  1.84s/it]
Outer training loop:  62%|######1   | 30877/50000 [19:46:31<9:51:56,  1.86s/it]
Outer training loop:  62%|######1   | 30878/50000 [19:46:33<9:44:26,  1.83s/it]
Outer training loop:  62%|######1   | 30879/50000 [19:46:35<9:41:51,  1.83s/it]
Outer training loop:  62%|######1   | 30880/50000 [19:46:37<10:08:54,  1.91s/it]
Outer training loop:  62%|######1   | 30881/50000 [19:46:39<10:06:00,  1.90s/it]
Outer training loop:  62%|######1   | 30882/50000 [19:46:40<10:05:54,  1.90s/it]
Outer training loop:  62%|######1   | 30883/50000 [19:46:42<10:07:42,  1.91s/it]
Outer training loop:  62%|######1   | 30884/50000 [19:46:45<10:28:07,  1.97s/it]
Outer training loop:  62%|######1   | 30885/50000 [19:46:46<10:16:01,  1.93s/it]
Outer training loop:  62%|######1   | 30886/50000 [19:46:48<10:18:04,  1.94s/it]
Outer training loop:  62%|######1   | 30887/50000 [19:46:51<10:42:12,  2.02s/it]
Outer training loop:  62%|######1   | 30888/50000 [19:46:52<10:23:02,  1.96s/it]
Outer training loop:  62%|######1   | 30889/50000 [19:46:54<10:10:04,  1.92s/it]
Outer training loop:  62%|######1   | 30890/50000 [19:46:56<10:22:50,  1.96s/it]
Outer training loop:  62%|######1   | 30891/50000 [19:46:58<10:19:10,  1.94s/it]
Outer training loop:  62%|######1   | 30892/50000 [19:47:00<10:08:01,  1.91s/it]
Outer training loop:  62%|######1   | 30893/50000 [19:47:02<10:00:40,  1.89s/it]
Outer training loop:  62%|######1   | 30894/50000 [19:47:04<10:25:21,  1.96s/it]
Outer training loop:  62%|######1   | 30895/50000 [19:47:06<10:26:34,  1.97s/it]
Outer training loop:  62%|######1   | 30896/50000 [19:47:08<10:07:14,  1.91s/it]
Outer training loop:  62%|######1   | 30897/50000 [19:47:10<10:11:59,  1.92s/it]
Outer training loop:  62%|######1   | 30898/50000 [19:47:12<10:28:21,  1.97s/it]
Outer training loop:  62%|######1   | 30899/50000 [19:47:14<10:16:40,  1.94s/it]
Outer training loop:  62%|######1   | 30900/50000 [19:47:15<10:00:10,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  62%|######1   | 30901/50000 [19:47:23<19:00:20,  3.58s/it]
Outer training loop:  62%|######1   | 30902/50000 [19:47:25<16:14:40,  3.06s/it]
Outer training loop:  62%|######1   | 30903/50000 [19:47:27<14:27:55,  2.73s/it]
Outer training loop:  62%|######1   | 30904/50000 [19:47:29<13:32:33,  2.55s/it]
Outer training loop:  62%|######1   | 30905/50000 [19:47:31<12:30:11,  2.36s/it]
Outer training loop:  62%|######1   | 30906/50000 [19:47:33<11:42:40,  2.21s/it]
Outer training loop:  62%|######1   | 30907/50000 [19:47:35<11:15:49,  2.12s/it]
Outer training loop:  62%|######1   | 30908/50000 [19:47:37<11:08:05,  2.10s/it]
Outer training loop:  62%|######1   | 30909/50000 [19:47:38<10:43:09,  2.02s/it]
Outer training loop:  62%|######1   | 30910/50000 [19:47:40<10:15:25,  1.93s/it]
Outer training loop:  62%|######1   | 30911/50000 [19:47:42<10:05:22,  1.90s/it]
Outer training loop:  62%|######1   | 30912/50000 [19:47:44<10:17:45,  1.94s/it]
Outer training loop:  62%|######1   | 30913/50000 [19:47:46<9:58:29,  1.88s/it] 
Outer training loop:  62%|######1   | 30914/50000 [19:47:48<9:58:19,  1.88s/it]
Outer training loop:  62%|######1   | 30915/50000 [19:47:50<10:18:25,  1.94s/it]
Outer training loop:  62%|######1   | 30916/50000 [19:47:52<10:17:42,  1.94s/it]
Outer training loop:  62%|######1   | 30917/50000 [19:47:53<10:00:29,  1.89s/it]
Outer training loop:  62%|######1   | 30918/50000 [19:47:55<10:03:06,  1.90s/it]
Outer training loop:  62%|######1   | 30919/50000 [19:47:57<10:23:36,  1.96s/it]
Outer training loop:  62%|######1   | 30920/50000 [19:47:59<10:19:34,  1.95s/it]
Outer training loop:  62%|######1   | 30921/50000 [19:48:01<10:04:57,  1.90s/it]
Outer training loop:  62%|######1   | 30922/50000 [19:48:03<10:19:19,  1.95s/it]
Outer training loop:  62%|######1   | 30923/50000 [19:48:05<10:07:53,  1.91s/it]
Outer training loop:  62%|######1   | 30924/50000 [19:48:07<9:57:24,  1.88s/it] 
Outer training loop:  62%|######1   | 30925/50000 [19:48:09<10:22:50,  1.96s/it]
Outer training loop:  62%|######1   | 30926/50000 [19:48:11<10:15:43,  1.94s/it]
Outer training loop:  62%|######1   | 30927/50000 [19:48:13<10:20:39,  1.95s/it]
Outer training loop:  62%|######1   | 30928/50000 [19:48:15<10:35:22,  2.00s/it]
Outer training loop:  62%|######1   | 30929/50000 [19:48:17<10:16:43,  1.94s/it]
Outer training loop:  62%|######1   | 30930/50000 [19:48:18<9:53:50,  1.87s/it] 
Outer training loop:  62%|######1   | 30931/50000 [19:48:20<9:53:15,  1.87s/it]
Outer training loop:  62%|######1   | 30932/50000 [19:48:22<10:05:01,  1.90s/it]
Outer training loop:  62%|######1   | 30933/50000 [19:48:24<9:58:10,  1.88s/it] 
Outer training loop:  62%|######1   | 30934/50000 [19:48:26<10:09:53,  1.92s/it]
Outer training loop:  62%|######1   | 30935/50000 [19:48:28<9:59:48,  1.89s/it] 
Outer training loop:  62%|######1   | 30936/50000 [19:48:30<10:07:52,  1.91s/it]
Outer training loop:  62%|######1   | 30937/50000 [19:48:32<10:05:10,  1.90s/it]
Outer training loop:  62%|######1   | 30938/50000 [19:48:34<9:49:26,  1.86s/it] 
Outer training loop:  62%|######1   | 30939/50000 [19:48:36<10:17:18,  1.94s/it]
Outer training loop:  62%|######1   | 30940/50000 [19:48:38<10:09:16,  1.92s/it]
Outer training loop:  62%|######1   | 30941/50000 [19:48:39<9:59:43,  1.89s/it] 
Outer training loop:  62%|######1   | 30942/50000 [19:48:41<10:08:39,  1.92s/it]
Outer training loop:  62%|######1   | 30943/50000 [19:48:43<10:02:52,  1.90s/it]
Outer training loop:  62%|######1   | 30944/50000 [19:48:45<9:59:38,  1.89s/it] 
Outer training loop:  62%|######1   | 30945/50000 [19:48:47<10:09:53,  1.92s/it]
Outer training loop:  62%|######1   | 30946/50000 [19:48:49<10:20:20,  1.95s/it]
Outer training loop:  62%|######1   | 30947/50000 [19:48:51<10:17:24,  1.94s/it]
Outer training loop:  62%|######1   | 30948/50000 [19:48:53<10:11:55,  1.93s/it]
Outer training loop:  62%|######1   | 30949/50000 [19:48:55<10:27:07,  1.98s/it]
Outer training loop:  62%|######1   | 30950/50000 [19:48:57<10:12:46,  1.93s/it]
Outer training loop:  62%|######1   | 30951/50000 [19:48:59<10:00:00,  1.89s/it]
Outer training loop:  62%|######1   | 30952/50000 [19:49:00<9:54:43,  1.87s/it] 
Outer training loop:  62%|######1   | 30953/50000 [19:49:03<10:11:03,  1.92s/it]
Outer training loop:  62%|######1   | 30954/50000 [19:49:04<9:59:44,  1.89s/it] 
Outer training loop:  62%|######1   | 30955/50000 [19:49:06<10:02:37,  1.90s/it]
Outer training loop:  62%|######1   | 30956/50000 [19:49:08<9:49:40,  1.86s/it] 
Outer training loop:  62%|######1   | 30957/50000 [19:49:10<10:06:54,  1.91s/it]
Outer training loop:  62%|######1   | 30958/50000 [19:49:12<10:04:15,  1.90s/it]
Outer training loop:  62%|######1   | 30959/50000 [19:49:14<9:57:57,  1.88s/it] 
Outer training loop:  62%|######1   | 30960/50000 [19:49:16<10:06:41,  1.91s/it]
Outer training loop:  62%|######1   | 30961/50000 [19:49:18<10:05:29,  1.91s/it]
Outer training loop:  62%|######1   | 30962/50000 [19:49:19<9:52:21,  1.87s/it] 
Outer training loop:  62%|######1   | 30963/50000 [19:49:21<10:08:45,  1.92s/it]
Outer training loop:  62%|######1   | 30964/50000 [19:49:23<10:01:44,  1.90s/it]
Outer training loop:  62%|######1   | 30965/50000 [19:49:25<10:03:45,  1.90s/it]
Outer training loop:  62%|######1   | 30966/50000 [19:49:27<9:55:23,  1.88s/it] 
Outer training loop:  62%|######1   | 30967/50000 [19:49:29<10:13:24,  1.93s/it]
Outer training loop:  62%|######1   | 30968/50000 [19:49:31<10:04:45,  1.91s/it]
Outer training loop:  62%|######1   | 30969/50000 [19:49:33<10:07:19,  1.91s/it]
Outer training loop:  62%|######1   | 30970/50000 [19:49:35<10:29:16,  1.98s/it]
Outer training loop:  62%|######1   | 30971/50000 [19:49:37<10:21:52,  1.96s/it]
Outer training loop:  62%|######1   | 30972/50000 [19:49:39<9:55:16,  1.88s/it] 
Outer training loop:  62%|######1   | 30973/50000 [19:49:41<9:57:49,  1.89s/it]
Outer training loop:  62%|######1   | 30974/50000 [19:49:42<10:05:17,  1.91s/it]
Outer training loop:  62%|######1   | 30975/50000 [19:49:44<10:01:19,  1.90s/it]
Outer training loop:  62%|######1   | 30976/50000 [19:49:46<9:52:48,  1.87s/it] 
Outer training loop:  62%|######1   | 30977/50000 [19:49:48<9:39:37,  1.83s/it]
Outer training loop:  62%|######1   | 30978/50000 [19:49:50<9:57:37,  1.89s/it]
Outer training loop:  62%|######1   | 30979/50000 [19:49:52<9:56:00,  1.88s/it]
Outer training loop:  62%|######1   | 30980/50000 [19:49:54<9:58:31,  1.89s/it]
Outer training loop:  62%|######1   | 30981/50000 [19:49:56<10:25:14,  1.97s/it]
Outer training loop:  62%|######1   | 30982/50000 [19:49:58<10:12:11,  1.93s/it]
Outer training loop:  62%|######1   | 30983/50000 [19:50:00<10:10:17,  1.93s/it]
Outer training loop:  62%|######1   | 30984/50000 [19:50:02<10:22:46,  1.96s/it]
Outer training loop:  62%|######1   | 30985/50000 [19:50:03<10:06:14,  1.91s/it]
Outer training loop:  62%|######1   | 30986/50000 [19:50:05<10:04:49,  1.91s/it]
Outer training loop:  62%|######1   | 30987/50000 [19:50:07<10:14:40,  1.94s/it]
Outer training loop:  62%|######1   | 30988/50000 [19:50:09<9:54:29,  1.88s/it] 
Outer training loop:  62%|######1   | 30989/50000 [19:50:11<9:46:15,  1.85s/it]
Outer training loop:  62%|######1   | 30990/50000 [19:50:13<9:54:10,  1.88s/it]
Outer training loop:  62%|######1   | 30991/50000 [19:50:15<10:03:15,  1.90s/it]
Outer training loop:  62%|######1   | 30992/50000 [19:50:17<9:57:35,  1.89s/it] 
Outer training loop:  62%|######1   | 30993/50000 [19:50:18<9:45:29,  1.85s/it]
Outer training loop:  62%|######1   | 30994/50000 [19:50:20<10:01:23,  1.90s/it]
Outer training loop:  62%|######1   | 30995/50000 [19:50:22<10:03:15,  1.90s/it]
Outer training loop:  62%|######1   | 30996/50000 [19:50:24<10:01:22,  1.90s/it]
Outer training loop:  62%|######1   | 30997/50000 [19:50:26<10:05:11,  1.91s/it]
Outer training loop:  62%|######1   | 30998/50000 [19:50:28<9:59:26,  1.89s/it] 
Outer training loop:  62%|######1   | 30999/50000 [19:50:30<9:54:06,  1.88s/it]
Outer training loop:  62%|######2   | 31000/50000 [19:50:32<10:05:25,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:33,  1.87s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:31,  1.87s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:34,  1.93s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:30,  1.92s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.94s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.97s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:13,  1.96s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.85s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:00,  1.85s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:57,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.86s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:50,  1.84s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:53,  1.96s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.92s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:42,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:42,  1.97s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.94s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:32,  1.89s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:32,  1.96s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:29,  1.94s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:22,  1.88s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.89s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.90s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:18,  1.97s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.91s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.87s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:08,  1.84s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:01,  1.81s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.92s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:49,  1.97s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:32,  1.94s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.89s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.89s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:17,  1.79s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.99s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.93s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.92s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.81s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:17,  1.68s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:33,  1.90s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.86s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.87s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.86s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.92s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:22,  1.95s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:14,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:10,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:14,  1.96s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.92s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:01,  1.85s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:02,  1.91s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:57,  1.95s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:42,  1.84s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.85s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:31,  1.82s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:33,  1.91s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.87s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:20,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:23,  1.94s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.90s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.87s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.88s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:13,  1.99s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:09,  1.93s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.87s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:58,  1.96s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:48,  1.96s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.89s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.88s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:43,  1.98s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:41,  1.95s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.89s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.89s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.85s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:26,  2.04s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.98s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.89s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.88s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.86s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.84s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.82s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:08,  2.04s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.92s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.89s/it]

Outer training loop:  62%|######2   | 31001/50000 [19:55:53<516:15:55, 97.82s/it]
Outer training loop:  62%|######2   | 31002/50000 [19:55:55<364:12:31, 69.02s/it]
Outer training loop:  62%|######2   | 31003/50000 [19:55:57<257:44:20, 48.84s/it]
Outer training loop:  62%|######2   | 31004/50000 [19:55:59<183:26:26, 34.76s/it]
Outer training loop:  62%|######2   | 31005/50000 [19:56:01<131:11:51, 24.87s/it]
Outer training loop:  62%|######2   | 31006/50000 [19:56:03<95:41:11, 18.14s/it] 
Outer training loop:  62%|######2   | 31007/50000 [19:56:05<69:58:41, 13.26s/it]
Outer training loop:  62%|######2   | 31008/50000 [19:56:07<52:04:48,  9.87s/it]
Outer training loop:  62%|######2   | 31009/50000 [19:56:09<39:43:16,  7.53s/it]
Outer training loop:  62%|######2   | 31010/50000 [19:56:11<30:39:00,  5.81s/it]
Outer training loop:  62%|######2   | 31011/50000 [19:56:13<24:34:19,  4.66s/it]
Outer training loop:  62%|######2   | 31012/50000 [19:56:15<20:13:31,  3.83s/it]
Outer training loop:  62%|######2   | 31013/50000 [19:56:17<17:25:26,  3.30s/it]
Outer training loop:  62%|######2   | 31014/50000 [19:56:19<15:01:04,  2.85s/it]
Outer training loop:  62%|######2   | 31015/50000 [19:56:20<13:23:47,  2.54s/it]
Outer training loop:  62%|######2   | 31016/50000 [19:56:22<12:30:24,  2.37s/it]
Outer training loop:  62%|######2   | 31017/50000 [19:56:24<11:32:14,  2.19s/it]
Outer training loop:  62%|######2   | 31018/50000 [19:56:26<11:01:54,  2.09s/it]
Outer training loop:  62%|######2   | 31019/50000 [19:56:28<11:05:47,  2.10s/it]
Outer training loop:  62%|######2   | 31020/50000 [19:56:30<10:28:25,  1.99s/it]
Outer training loop:  62%|######2   | 31021/50000 [19:56:32<10:07:11,  1.92s/it]
Outer training loop:  62%|######2   | 31022/50000 [19:56:34<10:16:13,  1.95s/it]
Outer training loop:  62%|######2   | 31023/50000 [19:56:35<10:00:46,  1.90s/it]
Outer training loop:  62%|######2   | 31024/50000 [19:56:37<10:09:26,  1.93s/it]
Outer training loop:  62%|######2   | 31025/50000 [19:56:39<9:51:58,  1.87s/it] 
Outer training loop:  62%|######2   | 31026/50000 [19:56:41<10:19:42,  1.96s/it]
Outer training loop:  62%|######2   | 31027/50000 [19:56:43<10:16:53,  1.95s/it]
Outer training loop:  62%|######2   | 31028/50000 [19:56:45<10:03:53,  1.91s/it]
Outer training loop:  62%|######2   | 31029/50000 [19:56:47<9:59:45,  1.90s/it] 
Outer training loop:  62%|######2   | 31030/50000 [19:56:49<9:51:04,  1.87s/it]
Outer training loop:  62%|######2   | 31031/50000 [19:56:51<9:53:22,  1.88s/it]
Outer training loop:  62%|######2   | 31032/50000 [19:56:53<10:17:52,  1.95s/it]
Outer training loop:  62%|######2   | 31033/50000 [19:56:55<10:05:15,  1.91s/it]
Outer training loop:  62%|######2   | 31034/50000 [19:56:56<10:06:03,  1.92s/it]
Outer training loop:  62%|######2   | 31035/50000 [19:56:58<10:10:14,  1.93s/it]
Outer training loop:  62%|######2   | 31036/50000 [19:57:01<10:23:38,  1.97s/it]
Outer training loop:  62%|######2   | 31037/50000 [19:57:02<10:11:51,  1.94s/it]
Outer training loop:  62%|######2   | 31038/50000 [19:57:04<10:13:36,  1.94s/it]
Outer training loop:  62%|######2   | 31039/50000 [19:57:06<10:26:50,  1.98s/it]
Outer training loop:  62%|######2   | 31040/50000 [19:57:08<10:11:49,  1.94s/it]
Outer training loop:  62%|######2   | 31041/50000 [19:57:10<9:56:39,  1.89s/it] 
Outer training loop:  62%|######2   | 31042/50000 [19:57:12<9:54:01,  1.88s/it]
Outer training loop:  62%|######2   | 31043/50000 [19:57:14<10:11:34,  1.94s/it]
Outer training loop:  62%|######2   | 31044/50000 [19:57:16<10:00:44,  1.90s/it]
Outer training loop:  62%|######2   | 31045/50000 [19:57:18<10:08:37,  1.93s/it]
Outer training loop:  62%|######2   | 31046/50000 [19:57:20<9:59:23,  1.90s/it] 
Outer training loop:  62%|######2   | 31047/50000 [19:57:21<9:51:28,  1.87s/it]
Outer training loop:  62%|######2   | 31048/50000 [19:57:23<9:45:39,  1.85s/it]
Outer training loop:  62%|######2   | 31049/50000 [19:57:25<10:17:26,  1.95s/it]
Outer training loop:  62%|######2   | 31050/50000 [19:57:27<10:00:29,  1.90s/it]
Outer training loop:  62%|######2   | 31051/50000 [19:57:29<9:57:43,  1.89s/it] 
Outer training loop:  62%|######2   | 31052/50000 [19:57:31<9:55:53,  1.89s/it]
Outer training loop:  62%|######2   | 31053/50000 [19:57:33<9:59:31,  1.90s/it]
Outer training loop:  62%|######2   | 31054/50000 [19:57:35<9:45:32,  1.85s/it]
Outer training loop:  62%|######2   | 31055/50000 [19:57:36<9:44:10,  1.85s/it]
Outer training loop:  62%|######2   | 31056/50000 [19:57:38<10:02:48,  1.91s/it]
Outer training loop:  62%|######2   | 31057/50000 [19:57:40<10:05:58,  1.92s/it]
Outer training loop:  62%|######2   | 31058/50000 [19:57:42<9:55:47,  1.89s/it] 
Outer training loop:  62%|######2   | 31059/50000 [19:57:44<10:05:12,  1.92s/it]
Outer training loop:  62%|######2   | 31060/50000 [19:57:46<9:47:13,  1.86s/it] 
Outer training loop:  62%|######2   | 31061/50000 [19:57:48<9:48:04,  1.86s/it]
Outer training loop:  62%|######2   | 31062/50000 [19:57:50<9:52:01,  1.88s/it]
Outer training loop:  62%|######2   | 31063/50000 [19:57:52<10:08:42,  1.93s/it]
Outer training loop:  62%|######2   | 31064/50000 [19:57:54<9:54:00,  1.88s/it] 
Outer training loop:  62%|######2   | 31065/50000 [19:57:55<9:51:30,  1.87s/it]
Outer training loop:  62%|######2   | 31066/50000 [19:57:57<10:09:52,  1.93s/it]
Outer training loop:  62%|######2   | 31067/50000 [19:57:59<9:50:24,  1.87s/it] 
Outer training loop:  62%|######2   | 31068/50000 [19:58:01<9:33:35,  1.82s/it]
Outer training loop:  62%|######2   | 31069/50000 [19:58:03<9:54:43,  1.88s/it]
Outer training loop:  62%|######2   | 31070/50000 [19:58:05<9:53:09,  1.88s/it]
Outer training loop:  62%|######2   | 31071/50000 [19:58:07<9:39:55,  1.84s/it]
Outer training loop:  62%|######2   | 31072/50000 [19:58:08<9:39:22,  1.84s/it]
Outer training loop:  62%|######2   | 31073/50000 [19:58:10<9:53:18,  1.88s/it]
Outer training loop:  62%|######2   | 31074/50000 [19:58:12<9:49:47,  1.87s/it]
Outer training loop:  62%|######2   | 31075/50000 [19:58:14<9:49:03,  1.87s/it]
Outer training loop:  62%|######2   | 31076/50000 [19:58:16<10:18:20,  1.96s/it]
Outer training loop:  62%|######2   | 31077/50000 [19:58:18<9:58:31,  1.90s/it] 
Outer training loop:  62%|######2   | 31078/50000 [19:58:20<9:56:22,  1.89s/it]
Outer training loop:  62%|######2   | 31079/50000 [19:58:22<9:44:56,  1.85s/it]
Outer training loop:  62%|######2   | 31080/50000 [19:58:24<9:45:49,  1.86s/it]
Outer training loop:  62%|######2   | 31081/50000 [19:58:25<9:44:14,  1.85s/it]
Outer training loop:  62%|######2   | 31082/50000 [19:58:27<9:40:52,  1.84s/it]
Outer training loop:  62%|######2   | 31083/50000 [19:58:29<9:58:10,  1.90s/it]
Outer training loop:  62%|######2   | 31084/50000 [19:58:31<9:43:50,  1.85s/it]
Outer training loop:  62%|######2   | 31085/50000 [19:58:33<9:33:54,  1.82s/it]
Outer training loop:  62%|######2   | 31086/50000 [19:58:35<9:45:51,  1.86s/it]
Outer training loop:  62%|######2   | 31087/50000 [19:58:37<9:58:44,  1.90s/it]
Outer training loop:  62%|######2   | 31088/50000 [19:58:39<9:57:54,  1.90s/it]
Outer training loop:  62%|######2   | 31089/50000 [19:58:40<9:51:55,  1.88s/it]
Outer training loop:  62%|######2   | 31090/50000 [19:58:42<10:01:00,  1.91s/it]
Outer training loop:  62%|######2   | 31091/50000 [19:58:44<9:50:38,  1.87s/it] 
Outer training loop:  62%|######2   | 31092/50000 [19:58:46<9:44:24,  1.85s/it]
Outer training loop:  62%|######2   | 31093/50000 [19:58:48<9:36:24,  1.83s/it]
Outer training loop:  62%|######2   | 31094/50000 [19:58:50<10:05:47,  1.92s/it]
Outer training loop:  62%|######2   | 31095/50000 [19:58:52<9:45:18,  1.86s/it] 
Outer training loop:  62%|######2   | 31096/50000 [19:58:53<9:40:06,  1.84s/it]
Outer training loop:  62%|######2   | 31097/50000 [19:58:56<10:13:54,  1.95s/it]
Outer training loop:  62%|######2   | 31098/50000 [19:58:57<10:11:00,  1.94s/it]
Outer training loop:  62%|######2   | 31099/50000 [19:58:59<10:09:20,  1.93s/it]
Outer training loop:  62%|######2   | 31100/50000 [19:59:01<10:01:08,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  62%|######2   | 31101/50000 [19:59:09<19:06:40,  3.64s/it]
Outer training loop:  62%|######2   | 31102/50000 [19:59:11<16:14:45,  3.09s/it]
Outer training loop:  62%|######2   | 31103/50000 [19:59:13<14:21:45,  2.74s/it]
Outer training loop:  62%|######2   | 31104/50000 [19:59:15<13:00:27,  2.48s/it]
Outer training loop:  62%|######2   | 31105/50000 [19:59:16<12:07:23,  2.31s/it]
Outer training loop:  62%|######2   | 31106/50000 [19:59:18<11:17:34,  2.15s/it]
Outer training loop:  62%|######2   | 31107/50000 [19:59:20<11:02:28,  2.10s/it]
Outer training loop:  62%|######2   | 31108/50000 [19:59:22<10:52:37,  2.07s/it]
Outer training loop:  62%|######2   | 31109/50000 [19:59:24<10:35:33,  2.02s/it]
Outer training loop:  62%|######2   | 31110/50000 [19:59:26<10:25:38,  1.99s/it]
Outer training loop:  62%|######2   | 31111/50000 [19:59:28<10:29:30,  2.00s/it]
Outer training loop:  62%|######2   | 31112/50000 [19:59:30<10:19:38,  1.97s/it]
Outer training loop:  62%|######2   | 31113/50000 [19:59:32<10:12:00,  1.94s/it]
Outer training loop:  62%|######2   | 31114/50000 [19:59:34<9:56:21,  1.89s/it] 
Outer training loop:  62%|######2   | 31115/50000 [19:59:36<10:20:00,  1.97s/it]
Outer training loop:  62%|######2   | 31116/50000 [19:59:38<10:04:49,  1.92s/it]
Outer training loop:  62%|######2   | 31117/50000 [19:59:39<9:52:50,  1.88s/it] 
Outer training loop:  62%|######2   | 31118/50000 [19:59:41<10:05:56,  1.93s/it]
Outer training loop:  62%|######2   | 31119/50000 [19:59:43<10:01:07,  1.91s/it]
Outer training loop:  62%|######2   | 31120/50000 [19:59:45<9:52:06,  1.88s/it] 
Outer training loop:  62%|######2   | 31121/50000 [19:59:47<9:59:17,  1.90s/it]
Outer training loop:  62%|######2   | 31122/50000 [19:59:49<10:03:08,  1.92s/it]
Outer training loop:  62%|######2   | 31123/50000 [19:59:51<10:12:47,  1.95s/it]
Outer training loop:  62%|######2   | 31124/50000 [19:59:53<9:53:44,  1.89s/it] 
Outer training loop:  62%|######2   | 31125/50000 [19:59:55<10:10:34,  1.94s/it]
Outer training loop:  62%|######2   | 31126/50000 [19:59:57<9:53:32,  1.89s/it] 
Outer training loop:  62%|######2   | 31127/50000 [19:59:58<9:42:06,  1.85s/it]
Outer training loop:  62%|######2   | 31128/50000 [20:00:00<10:01:14,  1.91s/it]
Outer training loop:  62%|######2   | 31129/50000 [20:00:02<9:55:39,  1.89s/it] 
Outer training loop:  62%|######2   | 31130/50000 [20:00:04<9:55:01,  1.89s/it]
Outer training loop:  62%|######2   | 31131/50000 [20:00:06<9:55:11,  1.89s/it]
Outer training loop:  62%|######2   | 31132/50000 [20:00:08<10:10:10,  1.94s/it]
Outer training loop:  62%|######2   | 31133/50000 [20:00:10<10:02:51,  1.92s/it]
Outer training loop:  62%|######2   | 31134/50000 [20:00:12<9:56:16,  1.90s/it] 
Outer training loop:  62%|######2   | 31135/50000 [20:00:14<10:10:47,  1.94s/it]
Outer training loop:  62%|######2   | 31136/50000 [20:00:16<10:09:11,  1.94s/it]
Outer training loop:  62%|######2   | 31137/50000 [20:00:18<10:03:41,  1.92s/it]
Outer training loop:  62%|######2   | 31138/50000 [20:00:20<10:35:16,  2.02s/it]
Outer training loop:  62%|######2   | 31139/50000 [20:00:22<10:31:35,  2.01s/it]
Outer training loop:  62%|######2   | 31140/50000 [20:00:24<10:23:31,  1.98s/it]
Outer training loop:  62%|######2   | 31141/50000 [20:00:26<10:25:43,  1.99s/it]
Outer training loop:  62%|######2   | 31142/50000 [20:00:28<10:30:48,  2.01s/it]
Outer training loop:  62%|######2   | 31143/50000 [20:00:30<10:16:15,  1.96s/it]
Outer training loop:  62%|######2   | 31144/50000 [20:00:32<10:03:04,  1.92s/it]
Outer training loop:  62%|######2   | 31145/50000 [20:00:34<10:12:43,  1.95s/it]
Outer training loop:  62%|######2   | 31146/50000 [20:00:35<10:08:10,  1.94s/it]
Outer training loop:  62%|######2   | 31147/50000 [20:00:37<9:59:59,  1.91s/it] 
Outer training loop:  62%|######2   | 31148/50000 [20:00:39<9:59:04,  1.91s/it]
Outer training loop:  62%|######2   | 31149/50000 [20:00:41<10:17:55,  1.97s/it]
Outer training loop:  62%|######2   | 31150/50000 [20:00:43<10:01:22,  1.91s/it]
Outer training loop:  62%|######2   | 31151/50000 [20:00:45<9:48:56,  1.87s/it] 
Outer training loop:  62%|######2   | 31152/50000 [20:00:47<10:01:11,  1.91s/it]
Outer training loop:  62%|######2   | 31153/50000 [20:00:49<10:01:19,  1.91s/it]
Outer training loop:  62%|######2   | 31154/50000 [20:00:51<10:06:00,  1.93s/it]
Outer training loop:  62%|######2   | 31155/50000 [20:00:53<10:15:48,  1.96s/it]
Outer training loop:  62%|######2   | 31156/50000 [20:00:55<10:10:35,  1.94s/it]
Outer training loop:  62%|######2   | 31157/50000 [20:00:57<10:03:34,  1.92s/it]
Outer training loop:  62%|######2   | 31158/50000 [20:00:58<9:59:32,  1.91s/it] 
Outer training loop:  62%|######2   | 31159/50000 [20:01:00<10:06:38,  1.93s/it]
Outer training loop:  62%|######2   | 31160/50000 [20:01:02<10:04:44,  1.93s/it]
Outer training loop:  62%|######2   | 31161/50000 [20:01:04<9:59:23,  1.91s/it] 
Outer training loop:  62%|######2   | 31162/50000 [20:01:06<10:16:30,  1.96s/it]
Outer training loop:  62%|######2   | 31163/50000 [20:01:08<10:08:08,  1.94s/it]
Outer training loop:  62%|######2   | 31164/50000 [20:01:10<10:07:37,  1.94s/it]
Outer training loop:  62%|######2   | 31165/50000 [20:01:12<10:00:24,  1.91s/it]
Outer training loop:  62%|######2   | 31166/50000 [20:01:14<10:29:10,  2.00s/it]
Outer training loop:  62%|######2   | 31167/50000 [20:01:16<10:03:04,  1.92s/it]
Outer training loop:  62%|######2   | 31168/50000 [20:01:18<10:04:06,  1.92s/it]
Outer training loop:  62%|######2   | 31169/50000 [20:01:20<10:23:05,  1.99s/it]
Outer training loop:  62%|######2   | 31170/50000 [20:01:22<10:10:41,  1.95s/it]
Outer training loop:  62%|######2   | 31171/50000 [20:01:24<9:59:06,  1.91s/it] 
Outer training loop:  62%|######2   | 31172/50000 [20:01:26<10:21:28,  1.98s/it]
Outer training loop:  62%|######2   | 31173/50000 [20:01:28<10:17:58,  1.97s/it]
Outer training loop:  62%|######2   | 31174/50000 [20:01:30<10:02:57,  1.92s/it]
Outer training loop:  62%|######2   | 31175/50000 [20:01:32<10:05:49,  1.93s/it]
Outer training loop:  62%|######2   | 31176/50000 [20:01:33<9:45:59,  1.87s/it] 
Outer training loop:  62%|######2   | 31177/50000 [20:01:35<9:55:56,  1.90s/it]
Outer training loop:  62%|######2   | 31178/50000 [20:01:37<9:51:50,  1.89s/it]
Outer training loop:  62%|######2   | 31179/50000 [20:01:39<10:00:35,  1.91s/it]
Outer training loop:  62%|######2   | 31180/50000 [20:01:41<10:05:20,  1.93s/it]
Outer training loop:  62%|######2   | 31181/50000 [20:01:43<10:05:03,  1.93s/it]
Outer training loop:  62%|######2   | 31182/50000 [20:01:45<10:12:27,  1.95s/it]
Outer training loop:  62%|######2   | 31183/50000 [20:01:47<10:03:10,  1.92s/it]
Outer training loop:  62%|######2   | 31184/50000 [20:01:49<10:03:54,  1.93s/it]
Outer training loop:  62%|######2   | 31185/50000 [20:01:51<9:57:42,  1.91s/it] 
Outer training loop:  62%|######2   | 31186/50000 [20:01:53<10:14:40,  1.96s/it]
Outer training loop:  62%|######2   | 31187/50000 [20:01:55<10:06:31,  1.93s/it]
Outer training loop:  62%|######2   | 31188/50000 [20:01:56<10:00:47,  1.92s/it]
Outer training loop:  62%|######2   | 31189/50000 [20:01:59<10:25:17,  1.99s/it]
Outer training loop:  62%|######2   | 31190/50000 [20:02:00<10:10:44,  1.95s/it]
Outer training loop:  62%|######2   | 31191/50000 [20:02:02<10:05:08,  1.93s/it]
Outer training loop:  62%|######2   | 31192/50000 [20:02:04<9:48:43,  1.88s/it] 
Outer training loop:  62%|######2   | 31193/50000 [20:02:06<10:08:22,  1.94s/it]
Outer training loop:  62%|######2   | 31194/50000 [20:02:08<10:00:44,  1.92s/it]
Outer training loop:  62%|######2   | 31195/50000 [20:02:10<10:09:38,  1.95s/it]
Outer training loop:  62%|######2   | 31196/50000 [20:02:12<9:47:58,  1.88s/it] 
Outer training loop:  62%|######2   | 31197/50000 [20:02:14<9:55:26,  1.90s/it]
Outer training loop:  62%|######2   | 31198/50000 [20:02:16<9:50:13,  1.88s/it]
Outer training loop:  62%|######2   | 31199/50000 [20:02:18<10:04:10,  1.93s/it]
Outer training loop:  62%|######2   | 31200/50000 [20:02:19<9:56:00,  1.90s/it] 

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.01s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.02s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.93s/it]

Outer training loop:  62%|######2   | 31201/50000 [20:02:27<18:54:19,  3.62s/it]
Outer training loop:  62%|######2   | 31202/50000 [20:02:29<16:39:22,  3.19s/it]
Outer training loop:  62%|######2   | 31203/50000 [20:02:31<14:35:01,  2.79s/it]
Outer training loop:  62%|######2   | 31204/50000 [20:02:33<13:06:43,  2.51s/it]
Outer training loop:  62%|######2   | 31205/50000 [20:02:35<12:22:18,  2.37s/it]
Outer training loop:  62%|######2   | 31206/50000 [20:02:37<11:39:38,  2.23s/it]
Outer training loop:  62%|######2   | 31207/50000 [20:02:39<10:45:40,  2.06s/it]
Outer training loop:  62%|######2   | 31208/50000 [20:02:40<10:23:53,  1.99s/it]
Outer training loop:  62%|######2   | 31209/50000 [20:02:42<10:25:35,  2.00s/it]
Outer training loop:  62%|######2   | 31210/50000 [20:02:44<10:13:32,  1.96s/it]
Outer training loop:  62%|######2   | 31211/50000 [20:02:46<9:59:52,  1.92s/it] 
Outer training loop:  62%|######2   | 31212/50000 [20:02:48<10:15:41,  1.97s/it]
Outer training loop:  62%|######2   | 31213/50000 [20:02:50<10:10:12,  1.95s/it]
Outer training loop:  62%|######2   | 31214/50000 [20:02:52<10:01:51,  1.92s/it]
Outer training loop:  62%|######2   | 31215/50000 [20:02:54<10:06:58,  1.94s/it]
Outer training loop:  62%|######2   | 31216/50000 [20:02:56<9:52:53,  1.89s/it] 
Outer training loop:  62%|######2   | 31217/50000 [20:02:58<9:58:43,  1.91s/it]
Outer training loop:  62%|######2   | 31218/50000 [20:03:00<9:56:28,  1.91s/it]
Outer training loop:  62%|######2   | 31219/50000 [20:03:02<10:06:17,  1.94s/it]
Outer training loop:  62%|######2   | 31220/50000 [20:03:04<10:13:35,  1.96s/it]
Outer training loop:  62%|######2   | 31221/50000 [20:03:06<10:19:43,  1.98s/it]
Outer training loop:  62%|######2   | 31222/50000 [20:03:08<10:27:01,  2.00s/it]
Outer training loop:  62%|######2   | 31223/50000 [20:03:09<10:03:28,  1.93s/it]
Outer training loop:  62%|######2   | 31224/50000 [20:03:11<9:55:48,  1.90s/it] 
Outer training loop:  62%|######2   | 31225/50000 [20:03:13<9:53:15,  1.90s/it]
Outer training loop:  62%|######2   | 31226/50000 [20:03:15<10:13:51,  1.96s/it]
Outer training loop:  62%|######2   | 31227/50000 [20:03:17<10:22:03,  1.99s/it]
Outer training loop:  62%|######2   | 31228/50000 [20:03:19<10:17:44,  1.97s/it]
Outer training loop:  62%|######2   | 31229/50000 [20:03:21<10:16:34,  1.97s/it]
Outer training loop:  62%|######2   | 31230/50000 [20:03:23<10:07:23,  1.94s/it]
Outer training loop:  62%|######2   | 31231/50000 [20:03:25<10:21:31,  1.99s/it]
Outer training loop:  62%|######2   | 31232/50000 [20:03:27<10:12:08,  1.96s/it]
Outer training loop:  62%|######2   | 31233/50000 [20:03:29<10:01:08,  1.92s/it]
Outer training loop:  62%|######2   | 31234/50000 [20:03:31<9:50:33,  1.89s/it] 
Outer training loop:  62%|######2   | 31235/50000 [20:03:33<9:59:10,  1.92s/it]
Outer training loop:  62%|######2   | 31236/50000 [20:03:35<10:13:12,  1.96s/it]
Outer training loop:  62%|######2   | 31237/50000 [20:03:37<10:00:29,  1.92s/it]
Outer training loop:  62%|######2   | 31238/50000 [20:03:38<9:48:51,  1.88s/it] 
Outer training loop:  62%|######2   | 31239/50000 [20:03:40<9:55:25,  1.90s/it]
Outer training loop:  62%|######2   | 31240/50000 [20:03:42<9:52:58,  1.90s/it]
Outer training loop:  62%|######2   | 31241/50000 [20:03:44<9:48:11,  1.88s/it]
Outer training loop:  62%|######2   | 31242/50000 [20:03:46<9:55:48,  1.91s/it]
Outer training loop:  62%|######2   | 31243/50000 [20:03:48<9:48:00,  1.88s/it]
Outer training loop:  62%|######2   | 31244/50000 [20:03:50<9:57:16,  1.91s/it]
Outer training loop:  62%|######2   | 31245/50000 [20:03:52<9:57:36,  1.91s/it]
Outer training loop:  62%|######2   | 31246/50000 [20:03:54<10:21:39,  1.99s/it]
Outer training loop:  62%|######2   | 31247/50000 [20:03:56<10:07:01,  1.94s/it]
Outer training loop:  62%|######2   | 31248/50000 [20:03:58<9:59:36,  1.92s/it] 
Outer training loop:  62%|######2   | 31249/50000 [20:04:00<10:05:48,  1.94s/it]
Outer training loop:  62%|######2   | 31250/50000 [20:04:01<9:51:55,  1.89s/it] 
Outer training loop:  63%|######2   | 31251/50000 [20:04:03<9:49:12,  1.89s/it]
Outer training loop:  63%|######2   | 31252/50000 [20:04:05<9:50:20,  1.89s/it]
Outer training loop:  63%|######2   | 31253/50000 [20:04:07<10:12:13,  1.96s/it]
Outer training loop:  63%|######2   | 31254/50000 [20:04:09<9:56:34,  1.91s/it] 
Outer training loop:  63%|######2   | 31255/50000 [20:04:11<9:51:52,  1.89s/it]
Outer training loop:  63%|######2   | 31256/50000 [20:04:13<10:01:21,  1.92s/it]
Outer training loop:  63%|######2   | 31257/50000 [20:04:15<9:48:08,  1.88s/it] 
Outer training loop:  63%|######2   | 31258/50000 [20:04:17<9:49:13,  1.89s/it]
Outer training loop:  63%|######2   | 31259/50000 [20:04:19<10:03:16,  1.93s/it]
Outer training loop:  63%|######2   | 31260/50000 [20:04:20<9:53:19,  1.90s/it] 
Outer training loop:  63%|######2   | 31261/50000 [20:04:22<9:48:17,  1.88s/it]
Outer training loop:  63%|######2   | 31262/50000 [20:04:24<9:37:41,  1.85s/it]
Outer training loop:  63%|######2   | 31263/50000 [20:04:26<9:47:51,  1.88s/it]
Outer training loop:  63%|######2   | 31264/50000 [20:04:28<9:53:22,  1.90s/it]
Outer training loop:  63%|######2   | 31265/50000 [20:04:30<9:46:44,  1.88s/it]
Outer training loop:  63%|######2   | 31266/50000 [20:04:32<10:03:23,  1.93s/it]
Outer training loop:  63%|######2   | 31267/50000 [20:04:34<9:39:24,  1.86s/it] 
Outer training loop:  63%|######2   | 31268/50000 [20:04:35<9:38:44,  1.85s/it]
Outer training loop:  63%|######2   | 31269/50000 [20:04:37<9:32:06,  1.83s/it]
Outer training loop:  63%|######2   | 31270/50000 [20:04:39<10:01:23,  1.93s/it]
Outer training loop:  63%|######2   | 31271/50000 [20:04:41<9:53:17,  1.90s/it] 
Outer training loop:  63%|######2   | 31272/50000 [20:04:43<9:41:59,  1.86s/it]
Outer training loop:  63%|######2   | 31273/50000 [20:04:45<9:56:40,  1.91s/it]
Outer training loop:  63%|######2   | 31274/50000 [20:04:47<9:41:52,  1.86s/it]
Outer training loop:  63%|######2   | 31275/50000 [20:04:49<9:34:37,  1.84s/it]
Outer training loop:  63%|######2   | 31276/50000 [20:04:51<10:06:35,  1.94s/it]
Outer training loop:  63%|######2   | 31277/50000 [20:04:52<9:51:23,  1.90s/it] 
Outer training loop:  63%|######2   | 31278/50000 [20:04:54<9:45:37,  1.88s/it]
Outer training loop:  63%|######2   | 31279/50000 [20:04:56<9:57:40,  1.92s/it]
Outer training loop:  63%|######2   | 31280/50000 [20:04:58<9:43:50,  1.87s/it]
Outer training loop:  63%|######2   | 31281/50000 [20:05:00<9:44:18,  1.87s/it]
Outer training loop:  63%|######2   | 31282/50000 [20:05:02<9:45:24,  1.88s/it]
Outer training loop:  63%|######2   | 31283/50000 [20:05:04<10:05:50,  1.94s/it]
Outer training loop:  63%|######2   | 31284/50000 [20:05:06<9:55:21,  1.91s/it] 
Outer training loop:  63%|######2   | 31285/50000 [20:05:08<9:42:07,  1.87s/it]
Outer training loop:  63%|######2   | 31286/50000 [20:05:10<9:56:44,  1.91s/it]
Outer training loop:  63%|######2   | 31287/50000 [20:05:11<9:44:31,  1.87s/it]
Outer training loop:  63%|######2   | 31288/50000 [20:05:13<9:46:00,  1.88s/it]
Outer training loop:  63%|######2   | 31289/50000 [20:05:15<10:02:59,  1.93s/it]
Outer training loop:  63%|######2   | 31290/50000 [20:05:17<10:01:54,  1.93s/it]
Outer training loop:  63%|######2   | 31291/50000 [20:05:19<9:50:56,  1.90s/it] 
Outer training loop:  63%|######2   | 31292/50000 [20:05:21<9:44:28,  1.87s/it]
Outer training loop:  63%|######2   | 31293/50000 [20:05:23<9:59:53,  1.92s/it]
Outer training loop:  63%|######2   | 31294/50000 [20:05:25<9:48:50,  1.89s/it]
Outer training loop:  63%|######2   | 31295/50000 [20:05:27<9:46:37,  1.88s/it]
Outer training loop:  63%|######2   | 31296/50000 [20:05:29<9:52:47,  1.90s/it]
Outer training loop:  63%|######2   | 31297/50000 [20:05:30<9:43:46,  1.87s/it]
Outer training loop:  63%|######2   | 31298/50000 [20:05:32<9:43:16,  1.87s/it]
Outer training loop:  63%|######2   | 31299/50000 [20:05:34<10:00:16,  1.93s/it]
Outer training loop:  63%|######2   | 31300/50000 [20:05:36<9:44:50,  1.88s/it] 

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  63%|######2   | 31301/50000 [20:05:44<18:30:08,  3.56s/it]
Outer training loop:  63%|######2   | 31302/50000 [20:05:45<15:43:46,  3.03s/it]
Outer training loop:  63%|######2   | 31303/50000 [20:05:47<13:51:17,  2.67s/it]
Outer training loop:  63%|######2   | 31304/50000 [20:05:49<12:34:03,  2.42s/it]
Outer training loop:  63%|######2   | 31305/50000 [20:05:51<11:51:04,  2.28s/it]
Outer training loop:  63%|######2   | 31306/50000 [20:05:53<11:06:21,  2.14s/it]
Outer training loop:  63%|######2   | 31307/50000 [20:05:55<11:01:21,  2.12s/it]
Outer training loop:  63%|######2   | 31308/50000 [20:05:57<10:36:54,  2.04s/it]
Outer training loop:  63%|######2   | 31309/50000 [20:05:59<10:22:02,  2.00s/it]
Outer training loop:  63%|######2   | 31310/50000 [20:06:01<10:18:47,  1.99s/it]
Outer training loop:  63%|######2   | 31311/50000 [20:06:02<10:08:39,  1.95s/it]
Outer training loop:  63%|######2   | 31312/50000 [20:06:04<9:53:50,  1.91s/it] 
Outer training loop:  63%|######2   | 31313/50000 [20:06:06<9:50:10,  1.89s/it]
Outer training loop:  63%|######2   | 31314/50000 [20:06:08<9:57:24,  1.92s/it]
Outer training loop:  63%|######2   | 31315/50000 [20:06:10<9:54:14,  1.91s/it]
Outer training loop:  63%|######2   | 31316/50000 [20:06:12<9:42:27,  1.87s/it]
Outer training loop:  63%|######2   | 31317/50000 [20:06:14<10:16:36,  1.98s/it]
Outer training loop:  63%|######2   | 31318/50000 [20:06:16<9:59:44,  1.93s/it] 
Outer training loop:  63%|######2   | 31319/50000 [20:06:18<9:47:57,  1.89s/it]
Outer training loop:  63%|######2   | 31320/50000 [20:06:20<10:00:42,  1.93s/it]
Outer training loop:  63%|######2   | 31321/50000 [20:06:21<9:45:51,  1.88s/it] 
Outer training loop:  63%|######2   | 31322/50000 [20:06:23<9:36:45,  1.85s/it]
Outer training loop:  63%|######2   | 31323/50000 [20:06:25<9:50:35,  1.90s/it]
Outer training loop:  63%|######2   | 31324/50000 [20:06:27<9:42:42,  1.87s/it]
Outer training loop:  63%|######2   | 31325/50000 [20:06:29<9:40:26,  1.86s/it]
Outer training loop:  63%|######2   | 31326/50000 [20:06:31<9:33:53,  1.84s/it]
Outer training loop:  63%|######2   | 31327/50000 [20:06:33<9:56:21,  1.92s/it]
Outer training loop:  63%|######2   | 31328/50000 [20:06:34<9:39:58,  1.86s/it]
Outer training loop:  63%|######2   | 31329/50000 [20:06:36<9:32:26,  1.84s/it]
Outer training loop:  63%|######2   | 31330/50000 [20:06:38<9:53:23,  1.91s/it]
Outer training loop:  63%|######2   | 31331/50000 [20:06:40<9:48:37,  1.89s/it]
Outer training loop:  63%|######2   | 31332/50000 [20:06:42<9:47:31,  1.89s/it]
Outer training loop:  63%|######2   | 31333/50000 [20:06:44<10:07:27,  1.95s/it]
Outer training loop:  63%|######2   | 31334/50000 [20:06:46<9:57:02,  1.92s/it] 
Outer training loop:  63%|######2   | 31335/50000 [20:06:48<9:46:30,  1.89s/it]
Outer training loop:  63%|######2   | 31336/50000 [20:06:50<9:49:28,  1.89s/it]
Outer training loop:  63%|######2   | 31337/50000 [20:06:52<9:51:04,  1.90s/it]
Outer training loop:  63%|######2   | 31338/50000 [20:06:53<9:44:08,  1.88s/it]
Outer training loop:  63%|######2   | 31339/50000 [20:06:55<9:38:11,  1.86s/it]
Outer training loop:  63%|######2   | 31340/50000 [20:06:57<9:50:55,  1.90s/it]
Outer training loop:  63%|######2   | 31341/50000 [20:06:59<9:34:06,  1.85s/it]
Outer training loop:  63%|######2   | 31342/50000 [20:07:01<9:28:05,  1.83s/it]
Outer training loop:  63%|######2   | 31343/50000 [20:07:03<9:35:00,  1.85s/it]
Outer training loop:  63%|######2   | 31344/50000 [20:07:04<9:30:31,  1.83s/it]
Outer training loop:  63%|######2   | 31345/50000 [20:07:06<9:38:47,  1.86s/it]
Outer training loop:  63%|######2   | 31346/50000 [20:07:08<9:31:54,  1.84s/it]
Outer training loop:  63%|######2   | 31347/50000 [20:07:10<9:40:30,  1.87s/it]
Outer training loop:  63%|######2   | 31348/50000 [20:07:12<9:36:20,  1.85s/it]
Outer training loop:  63%|######2   | 31349/50000 [20:07:14<9:27:52,  1.83s/it]
Outer training loop:  63%|######2   | 31350/50000 [20:07:16<9:44:31,  1.88s/it]
Outer training loop:  63%|######2   | 31351/50000 [20:07:17<9:32:54,  1.84s/it]
Outer training loop:  63%|######2   | 31352/50000 [20:07:19<9:33:15,  1.84s/it]
Outer training loop:  63%|######2   | 31353/50000 [20:07:21<9:26:57,  1.82s/it]
Outer training loop:  63%|######2   | 31354/50000 [20:07:23<9:39:37,  1.87s/it]
Outer training loop:  63%|######2   | 31355/50000 [20:07:25<9:25:13,  1.82s/it]
Outer training loop:  63%|######2   | 31356/50000 [20:07:26<9:15:23,  1.79s/it]
Outer training loop:  63%|######2   | 31357/50000 [20:07:28<9:39:24,  1.86s/it]
Outer training loop:  63%|######2   | 31358/50000 [20:07:30<9:31:39,  1.84s/it]
Outer training loop:  63%|######2   | 31359/50000 [20:07:32<9:28:15,  1.83s/it]
Outer training loop:  63%|######2   | 31360/50000 [20:07:34<9:38:25,  1.86s/it]
Outer training loop:  63%|######2   | 31361/50000 [20:07:36<9:56:36,  1.92s/it]
Outer training loop:  63%|######2   | 31362/50000 [20:07:38<9:48:56,  1.90s/it]
Outer training loop:  63%|######2   | 31363/50000 [20:07:40<9:39:02,  1.86s/it]
Outer training loop:  63%|######2   | 31364/50000 [20:07:42<9:45:23,  1.88s/it]
Outer training loop:  63%|######2   | 31365/50000 [20:07:43<9:35:07,  1.85s/it]
Outer training loop:  63%|######2   | 31366/50000 [20:07:45<9:35:37,  1.85s/it]
Outer training loop:  63%|######2   | 31367/50000 [20:07:47<9:52:53,  1.91s/it]
Outer training loop:  63%|######2   | 31368/50000 [20:07:49<10:10:53,  1.97s/it]
Outer training loop:  63%|######2   | 31369/50000 [20:07:51<9:48:24,  1.89s/it] 
Outer training loop:  63%|######2   | 31370/50000 [20:07:53<9:49:34,  1.90s/it]
Outer training loop:  63%|######2   | 31371/50000 [20:07:55<10:15:19,  1.98s/it]
Outer training loop:  63%|######2   | 31372/50000 [20:07:57<10:01:04,  1.94s/it]
Outer training loop:  63%|######2   | 31373/50000 [20:07:59<9:47:27,  1.89s/it] 
Outer training loop:  63%|######2   | 31374/50000 [20:08:01<9:38:03,  1.86s/it]
Outer training loop:  63%|######2   | 31375/50000 [20:08:03<9:42:58,  1.88s/it]
Outer training loop:  63%|######2   | 31376/50000 [20:08:04<9:46:59,  1.89s/it]
Outer training loop:  63%|######2   | 31377/50000 [20:08:06<9:43:55,  1.88s/it]
Outer training loop:  63%|######2   | 31378/50000 [20:08:08<9:51:04,  1.90s/it]
Outer training loop:  63%|######2   | 31379/50000 [20:08:10<9:53:25,  1.91s/it]
Outer training loop:  63%|######2   | 31380/50000 [20:08:12<9:30:53,  1.84s/it]
Outer training loop:  63%|######2   | 31381/50000 [20:08:14<9:50:02,  1.90s/it]
Outer training loop:  63%|######2   | 31382/50000 [20:08:16<9:40:58,  1.87s/it]
Outer training loop:  63%|######2   | 31383/50000 [20:08:18<9:39:07,  1.87s/it]
Outer training loop:  63%|######2   | 31384/50000 [20:08:20<9:55:29,  1.92s/it]
Outer training loop:  63%|######2   | 31385/50000 [20:08:21<9:45:40,  1.89s/it]
Outer training loop:  63%|######2   | 31386/50000 [20:08:23<9:37:24,  1.86s/it]
Outer training loop:  63%|######2   | 31387/50000 [20:08:25<9:47:11,  1.89s/it]
Outer training loop:  63%|######2   | 31388/50000 [20:08:27<10:16:54,  1.99s/it]
Outer training loop:  63%|######2   | 31389/50000 [20:08:29<10:04:31,  1.95s/it]
Outer training loop:  63%|######2   | 31390/50000 [20:08:31<9:51:12,  1.91s/it] 
Outer training loop:  63%|######2   | 31391/50000 [20:08:33<9:55:37,  1.92s/it]
Outer training loop:  63%|######2   | 31392/50000 [20:08:35<10:04:32,  1.95s/it]
Outer training loop:  63%|######2   | 31393/50000 [20:08:37<10:02:56,  1.94s/it]
Outer training loop:  63%|######2   | 31394/50000 [20:08:39<9:53:01,  1.91s/it] 
Outer training loop:  63%|######2   | 31395/50000 [20:08:41<10:00:00,  1.93s/it]
Outer training loop:  63%|######2   | 31396/50000 [20:08:43<9:57:37,  1.93s/it] 
Outer training loop:  63%|######2   | 31397/50000 [20:08:44<9:37:28,  1.86s/it]
Outer training loop:  63%|######2   | 31398/50000 [20:08:46<9:35:57,  1.86s/it]
Outer training loop:  63%|######2   | 31399/50000 [20:08:48<9:34:08,  1.85s/it]
Outer training loop:  63%|######2   | 31400/50000 [20:08:50<9:42:30,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  2.00s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  63%|######2   | 31401/50000 [20:08:57<18:04:19,  3.50s/it]
Outer training loop:  63%|######2   | 31402/50000 [20:08:59<15:33:09,  3.01s/it]
Outer training loop:  63%|######2   | 31403/50000 [20:09:01<13:59:16,  2.71s/it]
Outer training loop:  63%|######2   | 31404/50000 [20:09:03<12:48:08,  2.48s/it]
Outer training loop:  63%|######2   | 31405/50000 [20:09:05<11:57:03,  2.31s/it]
Outer training loop:  63%|######2   | 31406/50000 [20:09:07<11:32:12,  2.23s/it]
Outer training loop:  63%|######2   | 31407/50000 [20:09:09<10:51:03,  2.10s/it]
Outer training loop:  63%|######2   | 31408/50000 [20:09:11<10:33:04,  2.04s/it]
Outer training loop:  63%|######2   | 31409/50000 [20:09:13<10:09:08,  1.97s/it]
Outer training loop:  63%|######2   | 31410/50000 [20:09:15<10:26:45,  2.02s/it]
Outer training loop:  63%|######2   | 31411/50000 [20:09:16<10:01:40,  1.94s/it]
Outer training loop:  63%|######2   | 31412/50000 [20:09:18<9:46:57,  1.89s/it] 
Outer training loop:  63%|######2   | 31413/50000 [20:09:20<10:02:51,  1.95s/it]
Outer training loop:  63%|######2   | 31414/50000 [20:09:22<10:21:01,  2.00s/it]
Outer training loop:  63%|######2   | 31415/50000 [20:09:24<10:00:19,  1.94s/it]
Outer training loop:  63%|######2   | 31416/50000 [20:09:26<10:04:08,  1.95s/it]
Outer training loop:  63%|######2   | 31417/50000 [20:09:28<9:58:59,  1.93s/it] 
Outer training loop:  63%|######2   | 31418/50000 [20:09:30<10:02:53,  1.95s/it]
Outer training loop:  63%|######2   | 31419/50000 [20:09:32<10:01:59,  1.94s/it]
Outer training loop:  63%|######2   | 31420/50000 [20:09:34<10:19:26,  2.00s/it]
Outer training loop:  63%|######2   | 31421/50000 [20:09:36<10:16:27,  1.99s/it]
Outer training loop:  63%|######2   | 31422/50000 [20:09:38<9:58:25,  1.93s/it] 
Outer training loop:  63%|######2   | 31423/50000 [20:09:40<10:08:40,  1.97s/it]
Outer training loop:  63%|######2   | 31424/50000 [20:09:42<9:56:21,  1.93s/it] 
Outer training loop:  63%|######2   | 31425/50000 [20:09:44<9:56:08,  1.93s/it]
Outer training loop:  63%|######2   | 31426/50000 [20:09:46<10:03:40,  1.95s/it]
Outer training loop:  63%|######2   | 31427/50000 [20:09:48<10:16:10,  1.99s/it]
Outer training loop:  63%|######2   | 31428/50000 [20:09:50<10:09:34,  1.97s/it]
Outer training loop:  63%|######2   | 31429/50000 [20:09:52<9:57:27,  1.93s/it] 
Outer training loop:  63%|######2   | 31430/50000 [20:09:54<10:08:52,  1.97s/it]
Outer training loop:  63%|######2   | 31431/50000 [20:09:56<9:58:10,  1.93s/it] 
Outer training loop:  63%|######2   | 31432/50000 [20:09:57<9:51:02,  1.91s/it]
Outer training loop:  63%|######2   | 31433/50000 [20:09:59<10:01:29,  1.94s/it]
Outer training loop:  63%|######2   | 31434/50000 [20:10:01<9:49:50,  1.91s/it] 
Outer training loop:  63%|######2   | 31435/50000 [20:10:03<9:37:05,  1.87s/it]
Outer training loop:  63%|######2   | 31436/50000 [20:10:05<9:45:05,  1.89s/it]
Outer training loop:  63%|######2   | 31437/50000 [20:10:07<10:02:59,  1.95s/it]
Outer training loop:  63%|######2   | 31438/50000 [20:10:09<9:56:01,  1.93s/it] 
Outer training loop:  63%|######2   | 31439/50000 [20:10:11<9:43:09,  1.89s/it]
Outer training loop:  63%|######2   | 31440/50000 [20:10:13<9:42:19,  1.88s/it]
Outer training loop:  63%|######2   | 31441/50000 [20:10:14<9:46:48,  1.90s/it]
Outer training loop:  63%|######2   | 31442/50000 [20:10:16<9:33:03,  1.85s/it]
Outer training loop:  63%|######2   | 31443/50000 [20:10:18<9:35:42,  1.86s/it]
Outer training loop:  63%|######2   | 31444/50000 [20:10:20<9:58:33,  1.94s/it]
Outer training loop:  63%|######2   | 31445/50000 [20:10:22<9:41:59,  1.88s/it]
Outer training loop:  63%|######2   | 31446/50000 [20:10:24<9:40:09,  1.88s/it]
Outer training loop:  63%|######2   | 31447/50000 [20:10:26<9:42:28,  1.88s/it]
Outer training loop:  63%|######2   | 31448/50000 [20:10:28<9:58:13,  1.93s/it]
Outer training loop:  63%|######2   | 31449/50000 [20:10:30<9:50:16,  1.91s/it]
Outer training loop:  63%|######2   | 31450/50000 [20:10:31<9:41:36,  1.88s/it]
Outer training loop:  63%|######2   | 31451/50000 [20:10:33<9:38:53,  1.87s/it]
Outer training loop:  63%|######2   | 31452/50000 [20:10:35<9:52:51,  1.92s/it]
Outer training loop:  63%|######2   | 31453/50000 [20:10:37<9:39:47,  1.88s/it]
Outer training loop:  63%|######2   | 31454/50000 [20:10:39<9:30:38,  1.85s/it]
Outer training loop:  63%|######2   | 31455/50000 [20:10:41<9:41:53,  1.88s/it]
Outer training loop:  63%|######2   | 31456/50000 [20:10:43<9:36:34,  1.87s/it]
Outer training loop:  63%|######2   | 31457/50000 [20:10:45<9:34:03,  1.86s/it]
Outer training loop:  63%|######2   | 31458/50000 [20:10:47<9:47:59,  1.90s/it]
Outer training loop:  63%|######2   | 31459/50000 [20:10:48<9:33:18,  1.86s/it]
Outer training loop:  63%|######2   | 31460/50000 [20:10:50<9:45:25,  1.89s/it]
Outer training loop:  63%|######2   | 31461/50000 [20:10:52<9:39:36,  1.88s/it]
Outer training loop:  63%|######2   | 31462/50000 [20:10:54<10:08:28,  1.97s/it]
Outer training loop:  63%|######2   | 31463/50000 [20:10:56<9:48:53,  1.91s/it] 
Outer training loop:  63%|######2   | 31464/50000 [20:10:58<9:57:39,  1.93s/it]
Outer training loop:  63%|######2   | 31465/50000 [20:11:00<10:03:08,  1.95s/it]
Outer training loop:  63%|######2   | 31466/50000 [20:11:02<9:50:05,  1.91s/it] 
Outer training loop:  63%|######2   | 31467/50000 [20:11:04<9:38:51,  1.87s/it]
Outer training loop:  63%|######2   | 31468/50000 [20:11:06<9:51:34,  1.92s/it]
Outer training loop:  63%|######2   | 31469/50000 [20:11:07<9:30:51,  1.85s/it]
Outer training loop:  63%|######2   | 31470/50000 [20:11:09<9:39:01,  1.87s/it]
Outer training loop:  63%|######2   | 31471/50000 [20:11:11<9:31:39,  1.85s/it]
Outer training loop:  63%|######2   | 31472/50000 [20:11:13<9:46:15,  1.90s/it]
Outer training loop:  63%|######2   | 31473/50000 [20:11:15<9:34:33,  1.86s/it]
Outer training loop:  63%|######2   | 31474/50000 [20:11:17<9:38:25,  1.87s/it]
Outer training loop:  63%|######2   | 31475/50000 [20:11:19<9:49:12,  1.91s/it]
Outer training loop:  63%|######2   | 31476/50000 [20:11:21<9:39:06,  1.88s/it]
Outer training loop:  63%|######2   | 31477/50000 [20:11:22<9:35:14,  1.86s/it]
Outer training loop:  63%|######2   | 31478/50000 [20:11:24<9:54:13,  1.92s/it]
Outer training loop:  63%|######2   | 31479/50000 [20:11:26<9:45:53,  1.90s/it]
Outer training loop:  63%|######2   | 31480/50000 [20:11:28<9:42:31,  1.89s/it]
Outer training loop:  63%|######2   | 31481/50000 [20:11:30<9:56:04,  1.93s/it]
Outer training loop:  63%|######2   | 31482/50000 [20:11:32<9:42:52,  1.89s/it]
Outer training loop:  63%|######2   | 31483/50000 [20:11:34<9:38:24,  1.87s/it]
Outer training loop:  63%|######2   | 31484/50000 [20:11:36<9:30:22,  1.85s/it]
Outer training loop:  63%|######2   | 31485/50000 [20:11:38<9:38:48,  1.88s/it]
Outer training loop:  63%|######2   | 31486/50000 [20:11:39<9:33:47,  1.86s/it]
Outer training loop:  63%|######2   | 31487/50000 [20:11:41<9:38:32,  1.88s/it]
Outer training loop:  63%|######2   | 31488/50000 [20:11:43<9:52:15,  1.92s/it]
Outer training loop:  63%|######2   | 31489/50000 [20:11:45<9:43:41,  1.89s/it]
Outer training loop:  63%|######2   | 31490/50000 [20:11:47<9:48:26,  1.91s/it]
Outer training loop:  63%|######2   | 31491/50000 [20:11:49<9:49:03,  1.91s/it]
Outer training loop:  63%|######2   | 31492/50000 [20:11:51<9:59:13,  1.94s/it]
Outer training loop:  63%|######2   | 31493/50000 [20:11:53<9:42:48,  1.89s/it]
Outer training loop:  63%|######2   | 31494/50000 [20:11:55<9:44:41,  1.90s/it]
Outer training loop:  63%|######2   | 31495/50000 [20:11:57<9:48:04,  1.91s/it]
Outer training loop:  63%|######2   | 31496/50000 [20:11:58<9:38:54,  1.88s/it]
Outer training loop:  63%|######2   | 31497/50000 [20:12:00<9:26:53,  1.84s/it]
Outer training loop:  63%|######2   | 31498/50000 [20:12:02<9:38:55,  1.88s/it]
Outer training loop:  63%|######2   | 31499/50000 [20:12:04<9:30:41,  1.85s/it]
Outer training loop:  63%|######3   | 31500/50000 [20:12:06<9:25:10,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.79s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:40,  1.96s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:33,  1.90s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:29,  1.87s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.89s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.87s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:25,  1.94s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:21,  1.95s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:20,  1.98s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.88s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:06,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:02,  1.83s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:59,  1.84s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:58,  1.94s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:46,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:51,  2.00s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:48,  1.98s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:38,  1.90s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:41,  1.98s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:37,  1.95s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:32,  1.89s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:23,  1.90s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.85s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:14,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:11,  1.84s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<00:58,  1.78s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:57,  1.81s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.93s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.85s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:43,  1.82s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:46,  2.01s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:43,  1.96s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:40,  1.92s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.88s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.81s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:28,  1.80s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:29,  1.95s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.89s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.86s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  2.00s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.94s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.93s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.89s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:24,  1.76s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.75s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:20,  1.76s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:34,  1.95s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.91s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.87s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.89s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:25,  1.94s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:19,  1.89s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.88s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:15,  1.91s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:08,  1.86s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:05,  1.85s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.94s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:00,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:02,  1.94s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.86s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:51,  1.93s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.85s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:40,  1.93s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.90s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:33,  2.00s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:28,  1.93s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:21,  1.91s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.87s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:18,  1.97s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:15,  1.94s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:08,  1.86s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.83s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.94s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:50,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:51,  1.97s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:48,  1.95s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.88s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:38,  1.84s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.82s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:34,  1.94s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:32,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:21,  1.82s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:17,  1.96s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.92s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.89s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.91s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.87s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.87s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.86s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.81s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  2.04s/it]

Outer training loop:  63%|######3   | 31501/50000 [20:17:26<500:36:23, 97.42s/it]
Outer training loop:  63%|######3   | 31502/50000 [20:17:28<353:16:22, 68.75s/it]
Outer training loop:  63%|######3   | 31503/50000 [20:17:30<249:57:50, 48.65s/it]
Outer training loop:  63%|######3   | 31504/50000 [20:17:32<177:51:32, 34.62s/it]
Outer training loop:  63%|######3   | 31505/50000 [20:17:34<127:25:42, 24.80s/it]
Outer training loop:  63%|######3   | 31506/50000 [20:17:35<92:01:00, 17.91s/it] 
Outer training loop:  63%|######3   | 31507/50000 [20:17:37<67:23:54, 13.12s/it]
Outer training loop:  63%|######3   | 31508/50000 [20:17:39<49:55:47,  9.72s/it]
Outer training loop:  63%|######3   | 31509/50000 [20:17:41<37:52:20,  7.37s/it]
Outer training loop:  63%|######3   | 31510/50000 [20:17:43<29:23:27,  5.72s/it]
Outer training loop:  63%|######3   | 31511/50000 [20:17:45<23:29:56,  4.58s/it]
Outer training loop:  63%|######3   | 31512/50000 [20:17:47<20:23:21,  3.97s/it]
Outer training loop:  63%|######3   | 31513/50000 [20:17:49<17:02:58,  3.32s/it]
Outer training loop:  63%|######3   | 31514/50000 [20:17:51<14:46:18,  2.88s/it]
Outer training loop:  63%|######3   | 31515/50000 [20:17:53<13:13:05,  2.57s/it]
Outer training loop:  63%|######3   | 31516/50000 [20:17:55<12:00:26,  2.34s/it]
Outer training loop:  63%|######3   | 31517/50000 [20:17:57<11:16:25,  2.20s/it]
Outer training loop:  63%|######3   | 31518/50000 [20:17:58<10:47:10,  2.10s/it]
Outer training loop:  63%|######3   | 31519/50000 [20:18:00<10:39:45,  2.08s/it]
Outer training loop:  63%|######3   | 31520/50000 [20:18:02<10:22:50,  2.02s/it]
Outer training loop:  63%|######3   | 31521/50000 [20:18:04<10:05:15,  1.97s/it]
Outer training loop:  63%|######3   | 31522/50000 [20:18:06<10:07:57,  1.97s/it]
Outer training loop:  63%|######3   | 31523/50000 [20:18:08<9:53:14,  1.93s/it] 
Outer training loop:  63%|######3   | 31524/50000 [20:18:10<9:45:01,  1.90s/it]
Outer training loop:  63%|######3   | 31525/50000 [20:18:12<9:52:53,  1.93s/it]
Outer training loop:  63%|######3   | 31526/50000 [20:18:14<9:50:13,  1.92s/it]
Outer training loop:  63%|######3   | 31527/50000 [20:18:15<9:30:17,  1.85s/it]
Outer training loop:  63%|######3   | 31528/50000 [20:18:17<9:22:21,  1.83s/it]
Outer training loop:  63%|######3   | 31529/50000 [20:18:19<9:45:49,  1.90s/it]
Outer training loop:  63%|######3   | 31530/50000 [20:18:21<9:40:01,  1.88s/it]
Outer training loop:  63%|######3   | 31531/50000 [20:18:23<9:33:25,  1.86s/it]
Outer training loop:  63%|######3   | 31532/50000 [20:18:25<9:55:38,  1.94s/it]
Outer training loop:  63%|######3   | 31533/50000 [20:18:27<9:51:04,  1.92s/it]
Outer training loop:  63%|######3   | 31534/50000 [20:18:29<9:38:28,  1.88s/it]
Outer training loop:  63%|######3   | 31535/50000 [20:18:31<9:49:04,  1.91s/it]
Outer training loop:  63%|######3   | 31536/50000 [20:18:32<9:40:06,  1.89s/it]
Outer training loop:  63%|######3   | 31537/50000 [20:18:34<9:35:04,  1.87s/it]
Outer training loop:  63%|######3   | 31538/50000 [20:18:36<9:59:16,  1.95s/it]
Outer training loop:  63%|######3   | 31539/50000 [20:18:38<9:53:31,  1.93s/it]
Outer training loop:  63%|######3   | 31540/50000 [20:18:40<9:41:55,  1.89s/it]
Outer training loop:  63%|######3   | 31541/50000 [20:18:42<9:38:52,  1.88s/it]
Outer training loop:  63%|######3   | 31542/50000 [20:18:44<9:57:22,  1.94s/it]
Outer training loop:  63%|######3   | 31543/50000 [20:18:46<9:49:09,  1.92s/it]
Outer training loop:  63%|######3   | 31544/50000 [20:18:48<9:31:45,  1.86s/it]
Outer training loop:  63%|######3   | 31545/50000 [20:18:50<10:11:31,  1.99s/it]
Outer training loop:  63%|######3   | 31546/50000 [20:18:52<9:53:06,  1.93s/it] 
Outer training loop:  63%|######3   | 31547/50000 [20:18:53<9:35:04,  1.87s/it]
Outer training loop:  63%|######3   | 31548/50000 [20:18:56<9:53:10,  1.93s/it]
Outer training loop:  63%|######3   | 31549/50000 [20:18:57<9:38:19,  1.88s/it]
Outer training loop:  63%|######3   | 31550/50000 [20:18:59<9:42:31,  1.89s/it]
Outer training loop:  63%|######3   | 31551/50000 [20:19:01<9:49:23,  1.92s/it]
Outer training loop:  63%|######3   | 31552/50000 [20:19:03<9:44:13,  1.90s/it]
Outer training loop:  63%|######3   | 31553/50000 [20:19:05<9:41:56,  1.89s/it]
Outer training loop:  63%|######3   | 31554/50000 [20:19:07<9:32:41,  1.86s/it]
Outer training loop:  63%|######3   | 31555/50000 [20:19:09<9:52:13,  1.93s/it]
Outer training loop:  63%|######3   | 31556/50000 [20:19:11<9:56:12,  1.94s/it]
Outer training loop:  63%|######3   | 31557/50000 [20:19:13<9:44:18,  1.90s/it]
Outer training loop:  63%|######3   | 31558/50000 [20:19:15<10:00:07,  1.95s/it]
Outer training loop:  63%|######3   | 31559/50000 [20:19:16<9:43:31,  1.90s/it] 
Outer training loop:  63%|######3   | 31560/50000 [20:19:18<9:40:33,  1.89s/it]
Outer training loop:  63%|######3   | 31561/50000 [20:19:20<9:37:08,  1.88s/it]
Outer training loop:  63%|######3   | 31562/50000 [20:19:22<9:40:49,  1.89s/it]
Outer training loop:  63%|######3   | 31563/50000 [20:19:24<9:37:27,  1.88s/it]
Outer training loop:  63%|######3   | 31564/50000 [20:19:26<9:27:41,  1.85s/it]
Outer training loop:  63%|######3   | 31565/50000 [20:19:28<9:43:23,  1.90s/it]
Outer training loop:  63%|######3   | 31566/50000 [20:19:30<9:38:27,  1.88s/it]
Outer training loop:  63%|######3   | 31567/50000 [20:19:31<9:25:35,  1.84s/it]
Outer training loop:  63%|######3   | 31568/50000 [20:19:33<9:38:29,  1.88s/it]
Outer training loop:  63%|######3   | 31569/50000 [20:19:35<9:44:30,  1.90s/it]
Outer training loop:  63%|######3   | 31570/50000 [20:19:37<9:32:46,  1.86s/it]
Outer training loop:  63%|######3   | 31571/50000 [20:19:39<9:29:53,  1.86s/it]
Outer training loop:  63%|######3   | 31572/50000 [20:19:41<9:38:57,  1.89s/it]
Outer training loop:  63%|######3   | 31573/50000 [20:19:43<9:38:33,  1.88s/it]
Outer training loop:  63%|######3   | 31574/50000 [20:19:44<9:33:05,  1.87s/it]
Outer training loop:  63%|######3   | 31575/50000 [20:19:47<9:52:36,  1.93s/it]
Outer training loop:  63%|######3   | 31576/50000 [20:19:48<9:45:47,  1.91s/it]
Outer training loop:  63%|######3   | 31577/50000 [20:19:50<9:41:23,  1.89s/it]
Outer training loop:  63%|######3   | 31578/50000 [20:19:52<9:37:05,  1.88s/it]
Outer training loop:  63%|######3   | 31579/50000 [20:19:54<9:51:56,  1.93s/it]
Outer training loop:  63%|######3   | 31580/50000 [20:19:56<9:42:28,  1.90s/it]
Outer training loop:  63%|######3   | 31581/50000 [20:19:58<9:50:44,  1.92s/it]
Outer training loop:  63%|######3   | 31582/50000 [20:20:00<9:50:22,  1.92s/it]
Outer training loop:  63%|######3   | 31583/50000 [20:20:02<9:45:21,  1.91s/it]
Outer training loop:  63%|######3   | 31584/50000 [20:20:04<9:39:41,  1.89s/it]
Outer training loop:  63%|######3   | 31585/50000 [20:20:06<9:43:31,  1.90s/it]
Outer training loop:  63%|######3   | 31586/50000 [20:20:08<9:58:47,  1.95s/it]
Outer training loop:  63%|######3   | 31587/50000 [20:20:09<9:48:08,  1.92s/it]
Outer training loop:  63%|######3   | 31588/50000 [20:20:11<9:36:41,  1.88s/it]
Outer training loop:  63%|######3   | 31589/50000 [20:20:13<9:50:44,  1.93s/it]
Outer training loop:  63%|######3   | 31590/50000 [20:20:15<9:40:14,  1.89s/it]
Outer training loop:  63%|######3   | 31591/50000 [20:20:17<9:33:00,  1.87s/it]
Outer training loop:  63%|######3   | 31592/50000 [20:20:19<9:32:08,  1.86s/it]
Outer training loop:  63%|######3   | 31593/50000 [20:20:21<9:47:36,  1.92s/it]
Outer training loop:  63%|######3   | 31594/50000 [20:20:23<9:39:01,  1.89s/it]
Outer training loop:  63%|######3   | 31595/50000 [20:20:25<9:56:01,  1.94s/it]
Outer training loop:  63%|######3   | 31596/50000 [20:20:27<9:57:53,  1.95s/it]
Outer training loop:  63%|######3   | 31597/50000 [20:20:28<9:44:46,  1.91s/it]
Outer training loop:  63%|######3   | 31598/50000 [20:20:30<9:31:04,  1.86s/it]
Outer training loop:  63%|######3   | 31599/50000 [20:20:32<9:28:16,  1.85s/it]
Outer training loop:  63%|######3   | 31600/50000 [20:20:34<9:46:43,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  63%|######3   | 31601/50000 [20:20:42<18:24:35,  3.60s/it]
Outer training loop:  63%|######3   | 31602/50000 [20:20:44<15:45:25,  3.08s/it]
Outer training loop:  63%|######3   | 31603/50000 [20:20:45<13:38:43,  2.67s/it]
Outer training loop:  63%|######3   | 31604/50000 [20:20:47<12:33:28,  2.46s/it]
Outer training loop:  63%|######3   | 31605/50000 [20:20:49<11:34:31,  2.27s/it]
Outer training loop:  63%|######3   | 31606/50000 [20:20:51<10:57:02,  2.14s/it]
Outer training loop:  63%|######3   | 31607/50000 [20:20:53<11:00:18,  2.15s/it]
Outer training loop:  63%|######3   | 31608/50000 [20:20:55<10:34:49,  2.07s/it]
Outer training loop:  63%|######3   | 31609/50000 [20:20:57<10:01:59,  1.96s/it]
Outer training loop:  63%|######3   | 31610/50000 [20:20:59<10:07:22,  1.98s/it]
Outer training loop:  63%|######3   | 31611/50000 [20:21:00<9:51:15,  1.93s/it] 
Outer training loop:  63%|######3   | 31612/50000 [20:21:02<9:32:28,  1.87s/it]
Outer training loop:  63%|######3   | 31613/50000 [20:21:04<9:36:46,  1.88s/it]
Outer training loop:  63%|######3   | 31614/50000 [20:21:06<9:48:00,  1.92s/it]
Outer training loop:  63%|######3   | 31615/50000 [20:21:08<9:35:13,  1.88s/it]
Outer training loop:  63%|######3   | 31616/50000 [20:21:10<9:39:21,  1.89s/it]
Outer training loop:  63%|######3   | 31617/50000 [20:21:12<9:45:25,  1.91s/it]
Outer training loop:  63%|######3   | 31618/50000 [20:21:14<9:42:19,  1.90s/it]
Outer training loop:  63%|######3   | 31619/50000 [20:21:15<9:37:34,  1.89s/it]
Outer training loop:  63%|######3   | 31620/50000 [20:21:17<9:46:31,  1.91s/it]
Outer training loop:  63%|######3   | 31621/50000 [20:21:19<9:34:56,  1.88s/it]
Outer training loop:  63%|######3   | 31622/50000 [20:21:21<9:30:16,  1.86s/it]
Outer training loop:  63%|######3   | 31623/50000 [20:21:23<9:27:09,  1.85s/it]
Outer training loop:  63%|######3   | 31624/50000 [20:21:25<9:44:33,  1.91s/it]
Outer training loop:  63%|######3   | 31625/50000 [20:21:27<9:36:45,  1.88s/it]
Outer training loop:  63%|######3   | 31626/50000 [20:21:29<9:38:37,  1.89s/it]
Outer training loop:  63%|######3   | 31627/50000 [20:21:31<9:33:22,  1.87s/it]
Outer training loop:  63%|######3   | 31628/50000 [20:21:33<9:55:17,  1.94s/it]
Outer training loop:  63%|######3   | 31629/50000 [20:21:34<9:41:09,  1.90s/it]
Outer training loop:  63%|######3   | 31630/50000 [20:21:36<9:30:24,  1.86s/it]
Outer training loop:  63%|######3   | 31631/50000 [20:21:38<9:46:33,  1.92s/it]
Outer training loop:  63%|######3   | 31632/50000 [20:21:40<9:39:09,  1.89s/it]
Outer training loop:  63%|######3   | 31633/50000 [20:21:42<9:35:14,  1.88s/it]
Outer training loop:  63%|######3   | 31634/50000 [20:21:44<9:53:31,  1.94s/it]
Outer training loop:  63%|######3   | 31635/50000 [20:21:46<9:49:49,  1.93s/it]
Outer training loop:  63%|######3   | 31636/50000 [20:21:48<9:28:38,  1.86s/it]
Outer training loop:  63%|######3   | 31637/50000 [20:21:49<9:27:49,  1.86s/it]
Outer training loop:  63%|######3   | 31638/50000 [20:21:51<9:34:11,  1.88s/it]
Outer training loop:  63%|######3   | 31639/50000 [20:21:53<9:25:11,  1.85s/it]
Outer training loop:  63%|######3   | 31640/50000 [20:21:55<9:22:42,  1.84s/it]
Outer training loop:  63%|######3   | 31641/50000 [20:21:57<9:32:36,  1.87s/it]
Outer training loop:  63%|######3   | 31642/50000 [20:21:59<9:49:41,  1.93s/it]
Outer training loop:  63%|######3   | 31643/50000 [20:22:01<9:31:14,  1.87s/it]
Outer training loop:  63%|######3   | 31644/50000 [20:22:03<9:31:14,  1.87s/it]
Outer training loop:  63%|######3   | 31645/50000 [20:22:05<9:45:20,  1.91s/it]
Outer training loop:  63%|######3   | 31646/50000 [20:22:06<9:38:22,  1.89s/it]
Outer training loop:  63%|######3   | 31647/50000 [20:22:08<9:31:46,  1.87s/it]
Outer training loop:  63%|######3   | 31648/50000 [20:22:10<9:35:40,  1.88s/it]
Outer training loop:  63%|######3   | 31649/50000 [20:22:12<9:24:04,  1.84s/it]
Outer training loop:  63%|######3   | 31650/50000 [20:22:14<9:28:18,  1.86s/it]
Outer training loop:  63%|######3   | 31651/50000 [20:22:16<9:29:01,  1.86s/it]
Outer training loop:  63%|######3   | 31652/50000 [20:22:18<9:38:41,  1.89s/it]
Outer training loop:  63%|######3   | 31653/50000 [20:22:19<9:31:32,  1.87s/it]
Outer training loop:  63%|######3   | 31654/50000 [20:22:21<9:31:41,  1.87s/it]
Outer training loop:  63%|######3   | 31655/50000 [20:22:23<9:48:46,  1.93s/it]
Outer training loop:  63%|######3   | 31656/50000 [20:22:25<9:49:39,  1.93s/it]
Outer training loop:  63%|######3   | 31657/50000 [20:22:27<9:36:23,  1.89s/it]
Outer training loop:  63%|######3   | 31658/50000 [20:22:29<9:50:21,  1.93s/it]
Outer training loop:  63%|######3   | 31659/50000 [20:22:31<9:49:44,  1.93s/it]
Outer training loop:  63%|######3   | 31660/50000 [20:22:33<9:29:52,  1.86s/it]
Outer training loop:  63%|######3   | 31661/50000 [20:22:35<9:52:17,  1.94s/it]
Outer training loop:  63%|######3   | 31662/50000 [20:22:37<9:34:05,  1.88s/it]
Outer training loop:  63%|######3   | 31663/50000 [20:22:38<9:30:49,  1.87s/it]
Outer training loop:  63%|######3   | 31664/50000 [20:22:40<9:32:23,  1.87s/it]
Outer training loop:  63%|######3   | 31665/50000 [20:22:42<9:47:34,  1.92s/it]
Outer training loop:  63%|######3   | 31666/50000 [20:22:44<9:44:19,  1.91s/it]
Outer training loop:  63%|######3   | 31667/50000 [20:22:46<9:23:12,  1.84s/it]
Outer training loop:  63%|######3   | 31668/50000 [20:22:48<9:36:51,  1.89s/it]
Outer training loop:  63%|######3   | 31669/50000 [20:22:50<9:35:50,  1.88s/it]
Outer training loop:  63%|######3   | 31670/50000 [20:22:52<9:24:29,  1.85s/it]
Outer training loop:  63%|######3   | 31671/50000 [20:22:53<9:22:40,  1.84s/it]
Outer training loop:  63%|######3   | 31672/50000 [20:22:55<9:40:55,  1.90s/it]
Outer training loop:  63%|######3   | 31673/50000 [20:22:57<9:19:42,  1.83s/it]
Outer training loop:  63%|######3   | 31674/50000 [20:22:59<9:25:08,  1.85s/it]
Outer training loop:  63%|######3   | 31675/50000 [20:23:01<9:51:53,  1.94s/it]
Outer training loop:  63%|######3   | 31676/50000 [20:23:03<9:35:53,  1.89s/it]
Outer training loop:  63%|######3   | 31677/50000 [20:23:05<9:37:56,  1.89s/it]
Outer training loop:  63%|######3   | 31678/50000 [20:23:07<9:49:31,  1.93s/it]
Outer training loop:  63%|######3   | 31679/50000 [20:23:09<9:41:51,  1.91s/it]
Outer training loop:  63%|######3   | 31680/50000 [20:23:11<9:35:48,  1.89s/it]
Outer training loop:  63%|######3   | 31681/50000 [20:23:13<9:50:39,  1.93s/it]
Outer training loop:  63%|######3   | 31682/50000 [20:23:15<9:49:46,  1.93s/it]
Outer training loop:  63%|######3   | 31683/50000 [20:23:16<9:31:45,  1.87s/it]
Outer training loop:  63%|######3   | 31684/50000 [20:23:18<9:22:27,  1.84s/it]
Outer training loop:  63%|######3   | 31685/50000 [20:23:20<9:47:52,  1.93s/it]
Outer training loop:  63%|######3   | 31686/50000 [20:23:22<9:33:52,  1.88s/it]
Outer training loop:  63%|######3   | 31687/50000 [20:23:24<9:20:15,  1.84s/it]
Outer training loop:  63%|######3   | 31688/50000 [20:23:26<9:42:25,  1.91s/it]
Outer training loop:  63%|######3   | 31689/50000 [20:23:28<9:31:53,  1.87s/it]
Outer training loop:  63%|######3   | 31690/50000 [20:23:29<9:15:14,  1.82s/it]
Outer training loop:  63%|######3   | 31691/50000 [20:23:31<9:18:44,  1.83s/it]
Outer training loop:  63%|######3   | 31692/50000 [20:23:33<9:57:23,  1.96s/it]
Outer training loop:  63%|######3   | 31693/50000 [20:23:35<10:02:29,  1.97s/it]
Outer training loop:  63%|######3   | 31694/50000 [20:23:37<9:49:35,  1.93s/it] 
Outer training loop:  63%|######3   | 31695/50000 [20:23:39<9:47:18,  1.93s/it]
Outer training loop:  63%|######3   | 31696/50000 [20:23:41<9:40:43,  1.90s/it]
Outer training loop:  63%|######3   | 31697/50000 [20:23:43<9:39:46,  1.90s/it]
Outer training loop:  63%|######3   | 31698/50000 [20:23:45<9:59:32,  1.97s/it]
Outer training loop:  63%|######3   | 31699/50000 [20:23:47<9:47:47,  1.93s/it]
Outer training loop:  63%|######3   | 31700/50000 [20:23:49<9:31:24,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.94s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  63%|######3   | 31701/50000 [20:23:56<17:58:00,  3.53s/it]
Outer training loop:  63%|######3   | 31702/50000 [20:23:58<15:53:04,  3.13s/it]
Outer training loop:  63%|######3   | 31703/50000 [20:24:00<13:57:17,  2.75s/it]
Outer training loop:  63%|######3   | 31704/50000 [20:24:02<12:27:12,  2.45s/it]
Outer training loop:  63%|######3   | 31705/50000 [20:24:04<11:33:46,  2.28s/it]
Outer training loop:  63%|######3   | 31706/50000 [20:24:06<11:03:23,  2.18s/it]
Outer training loop:  63%|######3   | 31707/50000 [20:24:07<10:36:21,  2.09s/it]
Outer training loop:  63%|######3   | 31708/50000 [20:24:09<10:05:47,  1.99s/it]
Outer training loop:  63%|######3   | 31709/50000 [20:24:11<9:57:47,  1.96s/it] 
Outer training loop:  63%|######3   | 31710/50000 [20:24:13<9:40:02,  1.90s/it]
Outer training loop:  63%|######3   | 31711/50000 [20:24:15<9:31:49,  1.88s/it]
Outer training loop:  63%|######3   | 31712/50000 [20:24:17<9:41:18,  1.91s/it]
Outer training loop:  63%|######3   | 31713/50000 [20:24:18<9:26:34,  1.86s/it]
Outer training loop:  63%|######3   | 31714/50000 [20:24:20<9:22:45,  1.85s/it]
Outer training loop:  63%|######3   | 31715/50000 [20:24:22<9:23:00,  1.85s/it]
Outer training loop:  63%|######3   | 31716/50000 [20:24:24<9:44:08,  1.92s/it]
Outer training loop:  63%|######3   | 31717/50000 [20:24:26<9:41:31,  1.91s/it]
Outer training loop:  63%|######3   | 31718/50000 [20:24:28<9:38:50,  1.90s/it]
Outer training loop:  63%|######3   | 31719/50000 [20:24:30<9:43:18,  1.91s/it]
Outer training loop:  63%|######3   | 31720/50000 [20:24:32<9:40:03,  1.90s/it]
Outer training loop:  63%|######3   | 31721/50000 [20:24:34<9:33:11,  1.88s/it]
Outer training loop:  63%|######3   | 31722/50000 [20:24:35<9:32:25,  1.88s/it]
Outer training loop:  63%|######3   | 31723/50000 [20:24:37<9:37:10,  1.89s/it]
Outer training loop:  63%|######3   | 31724/50000 [20:24:39<9:39:37,  1.90s/it]
Outer training loop:  63%|######3   | 31725/50000 [20:24:41<9:29:27,  1.87s/it]
Outer training loop:  63%|######3   | 31726/50000 [20:24:43<9:42:12,  1.91s/it]
Outer training loop:  63%|######3   | 31727/50000 [20:24:45<9:32:52,  1.88s/it]
Outer training loop:  63%|######3   | 31728/50000 [20:24:47<9:31:30,  1.88s/it]
Outer training loop:  63%|######3   | 31729/50000 [20:24:49<9:43:05,  1.91s/it]
Outer training loop:  63%|######3   | 31730/50000 [20:24:51<9:35:07,  1.89s/it]
Outer training loop:  63%|######3   | 31731/50000 [20:24:52<9:25:16,  1.86s/it]
Outer training loop:  63%|######3   | 31732/50000 [20:24:54<9:04:28,  1.79s/it]
Outer training loop:  63%|######3   | 31733/50000 [20:24:56<9:15:19,  1.82s/it]
Outer training loop:  63%|######3   | 31734/50000 [20:24:58<9:12:06,  1.81s/it]
Outer training loop:  63%|######3   | 31735/50000 [20:25:00<9:15:24,  1.82s/it]
Outer training loop:  63%|######3   | 31736/50000 [20:25:02<9:28:22,  1.87s/it]
Outer training loop:  63%|######3   | 31737/50000 [20:25:03<9:16:53,  1.83s/it]
Outer training loop:  63%|######3   | 31738/50000 [20:25:05<9:13:39,  1.82s/it]
Outer training loop:  63%|######3   | 31739/50000 [20:25:07<9:37:20,  1.90s/it]
Outer training loop:  63%|######3   | 31740/50000 [20:25:09<9:41:28,  1.91s/it]
Outer training loop:  63%|######3   | 31741/50000 [20:25:11<9:40:52,  1.91s/it]
Outer training loop:  63%|######3   | 31742/50000 [20:25:13<9:21:48,  1.85s/it]
Outer training loop:  63%|######3   | 31743/50000 [20:25:15<9:34:28,  1.89s/it]
Outer training loop:  63%|######3   | 31744/50000 [20:25:16<9:29:12,  1.87s/it]
Outer training loop:  63%|######3   | 31745/50000 [20:25:18<9:30:26,  1.87s/it]
Outer training loop:  63%|######3   | 31746/50000 [20:25:20<9:47:28,  1.93s/it]
Outer training loop:  63%|######3   | 31747/50000 [20:25:22<9:40:57,  1.91s/it]
Outer training loop:  63%|######3   | 31748/50000 [20:25:24<9:22:58,  1.85s/it]
Outer training loop:  63%|######3   | 31749/50000 [20:25:26<9:30:06,  1.87s/it]
Outer training loop:  64%|######3   | 31750/50000 [20:25:28<9:40:20,  1.91s/it]
Outer training loop:  64%|######3   | 31751/50000 [20:25:30<9:33:09,  1.88s/it]
Outer training loop:  64%|######3   | 31752/50000 [20:25:32<9:27:25,  1.87s/it]
Outer training loop:  64%|######3   | 31753/50000 [20:25:34<9:39:03,  1.90s/it]
Outer training loop:  64%|######3   | 31754/50000 [20:25:35<9:29:47,  1.87s/it]
Outer training loop:  64%|######3   | 31755/50000 [20:25:37<9:32:02,  1.88s/it]
Outer training loop:  64%|######3   | 31756/50000 [20:25:39<9:36:57,  1.90s/it]
Outer training loop:  64%|######3   | 31757/50000 [20:25:41<9:25:04,  1.86s/it]
Outer training loop:  64%|######3   | 31758/50000 [20:25:43<9:14:47,  1.82s/it]
Outer training loop:  64%|######3   | 31759/50000 [20:25:44<9:09:33,  1.81s/it]
Outer training loop:  64%|######3   | 31760/50000 [20:25:46<9:22:59,  1.85s/it]
Outer training loop:  64%|######3   | 31761/50000 [20:25:48<9:12:04,  1.82s/it]
Outer training loop:  64%|######3   | 31762/50000 [20:25:50<9:10:18,  1.81s/it]
Outer training loop:  64%|######3   | 31763/50000 [20:25:52<9:41:46,  1.91s/it]
Outer training loop:  64%|######3   | 31764/50000 [20:25:54<9:28:17,  1.87s/it]
Outer training loop:  64%|######3   | 31765/50000 [20:25:56<9:21:48,  1.85s/it]
Outer training loop:  64%|######3   | 31766/50000 [20:25:58<9:31:54,  1.88s/it]
Outer training loop:  64%|######3   | 31767/50000 [20:25:59<9:14:29,  1.82s/it]
Outer training loop:  64%|######3   | 31768/50000 [20:26:01<9:10:02,  1.81s/it]
Outer training loop:  64%|######3   | 31769/50000 [20:26:03<9:05:17,  1.79s/it]
Outer training loop:  64%|######3   | 31770/50000 [20:26:05<9:31:43,  1.88s/it]
Outer training loop:  64%|######3   | 31771/50000 [20:26:07<9:18:40,  1.84s/it]
Outer training loop:  64%|######3   | 31772/50000 [20:26:09<9:14:53,  1.83s/it]
Outer training loop:  64%|######3   | 31773/50000 [20:26:10<9:25:54,  1.86s/it]
Outer training loop:  64%|######3   | 31774/50000 [20:26:12<9:21:03,  1.85s/it]
Outer training loop:  64%|######3   | 31775/50000 [20:26:14<9:19:11,  1.84s/it]
Outer training loop:  64%|######3   | 31776/50000 [20:26:16<9:09:13,  1.81s/it]
Outer training loop:  64%|######3   | 31777/50000 [20:26:18<9:17:51,  1.84s/it]
Outer training loop:  64%|######3   | 31778/50000 [20:26:20<9:16:15,  1.83s/it]
Outer training loop:  64%|######3   | 31779/50000 [20:26:21<9:08:59,  1.81s/it]
Outer training loop:  64%|######3   | 31780/50000 [20:26:23<9:17:47,  1.84s/it]
Outer training loop:  64%|######3   | 31781/50000 [20:26:25<9:28:43,  1.87s/it]
Outer training loop:  64%|######3   | 31782/50000 [20:26:27<9:18:46,  1.84s/it]
Outer training loop:  64%|######3   | 31783/50000 [20:26:29<9:34:23,  1.89s/it]
Outer training loop:  64%|######3   | 31784/50000 [20:26:31<9:23:25,  1.86s/it]
Outer training loop:  64%|######3   | 31785/50000 [20:26:33<9:25:45,  1.86s/it]
Outer training loop:  64%|######3   | 31786/50000 [20:26:34<9:26:45,  1.87s/it]
Outer training loop:  64%|######3   | 31787/50000 [20:26:36<9:37:51,  1.90s/it]
Outer training loop:  64%|######3   | 31788/50000 [20:26:38<9:27:58,  1.87s/it]
Outer training loop:  64%|######3   | 31789/50000 [20:26:40<9:53:16,  1.95s/it]
Outer training loop:  64%|######3   | 31790/50000 [20:26:42<10:06:11,  2.00s/it]
Outer training loop:  64%|######3   | 31791/50000 [20:26:44<9:54:42,  1.96s/it] 
Outer training loop:  64%|######3   | 31792/50000 [20:26:46<9:40:09,  1.91s/it]
Outer training loop:  64%|######3   | 31793/50000 [20:26:48<9:27:37,  1.87s/it]
Outer training loop:  64%|######3   | 31794/50000 [20:26:50<9:40:06,  1.91s/it]
Outer training loop:  64%|######3   | 31795/50000 [20:26:52<9:30:39,  1.88s/it]
Outer training loop:  64%|######3   | 31796/50000 [20:26:54<9:24:31,  1.86s/it]
Outer training loop:  64%|######3   | 31797/50000 [20:26:56<9:34:28,  1.89s/it]
Outer training loop:  64%|######3   | 31798/50000 [20:26:57<9:38:21,  1.91s/it]
Outer training loop:  64%|######3   | 31799/50000 [20:26:59<9:23:39,  1.86s/it]
Outer training loop:  64%|######3   | 31800/50000 [20:27:01<9:16:16,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.91s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  64%|######3   | 31801/50000 [20:27:09<18:02:15,  3.57s/it]
Outer training loop:  64%|######3   | 31802/50000 [20:27:11<15:44:10,  3.11s/it]
Outer training loop:  64%|######3   | 31803/50000 [20:27:12<13:46:38,  2.73s/it]
Outer training loop:  64%|######3   | 31804/50000 [20:27:14<12:19:47,  2.44s/it]
Outer training loop:  64%|######3   | 31805/50000 [20:27:16<11:34:45,  2.29s/it]
Outer training loop:  64%|######3   | 31806/50000 [20:27:18<10:49:06,  2.14s/it]
Outer training loop:  64%|######3   | 31807/50000 [20:27:20<10:21:50,  2.05s/it]
Outer training loop:  64%|######3   | 31808/50000 [20:27:22<9:57:39,  1.97s/it] 
Outer training loop:  64%|######3   | 31809/50000 [20:27:24<10:02:21,  1.99s/it]
Outer training loop:  64%|######3   | 31810/50000 [20:27:25<9:47:24,  1.94s/it] 
Outer training loop:  64%|######3   | 31811/50000 [20:27:27<9:45:03,  1.93s/it]
Outer training loop:  64%|######3   | 31812/50000 [20:27:29<9:59:11,  1.98s/it]
Outer training loop:  64%|######3   | 31813/50000 [20:27:31<9:49:44,  1.95s/it]
Outer training loop:  64%|######3   | 31814/50000 [20:27:33<9:41:46,  1.92s/it]
Outer training loop:  64%|######3   | 31815/50000 [20:27:35<9:54:11,  1.96s/it]
Outer training loop:  64%|######3   | 31816/50000 [20:27:37<9:34:08,  1.89s/it]
Outer training loop:  64%|######3   | 31817/50000 [20:27:39<9:27:38,  1.87s/it]
Outer training loop:  64%|######3   | 31818/50000 [20:27:41<9:46:09,  1.93s/it]
Outer training loop:  64%|######3   | 31819/50000 [20:27:43<9:56:08,  1.97s/it]
Outer training loop:  64%|######3   | 31820/50000 [20:27:45<9:44:40,  1.93s/it]
Outer training loop:  64%|######3   | 31821/50000 [20:27:47<9:56:34,  1.97s/it]
Outer training loop:  64%|######3   | 31822/50000 [20:27:49<9:55:39,  1.97s/it]
Outer training loop:  64%|######3   | 31823/50000 [20:27:51<9:45:18,  1.93s/it]
Outer training loop:  64%|######3   | 31824/50000 [20:27:53<9:46:15,  1.94s/it]
Outer training loop:  64%|######3   | 31825/50000 [20:27:54<9:38:15,  1.91s/it]
Outer training loop:  64%|######3   | 31826/50000 [20:27:56<9:32:56,  1.89s/it]
Outer training loop:  64%|######3   | 31827/50000 [20:27:58<9:25:07,  1.87s/it]
Outer training loop:  64%|######3   | 31828/50000 [20:28:00<9:42:49,  1.92s/it]
Outer training loop:  64%|######3   | 31829/50000 [20:28:02<9:27:40,  1.87s/it]
Outer training loop:  64%|######3   | 31830/50000 [20:28:04<9:11:58,  1.82s/it]
Outer training loop:  64%|######3   | 31831/50000 [20:28:06<9:29:27,  1.88s/it]
Outer training loop:  64%|######3   | 31832/50000 [20:28:08<9:29:49,  1.88s/it]
Outer training loop:  64%|######3   | 31833/50000 [20:28:09<9:21:03,  1.85s/it]
Outer training loop:  64%|######3   | 31834/50000 [20:28:11<9:25:53,  1.87s/it]
Outer training loop:  64%|######3   | 31835/50000 [20:28:13<9:44:10,  1.93s/it]
Outer training loop:  64%|######3   | 31836/50000 [20:28:15<9:47:23,  1.94s/it]
Outer training loop:  64%|######3   | 31837/50000 [20:28:17<9:40:03,  1.92s/it]
Outer training loop:  64%|######3   | 31838/50000 [20:28:19<9:47:27,  1.94s/it]
Outer training loop:  64%|######3   | 31839/50000 [20:28:21<9:33:12,  1.89s/it]
Outer training loop:  64%|######3   | 31840/50000 [20:28:23<9:27:50,  1.88s/it]
Outer training loop:  64%|######3   | 31841/50000 [20:28:25<9:44:11,  1.93s/it]
Outer training loop:  64%|######3   | 31842/50000 [20:28:27<9:38:05,  1.91s/it]
Outer training loop:  64%|######3   | 31843/50000 [20:28:28<9:31:15,  1.89s/it]
Outer training loop:  64%|######3   | 31844/50000 [20:28:30<9:20:59,  1.85s/it]
Outer training loop:  64%|######3   | 31845/50000 [20:28:32<9:41:45,  1.92s/it]
Outer training loop:  64%|######3   | 31846/50000 [20:28:34<9:32:24,  1.89s/it]
Outer training loop:  64%|######3   | 31847/50000 [20:28:36<9:38:16,  1.91s/it]
Outer training loop:  64%|######3   | 31848/50000 [20:28:38<9:45:47,  1.94s/it]
Outer training loop:  64%|######3   | 31849/50000 [20:28:40<9:51:43,  1.96s/it]
Outer training loop:  64%|######3   | 31850/50000 [20:28:42<9:48:59,  1.95s/it]
Outer training loop:  64%|######3   | 31851/50000 [20:28:44<10:00:57,  1.99s/it]
Outer training loop:  64%|######3   | 31852/50000 [20:28:46<9:42:36,  1.93s/it] 
Outer training loop:  64%|######3   | 31853/50000 [20:28:48<9:29:54,  1.88s/it]
Outer training loop:  64%|######3   | 31854/50000 [20:28:50<9:26:31,  1.87s/it]
Outer training loop:  64%|######3   | 31855/50000 [20:28:52<9:44:56,  1.93s/it]
Outer training loop:  64%|######3   | 31856/50000 [20:28:53<9:39:29,  1.92s/it]
Outer training loop:  64%|######3   | 31857/50000 [20:28:55<9:38:17,  1.91s/it]
Outer training loop:  64%|######3   | 31858/50000 [20:28:57<9:44:32,  1.93s/it]
Outer training loop:  64%|######3   | 31859/50000 [20:28:59<9:37:23,  1.91s/it]
Outer training loop:  64%|######3   | 31860/50000 [20:29:01<9:25:29,  1.87s/it]
Outer training loop:  64%|######3   | 31861/50000 [20:29:03<9:32:49,  1.89s/it]
Outer training loop:  64%|######3   | 31862/50000 [20:29:05<9:36:30,  1.91s/it]
Outer training loop:  64%|######3   | 31863/50000 [20:29:07<9:26:50,  1.88s/it]
Outer training loop:  64%|######3   | 31864/50000 [20:29:09<9:29:56,  1.89s/it]
Outer training loop:  64%|######3   | 31865/50000 [20:29:10<9:26:24,  1.87s/it]
Outer training loop:  64%|######3   | 31866/50000 [20:29:12<9:41:07,  1.92s/it]
Outer training loop:  64%|######3   | 31867/50000 [20:29:14<9:45:39,  1.94s/it]
Outer training loop:  64%|######3   | 31868/50000 [20:29:17<9:54:55,  1.97s/it]
Outer training loop:  64%|######3   | 31869/50000 [20:29:18<9:35:58,  1.91s/it]
Outer training loop:  64%|######3   | 31870/50000 [20:29:20<9:25:53,  1.87s/it]
Outer training loop:  64%|######3   | 31871/50000 [20:29:22<9:45:32,  1.94s/it]
Outer training loop:  64%|######3   | 31872/50000 [20:29:24<9:29:19,  1.88s/it]
Outer training loop:  64%|######3   | 31873/50000 [20:29:26<9:26:40,  1.88s/it]
Outer training loop:  64%|######3   | 31874/50000 [20:29:28<9:46:24,  1.94s/it]
Outer training loop:  64%|######3   | 31875/50000 [20:29:30<9:33:09,  1.90s/it]
Outer training loop:  64%|######3   | 31876/50000 [20:29:32<9:29:01,  1.88s/it]
Outer training loop:  64%|######3   | 31877/50000 [20:29:33<9:31:09,  1.89s/it]
Outer training loop:  64%|######3   | 31878/50000 [20:29:35<9:41:32,  1.93s/it]
Outer training loop:  64%|######3   | 31879/50000 [20:29:37<9:41:24,  1.93s/it]
Outer training loop:  64%|######3   | 31880/50000 [20:29:39<9:45:58,  1.94s/it]
Outer training loop:  64%|######3   | 31881/50000 [20:29:41<9:55:15,  1.97s/it]
Outer training loop:  64%|######3   | 31882/50000 [20:29:43<9:47:20,  1.95s/it]
Outer training loop:  64%|######3   | 31883/50000 [20:29:45<9:56:40,  1.98s/it]
Outer training loop:  64%|######3   | 31884/50000 [20:29:47<9:41:09,  1.92s/it]
Outer training loop:  64%|######3   | 31885/50000 [20:29:49<9:44:22,  1.94s/it]
Outer training loop:  64%|######3   | 31886/50000 [20:29:51<9:59:13,  1.98s/it]
Outer training loop:  64%|######3   | 31887/50000 [20:29:53<9:52:27,  1.96s/it]
Outer training loop:  64%|######3   | 31888/50000 [20:29:55<10:11:22,  2.03s/it]
Outer training loop:  64%|######3   | 31889/50000 [20:29:57<9:59:24,  1.99s/it] 
Outer training loop:  64%|######3   | 31890/50000 [20:29:59<9:51:24,  1.96s/it]
Outer training loop:  64%|######3   | 31891/50000 [20:30:01<10:01:51,  1.99s/it]
Outer training loop:  64%|######3   | 31892/50000 [20:30:03<9:51:04,  1.96s/it] 
Outer training loop:  64%|######3   | 31893/50000 [20:30:05<9:59:13,  1.99s/it]
Outer training loop:  64%|######3   | 31894/50000 [20:30:07<10:09:31,  2.02s/it]
Outer training loop:  64%|######3   | 31895/50000 [20:30:09<9:56:35,  1.98s/it] 
Outer training loop:  64%|######3   | 31896/50000 [20:30:11<9:52:24,  1.96s/it]
Outer training loop:  64%|######3   | 31897/50000 [20:30:13<9:53:51,  1.97s/it]
Outer training loop:  64%|######3   | 31898/50000 [20:30:15<10:08:43,  2.02s/it]
Outer training loop:  64%|######3   | 31899/50000 [20:30:17<9:57:31,  1.98s/it] 
Outer training loop:  64%|######3   | 31900/50000 [20:30:19<9:39:36,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  64%|######3   | 31901/50000 [20:30:26<17:56:20,  3.57s/it]
Outer training loop:  64%|######3   | 31902/50000 [20:30:28<15:31:11,  3.09s/it]
Outer training loop:  64%|######3   | 31903/50000 [20:30:30<13:44:42,  2.73s/it]
Outer training loop:  64%|######3   | 31904/50000 [20:30:32<12:21:59,  2.46s/it]
Outer training loop:  64%|######3   | 31905/50000 [20:30:34<12:00:34,  2.39s/it]
Outer training loop:  64%|######3   | 31906/50000 [20:30:36<11:23:41,  2.27s/it]
Outer training loop:  64%|######3   | 31907/50000 [20:30:38<10:50:05,  2.16s/it]
Outer training loop:  64%|######3   | 31908/50000 [20:30:40<10:18:30,  2.05s/it]
Outer training loop:  64%|######3   | 31909/50000 [20:30:42<10:20:49,  2.06s/it]
Outer training loop:  64%|######3   | 31910/50000 [20:30:44<10:03:13,  2.00s/it]
Outer training loop:  64%|######3   | 31911/50000 [20:30:46<9:52:27,  1.97s/it] 
Outer training loop:  64%|######3   | 31912/50000 [20:30:47<9:48:33,  1.95s/it]
Outer training loop:  64%|######3   | 31913/50000 [20:30:50<10:02:59,  2.00s/it]
Outer training loop:  64%|######3   | 31914/50000 [20:30:52<9:56:39,  1.98s/it] 
Outer training loop:  64%|######3   | 31915/50000 [20:30:53<9:53:56,  1.97s/it]
Outer training loop:  64%|######3   | 31916/50000 [20:30:55<9:55:20,  1.98s/it]
Outer training loop:  64%|######3   | 31917/50000 [20:30:57<9:43:28,  1.94s/it]
Outer training loop:  64%|######3   | 31918/50000 [20:30:59<9:28:05,  1.89s/it]
Outer training loop:  64%|######3   | 31919/50000 [20:31:01<9:34:23,  1.91s/it]
Outer training loop:  64%|######3   | 31920/50000 [20:31:03<9:32:12,  1.90s/it]
Outer training loop:  64%|######3   | 31921/50000 [20:31:05<9:40:15,  1.93s/it]
Outer training loop:  64%|######3   | 31922/50000 [20:31:07<9:29:30,  1.89s/it]
Outer training loop:  64%|######3   | 31923/50000 [20:31:09<9:37:52,  1.92s/it]
Outer training loop:  64%|######3   | 31924/50000 [20:31:11<9:39:13,  1.92s/it]
Outer training loop:  64%|######3   | 31925/50000 [20:31:13<9:36:34,  1.91s/it]
Outer training loop:  64%|######3   | 31926/50000 [20:31:15<9:49:33,  1.96s/it]
Outer training loop:  64%|######3   | 31927/50000 [20:31:16<9:37:09,  1.92s/it]
Outer training loop:  64%|######3   | 31928/50000 [20:31:18<9:26:47,  1.88s/it]
Outer training loop:  64%|######3   | 31929/50000 [20:31:20<9:32:41,  1.90s/it]
Outer training loop:  64%|######3   | 31930/50000 [20:31:22<9:44:35,  1.94s/it]
Outer training loop:  64%|######3   | 31931/50000 [20:31:24<9:32:14,  1.90s/it]
Outer training loop:  64%|######3   | 31932/50000 [20:31:26<9:30:26,  1.89s/it]
Outer training loop:  64%|######3   | 31933/50000 [20:31:28<10:10:05,  2.03s/it]
Outer training loop:  64%|######3   | 31934/50000 [20:31:30<9:40:28,  1.93s/it] 
Outer training loop:  64%|######3   | 31935/50000 [20:31:32<9:29:03,  1.89s/it]
Outer training loop:  64%|######3   | 31936/50000 [20:31:34<9:34:35,  1.91s/it]
Outer training loop:  64%|######3   | 31937/50000 [20:31:36<9:34:47,  1.91s/it]
Outer training loop:  64%|######3   | 31938/50000 [20:31:37<9:26:25,  1.88s/it]
Outer training loop:  64%|######3   | 31939/50000 [20:31:39<9:24:10,  1.87s/it]
Outer training loop:  64%|######3   | 31940/50000 [20:31:41<9:34:20,  1.91s/it]
Outer training loop:  64%|######3   | 31941/50000 [20:31:43<9:31:59,  1.90s/it]
Outer training loop:  64%|######3   | 31942/50000 [20:31:45<9:26:38,  1.88s/it]
Outer training loop:  64%|######3   | 31943/50000 [20:31:47<9:48:03,  1.95s/it]
Outer training loop:  64%|######3   | 31944/50000 [20:31:49<9:46:32,  1.95s/it]
Outer training loop:  64%|######3   | 31945/50000 [20:31:51<9:42:46,  1.94s/it]
Outer training loop:  64%|######3   | 31946/50000 [20:31:53<9:30:15,  1.90s/it]
Outer training loop:  64%|######3   | 31947/50000 [20:31:55<9:46:59,  1.95s/it]
Outer training loop:  64%|######3   | 31948/50000 [20:31:57<9:41:18,  1.93s/it]
Outer training loop:  64%|######3   | 31949/50000 [20:31:59<9:31:00,  1.90s/it]
Outer training loop:  64%|######3   | 31950/50000 [20:32:01<9:41:54,  1.93s/it]
Outer training loop:  64%|######3   | 31951/50000 [20:32:02<9:36:14,  1.92s/it]
Outer training loop:  64%|######3   | 31952/50000 [20:32:04<9:27:56,  1.89s/it]
Outer training loop:  64%|######3   | 31953/50000 [20:32:06<9:41:07,  1.93s/it]
Outer training loop:  64%|######3   | 31954/50000 [20:32:08<9:32:42,  1.90s/it]
Outer training loop:  64%|######3   | 31955/50000 [20:32:10<9:25:01,  1.88s/it]
Outer training loop:  64%|######3   | 31956/50000 [20:32:12<9:18:20,  1.86s/it]
Outer training loop:  64%|######3   | 31957/50000 [20:32:14<9:22:09,  1.87s/it]
Outer training loop:  64%|######3   | 31958/50000 [20:32:15<9:12:39,  1.84s/it]
Outer training loop:  64%|######3   | 31959/50000 [20:32:17<9:09:32,  1.83s/it]
Outer training loop:  64%|######3   | 31960/50000 [20:32:19<9:28:53,  1.89s/it]
Outer training loop:  64%|######3   | 31961/50000 [20:32:21<9:37:21,  1.92s/it]
Outer training loop:  64%|######3   | 31962/50000 [20:32:23<9:35:13,  1.91s/it]
Outer training loop:  64%|######3   | 31963/50000 [20:32:25<9:35:48,  1.92s/it]
Outer training loop:  64%|######3   | 31964/50000 [20:32:27<9:34:04,  1.91s/it]
Outer training loop:  64%|######3   | 31965/50000 [20:32:29<9:27:28,  1.89s/it]
Outer training loop:  64%|######3   | 31966/50000 [20:32:31<9:19:01,  1.86s/it]
Outer training loop:  64%|######3   | 31967/50000 [20:32:33<9:29:50,  1.90s/it]
Outer training loop:  64%|######3   | 31968/50000 [20:32:34<9:25:18,  1.88s/it]
Outer training loop:  64%|######3   | 31969/50000 [20:32:36<9:16:15,  1.85s/it]
Outer training loop:  64%|######3   | 31970/50000 [20:32:38<9:33:45,  1.91s/it]
Outer training loop:  64%|######3   | 31971/50000 [20:32:40<9:22:32,  1.87s/it]
Outer training loop:  64%|######3   | 31972/50000 [20:32:42<9:13:28,  1.84s/it]
Outer training loop:  64%|######3   | 31973/50000 [20:32:44<9:10:14,  1.83s/it]
Outer training loop:  64%|######3   | 31974/50000 [20:32:45<9:17:49,  1.86s/it]
Outer training loop:  64%|######3   | 31975/50000 [20:32:47<9:10:53,  1.83s/it]
Outer training loop:  64%|######3   | 31976/50000 [20:32:49<8:56:59,  1.79s/it]
Outer training loop:  64%|######3   | 31977/50000 [20:32:51<9:13:27,  1.84s/it]
Outer training loop:  64%|######3   | 31978/50000 [20:32:53<9:06:23,  1.82s/it]
Outer training loop:  64%|######3   | 31979/50000 [20:32:54<9:03:06,  1.81s/it]
Outer training loop:  64%|######3   | 31980/50000 [20:32:57<9:30:35,  1.90s/it]
Outer training loop:  64%|######3   | 31981/50000 [20:32:58<9:18:13,  1.86s/it]
Outer training loop:  64%|######3   | 31982/50000 [20:33:00<9:08:33,  1.83s/it]
Outer training loop:  64%|######3   | 31983/50000 [20:33:02<9:35:43,  1.92s/it]
Outer training loop:  64%|######3   | 31984/50000 [20:33:04<9:34:15,  1.91s/it]
Outer training loop:  64%|######3   | 31985/50000 [20:33:06<9:32:06,  1.91s/it]
Outer training loop:  64%|######3   | 31986/50000 [20:33:08<9:25:53,  1.88s/it]
Outer training loop:  64%|######3   | 31987/50000 [20:33:10<9:38:37,  1.93s/it]
Outer training loop:  64%|######3   | 31988/50000 [20:33:12<9:39:49,  1.93s/it]
Outer training loop:  64%|######3   | 31989/50000 [20:33:14<9:37:05,  1.92s/it]
Outer training loop:  64%|######3   | 31990/50000 [20:33:15<9:19:02,  1.86s/it]
Outer training loop:  64%|######3   | 31991/50000 [20:33:18<9:36:28,  1.92s/it]
Outer training loop:  64%|######3   | 31992/50000 [20:33:19<9:36:20,  1.92s/it]
Outer training loop:  64%|######3   | 31993/50000 [20:33:21<9:31:37,  1.90s/it]
Outer training loop:  64%|######3   | 31994/50000 [20:33:23<9:42:32,  1.94s/it]
Outer training loop:  64%|######3   | 31995/50000 [20:33:25<9:45:36,  1.95s/it]
Outer training loop:  64%|######3   | 31996/50000 [20:33:27<9:34:04,  1.91s/it]
Outer training loop:  64%|######3   | 31997/50000 [20:33:29<9:50:59,  1.97s/it]
Outer training loop:  64%|######3   | 31998/50000 [20:33:31<9:41:39,  1.94s/it]
Outer training loop:  64%|######3   | 31999/50000 [20:33:33<9:30:50,  1.90s/it]
Outer training loop:  64%|######4   | 32000/50000 [20:33:35<9:29:48,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.70s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.77s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:29,  1.87s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.90s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:22,  1.85s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:25,  1.92s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.94s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:25,  2.02s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:22,  2.00s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:18,  1.98s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:15,  1.96s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:10,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:31<02:10,  1.94s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.92s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.87s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:03,  1.96s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:56,  2.00s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:50,  1.94s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:48,  1.94s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:44,  2.00s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:39,  1.96s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:36,  1.93s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:33,  1.90s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:29,  1.94s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:24,  1.91s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:20,  1.87s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:18,  1.87s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:16,  1.96s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:12,  1.91s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:09,  1.87s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:02,  1.95s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:59,  1.93s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:56,  1.88s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:45,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:47,  1.97s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:44,  1.93s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:30,  1.82s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:31,  1.96s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.93s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:21,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:17,  1.79s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.81s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:14,  2.05s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.97s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.96s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:08,  2.01s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.94s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.87s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.73s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.74s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:35,  1.94s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.92s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.86s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.87s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:28,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:15,  1.86s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:07,  1.85s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:03,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:59,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:02,  1.94s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:50,  1.84s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:54,  1.94s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:42,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:41,  1.85s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:38,  1.81s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:35,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:32,  1.84s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:33,  1.98s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:29,  1.94s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:20,  1.87s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:20,  1.95s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:16,  1.92s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:11,  1.93s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:09,  1.92s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:10,  2.01s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:06,  1.96s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.93s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.92s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.88s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:55,  1.99s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:52,  1.95s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.90s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.91s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.90s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:38,  1.84s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.97s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.93s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.88s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  2.00s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.91s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.88s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.86s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.83s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.81s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:10,  1.81s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:08,  1.79s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.82s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  2.00s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.91s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.88s/it]

Outer training loop:  64%|######4   | 32001/50000 [20:38:57<490:26:46, 98.09s/it]
Outer training loop:  64%|######4   | 32002/50000 [20:38:59<345:56:02, 69.19s/it]
Outer training loop:  64%|######4   | 32003/50000 [20:39:01<244:51:06, 48.98s/it]
Outer training loop:  64%|######4   | 32004/50000 [20:39:03<174:01:43, 34.81s/it]
Outer training loop:  64%|######4   | 32005/50000 [20:39:05<124:34:47, 24.92s/it]
Outer training loop:  64%|######4   | 32006/50000 [20:39:06<90:00:28, 18.01s/it] 
Outer training loop:  64%|######4   | 32007/50000 [20:39:08<65:31:23, 13.11s/it]
Outer training loop:  64%|######4   | 32008/50000 [20:39:10<48:40:21,  9.74s/it]
Outer training loop:  64%|######4   | 32009/50000 [20:39:12<37:43:30,  7.55s/it]
Outer training loop:  64%|######4   | 32010/50000 [20:39:14<29:10:23,  5.84s/it]
Outer training loop:  64%|######4   | 32011/50000 [20:39:16<23:07:10,  4.63s/it]
Outer training loop:  64%|######4   | 32012/50000 [20:39:18<19:10:35,  3.84s/it]
Outer training loop:  64%|######4   | 32013/50000 [20:39:20<16:15:11,  3.25s/it]
Outer training loop:  64%|######4   | 32014/50000 [20:39:22<14:28:42,  2.90s/it]
Outer training loop:  64%|######4   | 32015/50000 [20:39:24<12:46:31,  2.56s/it]
Outer training loop:  64%|######4   | 32016/50000 [20:39:26<11:55:51,  2.39s/it]
Outer training loop:  64%|######4   | 32017/50000 [20:39:28<11:14:59,  2.25s/it]
Outer training loop:  64%|######4   | 32018/50000 [20:39:29<10:33:45,  2.11s/it]
Outer training loop:  64%|######4   | 32019/50000 [20:39:31<10:21:40,  2.07s/it]
Outer training loop:  64%|######4   | 32020/50000 [20:39:33<9:54:34,  1.98s/it] 
Outer training loop:  64%|######4   | 32021/50000 [20:39:35<9:52:51,  1.98s/it]
Outer training loop:  64%|######4   | 32022/50000 [20:39:37<9:59:17,  2.00s/it]
Outer training loop:  64%|######4   | 32023/50000 [20:39:39<9:47:28,  1.96s/it]
Outer training loop:  64%|######4   | 32024/50000 [20:39:41<9:30:28,  1.90s/it]
Outer training loop:  64%|######4   | 32025/50000 [20:39:43<9:21:11,  1.87s/it]
Outer training loop:  64%|######4   | 32026/50000 [20:39:45<9:37:45,  1.93s/it]
Outer training loop:  64%|######4   | 32027/50000 [20:39:47<9:25:00,  1.89s/it]
Outer training loop:  64%|######4   | 32028/50000 [20:39:48<9:19:27,  1.87s/it]
Outer training loop:  64%|######4   | 32029/50000 [20:39:51<9:46:22,  1.96s/it]
Outer training loop:  64%|######4   | 32030/50000 [20:39:52<9:30:31,  1.90s/it]
Outer training loop:  64%|######4   | 32031/50000 [20:39:54<9:28:29,  1.90s/it]
Outer training loop:  64%|######4   | 32032/50000 [20:39:56<9:52:01,  1.98s/it]
Outer training loop:  64%|######4   | 32033/50000 [20:39:58<9:48:44,  1.97s/it]
Outer training loop:  64%|######4   | 32034/50000 [20:40:00<9:39:03,  1.93s/it]
Outer training loop:  64%|######4   | 32035/50000 [20:40:02<9:26:23,  1.89s/it]
Outer training loop:  64%|######4   | 32036/50000 [20:40:04<9:45:41,  1.96s/it]
Outer training loop:  64%|######4   | 32037/50000 [20:40:06<9:44:23,  1.95s/it]
Outer training loop:  64%|######4   | 32038/50000 [20:40:08<9:47:29,  1.96s/it]
Outer training loop:  64%|######4   | 32039/50000 [20:40:10<9:57:05,  1.99s/it]
Outer training loop:  64%|######4   | 32040/50000 [20:40:12<9:36:05,  1.92s/it]
Outer training loop:  64%|######4   | 32041/50000 [20:40:14<9:45:28,  1.96s/it]
Outer training loop:  64%|######4   | 32042/50000 [20:40:16<9:48:18,  1.97s/it]
Outer training loop:  64%|######4   | 32043/50000 [20:40:18<9:35:47,  1.92s/it]
Outer training loop:  64%|######4   | 32044/50000 [20:40:20<9:33:59,  1.92s/it]
Outer training loop:  64%|######4   | 32045/50000 [20:40:21<9:29:09,  1.90s/it]
Outer training loop:  64%|######4   | 32046/50000 [20:40:23<9:40:42,  1.94s/it]
Outer training loop:  64%|######4   | 32047/50000 [20:40:25<9:33:34,  1.92s/it]
Outer training loop:  64%|######4   | 32048/50000 [20:40:27<9:18:30,  1.87s/it]
Outer training loop:  64%|######4   | 32049/50000 [20:40:29<9:28:33,  1.90s/it]
Outer training loop:  64%|######4   | 32050/50000 [20:40:31<9:22:16,  1.88s/it]
Outer training loop:  64%|######4   | 32051/50000 [20:40:33<9:08:53,  1.83s/it]
Outer training loop:  64%|######4   | 32052/50000 [20:40:35<9:32:28,  1.91s/it]
Outer training loop:  64%|######4   | 32053/50000 [20:40:37<9:28:50,  1.90s/it]
Outer training loop:  64%|######4   | 32054/50000 [20:40:38<9:24:29,  1.89s/it]
Outer training loop:  64%|######4   | 32055/50000 [20:40:40<9:40:05,  1.94s/it]
Outer training loop:  64%|######4   | 32056/50000 [20:40:42<9:32:50,  1.92s/it]
Outer training loop:  64%|######4   | 32057/50000 [20:40:44<9:20:22,  1.87s/it]
Outer training loop:  64%|######4   | 32058/50000 [20:40:46<9:23:58,  1.89s/it]
Outer training loop:  64%|######4   | 32059/50000 [20:40:48<9:24:33,  1.89s/it]
Outer training loop:  64%|######4   | 32060/50000 [20:40:50<9:25:54,  1.89s/it]
Outer training loop:  64%|######4   | 32061/50000 [20:40:52<9:21:34,  1.88s/it]
Outer training loop:  64%|######4   | 32062/50000 [20:40:54<9:37:30,  1.93s/it]
Outer training loop:  64%|######4   | 32063/50000 [20:40:56<9:26:19,  1.89s/it]
Outer training loop:  64%|######4   | 32064/50000 [20:40:58<9:37:18,  1.93s/it]
Outer training loop:  64%|######4   | 32065/50000 [20:40:59<9:11:59,  1.85s/it]
Outer training loop:  64%|######4   | 32066/50000 [20:41:01<9:16:06,  1.86s/it]
Outer training loop:  64%|######4   | 32067/50000 [20:41:03<9:03:52,  1.82s/it]
Outer training loop:  64%|######4   | 32068/50000 [20:41:05<9:05:38,  1.83s/it]
Outer training loop:  64%|######4   | 32069/50000 [20:41:07<9:15:21,  1.86s/it]
Outer training loop:  64%|######4   | 32070/50000 [20:41:08<9:14:22,  1.86s/it]
Outer training loop:  64%|######4   | 32071/50000 [20:41:10<9:20:08,  1.87s/it]
Outer training loop:  64%|######4   | 32072/50000 [20:41:12<9:38:18,  1.94s/it]
Outer training loop:  64%|######4   | 32073/50000 [20:41:14<9:27:05,  1.90s/it]
Outer training loop:  64%|######4   | 32074/50000 [20:41:16<9:22:22,  1.88s/it]
Outer training loop:  64%|######4   | 32075/50000 [20:41:18<9:24:12,  1.89s/it]
Outer training loop:  64%|######4   | 32076/50000 [20:41:20<9:43:19,  1.95s/it]
Outer training loop:  64%|######4   | 32077/50000 [20:41:22<9:35:46,  1.93s/it]
Outer training loop:  64%|######4   | 32078/50000 [20:41:24<9:18:41,  1.87s/it]
Outer training loop:  64%|######4   | 32079/50000 [20:41:26<9:38:35,  1.94s/it]
Outer training loop:  64%|######4   | 32080/50000 [20:41:28<9:46:30,  1.96s/it]
Outer training loop:  64%|######4   | 32081/50000 [20:41:30<9:26:58,  1.90s/it]
Outer training loop:  64%|######4   | 32082/50000 [20:41:31<9:22:40,  1.88s/it]
Outer training loop:  64%|######4   | 32083/50000 [20:41:34<9:39:23,  1.94s/it]
Outer training loop:  64%|######4   | 32084/50000 [20:41:35<9:31:02,  1.91s/it]
Outer training loop:  64%|######4   | 32085/50000 [20:41:37<9:26:20,  1.90s/it]
Outer training loop:  64%|######4   | 32086/50000 [20:41:39<9:37:37,  1.93s/it]
Outer training loop:  64%|######4   | 32087/50000 [20:41:41<9:31:00,  1.91s/it]
Outer training loop:  64%|######4   | 32088/50000 [20:41:43<9:25:10,  1.89s/it]
Outer training loop:  64%|######4   | 32089/50000 [20:41:45<9:20:53,  1.88s/it]
Outer training loop:  64%|######4   | 32090/50000 [20:41:47<9:43:18,  1.95s/it]
Outer training loop:  64%|######4   | 32091/50000 [20:41:49<9:30:13,  1.91s/it]
Outer training loop:  64%|######4   | 32092/50000 [20:41:51<9:20:48,  1.88s/it]
Outer training loop:  64%|######4   | 32093/50000 [20:41:53<9:34:45,  1.93s/it]
Outer training loop:  64%|######4   | 32094/50000 [20:41:54<9:20:44,  1.88s/it]
Outer training loop:  64%|######4   | 32095/50000 [20:41:56<9:25:03,  1.89s/it]
Outer training loop:  64%|######4   | 32096/50000 [20:41:58<9:27:22,  1.90s/it]
Outer training loop:  64%|######4   | 32097/50000 [20:42:00<9:44:04,  1.96s/it]
Outer training loop:  64%|######4   | 32098/50000 [20:42:02<9:37:50,  1.94s/it]
Outer training loop:  64%|######4   | 32099/50000 [20:42:04<9:31:12,  1.91s/it]
Outer training loop:  64%|######4   | 32100/50000 [20:42:06<9:47:34,  1.97s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  64%|######4   | 32101/50000 [20:42:14<18:01:47,  3.63s/it]
Outer training loop:  64%|######4   | 32102/50000 [20:42:15<15:23:00,  3.09s/it]
Outer training loop:  64%|######4   | 32103/50000 [20:42:17<13:24:33,  2.70s/it]
Outer training loop:  64%|######4   | 32104/50000 [20:42:19<12:18:04,  2.47s/it]
Outer training loop:  64%|######4   | 32105/50000 [20:42:21<11:22:56,  2.29s/it]
Outer training loop:  64%|######4   | 32106/50000 [20:42:23<10:34:48,  2.13s/it]
Outer training loop:  64%|######4   | 32107/50000 [20:42:25<10:21:19,  2.08s/it]
Outer training loop:  64%|######4   | 32108/50000 [20:42:27<10:11:38,  2.05s/it]
Outer training loop:  64%|######4   | 32109/50000 [20:42:29<9:50:17,  1.98s/it] 
Outer training loop:  64%|######4   | 32110/50000 [20:42:30<9:38:05,  1.94s/it]
Outer training loop:  64%|######4   | 32111/50000 [20:42:33<10:03:29,  2.02s/it]
Outer training loop:  64%|######4   | 32112/50000 [20:42:34<9:35:35,  1.93s/it] 
Outer training loop:  64%|######4   | 32113/50000 [20:42:36<9:21:47,  1.88s/it]
Outer training loop:  64%|######4   | 32114/50000 [20:42:38<9:18:20,  1.87s/it]
Outer training loop:  64%|######4   | 32115/50000 [20:42:40<9:34:13,  1.93s/it]
Outer training loop:  64%|######4   | 32116/50000 [20:42:42<9:25:09,  1.90s/it]
Outer training loop:  64%|######4   | 32117/50000 [20:42:44<9:21:06,  1.88s/it]
Outer training loop:  64%|######4   | 32118/50000 [20:42:46<9:32:32,  1.92s/it]
Outer training loop:  64%|######4   | 32119/50000 [20:42:48<9:31:24,  1.92s/it]
Outer training loop:  64%|######4   | 32120/50000 [20:42:49<9:21:22,  1.88s/it]
Outer training loop:  64%|######4   | 32121/50000 [20:42:51<9:12:23,  1.85s/it]
Outer training loop:  64%|######4   | 32122/50000 [20:42:53<9:19:08,  1.88s/it]
Outer training loop:  64%|######4   | 32123/50000 [20:42:55<9:16:57,  1.87s/it]
Outer training loop:  64%|######4   | 32124/50000 [20:42:57<9:08:51,  1.84s/it]
Outer training loop:  64%|######4   | 32125/50000 [20:42:59<9:30:44,  1.92s/it]
Outer training loop:  64%|######4   | 32126/50000 [20:43:01<9:16:31,  1.87s/it]
Outer training loop:  64%|######4   | 32127/50000 [20:43:02<9:06:06,  1.83s/it]
Outer training loop:  64%|######4   | 32128/50000 [20:43:04<8:54:17,  1.79s/it]
Outer training loop:  64%|######4   | 32129/50000 [20:43:06<9:19:04,  1.88s/it]
Outer training loop:  64%|######4   | 32130/50000 [20:43:08<9:15:33,  1.87s/it]
Outer training loop:  64%|######4   | 32131/50000 [20:43:10<9:19:24,  1.88s/it]
Outer training loop:  64%|######4   | 32132/50000 [20:43:12<9:48:01,  1.97s/it]
Outer training loop:  64%|######4   | 32133/50000 [20:43:14<9:26:38,  1.90s/it]
Outer training loop:  64%|######4   | 32134/50000 [20:43:16<9:13:07,  1.86s/it]
Outer training loop:  64%|######4   | 32135/50000 [20:43:18<9:22:28,  1.89s/it]
Outer training loop:  64%|######4   | 32136/50000 [20:43:19<9:13:28,  1.86s/it]
Outer training loop:  64%|######4   | 32137/50000 [20:43:21<9:20:42,  1.88s/it]
Outer training loop:  64%|######4   | 32138/50000 [20:43:23<9:19:42,  1.88s/it]
Outer training loop:  64%|######4   | 32139/50000 [20:43:25<9:39:39,  1.95s/it]
Outer training loop:  64%|######4   | 32140/50000 [20:43:27<9:26:21,  1.90s/it]
Outer training loop:  64%|######4   | 32141/50000 [20:43:29<9:14:32,  1.86s/it]
Outer training loop:  64%|######4   | 32142/50000 [20:43:31<9:22:30,  1.89s/it]
Outer training loop:  64%|######4   | 32143/50000 [20:43:33<9:17:50,  1.87s/it]
Outer training loop:  64%|######4   | 32144/50000 [20:43:34<9:15:55,  1.87s/it]
Outer training loop:  64%|######4   | 32145/50000 [20:43:36<9:08:12,  1.84s/it]
Outer training loop:  64%|######4   | 32146/50000 [20:43:38<9:29:03,  1.91s/it]
Outer training loop:  64%|######4   | 32147/50000 [20:43:40<9:16:11,  1.87s/it]
Outer training loop:  64%|######4   | 32148/50000 [20:43:42<9:08:51,  1.84s/it]
Outer training loop:  64%|######4   | 32149/50000 [20:43:44<9:17:58,  1.88s/it]
Outer training loop:  64%|######4   | 32150/50000 [20:43:46<9:08:52,  1.84s/it]
Outer training loop:  64%|######4   | 32151/50000 [20:43:47<9:04:26,  1.83s/it]
Outer training loop:  64%|######4   | 32152/50000 [20:43:49<9:22:23,  1.89s/it]
Outer training loop:  64%|######4   | 32153/50000 [20:43:51<9:23:40,  1.90s/it]
Outer training loop:  64%|######4   | 32154/50000 [20:43:53<9:21:20,  1.89s/it]
Outer training loop:  64%|######4   | 32155/50000 [20:43:55<9:17:23,  1.87s/it]
Outer training loop:  64%|######4   | 32156/50000 [20:43:57<9:28:29,  1.91s/it]
Outer training loop:  64%|######4   | 32157/50000 [20:43:59<9:23:48,  1.90s/it]
Outer training loop:  64%|######4   | 32158/50000 [20:44:01<9:32:43,  1.93s/it]
Outer training loop:  64%|######4   | 32159/50000 [20:44:03<9:34:06,  1.93s/it]
Outer training loop:  64%|######4   | 32160/50000 [20:44:05<9:29:42,  1.92s/it]
Outer training loop:  64%|######4   | 32161/50000 [20:44:07<9:41:53,  1.96s/it]
Outer training loop:  64%|######4   | 32162/50000 [20:44:09<9:55:43,  2.00s/it]
Outer training loop:  64%|######4   | 32163/50000 [20:44:11<9:39:18,  1.95s/it]
Outer training loop:  64%|######4   | 32164/50000 [20:44:13<9:31:34,  1.92s/it]
Outer training loop:  64%|######4   | 32165/50000 [20:44:15<9:46:06,  1.97s/it]
Outer training loop:  64%|######4   | 32166/50000 [20:44:16<9:35:01,  1.93s/it]
Outer training loop:  64%|######4   | 32167/50000 [20:44:18<9:14:02,  1.86s/it]
Outer training loop:  64%|######4   | 32168/50000 [20:44:20<9:26:56,  1.91s/it]
Outer training loop:  64%|######4   | 32169/50000 [20:44:22<9:17:51,  1.88s/it]
Outer training loop:  64%|######4   | 32170/50000 [20:44:24<9:14:02,  1.86s/it]
Outer training loop:  64%|######4   | 32171/50000 [20:44:26<9:29:48,  1.92s/it]
Outer training loop:  64%|######4   | 32172/50000 [20:44:28<9:24:26,  1.90s/it]
Outer training loop:  64%|######4   | 32173/50000 [20:44:30<9:20:29,  1.89s/it]
Outer training loop:  64%|######4   | 32174/50000 [20:44:31<9:08:10,  1.85s/it]
Outer training loop:  64%|######4   | 32175/50000 [20:44:33<9:28:26,  1.91s/it]
Outer training loop:  64%|######4   | 32176/50000 [20:44:35<9:24:14,  1.90s/it]
Outer training loop:  64%|######4   | 32177/50000 [20:44:37<9:19:50,  1.88s/it]
Outer training loop:  64%|######4   | 32178/50000 [20:44:39<9:43:12,  1.96s/it]
Outer training loop:  64%|######4   | 32179/50000 [20:44:41<9:24:33,  1.90s/it]
Outer training loop:  64%|######4   | 32180/50000 [20:44:43<9:18:35,  1.88s/it]
Outer training loop:  64%|######4   | 32181/50000 [20:44:45<9:43:43,  1.97s/it]
Outer training loop:  64%|######4   | 32182/50000 [20:44:47<9:25:58,  1.91s/it]
Outer training loop:  64%|######4   | 32183/50000 [20:44:49<9:17:49,  1.88s/it]
Outer training loop:  64%|######4   | 32184/50000 [20:44:51<9:22:43,  1.90s/it]
Outer training loop:  64%|######4   | 32185/50000 [20:44:53<9:34:07,  1.93s/it]
Outer training loop:  64%|######4   | 32186/50000 [20:44:54<9:26:07,  1.91s/it]
Outer training loop:  64%|######4   | 32187/50000 [20:44:56<9:23:31,  1.90s/it]
Outer training loop:  64%|######4   | 32188/50000 [20:44:58<9:32:35,  1.93s/it]
Outer training loop:  64%|######4   | 32189/50000 [20:45:00<9:20:12,  1.89s/it]
Outer training loop:  64%|######4   | 32190/50000 [20:45:02<9:21:13,  1.89s/it]
Outer training loop:  64%|######4   | 32191/50000 [20:45:04<9:13:12,  1.86s/it]
Outer training loop:  64%|######4   | 32192/50000 [20:45:06<9:26:53,  1.91s/it]
Outer training loop:  64%|######4   | 32193/50000 [20:45:08<9:19:33,  1.89s/it]
Outer training loop:  64%|######4   | 32194/50000 [20:45:10<9:27:26,  1.91s/it]
Outer training loop:  64%|######4   | 32195/50000 [20:45:12<9:27:12,  1.91s/it]
Outer training loop:  64%|######4   | 32196/50000 [20:45:13<9:19:59,  1.89s/it]
Outer training loop:  64%|######4   | 32197/50000 [20:45:15<9:08:05,  1.85s/it]
Outer training loop:  64%|######4   | 32198/50000 [20:45:17<9:16:27,  1.88s/it]
Outer training loop:  64%|######4   | 32199/50000 [20:45:19<9:22:11,  1.89s/it]
Outer training loop:  64%|######4   | 32200/50000 [20:45:21<9:26:33,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  64%|######4   | 32201/50000 [20:45:28<17:40:22,  3.57s/it]
Outer training loop:  64%|######4   | 32202/50000 [20:45:30<15:22:27,  3.11s/it]
Outer training loop:  64%|######4   | 32203/50000 [20:45:32<13:17:45,  2.69s/it]
Outer training loop:  64%|######4   | 32204/50000 [20:45:34<12:02:33,  2.44s/it]
Outer training loop:  64%|######4   | 32205/50000 [20:45:36<11:32:28,  2.33s/it]
Outer training loop:  64%|######4   | 32206/50000 [20:45:38<10:46:14,  2.18s/it]
Outer training loop:  64%|######4   | 32207/50000 [20:45:40<10:18:01,  2.08s/it]
Outer training loop:  64%|######4   | 32208/50000 [20:45:42<10:13:16,  2.07s/it]
Outer training loop:  64%|######4   | 32209/50000 [20:45:44<10:19:04,  2.09s/it]
Outer training loop:  64%|######4   | 32210/50000 [20:45:46<9:50:05,  1.99s/it] 
Outer training loop:  64%|######4   | 32211/50000 [20:45:48<9:47:08,  1.98s/it]
Outer training loop:  64%|######4   | 32212/50000 [20:45:50<10:08:20,  2.05s/it]
Outer training loop:  64%|######4   | 32213/50000 [20:45:52<9:48:49,  1.99s/it] 
Outer training loop:  64%|######4   | 32214/50000 [20:45:53<9:31:28,  1.93s/it]
Outer training loop:  64%|######4   | 32215/50000 [20:45:55<9:17:43,  1.88s/it]
Outer training loop:  64%|######4   | 32216/50000 [20:45:57<9:39:24,  1.95s/it]
Outer training loop:  64%|######4   | 32217/50000 [20:45:59<9:30:56,  1.93s/it]
Outer training loop:  64%|######4   | 32218/50000 [20:46:01<9:12:33,  1.86s/it]
Outer training loop:  64%|######4   | 32219/50000 [20:46:03<9:28:59,  1.92s/it]
Outer training loop:  64%|######4   | 32220/50000 [20:46:05<9:39:22,  1.96s/it]
Outer training loop:  64%|######4   | 32221/50000 [20:46:07<9:37:48,  1.95s/it]
Outer training loop:  64%|######4   | 32222/50000 [20:46:09<9:21:40,  1.90s/it]
Outer training loop:  64%|######4   | 32223/50000 [20:46:11<9:40:28,  1.96s/it]
Outer training loop:  64%|######4   | 32224/50000 [20:46:13<9:35:10,  1.94s/it]
Outer training loop:  64%|######4   | 32225/50000 [20:46:15<9:20:33,  1.89s/it]
Outer training loop:  64%|######4   | 32226/50000 [20:46:17<9:36:03,  1.94s/it]
Outer training loop:  64%|######4   | 32227/50000 [20:46:18<9:18:11,  1.88s/it]
Outer training loop:  64%|######4   | 32228/50000 [20:46:20<9:09:07,  1.85s/it]
Outer training loop:  64%|######4   | 32229/50000 [20:46:22<9:14:28,  1.87s/it]
Outer training loop:  64%|######4   | 32230/50000 [20:46:24<9:08:15,  1.85s/it]
Outer training loop:  64%|######4   | 32231/50000 [20:46:26<9:15:29,  1.88s/it]
Outer training loop:  64%|######4   | 32232/50000 [20:46:28<9:23:02,  1.90s/it]
Outer training loop:  64%|######4   | 32233/50000 [20:46:30<9:28:55,  1.92s/it]
Outer training loop:  64%|######4   | 32234/50000 [20:46:31<9:16:46,  1.88s/it]
Outer training loop:  64%|######4   | 32235/50000 [20:46:33<9:16:07,  1.88s/it]
Outer training loop:  64%|######4   | 32236/50000 [20:46:35<9:25:44,  1.91s/it]
Outer training loop:  64%|######4   | 32237/50000 [20:46:37<9:06:43,  1.85s/it]
Outer training loop:  64%|######4   | 32238/50000 [20:46:39<9:13:23,  1.87s/it]
Outer training loop:  64%|######4   | 32239/50000 [20:46:41<9:34:07,  1.94s/it]
Outer training loop:  64%|######4   | 32240/50000 [20:46:43<9:32:44,  1.93s/it]
Outer training loop:  64%|######4   | 32241/50000 [20:46:45<9:24:38,  1.91s/it]
Outer training loop:  64%|######4   | 32242/50000 [20:46:47<9:09:46,  1.86s/it]
Outer training loop:  64%|######4   | 32243/50000 [20:46:49<9:28:58,  1.92s/it]
Outer training loop:  64%|######4   | 32244/50000 [20:46:50<9:20:27,  1.89s/it]
Outer training loop:  64%|######4   | 32245/50000 [20:46:52<9:10:54,  1.86s/it]
Outer training loop:  64%|######4   | 32246/50000 [20:46:54<9:21:53,  1.90s/it]
Outer training loop:  64%|######4   | 32247/50000 [20:46:56<9:15:08,  1.88s/it]
Outer training loop:  64%|######4   | 32248/50000 [20:46:58<9:14:53,  1.88s/it]
Outer training loop:  64%|######4   | 32249/50000 [20:47:00<9:18:19,  1.89s/it]
Outer training loop:  64%|######4   | 32250/50000 [20:47:02<9:28:15,  1.92s/it]
Outer training loop:  65%|######4   | 32251/50000 [20:47:04<9:16:31,  1.88s/it]
Outer training loop:  65%|######4   | 32252/50000 [20:47:05<9:12:10,  1.87s/it]
Outer training loop:  65%|######4   | 32253/50000 [20:47:07<9:21:02,  1.90s/it]
Outer training loop:  65%|######4   | 32254/50000 [20:47:09<9:16:56,  1.88s/it]
Outer training loop:  65%|######4   | 32255/50000 [20:47:11<9:15:29,  1.88s/it]
Outer training loop:  65%|######4   | 32256/50000 [20:47:13<9:29:05,  1.92s/it]
Outer training loop:  65%|######4   | 32257/50000 [20:47:15<9:14:11,  1.87s/it]
Outer training loop:  65%|######4   | 32258/50000 [20:47:17<9:20:21,  1.90s/it]
Outer training loop:  65%|######4   | 32259/50000 [20:47:19<9:07:54,  1.85s/it]
Outer training loop:  65%|######4   | 32260/50000 [20:47:21<9:24:49,  1.91s/it]
Outer training loop:  65%|######4   | 32261/50000 [20:47:22<9:03:30,  1.84s/it]
Outer training loop:  65%|######4   | 32262/50000 [20:47:24<9:03:29,  1.84s/it]
Outer training loop:  65%|######4   | 32263/50000 [20:47:26<9:26:25,  1.92s/it]
Outer training loop:  65%|######4   | 32264/50000 [20:47:28<9:26:34,  1.92s/it]
Outer training loop:  65%|######4   | 32265/50000 [20:47:30<9:19:43,  1.89s/it]
Outer training loop:  65%|######4   | 32266/50000 [20:47:32<9:04:22,  1.84s/it]
Outer training loop:  65%|######4   | 32267/50000 [20:47:34<9:24:18,  1.91s/it]
Outer training loop:  65%|######4   | 32268/50000 [20:47:36<9:18:11,  1.89s/it]
Outer training loop:  65%|######4   | 32269/50000 [20:47:38<9:17:06,  1.89s/it]
Outer training loop:  65%|######4   | 32270/50000 [20:47:40<9:30:41,  1.93s/it]
Outer training loop:  65%|######4   | 32271/50000 [20:47:42<9:31:06,  1.93s/it]
Outer training loop:  65%|######4   | 32272/50000 [20:47:43<9:31:19,  1.93s/it]
Outer training loop:  65%|######4   | 32273/50000 [20:47:45<9:37:06,  1.95s/it]
Outer training loop:  65%|######4   | 32274/50000 [20:47:47<9:25:59,  1.92s/it]
Outer training loop:  65%|######4   | 32275/50000 [20:47:49<9:15:33,  1.88s/it]
Outer training loop:  65%|######4   | 32276/50000 [20:47:51<9:44:00,  1.98s/it]
Outer training loop:  65%|######4   | 32277/50000 [20:47:53<9:29:39,  1.93s/it]
Outer training loop:  65%|######4   | 32278/50000 [20:47:55<9:24:33,  1.91s/it]
Outer training loop:  65%|######4   | 32279/50000 [20:47:57<9:44:23,  1.98s/it]
Outer training loop:  65%|######4   | 32280/50000 [20:47:59<9:31:03,  1.93s/it]
Outer training loop:  65%|######4   | 32281/50000 [20:48:01<9:15:42,  1.88s/it]
Outer training loop:  65%|######4   | 32282/50000 [20:48:02<8:59:12,  1.83s/it]
Outer training loop:  65%|######4   | 32283/50000 [20:48:05<9:23:25,  1.91s/it]
Outer training loop:  65%|######4   | 32284/50000 [20:48:06<9:24:38,  1.91s/it]
Outer training loop:  65%|######4   | 32285/50000 [20:48:08<9:15:01,  1.88s/it]
Outer training loop:  65%|######4   | 32286/50000 [20:48:10<9:32:32,  1.94s/it]
Outer training loop:  65%|######4   | 32287/50000 [20:48:12<9:31:55,  1.94s/it]
Outer training loop:  65%|######4   | 32288/50000 [20:48:14<9:07:35,  1.85s/it]
Outer training loop:  65%|######4   | 32289/50000 [20:48:16<9:02:57,  1.84s/it]
Outer training loop:  65%|######4   | 32290/50000 [20:48:18<9:12:24,  1.87s/it]
Outer training loop:  65%|######4   | 32291/50000 [20:48:20<9:23:51,  1.91s/it]
Outer training loop:  65%|######4   | 32292/50000 [20:48:22<9:20:26,  1.90s/it]
Outer training loop:  65%|######4   | 32293/50000 [20:48:23<9:23:02,  1.91s/it]
Outer training loop:  65%|######4   | 32294/50000 [20:48:25<9:20:28,  1.90s/it]
Outer training loop:  65%|######4   | 32295/50000 [20:48:27<9:15:59,  1.88s/it]
Outer training loop:  65%|######4   | 32296/50000 [20:48:29<9:13:19,  1.88s/it]
Outer training loop:  65%|######4   | 32297/50000 [20:48:31<9:21:55,  1.90s/it]
Outer training loop:  65%|######4   | 32298/50000 [20:48:33<9:14:45,  1.88s/it]
Outer training loop:  65%|######4   | 32299/50000 [20:48:35<9:13:27,  1.88s/it]
Outer training loop:  65%|######4   | 32300/50000 [20:48:37<9:16:31,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  65%|######4   | 32301/50000 [20:48:44<17:07:37,  3.48s/it]
Outer training loop:  65%|######4   | 32302/50000 [20:48:46<14:48:15,  3.01s/it]
Outer training loop:  65%|######4   | 32303/50000 [20:48:47<12:53:44,  2.62s/it]
Outer training loop:  65%|######4   | 32304/50000 [20:48:49<11:58:01,  2.43s/it]
Outer training loop:  65%|######4   | 32305/50000 [20:48:51<11:20:01,  2.31s/it]
Outer training loop:  65%|######4   | 32306/50000 [20:48:53<10:43:41,  2.18s/it]
Outer training loop:  65%|######4   | 32307/50000 [20:48:55<10:23:40,  2.12s/it]
Outer training loop:  65%|######4   | 32308/50000 [20:48:57<9:53:17,  2.01s/it] 
Outer training loop:  65%|######4   | 32309/50000 [20:48:59<9:40:02,  1.97s/it]
Outer training loop:  65%|######4   | 32310/50000 [20:49:01<9:31:19,  1.94s/it]
Outer training loop:  65%|######4   | 32311/50000 [20:49:03<9:47:55,  1.99s/it]
Outer training loop:  65%|######4   | 32312/50000 [20:49:05<9:35:28,  1.95s/it]
Outer training loop:  65%|######4   | 32313/50000 [20:49:07<9:31:17,  1.94s/it]
Outer training loop:  65%|######4   | 32314/50000 [20:49:09<9:29:31,  1.93s/it]
Outer training loop:  65%|######4   | 32315/50000 [20:49:11<9:25:35,  1.92s/it]
Outer training loop:  65%|######4   | 32316/50000 [20:49:12<9:18:06,  1.89s/it]
Outer training loop:  65%|######4   | 32317/50000 [20:49:14<9:11:44,  1.87s/it]
Outer training loop:  65%|######4   | 32318/50000 [20:49:16<9:13:20,  1.88s/it]
Outer training loop:  65%|######4   | 32319/50000 [20:49:18<9:11:42,  1.87s/it]
Outer training loop:  65%|######4   | 32320/50000 [20:49:20<9:07:26,  1.86s/it]
Outer training loop:  65%|######4   | 32321/50000 [20:49:22<9:13:53,  1.88s/it]
Outer training loop:  65%|######4   | 32322/50000 [20:49:24<9:27:47,  1.93s/it]
Outer training loop:  65%|######4   | 32323/50000 [20:49:26<9:27:43,  1.93s/it]
Outer training loop:  65%|######4   | 32324/50000 [20:49:28<9:26:30,  1.92s/it]
Outer training loop:  65%|######4   | 32325/50000 [20:49:30<9:35:46,  1.95s/it]
Outer training loop:  65%|######4   | 32326/50000 [20:49:31<9:27:27,  1.93s/it]
Outer training loop:  65%|######4   | 32327/50000 [20:49:33<9:20:12,  1.90s/it]
Outer training loop:  65%|######4   | 32328/50000 [20:49:35<9:44:03,  1.98s/it]
Outer training loop:  65%|######4   | 32329/50000 [20:49:37<9:41:41,  1.98s/it]
Outer training loop:  65%|######4   | 32330/50000 [20:49:39<9:16:54,  1.89s/it]
Outer training loop:  65%|######4   | 32331/50000 [20:49:41<9:07:59,  1.86s/it]
Outer training loop:  65%|######4   | 32332/50000 [20:49:43<9:31:36,  1.94s/it]
Outer training loop:  65%|######4   | 32333/50000 [20:49:45<9:19:38,  1.90s/it]
Outer training loop:  65%|######4   | 32334/50000 [20:49:47<9:01:46,  1.84s/it]
Outer training loop:  65%|######4   | 32335/50000 [20:49:48<9:01:40,  1.84s/it]
Outer training loop:  65%|######4   | 32336/50000 [20:49:51<9:29:18,  1.93s/it]
Outer training loop:  65%|######4   | 32337/50000 [20:49:52<9:14:48,  1.88s/it]
Outer training loop:  65%|######4   | 32338/50000 [20:49:54<9:09:25,  1.87s/it]
Outer training loop:  65%|######4   | 32339/50000 [20:49:56<9:33:18,  1.95s/it]
Outer training loop:  65%|######4   | 32340/50000 [20:49:58<9:24:07,  1.92s/it]
Outer training loop:  65%|######4   | 32341/50000 [20:50:00<9:17:45,  1.90s/it]
Outer training loop:  65%|######4   | 32342/50000 [20:50:02<8:57:38,  1.83s/it]
Outer training loop:  65%|######4   | 32343/50000 [20:50:04<9:06:15,  1.86s/it]
Outer training loop:  65%|######4   | 32344/50000 [20:50:05<9:07:42,  1.86s/it]
Outer training loop:  65%|######4   | 32345/50000 [20:50:07<9:06:20,  1.86s/it]
Outer training loop:  65%|######4   | 32346/50000 [20:50:09<9:21:29,  1.91s/it]
Outer training loop:  65%|######4   | 32347/50000 [20:50:11<9:04:03,  1.85s/it]
Outer training loop:  65%|######4   | 32348/50000 [20:50:13<9:12:42,  1.88s/it]
Outer training loop:  65%|######4   | 32349/50000 [20:50:15<9:17:49,  1.90s/it]
Outer training loop:  65%|######4   | 32350/50000 [20:50:17<9:09:38,  1.87s/it]
Outer training loop:  65%|######4   | 32351/50000 [20:50:19<9:14:00,  1.88s/it]
Outer training loop:  65%|######4   | 32352/50000 [20:50:20<9:12:04,  1.88s/it]
Outer training loop:  65%|######4   | 32353/50000 [20:50:22<9:17:36,  1.90s/it]
Outer training loop:  65%|######4   | 32354/50000 [20:50:24<9:05:04,  1.85s/it]
Outer training loop:  65%|######4   | 32355/50000 [20:50:26<9:30:37,  1.94s/it]
Outer training loop:  65%|######4   | 32356/50000 [20:50:28<9:37:33,  1.96s/it]
Outer training loop:  65%|######4   | 32357/50000 [20:50:30<9:18:26,  1.90s/it]
Outer training loop:  65%|######4   | 32358/50000 [20:50:32<9:05:33,  1.86s/it]
Outer training loop:  65%|######4   | 32359/50000 [20:50:34<8:53:26,  1.81s/it]
Outer training loop:  65%|######4   | 32360/50000 [20:50:36<9:14:08,  1.88s/it]
Outer training loop:  65%|######4   | 32361/50000 [20:50:37<9:09:56,  1.87s/it]
Outer training loop:  65%|######4   | 32362/50000 [20:50:39<9:05:38,  1.86s/it]
Outer training loop:  65%|######4   | 32363/50000 [20:50:41<9:23:49,  1.92s/it]
Outer training loop:  65%|######4   | 32364/50000 [20:50:43<9:24:44,  1.92s/it]
Outer training loop:  65%|######4   | 32365/50000 [20:50:45<9:13:13,  1.88s/it]
Outer training loop:  65%|######4   | 32366/50000 [20:50:47<9:19:23,  1.90s/it]
Outer training loop:  65%|######4   | 32367/50000 [20:50:49<9:04:55,  1.85s/it]
Outer training loop:  65%|######4   | 32368/50000 [20:50:51<9:12:13,  1.88s/it]
Outer training loop:  65%|######4   | 32369/50000 [20:50:52<9:00:59,  1.84s/it]
Outer training loop:  65%|######4   | 32370/50000 [20:50:54<9:09:31,  1.87s/it]
Outer training loop:  65%|######4   | 32371/50000 [20:50:56<9:07:11,  1.86s/it]
Outer training loop:  65%|######4   | 32372/50000 [20:50:58<9:08:52,  1.87s/it]
Outer training loop:  65%|######4   | 32373/50000 [20:51:00<9:21:18,  1.91s/it]
Outer training loop:  65%|######4   | 32374/50000 [20:51:02<9:10:55,  1.88s/it]
Outer training loop:  65%|######4   | 32375/50000 [20:51:04<9:00:12,  1.84s/it]
Outer training loop:  65%|######4   | 32376/50000 [20:51:06<9:04:03,  1.85s/it]
Outer training loop:  65%|######4   | 32377/50000 [20:51:08<9:21:52,  1.91s/it]
Outer training loop:  65%|######4   | 32378/50000 [20:51:09<9:12:37,  1.88s/it]
Outer training loop:  65%|######4   | 32379/50000 [20:51:11<9:10:14,  1.87s/it]
Outer training loop:  65%|######4   | 32380/50000 [20:51:13<9:17:00,  1.90s/it]
Outer training loop:  65%|######4   | 32381/50000 [20:51:15<9:08:24,  1.87s/it]
Outer training loop:  65%|######4   | 32382/50000 [20:51:17<9:02:03,  1.85s/it]
Outer training loop:  65%|######4   | 32383/50000 [20:51:19<9:08:53,  1.87s/it]
Outer training loop:  65%|######4   | 32384/50000 [20:51:21<9:05:22,  1.86s/it]
Outer training loop:  65%|######4   | 32385/50000 [20:51:22<9:08:36,  1.87s/it]
Outer training loop:  65%|######4   | 32386/50000 [20:51:24<8:59:56,  1.84s/it]
Outer training loop:  65%|######4   | 32387/50000 [20:51:26<9:16:07,  1.89s/it]
Outer training loop:  65%|######4   | 32388/50000 [20:51:28<9:14:50,  1.89s/it]
Outer training loop:  65%|######4   | 32389/50000 [20:51:30<9:04:39,  1.86s/it]
Outer training loop:  65%|######4   | 32390/50000 [20:51:32<9:12:15,  1.88s/it]
Outer training loop:  65%|######4   | 32391/50000 [20:51:34<9:13:06,  1.88s/it]
Outer training loop:  65%|######4   | 32392/50000 [20:51:36<9:05:07,  1.86s/it]
Outer training loop:  65%|######4   | 32393/50000 [20:51:37<9:08:05,  1.87s/it]
Outer training loop:  65%|######4   | 32394/50000 [20:51:39<9:04:13,  1.85s/it]
Outer training loop:  65%|######4   | 32395/50000 [20:51:41<9:00:14,  1.84s/it]
Outer training loop:  65%|######4   | 32396/50000 [20:51:43<8:59:35,  1.84s/it]
Outer training loop:  65%|######4   | 32397/50000 [20:51:45<9:10:02,  1.87s/it]
Outer training loop:  65%|######4   | 32398/50000 [20:51:47<9:06:17,  1.86s/it]
Outer training loop:  65%|######4   | 32399/50000 [20:51:49<9:10:01,  1.87s/it]
Outer training loop:  65%|######4   | 32400/50000 [20:51:51<9:18:58,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  65%|######4   | 32401/50000 [20:51:58<17:32:39,  3.59s/it]
Outer training loop:  65%|######4   | 32402/50000 [20:52:00<15:12:18,  3.11s/it]
Outer training loop:  65%|######4   | 32403/50000 [20:52:02<13:14:33,  2.71s/it]
Outer training loop:  65%|######4   | 32404/50000 [20:52:04<12:18:34,  2.52s/it]
Outer training loop:  65%|######4   | 32405/50000 [20:52:06<11:26:59,  2.34s/it]
Outer training loop:  65%|######4   | 32406/50000 [20:52:08<10:45:17,  2.20s/it]
Outer training loop:  65%|######4   | 32407/50000 [20:52:10<10:11:48,  2.09s/it]
Outer training loop:  65%|######4   | 32408/50000 [20:52:12<10:07:34,  2.07s/it]
Outer training loop:  65%|######4   | 32409/50000 [20:52:13<9:41:53,  1.98s/it] 
Outer training loop:  65%|######4   | 32410/50000 [20:52:15<9:39:37,  1.98s/it]
Outer training loop:  65%|######4   | 32411/50000 [20:52:17<9:26:18,  1.93s/it]
Outer training loop:  65%|######4   | 32412/50000 [20:52:19<9:30:36,  1.95s/it]
Outer training loop:  65%|######4   | 32413/50000 [20:52:21<9:33:18,  1.96s/it]
Outer training loop:  65%|######4   | 32414/50000 [20:52:23<9:33:06,  1.96s/it]
Outer training loop:  65%|######4   | 32415/50000 [20:52:25<9:31:17,  1.95s/it]
Outer training loop:  65%|######4   | 32416/50000 [20:52:27<9:24:39,  1.93s/it]
Outer training loop:  65%|######4   | 32417/50000 [20:52:29<9:06:54,  1.87s/it]
Outer training loop:  65%|######4   | 32418/50000 [20:52:31<9:15:29,  1.90s/it]
Outer training loop:  65%|######4   | 32419/50000 [20:52:32<9:02:38,  1.85s/it]
Outer training loop:  65%|######4   | 32420/50000 [20:52:34<8:59:37,  1.84s/it]
Outer training loop:  65%|######4   | 32421/50000 [20:52:36<9:12:02,  1.88s/it]
Outer training loop:  65%|######4   | 32422/50000 [20:52:38<9:17:11,  1.90s/it]
Outer training loop:  65%|######4   | 32423/50000 [20:52:40<9:20:30,  1.91s/it]
Outer training loop:  65%|######4   | 32424/50000 [20:52:42<9:13:12,  1.89s/it]
Outer training loop:  65%|######4   | 32425/50000 [20:52:44<9:22:25,  1.92s/it]
Outer training loop:  65%|######4   | 32426/50000 [20:52:46<9:16:56,  1.90s/it]
Outer training loop:  65%|######4   | 32427/50000 [20:52:48<9:17:19,  1.90s/it]
Outer training loop:  65%|######4   | 32428/50000 [20:52:50<9:33:51,  1.96s/it]
Outer training loop:  65%|######4   | 32429/50000 [20:52:51<9:18:01,  1.91s/it]
Outer training loop:  65%|######4   | 32430/50000 [20:52:53<9:09:16,  1.88s/it]
Outer training loop:  65%|######4   | 32431/50000 [20:52:55<9:09:36,  1.88s/it]
Outer training loop:  65%|######4   | 32432/50000 [20:52:57<9:12:27,  1.89s/it]
Outer training loop:  65%|######4   | 32433/50000 [20:52:59<9:14:52,  1.90s/it]
Outer training loop:  65%|######4   | 32434/50000 [20:53:01<9:02:26,  1.85s/it]
Outer training loop:  65%|######4   | 32435/50000 [20:53:03<9:11:13,  1.88s/it]
Outer training loop:  65%|######4   | 32436/50000 [20:53:05<9:13:54,  1.89s/it]
Outer training loop:  65%|######4   | 32437/50000 [20:53:06<9:06:22,  1.87s/it]
Outer training loop:  65%|######4   | 32438/50000 [20:53:08<9:21:11,  1.92s/it]
Outer training loop:  65%|######4   | 32439/50000 [20:53:10<9:10:01,  1.88s/it]
Outer training loop:  65%|######4   | 32440/50000 [20:53:12<9:06:04,  1.87s/it]
Outer training loop:  65%|######4   | 32441/50000 [20:53:14<9:08:24,  1.87s/it]
Outer training loop:  65%|######4   | 32442/50000 [20:53:16<9:16:33,  1.90s/it]
Outer training loop:  65%|######4   | 32443/50000 [20:53:18<9:08:19,  1.87s/it]
Outer training loop:  65%|######4   | 32444/50000 [20:53:19<9:01:01,  1.85s/it]
Outer training loop:  65%|######4   | 32445/50000 [20:53:22<9:28:30,  1.94s/it]
Outer training loop:  65%|######4   | 32446/50000 [20:53:24<9:23:39,  1.93s/it]
Outer training loop:  65%|######4   | 32447/50000 [20:53:25<9:13:47,  1.89s/it]
Outer training loop:  65%|######4   | 32448/50000 [20:53:27<9:11:03,  1.88s/it]
Outer training loop:  65%|######4   | 32449/50000 [20:53:29<9:31:23,  1.95s/it]
Outer training loop:  65%|######4   | 32450/50000 [20:53:31<9:14:09,  1.89s/it]
Outer training loop:  65%|######4   | 32451/50000 [20:53:33<9:11:14,  1.88s/it]
Outer training loop:  65%|######4   | 32452/50000 [20:53:35<9:45:09,  2.00s/it]
Outer training loop:  65%|######4   | 32453/50000 [20:53:37<9:27:22,  1.94s/it]
Outer training loop:  65%|######4   | 32454/50000 [20:53:39<9:16:46,  1.90s/it]
Outer training loop:  65%|######4   | 32455/50000 [20:53:41<9:24:10,  1.93s/it]
Outer training loop:  65%|######4   | 32456/50000 [20:53:43<9:36:34,  1.97s/it]
Outer training loop:  65%|######4   | 32457/50000 [20:53:45<9:12:21,  1.89s/it]
Outer training loop:  65%|######4   | 32458/50000 [20:53:47<9:19:30,  1.91s/it]
Outer training loop:  65%|######4   | 32459/50000 [20:53:49<9:32:20,  1.96s/it]
Outer training loop:  65%|######4   | 32460/50000 [20:53:50<9:23:40,  1.93s/it]
Outer training loop:  65%|######4   | 32461/50000 [20:53:52<9:16:08,  1.90s/it]
Outer training loop:  65%|######4   | 32462/50000 [20:53:54<9:16:09,  1.90s/it]
Outer training loop:  65%|######4   | 32463/50000 [20:53:56<9:08:54,  1.88s/it]
Outer training loop:  65%|######4   | 32464/50000 [20:53:58<9:13:24,  1.89s/it]
Outer training loop:  65%|######4   | 32465/50000 [20:54:00<9:01:34,  1.85s/it]
Outer training loop:  65%|######4   | 32466/50000 [20:54:02<9:15:55,  1.90s/it]
Outer training loop:  65%|######4   | 32467/50000 [20:54:04<9:14:43,  1.90s/it]
Outer training loop:  65%|######4   | 32468/50000 [20:54:06<9:11:36,  1.89s/it]
Outer training loop:  65%|######4   | 32469/50000 [20:54:08<9:27:30,  1.94s/it]
Outer training loop:  65%|######4   | 32470/50000 [20:54:09<9:04:05,  1.86s/it]
Outer training loop:  65%|######4   | 32471/50000 [20:54:11<9:06:40,  1.87s/it]
Outer training loop:  65%|######4   | 32472/50000 [20:54:13<9:31:07,  1.96s/it]
Outer training loop:  65%|######4   | 32473/50000 [20:54:15<9:21:19,  1.92s/it]
Outer training loop:  65%|######4   | 32474/50000 [20:54:17<9:19:04,  1.91s/it]
Outer training loop:  65%|######4   | 32475/50000 [20:54:19<9:12:45,  1.89s/it]
Outer training loop:  65%|######4   | 32476/50000 [20:54:21<9:31:23,  1.96s/it]
Outer training loop:  65%|######4   | 32477/50000 [20:54:23<9:22:01,  1.92s/it]
Outer training loop:  65%|######4   | 32478/50000 [20:54:25<9:15:16,  1.90s/it]
Outer training loop:  65%|######4   | 32479/50000 [20:54:27<9:16:43,  1.91s/it]
Outer training loop:  65%|######4   | 32480/50000 [20:54:28<9:16:18,  1.91s/it]
Outer training loop:  65%|######4   | 32481/50000 [20:54:30<9:12:08,  1.89s/it]
Outer training loop:  65%|######4   | 32482/50000 [20:54:32<9:20:52,  1.92s/it]
Outer training loop:  65%|######4   | 32483/50000 [20:54:34<9:12:23,  1.89s/it]
Outer training loop:  65%|######4   | 32484/50000 [20:54:36<9:07:14,  1.87s/it]
Outer training loop:  65%|######4   | 32485/50000 [20:54:38<9:08:14,  1.88s/it]
Outer training loop:  65%|######4   | 32486/50000 [20:54:40<9:26:13,  1.94s/it]
Outer training loop:  65%|######4   | 32487/50000 [20:54:42<9:14:14,  1.90s/it]
Outer training loop:  65%|######4   | 32488/50000 [20:54:44<9:01:04,  1.85s/it]
Outer training loop:  65%|######4   | 32489/50000 [20:54:46<9:20:29,  1.92s/it]
Outer training loop:  65%|######4   | 32490/50000 [20:54:47<9:10:19,  1.89s/it]
Outer training loop:  65%|######4   | 32491/50000 [20:54:49<9:04:24,  1.87s/it]
Outer training loop:  65%|######4   | 32492/50000 [20:54:51<9:10:59,  1.89s/it]
Outer training loop:  65%|######4   | 32493/50000 [20:54:53<9:01:41,  1.86s/it]
Outer training loop:  65%|######4   | 32494/50000 [20:54:55<9:05:23,  1.87s/it]
Outer training loop:  65%|######4   | 32495/50000 [20:54:57<9:06:34,  1.87s/it]
Outer training loop:  65%|######4   | 32496/50000 [20:54:59<9:22:53,  1.93s/it]
Outer training loop:  65%|######4   | 32497/50000 [20:55:01<9:06:40,  1.87s/it]
Outer training loop:  65%|######4   | 32498/50000 [20:55:02<9:03:42,  1.86s/it]
Outer training loop:  65%|######4   | 32499/50000 [20:55:04<9:19:05,  1.92s/it]
Outer training loop:  65%|######5   | 32500/50000 [20:55:06<9:16:42,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.05s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.03s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.92s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:27,  1.80s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:29,  1.85s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.88s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:29,  1.94s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:25,  1.92s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.94s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:23,  1.97s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:22,  1.98s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:12,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:15,  1.97s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:11,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.85s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<01:57,  1.81s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.93s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:54,  1.94s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:51,  1.93s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:42,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:38,  1.90s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:32,  1.85s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:29,  1.82s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:32,  1.93s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.89s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:21,  1.82s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:17,  1.81s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:14,  1.78s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:11,  1.85s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:09,  1.82s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:06,  1.80s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:05,  1.81s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:05,  1.86s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:06,  1.95s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:02,  1.90s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:52,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:53,  1.98s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.95s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.91s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.89s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:37,  1.95s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:34,  1.89s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.84s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.81s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:19,  1.96s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:17,  1.95s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.95s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.89s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.89s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:04,  2.00s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.98s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.89s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:18,  1.69s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:19,  1.72s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:16,  1.71s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:20,  1.78s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.77s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:18,  1.80s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:30,  2.00s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:22,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:18,  1.90s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.89s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.94s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:15,  2.02s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.94s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<02:01,  1.95s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:53,  1.99s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.94s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:42,  1.89s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.86s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:40,  1.97s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:37,  1.95s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:34,  1.94s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.92s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:28,  1.97s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:24,  1.93s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:22,  1.93s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:20,  1.93s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:19,  1.95s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:17,  1.95s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:20,  2.07s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:17,  2.03s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:12,  1.95s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:09,  1.92s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.86s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:05,  1.91s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:01,  1.88s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<01:05,  2.05s/it]

Evaluating policy:  63%|######2   | 52/83 [01:40<01:01,  1.99s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:59,  1.97s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:56,  1.95s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:53,  1.92s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:51,  1.90s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:49,  1.90s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:51,  2.05s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:47,  1.98s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:44,  1.93s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:37,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:34,  1.93s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:07<00:34,  2.03s/it]

Evaluating policy:  81%|########  | 67/83 [02:09<00:31,  1.97s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:28,  1.93s/it]

Evaluating policy:  83%|########3 | 69/83 [02:12<00:27,  1.95s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:25,  1.95s/it]

Evaluating policy:  86%|########5 | 71/83 [02:16<00:23,  1.93s/it]

Evaluating policy:  87%|########6 | 72/83 [02:18<00:20,  1.88s/it]

Evaluating policy:  88%|########7 | 73/83 [02:20<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:22<00:16,  1.88s/it]

Evaluating policy:  90%|######### | 75/83 [02:24<00:16,  2.06s/it]

Evaluating policy:  92%|#########1| 76/83 [02:26<00:13,  1.99s/it]

Evaluating policy:  93%|#########2| 77/83 [02:28<00:11,  1.94s/it]

Evaluating policy:  94%|#########3| 78/83 [02:30<00:09,  1.90s/it]

Evaluating policy:  95%|#########5| 79/83 [02:32<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:33<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:35<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:37<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 83/83 [02:39<00:00,  1.86s/it]

Outer training loop:  65%|######5   | 32501/50000 [21:00:30<479:01:14, 98.55s/it]
Outer training loop:  65%|######5   | 32502/50000 [21:00:32<338:00:20, 69.54s/it]
Outer training loop:  65%|######5   | 32503/50000 [21:00:35<240:10:22, 49.42s/it]
Outer training loop:  65%|######5   | 32504/50000 [21:00:37<170:57:12, 35.18s/it]
Outer training loop:  65%|######5   | 32505/50000 [21:00:39<122:27:04, 25.20s/it]
Outer training loop:  65%|######5   | 32506/50000 [21:00:41<88:51:45, 18.29s/it] 
Outer training loop:  65%|######5   | 32507/50000 [21:00:43<65:03:45, 13.39s/it]
Outer training loop:  65%|######5   | 32508/50000 [21:00:44<48:08:56,  9.91s/it]
Outer training loop:  65%|######5   | 32509/50000 [21:00:46<36:23:38,  7.49s/it]
Outer training loop:  65%|######5   | 32510/50000 [21:00:48<28:29:30,  5.86s/it]
Outer training loop:  65%|######5   | 32511/50000 [21:00:50<22:42:30,  4.67s/it]
Outer training loop:  65%|######5   | 32512/50000 [21:00:52<18:30:24,  3.81s/it]
Outer training loop:  65%|######5   | 32513/50000 [21:00:54<15:47:46,  3.25s/it]
Outer training loop:  65%|######5   | 32514/50000 [21:00:56<13:48:26,  2.84s/it]
Outer training loop:  65%|######5   | 32515/50000 [21:00:58<12:21:34,  2.54s/it]
Outer training loop:  65%|######5   | 32516/50000 [21:01:00<11:30:12,  2.37s/it]
Outer training loop:  65%|######5   | 32517/50000 [21:01:02<10:49:47,  2.23s/it]
Outer training loop:  65%|######5   | 32518/50000 [21:01:03<10:08:48,  2.09s/it]
Outer training loop:  65%|######5   | 32519/50000 [21:01:05<9:47:22,  2.02s/it] 
Outer training loop:  65%|######5   | 32520/50000 [21:01:07<10:01:02,  2.06s/it]
Outer training loop:  65%|######5   | 32521/50000 [21:01:09<9:30:43,  1.96s/it] 
Outer training loop:  65%|######5   | 32522/50000 [21:01:11<9:20:21,  1.92s/it]
Outer training loop:  65%|######5   | 32523/50000 [21:01:13<9:26:19,  1.94s/it]
Outer training loop:  65%|######5   | 32524/50000 [21:01:15<9:14:14,  1.90s/it]
Outer training loop:  65%|######5   | 32525/50000 [21:01:16<8:59:20,  1.85s/it]
Outer training loop:  65%|######5   | 32526/50000 [21:01:18<9:08:26,  1.88s/it]
Outer training loop:  65%|######5   | 32527/50000 [21:01:20<9:06:03,  1.88s/it]
Outer training loop:  65%|######5   | 32528/50000 [21:01:22<9:01:40,  1.86s/it]
Outer training loop:  65%|######5   | 32529/50000 [21:01:24<9:11:50,  1.90s/it]
Outer training loop:  65%|######5   | 32530/50000 [21:01:26<9:05:03,  1.87s/it]
Outer training loop:  65%|######5   | 32531/50000 [21:01:28<9:02:57,  1.86s/it]
Outer training loop:  65%|######5   | 32532/50000 [21:01:30<8:53:44,  1.83s/it]
Outer training loop:  65%|######5   | 32533/50000 [21:01:32<9:31:18,  1.96s/it]
Outer training loop:  65%|######5   | 32534/50000 [21:01:33<9:10:02,  1.89s/it]
Outer training loop:  65%|######5   | 32535/50000 [21:01:35<9:01:53,  1.86s/it]
Outer training loop:  65%|######5   | 32536/50000 [21:01:37<9:13:22,  1.90s/it]
Outer training loop:  65%|######5   | 32537/50000 [21:01:39<9:04:51,  1.87s/it]
Outer training loop:  65%|######5   | 32538/50000 [21:01:41<9:03:25,  1.87s/it]
Outer training loop:  65%|######5   | 32539/50000 [21:01:43<9:17:45,  1.92s/it]
Outer training loop:  65%|######5   | 32540/50000 [21:01:45<9:10:50,  1.89s/it]
Outer training loop:  65%|######5   | 32541/50000 [21:01:47<8:55:28,  1.84s/it]
Outer training loop:  65%|######5   | 32542/50000 [21:01:48<8:48:17,  1.82s/it]
Outer training loop:  65%|######5   | 32543/50000 [21:01:50<9:10:55,  1.89s/it]
Outer training loop:  65%|######5   | 32544/50000 [21:01:52<9:02:21,  1.86s/it]
Outer training loop:  65%|######5   | 32545/50000 [21:01:54<9:00:07,  1.86s/it]
Outer training loop:  65%|######5   | 32546/50000 [21:01:56<9:10:09,  1.89s/it]
Outer training loop:  65%|######5   | 32547/50000 [21:01:58<9:08:19,  1.89s/it]
Outer training loop:  65%|######5   | 32548/50000 [21:02:00<9:09:30,  1.89s/it]
Outer training loop:  65%|######5   | 32549/50000 [21:02:02<9:13:39,  1.90s/it]
Outer training loop:  65%|######5   | 32550/50000 [21:02:03<8:50:57,  1.83s/it]
Outer training loop:  65%|######5   | 32551/50000 [21:02:05<8:48:02,  1.82s/it]
Outer training loop:  65%|######5   | 32552/50000 [21:02:07<8:51:57,  1.83s/it]
Outer training loop:  65%|######5   | 32553/50000 [21:02:09<9:07:55,  1.88s/it]
Outer training loop:  65%|######5   | 32554/50000 [21:02:11<9:10:22,  1.89s/it]
Outer training loop:  65%|######5   | 32555/50000 [21:02:13<9:18:43,  1.92s/it]
Outer training loop:  65%|######5   | 32556/50000 [21:02:15<9:20:56,  1.93s/it]
Outer training loop:  65%|######5   | 32557/50000 [21:02:17<9:26:30,  1.95s/it]
Outer training loop:  65%|######5   | 32558/50000 [21:02:19<9:15:26,  1.91s/it]
Outer training loop:  65%|######5   | 32559/50000 [21:02:20<9:09:18,  1.89s/it]
Outer training loop:  65%|######5   | 32560/50000 [21:02:22<9:17:30,  1.92s/it]
Outer training loop:  65%|######5   | 32561/50000 [21:02:24<9:15:19,  1.91s/it]
Outer training loop:  65%|######5   | 32562/50000 [21:02:26<9:03:09,  1.87s/it]
Outer training loop:  65%|######5   | 32563/50000 [21:02:28<9:27:53,  1.95s/it]
Outer training loop:  65%|######5   | 32564/50000 [21:02:30<9:23:15,  1.94s/it]
Outer training loop:  65%|######5   | 32565/50000 [21:02:32<9:03:01,  1.87s/it]
Outer training loop:  65%|######5   | 32566/50000 [21:02:34<9:17:45,  1.92s/it]
Outer training loop:  65%|######5   | 32567/50000 [21:02:36<9:13:55,  1.91s/it]
Outer training loop:  65%|######5   | 32568/50000 [21:02:38<9:01:57,  1.87s/it]
Outer training loop:  65%|######5   | 32569/50000 [21:02:39<8:53:14,  1.84s/it]
Outer training loop:  65%|######5   | 32570/50000 [21:02:41<9:15:56,  1.91s/it]
Outer training loop:  65%|######5   | 32571/50000 [21:02:43<9:06:35,  1.88s/it]
Outer training loop:  65%|######5   | 32572/50000 [21:02:45<8:56:20,  1.85s/it]
Outer training loop:  65%|######5   | 32573/50000 [21:02:47<9:08:18,  1.89s/it]
Outer training loop:  65%|######5   | 32574/50000 [21:02:49<9:07:14,  1.88s/it]
Outer training loop:  65%|######5   | 32575/50000 [21:02:51<9:02:08,  1.87s/it]
Outer training loop:  65%|######5   | 32576/50000 [21:02:53<9:09:06,  1.89s/it]
Outer training loop:  65%|######5   | 32577/50000 [21:02:54<8:55:42,  1.84s/it]
Outer training loop:  65%|######5   | 32578/50000 [21:02:56<9:02:17,  1.87s/it]
Outer training loop:  65%|######5   | 32579/50000 [21:02:58<9:26:53,  1.95s/it]
Outer training loop:  65%|######5   | 32580/50000 [21:03:00<9:22:50,  1.94s/it]
Outer training loop:  65%|######5   | 32581/50000 [21:03:02<9:08:53,  1.89s/it]
Outer training loop:  65%|######5   | 32582/50000 [21:03:04<9:04:15,  1.87s/it]
Outer training loop:  65%|######5   | 32583/50000 [21:03:06<9:39:02,  1.99s/it]
Outer training loop:  65%|######5   | 32584/50000 [21:03:08<9:16:42,  1.92s/it]
Outer training loop:  65%|######5   | 32585/50000 [21:03:10<8:58:50,  1.86s/it]
Outer training loop:  65%|######5   | 32586/50000 [21:03:12<9:06:30,  1.88s/it]
Outer training loop:  65%|######5   | 32587/50000 [21:03:14<9:06:10,  1.88s/it]
Outer training loop:  65%|######5   | 32588/50000 [21:03:15<9:03:20,  1.87s/it]
Outer training loop:  65%|######5   | 32589/50000 [21:03:17<9:12:31,  1.90s/it]
Outer training loop:  65%|######5   | 32590/50000 [21:03:19<9:05:56,  1.88s/it]
Outer training loop:  65%|######5   | 32591/50000 [21:03:21<8:53:14,  1.84s/it]
Outer training loop:  65%|######5   | 32592/50000 [21:03:23<8:49:24,  1.82s/it]
Outer training loop:  65%|######5   | 32593/50000 [21:03:25<9:23:45,  1.94s/it]
Outer training loop:  65%|######5   | 32594/50000 [21:03:27<9:17:36,  1.92s/it]
Outer training loop:  65%|######5   | 32595/50000 [21:03:29<9:14:05,  1.91s/it]
Outer training loop:  65%|######5   | 32596/50000 [21:03:31<9:27:17,  1.96s/it]
Outer training loop:  65%|######5   | 32597/50000 [21:03:33<9:22:46,  1.94s/it]
Outer training loop:  65%|######5   | 32598/50000 [21:03:35<9:20:59,  1.93s/it]
Outer training loop:  65%|######5   | 32599/50000 [21:03:37<9:26:39,  1.95s/it]
Outer training loop:  65%|######5   | 32600/50000 [21:03:38<9:17:17,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  65%|######5   | 32601/50000 [21:03:46<17:01:46,  3.52s/it]
Outer training loop:  65%|######5   | 32602/50000 [21:03:47<14:30:37,  3.00s/it]
Outer training loop:  65%|######5   | 32603/50000 [21:03:50<13:19:20,  2.76s/it]
Outer training loop:  65%|######5   | 32604/50000 [21:03:52<12:03:21,  2.49s/it]
Outer training loop:  65%|######5   | 32605/50000 [21:03:53<10:58:23,  2.27s/it]
Outer training loop:  65%|######5   | 32606/50000 [21:03:55<10:32:20,  2.18s/it]
Outer training loop:  65%|######5   | 32607/50000 [21:03:57<10:07:51,  2.10s/it]
Outer training loop:  65%|######5   | 32608/50000 [21:03:59<9:42:24,  2.01s/it] 
Outer training loop:  65%|######5   | 32609/50000 [21:04:01<9:27:17,  1.96s/it]
Outer training loop:  65%|######5   | 32610/50000 [21:04:03<9:24:45,  1.95s/it]
Outer training loop:  65%|######5   | 32611/50000 [21:04:05<9:16:35,  1.92s/it]
Outer training loop:  65%|######5   | 32612/50000 [21:04:06<9:09:42,  1.90s/it]
Outer training loop:  65%|######5   | 32613/50000 [21:04:08<9:12:32,  1.91s/it]
Outer training loop:  65%|######5   | 32614/50000 [21:04:10<9:06:52,  1.89s/it]
Outer training loop:  65%|######5   | 32615/50000 [21:04:12<8:55:04,  1.85s/it]
Outer training loop:  65%|######5   | 32616/50000 [21:04:14<9:08:08,  1.89s/it]
Outer training loop:  65%|######5   | 32617/50000 [21:04:16<8:54:52,  1.85s/it]
Outer training loop:  65%|######5   | 32618/50000 [21:04:17<8:48:48,  1.83s/it]
Outer training loop:  65%|######5   | 32619/50000 [21:04:19<9:02:30,  1.87s/it]
Outer training loop:  65%|######5   | 32620/50000 [21:04:21<9:06:05,  1.89s/it]
Outer training loop:  65%|######5   | 32621/50000 [21:04:23<8:46:41,  1.82s/it]
Outer training loop:  65%|######5   | 32622/50000 [21:04:25<8:41:44,  1.80s/it]
Outer training loop:  65%|######5   | 32623/50000 [21:04:27<9:07:30,  1.89s/it]
Outer training loop:  65%|######5   | 32624/50000 [21:04:29<9:10:01,  1.90s/it]
Outer training loop:  65%|######5   | 32625/50000 [21:04:31<9:12:28,  1.91s/it]
Outer training loop:  65%|######5   | 32626/50000 [21:04:33<9:11:14,  1.90s/it]
Outer training loop:  65%|######5   | 32627/50000 [21:04:34<9:02:59,  1.88s/it]
Outer training loop:  65%|######5   | 32628/50000 [21:04:36<8:59:34,  1.86s/it]
Outer training loop:  65%|######5   | 32629/50000 [21:04:38<8:52:27,  1.84s/it]
Outer training loop:  65%|######5   | 32630/50000 [21:04:40<9:19:37,  1.93s/it]
Outer training loop:  65%|######5   | 32631/50000 [21:04:42<9:07:34,  1.89s/it]
Outer training loop:  65%|######5   | 32632/50000 [21:04:44<9:04:22,  1.88s/it]
Outer training loop:  65%|######5   | 32633/50000 [21:04:46<9:09:48,  1.90s/it]
Outer training loop:  65%|######5   | 32634/50000 [21:04:48<9:04:35,  1.88s/it]
Outer training loop:  65%|######5   | 32635/50000 [21:04:49<8:58:11,  1.86s/it]
Outer training loop:  65%|######5   | 32636/50000 [21:04:51<8:57:54,  1.86s/it]
Outer training loop:  65%|######5   | 32637/50000 [21:04:53<9:14:56,  1.92s/it]
Outer training loop:  65%|######5   | 32638/50000 [21:04:55<9:11:26,  1.91s/it]
Outer training loop:  65%|######5   | 32639/50000 [21:04:57<9:05:51,  1.89s/it]
Outer training loop:  65%|######5   | 32640/50000 [21:04:59<9:25:55,  1.96s/it]
Outer training loop:  65%|######5   | 32641/50000 [21:05:01<9:22:17,  1.94s/it]
Outer training loop:  65%|######5   | 32642/50000 [21:05:03<9:14:41,  1.92s/it]
Outer training loop:  65%|######5   | 32643/50000 [21:05:05<9:15:06,  1.92s/it]
Outer training loop:  65%|######5   | 32644/50000 [21:05:07<9:35:53,  1.99s/it]
Outer training loop:  65%|######5   | 32645/50000 [21:05:09<9:16:35,  1.92s/it]
Outer training loop:  65%|######5   | 32646/50000 [21:05:11<9:10:51,  1.90s/it]
Outer training loop:  65%|######5   | 32647/50000 [21:05:13<9:20:12,  1.94s/it]
Outer training loop:  65%|######5   | 32648/50000 [21:05:14<9:05:58,  1.89s/it]
Outer training loop:  65%|######5   | 32649/50000 [21:05:16<8:56:50,  1.86s/it]
Outer training loop:  65%|######5   | 32650/50000 [21:05:18<8:43:30,  1.81s/it]
Outer training loop:  65%|######5   | 32651/50000 [21:05:20<9:04:37,  1.88s/it]
Outer training loop:  65%|######5   | 32652/50000 [21:05:22<8:59:23,  1.87s/it]
Outer training loop:  65%|######5   | 32653/50000 [21:05:24<8:58:39,  1.86s/it]
Outer training loop:  65%|######5   | 32654/50000 [21:05:26<9:15:18,  1.92s/it]
Outer training loop:  65%|######5   | 32655/50000 [21:05:28<9:12:44,  1.91s/it]
Outer training loop:  65%|######5   | 32656/50000 [21:05:29<9:03:49,  1.88s/it]
Outer training loop:  65%|######5   | 32657/50000 [21:05:31<9:09:53,  1.90s/it]
Outer training loop:  65%|######5   | 32658/50000 [21:05:33<9:03:03,  1.88s/it]
Outer training loop:  65%|######5   | 32659/50000 [21:05:35<9:04:52,  1.89s/it]
Outer training loop:  65%|######5   | 32660/50000 [21:05:37<8:57:22,  1.86s/it]
Outer training loop:  65%|######5   | 32661/50000 [21:05:39<9:10:35,  1.91s/it]
Outer training loop:  65%|######5   | 32662/50000 [21:05:41<9:08:22,  1.90s/it]
Outer training loop:  65%|######5   | 32663/50000 [21:05:43<9:06:13,  1.89s/it]
Outer training loop:  65%|######5   | 32664/50000 [21:05:45<9:12:46,  1.91s/it]
Outer training loop:  65%|######5   | 32665/50000 [21:05:46<9:03:41,  1.88s/it]
Outer training loop:  65%|######5   | 32666/50000 [21:05:48<9:02:19,  1.88s/it]
Outer training loop:  65%|######5   | 32667/50000 [21:05:50<9:28:21,  1.97s/it]
Outer training loop:  65%|######5   | 32668/50000 [21:05:52<9:09:45,  1.90s/it]
Outer training loop:  65%|######5   | 32669/50000 [21:05:54<8:50:33,  1.84s/it]
Outer training loop:  65%|######5   | 32670/50000 [21:05:56<8:42:52,  1.81s/it]
Outer training loop:  65%|######5   | 32671/50000 [21:05:58<9:00:29,  1.87s/it]
Outer training loop:  65%|######5   | 32672/50000 [21:06:00<9:01:12,  1.87s/it]
Outer training loop:  65%|######5   | 32673/50000 [21:06:01<9:04:08,  1.88s/it]
Outer training loop:  65%|######5   | 32674/50000 [21:06:03<8:50:45,  1.84s/it]
Outer training loop:  65%|######5   | 32675/50000 [21:06:05<9:08:13,  1.90s/it]
Outer training loop:  65%|######5   | 32676/50000 [21:06:07<8:55:41,  1.86s/it]
Outer training loop:  65%|######5   | 32677/50000 [21:06:09<8:51:30,  1.84s/it]
Outer training loop:  65%|######5   | 32678/50000 [21:06:11<9:04:49,  1.89s/it]
Outer training loop:  65%|######5   | 32679/50000 [21:06:13<9:04:33,  1.89s/it]
Outer training loop:  65%|######5   | 32680/50000 [21:06:15<9:04:59,  1.89s/it]
Outer training loop:  65%|######5   | 32681/50000 [21:06:17<9:13:22,  1.92s/it]
Outer training loop:  65%|######5   | 32682/50000 [21:06:18<9:00:12,  1.87s/it]
Outer training loop:  65%|######5   | 32683/50000 [21:06:20<8:50:23,  1.84s/it]
Outer training loop:  65%|######5   | 32684/50000 [21:06:22<8:46:12,  1.82s/it]
Outer training loop:  65%|######5   | 32685/50000 [21:06:24<9:05:30,  1.89s/it]
Outer training loop:  65%|######5   | 32686/50000 [21:06:26<9:06:27,  1.89s/it]
Outer training loop:  65%|######5   | 32687/50000 [21:06:28<9:05:18,  1.89s/it]
Outer training loop:  65%|######5   | 32688/50000 [21:06:30<9:07:42,  1.90s/it]
Outer training loop:  65%|######5   | 32689/50000 [21:06:31<8:57:14,  1.86s/it]
Outer training loop:  65%|######5   | 32690/50000 [21:06:33<8:49:06,  1.83s/it]
Outer training loop:  65%|######5   | 32691/50000 [21:06:35<9:14:13,  1.92s/it]
Outer training loop:  65%|######5   | 32692/50000 [21:06:37<8:58:16,  1.87s/it]
Outer training loop:  65%|######5   | 32693/50000 [21:06:39<8:55:43,  1.86s/it]
Outer training loop:  65%|######5   | 32694/50000 [21:06:41<8:47:24,  1.83s/it]
Outer training loop:  65%|######5   | 32695/50000 [21:06:43<9:03:06,  1.88s/it]
Outer training loop:  65%|######5   | 32696/50000 [21:06:44<8:45:16,  1.82s/it]
Outer training loop:  65%|######5   | 32697/50000 [21:06:46<8:41:35,  1.81s/it]
Outer training loop:  65%|######5   | 32698/50000 [21:06:48<8:58:03,  1.87s/it]
Outer training loop:  65%|######5   | 32699/50000 [21:06:50<8:54:50,  1.85s/it]
Outer training loop:  65%|######5   | 32700/50000 [21:06:52<8:44:58,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.00s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.99s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.90s/it]

Outer training loop:  65%|######5   | 32701/50000 [21:06:59<16:42:10,  3.48s/it]
Outer training loop:  65%|######5   | 32702/50000 [21:07:01<14:38:55,  3.05s/it]
Outer training loop:  65%|######5   | 32703/50000 [21:07:03<12:49:43,  2.67s/it]
Outer training loop:  65%|######5   | 32704/50000 [21:07:05<11:35:08,  2.41s/it]
Outer training loop:  65%|######5   | 32705/50000 [21:07:07<10:49:48,  2.25s/it]
Outer training loop:  65%|######5   | 32706/50000 [21:07:09<10:35:15,  2.20s/it]
Outer training loop:  65%|######5   | 32707/50000 [21:07:10<9:57:29,  2.07s/it] 
Outer training loop:  65%|######5   | 32708/50000 [21:07:12<9:28:00,  1.97s/it]
Outer training loop:  65%|######5   | 32709/50000 [21:07:14<9:22:07,  1.95s/it]
Outer training loop:  65%|######5   | 32710/50000 [21:07:16<9:19:47,  1.94s/it]
Outer training loop:  65%|######5   | 32711/50000 [21:07:18<9:09:15,  1.91s/it]
Outer training loop:  65%|######5   | 32712/50000 [21:07:20<8:55:31,  1.86s/it]
Outer training loop:  65%|######5   | 32713/50000 [21:07:22<9:09:57,  1.91s/it]
Outer training loop:  65%|######5   | 32714/50000 [21:07:23<8:58:15,  1.87s/it]
Outer training loop:  65%|######5   | 32715/50000 [21:07:25<9:06:08,  1.90s/it]
Outer training loop:  65%|######5   | 32716/50000 [21:07:27<9:17:16,  1.93s/it]
Outer training loop:  65%|######5   | 32717/50000 [21:07:29<9:05:57,  1.90s/it]
Outer training loop:  65%|######5   | 32718/50000 [21:07:31<8:47:32,  1.83s/it]
Outer training loop:  65%|######5   | 32719/50000 [21:07:33<8:40:35,  1.81s/it]
Outer training loop:  65%|######5   | 32720/50000 [21:07:35<8:53:25,  1.85s/it]
Outer training loop:  65%|######5   | 32721/50000 [21:07:36<8:54:16,  1.86s/it]
Outer training loop:  65%|######5   | 32722/50000 [21:07:38<8:43:04,  1.82s/it]
Outer training loop:  65%|######5   | 32723/50000 [21:07:40<9:02:36,  1.88s/it]
Outer training loop:  65%|######5   | 32724/50000 [21:07:42<8:57:54,  1.87s/it]
Outer training loop:  65%|######5   | 32725/50000 [21:07:44<8:53:22,  1.85s/it]
Outer training loop:  65%|######5   | 32726/50000 [21:07:46<8:49:40,  1.84s/it]
Outer training loop:  65%|######5   | 32727/50000 [21:07:48<9:11:35,  1.92s/it]
Outer training loop:  65%|######5   | 32728/50000 [21:07:50<9:08:26,  1.91s/it]
Outer training loop:  65%|######5   | 32729/50000 [21:07:51<8:53:44,  1.85s/it]
Outer training loop:  65%|######5   | 32730/50000 [21:07:53<9:02:39,  1.89s/it]
Outer training loop:  65%|######5   | 32731/50000 [21:07:55<9:00:40,  1.88s/it]
Outer training loop:  65%|######5   | 32732/50000 [21:07:57<9:02:39,  1.89s/it]
Outer training loop:  65%|######5   | 32733/50000 [21:07:59<8:53:56,  1.86s/it]
Outer training loop:  65%|######5   | 32734/50000 [21:08:01<8:57:14,  1.87s/it]
Outer training loop:  65%|######5   | 32735/50000 [21:08:03<8:56:34,  1.86s/it]
Outer training loop:  65%|######5   | 32736/50000 [21:08:04<8:59:57,  1.88s/it]
Outer training loop:  65%|######5   | 32737/50000 [21:08:06<9:10:52,  1.91s/it]
Outer training loop:  65%|######5   | 32738/50000 [21:08:08<9:04:26,  1.89s/it]
Outer training loop:  65%|######5   | 32739/50000 [21:08:10<8:49:09,  1.84s/it]
Outer training loop:  65%|######5   | 32740/50000 [21:08:12<9:04:45,  1.89s/it]
Outer training loop:  65%|######5   | 32741/50000 [21:08:14<9:02:30,  1.89s/it]
Outer training loop:  65%|######5   | 32742/50000 [21:08:16<8:54:36,  1.86s/it]
Outer training loop:  65%|######5   | 32743/50000 [21:08:17<8:47:22,  1.83s/it]
Outer training loop:  65%|######5   | 32744/50000 [21:08:20<9:05:34,  1.90s/it]
Outer training loop:  65%|######5   | 32745/50000 [21:08:21<9:09:58,  1.91s/it]
Outer training loop:  65%|######5   | 32746/50000 [21:08:23<9:03:46,  1.89s/it]
Outer training loop:  65%|######5   | 32747/50000 [21:08:26<9:28:31,  1.98s/it]
Outer training loop:  65%|######5   | 32748/50000 [21:08:27<9:18:04,  1.94s/it]
Outer training loop:  65%|######5   | 32749/50000 [21:08:29<9:07:08,  1.90s/it]
Outer training loop:  66%|######5   | 32750/50000 [21:08:31<9:08:10,  1.91s/it]
Outer training loop:  66%|######5   | 32751/50000 [21:08:33<9:16:16,  1.93s/it]
Outer training loop:  66%|######5   | 32752/50000 [21:08:35<9:03:14,  1.89s/it]
Outer training loop:  66%|######5   | 32753/50000 [21:08:37<9:03:48,  1.89s/it]
Outer training loop:  66%|######5   | 32754/50000 [21:08:39<9:15:18,  1.93s/it]
Outer training loop:  66%|######5   | 32755/50000 [21:08:41<9:01:51,  1.89s/it]
Outer training loop:  66%|######5   | 32756/50000 [21:08:42<8:58:55,  1.88s/it]
Outer training loop:  66%|######5   | 32757/50000 [21:08:45<9:19:40,  1.95s/it]
Outer training loop:  66%|######5   | 32758/50000 [21:08:46<9:17:35,  1.94s/it]
Outer training loop:  66%|######5   | 32759/50000 [21:08:48<9:05:13,  1.90s/it]
Outer training loop:  66%|######5   | 32760/50000 [21:08:50<9:00:05,  1.88s/it]
Outer training loop:  66%|######5   | 32761/50000 [21:08:52<9:04:59,  1.90s/it]
Outer training loop:  66%|######5   | 32762/50000 [21:08:54<8:54:07,  1.86s/it]
Outer training loop:  66%|######5   | 32763/50000 [21:08:56<8:49:59,  1.84s/it]
Outer training loop:  66%|######5   | 32764/50000 [21:08:58<9:11:04,  1.92s/it]
Outer training loop:  66%|######5   | 32765/50000 [21:09:00<9:03:23,  1.89s/it]
Outer training loop:  66%|######5   | 32766/50000 [21:09:01<8:58:00,  1.87s/it]
Outer training loop:  66%|######5   | 32767/50000 [21:09:03<9:03:05,  1.89s/it]
Outer training loop:  66%|######5   | 32768/50000 [21:09:05<8:58:01,  1.87s/it]
Outer training loop:  66%|######5   | 32769/50000 [21:09:07<9:02:58,  1.89s/it]
Outer training loop:  66%|######5   | 32770/50000 [21:09:09<9:01:31,  1.89s/it]
Outer training loop:  66%|######5   | 32771/50000 [21:09:11<9:08:39,  1.91s/it]
Outer training loop:  66%|######5   | 32772/50000 [21:09:13<9:03:49,  1.89s/it]
Outer training loop:  66%|######5   | 32773/50000 [21:09:15<8:50:31,  1.85s/it]
Outer training loop:  66%|######5   | 32774/50000 [21:09:17<9:07:09,  1.91s/it]
Outer training loop:  66%|######5   | 32775/50000 [21:09:18<8:51:05,  1.85s/it]
Outer training loop:  66%|######5   | 32776/50000 [21:09:20<8:51:01,  1.85s/it]
Outer training loop:  66%|######5   | 32777/50000 [21:09:22<8:57:13,  1.87s/it]
Outer training loop:  66%|######5   | 32778/50000 [21:09:24<9:09:08,  1.91s/it]
Outer training loop:  66%|######5   | 32779/50000 [21:09:26<9:07:52,  1.91s/it]
Outer training loop:  66%|######5   | 32780/50000 [21:09:28<9:05:22,  1.90s/it]
Outer training loop:  66%|######5   | 32781/50000 [21:09:30<9:17:45,  1.94s/it]
Outer training loop:  66%|######5   | 32782/50000 [21:09:32<9:07:08,  1.91s/it]
Outer training loop:  66%|######5   | 32783/50000 [21:09:34<9:10:55,  1.92s/it]
Outer training loop:  66%|######5   | 32784/50000 [21:09:36<9:25:00,  1.97s/it]
Outer training loop:  66%|######5   | 32785/50000 [21:09:38<9:14:49,  1.93s/it]
Outer training loop:  66%|######5   | 32786/50000 [21:09:39<9:03:04,  1.89s/it]
Outer training loop:  66%|######5   | 32787/50000 [21:09:41<8:50:17,  1.85s/it]
Outer training loop:  66%|######5   | 32788/50000 [21:09:43<9:00:19,  1.88s/it]
Outer training loop:  66%|######5   | 32789/50000 [21:09:45<8:57:35,  1.87s/it]
Outer training loop:  66%|######5   | 32790/50000 [21:09:47<8:44:49,  1.83s/it]
Outer training loop:  66%|######5   | 32791/50000 [21:09:49<8:56:16,  1.87s/it]
Outer training loop:  66%|######5   | 32792/50000 [21:09:50<8:50:02,  1.85s/it]
Outer training loop:  66%|######5   | 32793/50000 [21:09:52<8:49:56,  1.85s/it]
Outer training loop:  66%|######5   | 32794/50000 [21:09:54<9:05:13,  1.90s/it]
Outer training loop:  66%|######5   | 32795/50000 [21:09:56<8:51:51,  1.85s/it]
Outer training loop:  66%|######5   | 32796/50000 [21:09:58<8:53:15,  1.86s/it]
Outer training loop:  66%|######5   | 32797/50000 [21:10:00<8:53:28,  1.86s/it]
Outer training loop:  66%|######5   | 32798/50000 [21:10:02<9:10:12,  1.92s/it]
Outer training loop:  66%|######5   | 32799/50000 [21:10:04<9:04:14,  1.90s/it]
Outer training loop:  66%|######5   | 32800/50000 [21:10:06<9:01:25,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  66%|######5   | 32801/50000 [21:10:13<16:48:36,  3.52s/it]
Outer training loop:  66%|######5   | 32802/50000 [21:10:15<14:40:56,  3.07s/it]
Outer training loop:  66%|######5   | 32803/50000 [21:10:17<12:40:30,  2.65s/it]
Outer training loop:  66%|######5   | 32804/50000 [21:10:19<11:40:45,  2.45s/it]
Outer training loop:  66%|######5   | 32805/50000 [21:10:21<11:08:11,  2.33s/it]
Outer training loop:  66%|######5   | 32806/50000 [21:10:22<10:17:17,  2.15s/it]
Outer training loop:  66%|######5   | 32807/50000 [21:10:24<9:45:37,  2.04s/it] 
Outer training loop:  66%|######5   | 32808/50000 [21:10:26<9:36:51,  2.01s/it]
Outer training loop:  66%|######5   | 32809/50000 [21:10:28<9:27:39,  1.98s/it]
Outer training loop:  66%|######5   | 32810/50000 [21:10:30<9:09:31,  1.92s/it]
Outer training loop:  66%|######5   | 32811/50000 [21:10:32<9:08:26,  1.91s/it]
Outer training loop:  66%|######5   | 32812/50000 [21:10:34<9:17:54,  1.95s/it]
Outer training loop:  66%|######5   | 32813/50000 [21:10:35<9:02:34,  1.89s/it]
Outer training loop:  66%|######5   | 32814/50000 [21:10:37<9:02:03,  1.89s/it]
Outer training loop:  66%|######5   | 32815/50000 [21:10:39<9:16:17,  1.94s/it]
Outer training loop:  66%|######5   | 32816/50000 [21:10:41<9:00:09,  1.89s/it]
Outer training loop:  66%|######5   | 32817/50000 [21:10:43<8:55:00,  1.87s/it]
Outer training loop:  66%|######5   | 32818/50000 [21:10:45<8:59:48,  1.89s/it]
Outer training loop:  66%|######5   | 32819/50000 [21:10:47<8:45:34,  1.84s/it]
Outer training loop:  66%|######5   | 32820/50000 [21:10:48<8:44:29,  1.83s/it]
Outer training loop:  66%|######5   | 32821/50000 [21:10:51<9:10:45,  1.92s/it]
Outer training loop:  66%|######5   | 32822/50000 [21:10:52<9:00:12,  1.89s/it]
Outer training loop:  66%|######5   | 32823/50000 [21:10:54<8:48:08,  1.84s/it]
Outer training loop:  66%|######5   | 32824/50000 [21:10:56<9:06:24,  1.91s/it]
Outer training loop:  66%|######5   | 32825/50000 [21:10:58<9:15:14,  1.94s/it]
Outer training loop:  66%|######5   | 32826/50000 [21:11:00<9:06:29,  1.91s/it]
Outer training loop:  66%|######5   | 32827/50000 [21:11:02<8:53:49,  1.87s/it]
Outer training loop:  66%|######5   | 32828/50000 [21:11:04<9:10:24,  1.92s/it]
Outer training loop:  66%|######5   | 32829/50000 [21:11:06<9:03:54,  1.90s/it]
Outer training loop:  66%|######5   | 32830/50000 [21:11:08<9:00:38,  1.89s/it]
Outer training loop:  66%|######5   | 32831/50000 [21:11:10<9:15:27,  1.94s/it]
Outer training loop:  66%|######5   | 32832/50000 [21:11:11<9:08:26,  1.92s/it]
Outer training loop:  66%|######5   | 32833/50000 [21:11:13<9:03:03,  1.90s/it]
Outer training loop:  66%|######5   | 32834/50000 [21:11:15<8:57:44,  1.88s/it]
Outer training loop:  66%|######5   | 32835/50000 [21:11:17<9:01:11,  1.89s/it]
Outer training loop:  66%|######5   | 32836/50000 [21:11:19<9:10:42,  1.93s/it]
Outer training loop:  66%|######5   | 32837/50000 [21:11:21<8:59:35,  1.89s/it]
Outer training loop:  66%|######5   | 32838/50000 [21:11:23<9:17:29,  1.95s/it]
Outer training loop:  66%|######5   | 32839/50000 [21:11:25<9:17:34,  1.95s/it]
Outer training loop:  66%|######5   | 32840/50000 [21:11:27<9:08:54,  1.92s/it]
Outer training loop:  66%|######5   | 32841/50000 [21:11:29<9:06:40,  1.91s/it]
Outer training loop:  66%|######5   | 32842/50000 [21:11:31<9:20:42,  1.96s/it]
Outer training loop:  66%|######5   | 32843/50000 [21:11:33<9:11:49,  1.93s/it]
Outer training loop:  66%|######5   | 32844/50000 [21:11:34<8:56:42,  1.88s/it]
Outer training loop:  66%|######5   | 32845/50000 [21:11:36<9:11:06,  1.93s/it]
Outer training loop:  66%|######5   | 32846/50000 [21:11:38<8:58:21,  1.88s/it]
Outer training loop:  66%|######5   | 32847/50000 [21:11:40<8:56:14,  1.88s/it]
Outer training loop:  66%|######5   | 32848/50000 [21:11:42<8:58:43,  1.88s/it]
Outer training loop:  66%|######5   | 32849/50000 [21:11:44<9:09:51,  1.92s/it]
Outer training loop:  66%|######5   | 32850/50000 [21:11:46<9:02:21,  1.90s/it]
Outer training loop:  66%|######5   | 32851/50000 [21:11:48<8:55:01,  1.87s/it]
Outer training loop:  66%|######5   | 32852/50000 [21:11:50<9:17:03,  1.95s/it]
Outer training loop:  66%|######5   | 32853/50000 [21:11:52<9:07:06,  1.91s/it]
Outer training loop:  66%|######5   | 32854/50000 [21:11:53<8:50:58,  1.86s/it]
Outer training loop:  66%|######5   | 32855/50000 [21:11:55<8:57:39,  1.88s/it]
Outer training loop:  66%|######5   | 32856/50000 [21:11:57<9:08:35,  1.92s/it]
Outer training loop:  66%|######5   | 32857/50000 [21:11:59<9:00:14,  1.89s/it]
Outer training loop:  66%|######5   | 32858/50000 [21:12:01<8:50:16,  1.86s/it]
Outer training loop:  66%|######5   | 32859/50000 [21:12:03<9:07:43,  1.92s/it]
Outer training loop:  66%|######5   | 32860/50000 [21:12:05<8:47:57,  1.85s/it]
Outer training loop:  66%|######5   | 32861/50000 [21:12:06<8:49:39,  1.85s/it]
Outer training loop:  66%|######5   | 32862/50000 [21:12:09<9:07:43,  1.92s/it]
Outer training loop:  66%|######5   | 32863/50000 [21:12:10<8:54:59,  1.87s/it]
Outer training loop:  66%|######5   | 32864/50000 [21:12:12<8:45:36,  1.84s/it]
Outer training loop:  66%|######5   | 32865/50000 [21:12:14<8:35:53,  1.81s/it]
Outer training loop:  66%|######5   | 32866/50000 [21:12:16<9:00:12,  1.89s/it]
Outer training loop:  66%|######5   | 32867/50000 [21:12:18<8:58:43,  1.89s/it]
Outer training loop:  66%|######5   | 32868/50000 [21:12:19<8:38:24,  1.82s/it]
Outer training loop:  66%|######5   | 32869/50000 [21:12:21<8:51:16,  1.86s/it]
Outer training loop:  66%|######5   | 32870/50000 [21:12:23<8:52:02,  1.86s/it]
Outer training loop:  66%|######5   | 32871/50000 [21:12:25<8:53:55,  1.87s/it]
Outer training loop:  66%|######5   | 32872/50000 [21:12:27<9:04:06,  1.91s/it]
Outer training loop:  66%|######5   | 32873/50000 [21:12:29<8:54:59,  1.87s/it]
Outer training loop:  66%|######5   | 32874/50000 [21:12:31<9:09:37,  1.93s/it]
Outer training loop:  66%|######5   | 32875/50000 [21:12:33<8:51:10,  1.86s/it]
Outer training loop:  66%|######5   | 32876/50000 [21:12:35<9:11:52,  1.93s/it]
Outer training loop:  66%|######5   | 32877/50000 [21:12:37<9:01:15,  1.90s/it]
Outer training loop:  66%|######5   | 32878/50000 [21:12:38<8:53:30,  1.87s/it]
Outer training loop:  66%|######5   | 32879/50000 [21:12:40<9:10:36,  1.93s/it]
Outer training loop:  66%|######5   | 32880/50000 [21:12:42<9:03:37,  1.91s/it]
Outer training loop:  66%|######5   | 32881/50000 [21:12:44<9:06:37,  1.92s/it]
Outer training loop:  66%|######5   | 32882/50000 [21:12:46<9:12:36,  1.94s/it]
Outer training loop:  66%|######5   | 32883/50000 [21:12:48<9:07:25,  1.92s/it]
Outer training loop:  66%|######5   | 32884/50000 [21:12:50<9:06:58,  1.92s/it]
Outer training loop:  66%|######5   | 32885/50000 [21:12:52<8:58:03,  1.89s/it]
Outer training loop:  66%|######5   | 32886/50000 [21:12:54<9:10:10,  1.93s/it]
Outer training loop:  66%|######5   | 32887/50000 [21:12:56<9:06:19,  1.92s/it]
Outer training loop:  66%|######5   | 32888/50000 [21:12:58<9:00:13,  1.89s/it]
Outer training loop:  66%|######5   | 32889/50000 [21:13:00<9:22:34,  1.97s/it]
Outer training loop:  66%|######5   | 32890/50000 [21:13:02<9:05:46,  1.91s/it]
Outer training loop:  66%|######5   | 32891/50000 [21:13:03<8:44:15,  1.84s/it]
Outer training loop:  66%|######5   | 32892/50000 [21:13:05<9:04:51,  1.91s/it]
Outer training loop:  66%|######5   | 32893/50000 [21:13:07<8:56:51,  1.88s/it]
Outer training loop:  66%|######5   | 32894/50000 [21:13:09<8:50:50,  1.86s/it]
Outer training loop:  66%|######5   | 32895/50000 [21:13:11<9:04:32,  1.91s/it]
Outer training loop:  66%|######5   | 32896/50000 [21:13:13<8:59:01,  1.89s/it]
Outer training loop:  66%|######5   | 32897/50000 [21:13:14<8:43:10,  1.84s/it]
Outer training loop:  66%|######5   | 32898/50000 [21:13:17<9:05:32,  1.91s/it]
Outer training loop:  66%|######5   | 32899/50000 [21:13:19<9:05:30,  1.91s/it]
Outer training loop:  66%|######5   | 32900/50000 [21:13:20<8:53:38,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.94s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  66%|######5   | 32901/50000 [21:13:28<16:41:46,  3.52s/it]
Outer training loop:  66%|######5   | 32902/50000 [21:13:29<14:09:50,  2.98s/it]
Outer training loop:  66%|######5   | 32903/50000 [21:13:31<12:42:05,  2.67s/it]
Outer training loop:  66%|######5   | 32904/50000 [21:13:33<11:20:26,  2.39s/it]
Outer training loop:  66%|######5   | 32905/50000 [21:13:35<10:37:20,  2.24s/it]
Outer training loop:  66%|######5   | 32906/50000 [21:13:37<10:21:47,  2.18s/it]
Outer training loop:  66%|######5   | 32907/50000 [21:13:39<9:50:05,  2.07s/it] 
Outer training loop:  66%|######5   | 32908/50000 [21:13:41<9:25:50,  1.99s/it]
Outer training loop:  66%|######5   | 32909/50000 [21:13:43<9:24:15,  1.98s/it]
Outer training loop:  66%|######5   | 32910/50000 [21:13:44<9:07:46,  1.92s/it]
Outer training loop:  66%|######5   | 32911/50000 [21:13:46<8:59:06,  1.89s/it]
Outer training loop:  66%|######5   | 32912/50000 [21:13:48<8:48:00,  1.85s/it]
Outer training loop:  66%|######5   | 32913/50000 [21:13:50<9:14:04,  1.95s/it]
Outer training loop:  66%|######5   | 32914/50000 [21:13:52<8:59:31,  1.89s/it]
Outer training loop:  66%|######5   | 32915/50000 [21:13:54<8:49:41,  1.86s/it]
Outer training loop:  66%|######5   | 32916/50000 [21:13:56<9:08:42,  1.93s/it]
Outer training loop:  66%|######5   | 32917/50000 [21:13:58<9:03:34,  1.91s/it]
Outer training loop:  66%|######5   | 32918/50000 [21:13:59<8:54:42,  1.88s/it]
Outer training loop:  66%|######5   | 32919/50000 [21:14:01<8:43:23,  1.84s/it]
Outer training loop:  66%|######5   | 32920/50000 [21:14:03<8:55:47,  1.88s/it]
Outer training loop:  66%|######5   | 32921/50000 [21:14:05<9:07:30,  1.92s/it]
Outer training loop:  66%|######5   | 32922/50000 [21:14:07<9:03:17,  1.91s/it]
Outer training loop:  66%|######5   | 32923/50000 [21:14:09<9:07:36,  1.92s/it]
Outer training loop:  66%|######5   | 32924/50000 [21:14:11<8:50:44,  1.86s/it]
Outer training loop:  66%|######5   | 32925/50000 [21:14:13<9:04:53,  1.91s/it]
Outer training loop:  66%|######5   | 32926/50000 [21:14:15<9:02:21,  1.91s/it]
Outer training loop:  66%|######5   | 32927/50000 [21:14:17<9:11:59,  1.94s/it]
Outer training loop:  66%|######5   | 32928/50000 [21:14:18<9:04:05,  1.91s/it]
Outer training loop:  66%|######5   | 32929/50000 [21:14:20<9:12:03,  1.94s/it]
Outer training loop:  66%|######5   | 32930/50000 [21:14:22<9:13:36,  1.95s/it]
Outer training loop:  66%|######5   | 32931/50000 [21:14:24<9:06:19,  1.92s/it]
Outer training loop:  66%|######5   | 32932/50000 [21:14:26<9:00:49,  1.90s/it]
Outer training loop:  66%|######5   | 32933/50000 [21:14:28<9:07:47,  1.93s/it]
Outer training loop:  66%|######5   | 32934/50000 [21:14:30<8:56:26,  1.89s/it]
Outer training loop:  66%|######5   | 32935/50000 [21:14:32<8:52:23,  1.87s/it]
Outer training loop:  66%|######5   | 32936/50000 [21:14:34<9:06:02,  1.92s/it]
Outer training loop:  66%|######5   | 32937/50000 [21:14:36<8:58:54,  1.89s/it]
Outer training loop:  66%|######5   | 32938/50000 [21:14:37<8:50:39,  1.87s/it]
Outer training loop:  66%|######5   | 32939/50000 [21:14:39<9:04:01,  1.91s/it]
Outer training loop:  66%|######5   | 32940/50000 [21:14:41<8:53:08,  1.88s/it]
Outer training loop:  66%|######5   | 32941/50000 [21:14:43<8:53:00,  1.87s/it]
Outer training loop:  66%|######5   | 32942/50000 [21:14:45<8:42:13,  1.84s/it]
Outer training loop:  66%|######5   | 32943/50000 [21:14:47<8:54:11,  1.88s/it]
Outer training loop:  66%|######5   | 32944/50000 [21:14:49<8:57:11,  1.89s/it]
Outer training loop:  66%|######5   | 32945/50000 [21:14:51<8:47:46,  1.86s/it]
Outer training loop:  66%|######5   | 32946/50000 [21:14:52<8:43:54,  1.84s/it]
Outer training loop:  66%|######5   | 32947/50000 [21:14:54<8:52:50,  1.87s/it]
Outer training loop:  66%|######5   | 32948/50000 [21:14:56<8:44:39,  1.85s/it]
Outer training loop:  66%|######5   | 32949/50000 [21:14:58<8:56:33,  1.89s/it]
Outer training loop:  66%|######5   | 32950/50000 [21:15:00<8:44:19,  1.85s/it]
Outer training loop:  66%|######5   | 32951/50000 [21:15:02<8:41:44,  1.84s/it]
Outer training loop:  66%|######5   | 32952/50000 [21:15:03<8:35:49,  1.82s/it]
Outer training loop:  66%|######5   | 32953/50000 [21:15:05<8:52:36,  1.87s/it]
Outer training loop:  66%|######5   | 32954/50000 [21:15:07<8:46:58,  1.85s/it]
Outer training loop:  66%|######5   | 32955/50000 [21:15:09<8:48:27,  1.86s/it]
Outer training loop:  66%|######5   | 32956/50000 [21:15:11<8:33:49,  1.81s/it]
Outer training loop:  66%|######5   | 32957/50000 [21:15:13<8:48:41,  1.86s/it]
Outer training loop:  66%|######5   | 32958/50000 [21:15:15<8:37:59,  1.82s/it]
Outer training loop:  66%|######5   | 32959/50000 [21:15:16<8:45:16,  1.85s/it]
Outer training loop:  66%|######5   | 32960/50000 [21:15:18<8:52:45,  1.88s/it]
Outer training loop:  66%|######5   | 32961/50000 [21:15:20<9:10:06,  1.94s/it]
Outer training loop:  66%|######5   | 32962/50000 [21:15:22<9:01:53,  1.91s/it]
Outer training loop:  66%|######5   | 32963/50000 [21:15:24<9:13:42,  1.95s/it]
Outer training loop:  66%|######5   | 32964/50000 [21:15:26<9:09:51,  1.94s/it]
Outer training loop:  66%|######5   | 32965/50000 [21:15:28<9:16:30,  1.96s/it]
Outer training loop:  66%|######5   | 32966/50000 [21:15:30<8:59:49,  1.90s/it]
Outer training loop:  66%|######5   | 32967/50000 [21:15:32<9:16:50,  1.96s/it]
Outer training loop:  66%|######5   | 32968/50000 [21:15:34<9:14:42,  1.95s/it]
Outer training loop:  66%|######5   | 32969/50000 [21:15:36<9:05:29,  1.92s/it]
Outer training loop:  66%|######5   | 32970/50000 [21:15:38<9:09:21,  1.94s/it]
Outer training loop:  66%|######5   | 32971/50000 [21:15:40<9:18:36,  1.97s/it]
Outer training loop:  66%|######5   | 32972/50000 [21:15:42<9:11:36,  1.94s/it]
Outer training loop:  66%|######5   | 32973/50000 [21:15:44<9:11:34,  1.94s/it]
Outer training loop:  66%|######5   | 32974/50000 [21:15:46<9:24:54,  1.99s/it]
Outer training loop:  66%|######5   | 32975/50000 [21:15:48<9:15:51,  1.96s/it]
Outer training loop:  66%|######5   | 32976/50000 [21:15:50<9:04:10,  1.92s/it]
Outer training loop:  66%|######5   | 32977/50000 [21:15:52<9:18:16,  1.97s/it]
Outer training loop:  66%|######5   | 32978/50000 [21:15:54<9:17:11,  1.96s/it]
Outer training loop:  66%|######5   | 32979/50000 [21:15:56<9:18:38,  1.97s/it]
Outer training loop:  66%|######5   | 32980/50000 [21:15:58<9:23:59,  1.99s/it]
Outer training loop:  66%|######5   | 32981/50000 [21:16:00<9:17:54,  1.97s/it]
Outer training loop:  66%|######5   | 32982/50000 [21:16:01<9:06:31,  1.93s/it]
Outer training loop:  66%|######5   | 32983/50000 [21:16:03<9:18:48,  1.97s/it]
Outer training loop:  66%|######5   | 32984/50000 [21:16:05<9:27:42,  2.00s/it]
Outer training loop:  66%|######5   | 32985/50000 [21:16:07<9:18:15,  1.97s/it]
Outer training loop:  66%|######5   | 32986/50000 [21:16:09<9:11:47,  1.95s/it]
Outer training loop:  66%|######5   | 32987/50000 [21:16:11<9:19:51,  1.97s/it]
Outer training loop:  66%|######5   | 32988/50000 [21:16:13<9:17:05,  1.96s/it]
Outer training loop:  66%|######5   | 32989/50000 [21:16:15<9:10:21,  1.94s/it]
Outer training loop:  66%|######5   | 32990/50000 [21:16:17<9:16:35,  1.96s/it]
Outer training loop:  66%|######5   | 32991/50000 [21:16:19<9:06:28,  1.93s/it]
Outer training loop:  66%|######5   | 32992/50000 [21:16:21<9:07:18,  1.93s/it]
Outer training loop:  66%|######5   | 32993/50000 [21:16:23<8:54:14,  1.88s/it]
Outer training loop:  66%|######5   | 32994/50000 [21:16:25<9:23:30,  1.99s/it]
Outer training loop:  66%|######5   | 32995/50000 [21:16:27<9:12:33,  1.95s/it]
Outer training loop:  66%|######5   | 32996/50000 [21:16:29<9:04:05,  1.92s/it]
Outer training loop:  66%|######5   | 32997/50000 [21:16:31<9:08:06,  1.93s/it]
Outer training loop:  66%|######5   | 32998/50000 [21:16:32<9:01:36,  1.91s/it]
Outer training loop:  66%|######5   | 32999/50000 [21:16:34<8:54:40,  1.89s/it]
Outer training loop:  66%|######6   | 33000/50000 [21:16:36<9:07:22,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.96s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.92s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:26,  1.79s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:26,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:25,  1.81s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.87s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:24,  1.85s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:22,  1.84s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:21,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:19,  1.89s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:17,  1.88s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.93s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:13,  1.96s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.94s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:00,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:04,  1.97s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<02:00,  1.94s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:57,  1.92s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:57,  1.96s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:56,  2.02s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:52,  1.98s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:49,  1.96s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:45,  1.98s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:43,  1.99s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.94s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.92s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.86s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:32,  1.96s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:22,  1.87s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:16,  1.82s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:21,  1.99s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:17,  1.94s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:10,  1.85s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:07,  1.82s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:04,  1.79s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:02,  1.80s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.90s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:02,  1.88s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.85s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.84s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:52,  1.95s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:50,  1.93s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:38,  1.82s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.82s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.79s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:34,  1.92s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.90s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.87s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.82s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:23,  1.84s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:17,  1.78s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:17,  1.97s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.95s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.91s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.87s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.90s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.82s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<03:01,  2.21s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:49,  2.09s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:40,  2.00s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:41,  2.04s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:32,  1.96s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.93s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:26,  1.95s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:16,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:19,  1.97s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.88s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:11,  1.96s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.90s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.90s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:57,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:59,  1.93s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.88s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:47,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:51,  1.95s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:42,  1.97s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.92s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:31,  2.00s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:26,  1.93s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:24,  1.91s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:22,  1.91s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:19,  1.90s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:16,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:19,  1.99s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.90s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:09,  1.92s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.87s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:02,  1.85s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:03,  1.92s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:01,  1.92s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:52,  1.80s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:50,  1.81s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:49,  1.84s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:47,  1.82s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:49,  1.98s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:38,  1.84s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:33,  1.98s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:31,  1.94s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.91s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.90s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.87s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.99s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.93s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.80s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.83s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.83s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.80s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.84s/it]

Outer training loop:  66%|######6   | 33001/50000 [21:21:57<461:14:04, 97.68s/it]
Outer training loop:  66%|######6   | 33002/50000 [21:22:00<326:20:31, 69.12s/it]
Outer training loop:  66%|######6   | 33003/50000 [21:22:02<231:13:06, 48.97s/it]
Outer training loop:  66%|######6   | 33004/50000 [21:22:04<164:15:51, 34.79s/it]
Outer training loop:  66%|######6   | 33005/50000 [21:22:05<117:39:45, 24.92s/it]
Outer training loop:  66%|######6   | 33006/50000 [21:22:07<85:04:03, 18.02s/it] 
Outer training loop:  66%|######6   | 33007/50000 [21:22:09<62:10:46, 13.17s/it]
Outer training loop:  66%|######6   | 33008/50000 [21:22:11<46:07:30,  9.77s/it]
Outer training loop:  66%|######6   | 33009/50000 [21:22:13<35:16:07,  7.47s/it]
Outer training loop:  66%|######6   | 33010/50000 [21:22:15<27:19:45,  5.79s/it]
Outer training loop:  66%|######6   | 33011/50000 [21:22:17<21:39:34,  4.59s/it]
Outer training loop:  66%|######6   | 33012/50000 [21:22:19<18:07:25,  3.84s/it]
Outer training loop:  66%|######6   | 33013/50000 [21:22:21<15:21:02,  3.25s/it]
Outer training loop:  66%|######6   | 33014/50000 [21:22:23<13:19:20,  2.82s/it]
Outer training loop:  66%|######6   | 33015/50000 [21:22:25<11:56:25,  2.53s/it]
Outer training loop:  66%|######6   | 33016/50000 [21:22:26<11:07:49,  2.36s/it]
Outer training loop:  66%|######6   | 33017/50000 [21:22:28<10:31:00,  2.23s/it]
Outer training loop:  66%|######6   | 33018/50000 [21:22:30<9:47:08,  2.07s/it] 
Outer training loop:  66%|######6   | 33019/50000 [21:22:32<9:23:01,  1.99s/it]
Outer training loop:  66%|######6   | 33020/50000 [21:22:34<9:30:36,  2.02s/it]
Outer training loop:  66%|######6   | 33021/50000 [21:22:36<9:21:15,  1.98s/it]
Outer training loop:  66%|######6   | 33022/50000 [21:22:38<9:03:13,  1.92s/it]
Outer training loop:  66%|######6   | 33023/50000 [21:22:40<9:13:13,  1.96s/it]
Outer training loop:  66%|######6   | 33024/50000 [21:22:42<9:13:00,  1.95s/it]
Outer training loop:  66%|######6   | 33025/50000 [21:22:43<9:02:59,  1.92s/it]
Outer training loop:  66%|######6   | 33026/50000 [21:22:45<9:05:24,  1.93s/it]
Outer training loop:  66%|######6   | 33027/50000 [21:22:47<8:43:56,  1.85s/it]
Outer training loop:  66%|######6   | 33028/50000 [21:22:49<8:52:30,  1.88s/it]
Outer training loop:  66%|######6   | 33029/50000 [21:22:51<8:37:11,  1.83s/it]
Outer training loop:  66%|######6   | 33030/50000 [21:22:53<8:53:39,  1.89s/it]
Outer training loop:  66%|######6   | 33031/50000 [21:22:55<8:51:19,  1.88s/it]
Outer training loop:  66%|######6   | 33032/50000 [21:22:56<8:47:08,  1.86s/it]
Outer training loop:  66%|######6   | 33033/50000 [21:22:58<8:57:21,  1.90s/it]
Outer training loop:  66%|######6   | 33034/50000 [21:23:00<8:55:45,  1.89s/it]
Outer training loop:  66%|######6   | 33035/50000 [21:23:02<8:45:25,  1.86s/it]
Outer training loop:  66%|######6   | 33036/50000 [21:23:04<9:03:36,  1.92s/it]
Outer training loop:  66%|######6   | 33037/50000 [21:23:06<8:56:07,  1.90s/it]
Outer training loop:  66%|######6   | 33038/50000 [21:23:08<8:44:41,  1.86s/it]
Outer training loop:  66%|######6   | 33039/50000 [21:23:10<9:02:00,  1.92s/it]
Outer training loop:  66%|######6   | 33040/50000 [21:23:12<8:50:17,  1.88s/it]
Outer training loop:  66%|######6   | 33041/50000 [21:23:14<8:53:59,  1.89s/it]
Outer training loop:  66%|######6   | 33042/50000 [21:23:15<8:40:37,  1.84s/it]
Outer training loop:  66%|######6   | 33043/50000 [21:23:17<8:47:32,  1.87s/it]
Outer training loop:  66%|######6   | 33044/50000 [21:23:19<8:52:23,  1.88s/it]
Outer training loop:  66%|######6   | 33045/50000 [21:23:21<8:54:53,  1.89s/it]
Outer training loop:  66%|######6   | 33046/50000 [21:23:23<9:01:16,  1.92s/it]
Outer training loop:  66%|######6   | 33047/50000 [21:23:25<8:55:07,  1.89s/it]
Outer training loop:  66%|######6   | 33048/50000 [21:23:27<8:46:05,  1.86s/it]
Outer training loop:  66%|######6   | 33049/50000 [21:23:29<8:57:57,  1.90s/it]
Outer training loop:  66%|######6   | 33050/50000 [21:23:30<8:48:19,  1.87s/it]
Outer training loop:  66%|######6   | 33051/50000 [21:23:32<8:43:30,  1.85s/it]
Outer training loop:  66%|######6   | 33052/50000 [21:23:34<8:53:09,  1.89s/it]
Outer training loop:  66%|######6   | 33053/50000 [21:23:36<9:08:19,  1.94s/it]
Outer training loop:  66%|######6   | 33054/50000 [21:23:38<9:01:15,  1.92s/it]
Outer training loop:  66%|######6   | 33055/50000 [21:23:40<8:52:47,  1.89s/it]
Outer training loop:  66%|######6   | 33056/50000 [21:23:42<9:05:56,  1.93s/it]
Outer training loop:  66%|######6   | 33057/50000 [21:23:44<8:59:47,  1.91s/it]
Outer training loop:  66%|######6   | 33058/50000 [21:23:46<8:54:17,  1.89s/it]
Outer training loop:  66%|######6   | 33059/50000 [21:23:48<8:49:04,  1.87s/it]
Outer training loop:  66%|######6   | 33060/50000 [21:23:50<9:11:30,  1.95s/it]
Outer training loop:  66%|######6   | 33061/50000 [21:23:51<8:55:27,  1.90s/it]
Outer training loop:  66%|######6   | 33062/50000 [21:23:53<8:53:39,  1.89s/it]
Outer training loop:  66%|######6   | 33063/50000 [21:23:55<9:04:10,  1.93s/it]
Outer training loop:  66%|######6   | 33064/50000 [21:23:57<8:59:54,  1.91s/it]
Outer training loop:  66%|######6   | 33065/50000 [21:23:59<8:45:56,  1.86s/it]
Outer training loop:  66%|######6   | 33066/50000 [21:24:01<8:55:04,  1.90s/it]
Outer training loop:  66%|######6   | 33067/50000 [21:24:03<8:42:53,  1.85s/it]
Outer training loop:  66%|######6   | 33068/50000 [21:24:04<8:38:05,  1.84s/it]
Outer training loop:  66%|######6   | 33069/50000 [21:24:06<8:33:13,  1.82s/it]
Outer training loop:  66%|######6   | 33070/50000 [21:24:08<8:44:36,  1.86s/it]
Outer training loop:  66%|######6   | 33071/50000 [21:24:10<8:40:47,  1.85s/it]
Outer training loop:  66%|######6   | 33072/50000 [21:24:12<8:40:35,  1.85s/it]
Outer training loop:  66%|######6   | 33073/50000 [21:24:14<8:43:05,  1.85s/it]
Outer training loop:  66%|######6   | 33074/50000 [21:24:16<8:53:32,  1.89s/it]
Outer training loop:  66%|######6   | 33075/50000 [21:24:17<8:34:56,  1.83s/it]
Outer training loop:  66%|######6   | 33076/50000 [21:24:19<8:43:40,  1.86s/it]
Outer training loop:  66%|######6   | 33077/50000 [21:24:21<8:49:08,  1.88s/it]
Outer training loop:  66%|######6   | 33078/50000 [21:24:23<8:37:43,  1.84s/it]
Outer training loop:  66%|######6   | 33079/50000 [21:24:25<8:33:47,  1.82s/it]
Outer training loop:  66%|######6   | 33080/50000 [21:24:27<8:52:31,  1.89s/it]
Outer training loop:  66%|######6   | 33081/50000 [21:24:29<8:48:53,  1.88s/it]
Outer training loop:  66%|######6   | 33082/50000 [21:24:30<8:42:09,  1.85s/it]
Outer training loop:  66%|######6   | 33083/50000 [21:24:32<8:52:33,  1.89s/it]
Outer training loop:  66%|######6   | 33084/50000 [21:24:34<8:50:29,  1.88s/it]
Outer training loop:  66%|######6   | 33085/50000 [21:24:36<8:36:03,  1.83s/it]
Outer training loop:  66%|######6   | 33086/50000 [21:24:38<8:23:35,  1.79s/it]
Outer training loop:  66%|######6   | 33087/50000 [21:24:40<8:42:53,  1.86s/it]
Outer training loop:  66%|######6   | 33088/50000 [21:24:42<8:41:26,  1.85s/it]
Outer training loop:  66%|######6   | 33089/50000 [21:24:43<8:37:45,  1.84s/it]
Outer training loop:  66%|######6   | 33090/50000 [21:24:45<8:57:29,  1.91s/it]
Outer training loop:  66%|######6   | 33091/50000 [21:24:47<8:46:25,  1.87s/it]
Outer training loop:  66%|######6   | 33092/50000 [21:24:49<8:37:16,  1.84s/it]
Outer training loop:  66%|######6   | 33093/50000 [21:24:51<8:49:13,  1.88s/it]
Outer training loop:  66%|######6   | 33094/50000 [21:24:53<8:40:08,  1.85s/it]
Outer training loop:  66%|######6   | 33095/50000 [21:24:55<8:40:27,  1.85s/it]
Outer training loop:  66%|######6   | 33096/50000 [21:24:57<8:54:16,  1.90s/it]
Outer training loop:  66%|######6   | 33097/50000 [21:24:58<8:36:36,  1.83s/it]
Outer training loop:  66%|######6   | 33098/50000 [21:25:00<8:35:10,  1.83s/it]
Outer training loop:  66%|######6   | 33099/50000 [21:25:02<8:25:12,  1.79s/it]
Outer training loop:  66%|######6   | 33100/50000 [21:25:04<8:34:22,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  66%|######6   | 33101/50000 [21:25:11<16:24:48,  3.50s/it]
Outer training loop:  66%|######6   | 33102/50000 [21:25:13<14:01:29,  2.99s/it]
Outer training loop:  66%|######6   | 33103/50000 [21:25:15<12:31:46,  2.67s/it]
Outer training loop:  66%|######6   | 33104/50000 [21:25:17<11:34:44,  2.47s/it]
Outer training loop:  66%|######6   | 33105/50000 [21:25:19<10:39:41,  2.27s/it]
Outer training loop:  66%|######6   | 33106/50000 [21:25:20<9:54:03,  2.11s/it] 
Outer training loop:  66%|######6   | 33107/50000 [21:25:22<9:39:04,  2.06s/it]
Outer training loop:  66%|######6   | 33108/50000 [21:25:24<9:14:13,  1.97s/it]
Outer training loop:  66%|######6   | 33109/50000 [21:25:26<9:08:34,  1.95s/it]
Outer training loop:  66%|######6   | 33110/50000 [21:25:28<9:02:35,  1.93s/it]
Outer training loop:  66%|######6   | 33111/50000 [21:25:30<8:48:36,  1.88s/it]
Outer training loop:  66%|######6   | 33112/50000 [21:25:31<8:32:11,  1.82s/it]
Outer training loop:  66%|######6   | 33113/50000 [21:25:33<8:30:04,  1.81s/it]
Outer training loop:  66%|######6   | 33114/50000 [21:25:35<8:44:25,  1.86s/it]
Outer training loop:  66%|######6   | 33115/50000 [21:25:37<8:32:03,  1.82s/it]
Outer training loop:  66%|######6   | 33116/50000 [21:25:39<8:27:05,  1.80s/it]
Outer training loop:  66%|######6   | 33117/50000 [21:25:40<8:32:05,  1.82s/it]
Outer training loop:  66%|######6   | 33118/50000 [21:25:42<8:42:04,  1.86s/it]
Outer training loop:  66%|######6   | 33119/50000 [21:25:44<8:35:46,  1.83s/it]
Outer training loop:  66%|######6   | 33120/50000 [21:25:46<8:39:12,  1.85s/it]
Outer training loop:  66%|######6   | 33121/50000 [21:25:48<8:53:12,  1.90s/it]
Outer training loop:  66%|######6   | 33122/50000 [21:25:50<8:51:03,  1.89s/it]
Outer training loop:  66%|######6   | 33123/50000 [21:25:52<8:37:55,  1.84s/it]
Outer training loop:  66%|######6   | 33124/50000 [21:25:53<8:42:05,  1.86s/it]
Outer training loop:  66%|######6   | 33125/50000 [21:25:55<8:36:43,  1.84s/it]
Outer training loop:  66%|######6   | 33126/50000 [21:25:57<8:33:47,  1.83s/it]
Outer training loop:  66%|######6   | 33127/50000 [21:25:59<8:28:04,  1.81s/it]
Outer training loop:  66%|######6   | 33128/50000 [21:26:01<8:50:27,  1.89s/it]
Outer training loop:  66%|######6   | 33129/50000 [21:26:03<8:36:46,  1.84s/it]
Outer training loop:  66%|######6   | 33130/50000 [21:26:04<8:36:01,  1.84s/it]
Outer training loop:  66%|######6   | 33131/50000 [21:26:06<8:47:13,  1.88s/it]
Outer training loop:  66%|######6   | 33132/50000 [21:26:08<8:50:48,  1.89s/it]
Outer training loop:  66%|######6   | 33133/50000 [21:26:10<8:51:16,  1.89s/it]
Outer training loop:  66%|######6   | 33134/50000 [21:26:12<8:34:24,  1.83s/it]
Outer training loop:  66%|######6   | 33135/50000 [21:26:14<8:53:06,  1.90s/it]
Outer training loop:  66%|######6   | 33136/50000 [21:26:16<8:43:31,  1.86s/it]
Outer training loop:  66%|######6   | 33137/50000 [21:26:17<8:29:09,  1.81s/it]
Outer training loop:  66%|######6   | 33138/50000 [21:26:19<8:36:57,  1.84s/it]
Outer training loop:  66%|######6   | 33139/50000 [21:26:21<8:59:32,  1.92s/it]
Outer training loop:  66%|######6   | 33140/50000 [21:26:23<8:42:30,  1.86s/it]
Outer training loop:  66%|######6   | 33141/50000 [21:26:25<8:40:42,  1.85s/it]
Outer training loop:  66%|######6   | 33142/50000 [21:26:27<8:40:56,  1.85s/it]
Outer training loop:  66%|######6   | 33143/50000 [21:26:29<8:38:39,  1.85s/it]
Outer training loop:  66%|######6   | 33144/50000 [21:26:31<8:37:46,  1.84s/it]
Outer training loop:  66%|######6   | 33145/50000 [21:26:32<8:27:43,  1.81s/it]
Outer training loop:  66%|######6   | 33146/50000 [21:26:34<8:47:06,  1.88s/it]
Outer training loop:  66%|######6   | 33147/50000 [21:26:36<8:43:34,  1.86s/it]
Outer training loop:  66%|######6   | 33148/50000 [21:26:38<8:35:51,  1.84s/it]
Outer training loop:  66%|######6   | 33149/50000 [21:26:40<8:53:47,  1.90s/it]
Outer training loop:  66%|######6   | 33150/50000 [21:26:42<8:52:42,  1.90s/it]
Outer training loop:  66%|######6   | 33151/50000 [21:26:44<8:44:27,  1.87s/it]
Outer training loop:  66%|######6   | 33152/50000 [21:26:46<8:48:11,  1.88s/it]
Outer training loop:  66%|######6   | 33153/50000 [21:26:47<8:37:26,  1.84s/it]
Outer training loop:  66%|######6   | 33154/50000 [21:26:49<8:36:11,  1.84s/it]
Outer training loop:  66%|######6   | 33155/50000 [21:26:51<8:43:00,  1.86s/it]
Outer training loop:  66%|######6   | 33156/50000 [21:26:53<8:46:03,  1.87s/it]
Outer training loop:  66%|######6   | 33157/50000 [21:26:55<8:44:03,  1.87s/it]
Outer training loop:  66%|######6   | 33158/50000 [21:26:57<8:41:21,  1.86s/it]
Outer training loop:  66%|######6   | 33159/50000 [21:26:59<8:55:27,  1.91s/it]
Outer training loop:  66%|######6   | 33160/50000 [21:27:01<8:48:51,  1.88s/it]
Outer training loop:  66%|######6   | 33161/50000 [21:27:02<8:41:38,  1.86s/it]
Outer training loop:  66%|######6   | 33162/50000 [21:27:04<8:56:40,  1.91s/it]
Outer training loop:  66%|######6   | 33163/50000 [21:27:06<8:44:40,  1.87s/it]
Outer training loop:  66%|######6   | 33164/50000 [21:27:08<8:38:59,  1.85s/it]
Outer training loop:  66%|######6   | 33165/50000 [21:27:10<8:32:27,  1.83s/it]
Outer training loop:  66%|######6   | 33166/50000 [21:27:12<8:48:56,  1.89s/it]
Outer training loop:  66%|######6   | 33167/50000 [21:27:14<8:52:15,  1.90s/it]
Outer training loop:  66%|######6   | 33168/50000 [21:27:15<8:38:07,  1.85s/it]
Outer training loop:  66%|######6   | 33169/50000 [21:27:17<8:45:49,  1.87s/it]
Outer training loop:  66%|######6   | 33170/50000 [21:27:19<8:48:37,  1.88s/it]
Outer training loop:  66%|######6   | 33171/50000 [21:27:21<8:43:24,  1.87s/it]
Outer training loop:  66%|######6   | 33172/50000 [21:27:23<8:45:39,  1.87s/it]
Outer training loop:  66%|######6   | 33173/50000 [21:27:25<8:52:17,  1.90s/it]
Outer training loop:  66%|######6   | 33174/50000 [21:27:27<8:51:37,  1.90s/it]
Outer training loop:  66%|######6   | 33175/50000 [21:27:29<8:47:34,  1.88s/it]
Outer training loop:  66%|######6   | 33176/50000 [21:27:31<8:54:21,  1.91s/it]
Outer training loop:  66%|######6   | 33177/50000 [21:27:32<8:50:47,  1.89s/it]
Outer training loop:  66%|######6   | 33178/50000 [21:27:34<8:46:08,  1.88s/it]
Outer training loop:  66%|######6   | 33179/50000 [21:27:36<8:40:47,  1.86s/it]
Outer training loop:  66%|######6   | 33180/50000 [21:27:38<8:46:11,  1.88s/it]
Outer training loop:  66%|######6   | 33181/50000 [21:27:40<8:38:01,  1.85s/it]
Outer training loop:  66%|######6   | 33182/50000 [21:27:42<8:27:17,  1.81s/it]
Outer training loop:  66%|######6   | 33183/50000 [21:27:44<8:42:27,  1.86s/it]
Outer training loop:  66%|######6   | 33184/50000 [21:27:45<8:44:06,  1.87s/it]
Outer training loop:  66%|######6   | 33185/50000 [21:27:47<8:49:25,  1.89s/it]
Outer training loop:  66%|######6   | 33186/50000 [21:27:49<9:02:33,  1.94s/it]
Outer training loop:  66%|######6   | 33187/50000 [21:27:51<9:01:56,  1.93s/it]
Outer training loop:  66%|######6   | 33188/50000 [21:27:53<8:45:10,  1.87s/it]
Outer training loop:  66%|######6   | 33189/50000 [21:27:55<8:58:59,  1.92s/it]
Outer training loop:  66%|######6   | 33190/50000 [21:27:57<8:55:43,  1.91s/it]
Outer training loop:  66%|######6   | 33191/50000 [21:27:59<8:52:49,  1.90s/it]
Outer training loop:  66%|######6   | 33192/50000 [21:28:01<8:59:35,  1.93s/it]
Outer training loop:  66%|######6   | 33193/50000 [21:28:03<8:54:14,  1.91s/it]
Outer training loop:  66%|######6   | 33194/50000 [21:28:05<8:50:35,  1.89s/it]
Outer training loop:  66%|######6   | 33195/50000 [21:28:06<8:50:00,  1.89s/it]
Outer training loop:  66%|######6   | 33196/50000 [21:28:08<8:52:09,  1.90s/it]
Outer training loop:  66%|######6   | 33197/50000 [21:28:10<8:47:58,  1.89s/it]
Outer training loop:  66%|######6   | 33198/50000 [21:28:12<8:43:12,  1.87s/it]
Outer training loop:  66%|######6   | 33199/50000 [21:28:14<9:21:30,  2.01s/it]
Outer training loop:  66%|######6   | 33200/50000 [21:28:16<9:03:31,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  66%|######6   | 33201/50000 [21:28:24<16:44:45,  3.59s/it]
Outer training loop:  66%|######6   | 33202/50000 [21:28:25<14:22:32,  3.08s/it]
Outer training loop:  66%|######6   | 33203/50000 [21:28:27<12:36:55,  2.70s/it]
Outer training loop:  66%|######6   | 33204/50000 [21:28:29<11:39:15,  2.50s/it]
Outer training loop:  66%|######6   | 33205/50000 [21:28:31<10:39:29,  2.28s/it]
Outer training loop:  66%|######6   | 33206/50000 [21:28:33<10:10:45,  2.18s/it]
Outer training loop:  66%|######6   | 33207/50000 [21:28:35<9:59:29,  2.14s/it] 
Outer training loop:  66%|######6   | 33208/50000 [21:28:37<9:28:32,  2.03s/it]
Outer training loop:  66%|######6   | 33209/50000 [21:28:39<9:07:19,  1.96s/it]
Outer training loop:  66%|######6   | 33210/50000 [21:28:41<9:14:13,  1.98s/it]
Outer training loop:  66%|######6   | 33211/50000 [21:28:43<9:05:13,  1.95s/it]
Outer training loop:  66%|######6   | 33212/50000 [21:28:44<8:53:09,  1.91s/it]
Outer training loop:  66%|######6   | 33213/50000 [21:28:46<8:51:14,  1.90s/it]
Outer training loop:  66%|######6   | 33214/50000 [21:28:48<9:03:47,  1.94s/it]
Outer training loop:  66%|######6   | 33215/50000 [21:28:50<8:53:00,  1.91s/it]
Outer training loop:  66%|######6   | 33216/50000 [21:28:52<8:46:13,  1.88s/it]
Outer training loop:  66%|######6   | 33217/50000 [21:28:54<8:54:52,  1.91s/it]
Outer training loop:  66%|######6   | 33218/50000 [21:28:56<8:43:35,  1.87s/it]
Outer training loop:  66%|######6   | 33219/50000 [21:28:58<8:44:45,  1.88s/it]
Outer training loop:  66%|######6   | 33220/50000 [21:29:00<8:53:01,  1.91s/it]
Outer training loop:  66%|######6   | 33221/50000 [21:29:01<8:39:47,  1.86s/it]
Outer training loop:  66%|######6   | 33222/50000 [21:29:03<8:25:31,  1.81s/it]
Outer training loop:  66%|######6   | 33223/50000 [21:29:05<8:23:33,  1.80s/it]
Outer training loop:  66%|######6   | 33224/50000 [21:29:07<8:47:03,  1.89s/it]
Outer training loop:  66%|######6   | 33225/50000 [21:29:09<8:36:21,  1.85s/it]
Outer training loop:  66%|######6   | 33226/50000 [21:29:10<8:27:31,  1.82s/it]
Outer training loop:  66%|######6   | 33227/50000 [21:29:12<8:25:11,  1.81s/it]
Outer training loop:  66%|######6   | 33228/50000 [21:29:14<8:48:04,  1.89s/it]
Outer training loop:  66%|######6   | 33229/50000 [21:29:16<8:40:19,  1.86s/it]
Outer training loop:  66%|######6   | 33230/50000 [21:29:18<8:30:20,  1.83s/it]
Outer training loop:  66%|######6   | 33231/50000 [21:29:20<8:48:45,  1.89s/it]
Outer training loop:  66%|######6   | 33232/50000 [21:29:22<8:50:44,  1.90s/it]
Outer training loop:  66%|######6   | 33233/50000 [21:29:24<8:39:53,  1.86s/it]
Outer training loop:  66%|######6   | 33234/50000 [21:29:26<8:57:13,  1.92s/it]
Outer training loop:  66%|######6   | 33235/50000 [21:29:27<8:51:54,  1.90s/it]
Outer training loop:  66%|######6   | 33236/50000 [21:29:29<8:46:31,  1.88s/it]
Outer training loop:  66%|######6   | 33237/50000 [21:29:31<8:32:15,  1.83s/it]
Outer training loop:  66%|######6   | 33238/50000 [21:29:33<8:39:10,  1.86s/it]
Outer training loop:  66%|######6   | 33239/50000 [21:29:35<8:46:31,  1.88s/it]
Outer training loop:  66%|######6   | 33240/50000 [21:29:37<8:35:48,  1.85s/it]
Outer training loop:  66%|######6   | 33241/50000 [21:29:39<8:46:38,  1.89s/it]
Outer training loop:  66%|######6   | 33242/50000 [21:29:41<8:47:37,  1.89s/it]
Outer training loop:  66%|######6   | 33243/50000 [21:29:42<8:41:31,  1.87s/it]
Outer training loop:  66%|######6   | 33244/50000 [21:29:44<8:55:31,  1.92s/it]
Outer training loop:  66%|######6   | 33245/50000 [21:29:46<8:45:57,  1.88s/it]
Outer training loop:  66%|######6   | 33246/50000 [21:29:48<8:53:19,  1.91s/it]
Outer training loop:  66%|######6   | 33247/50000 [21:29:50<8:49:13,  1.90s/it]
Outer training loop:  66%|######6   | 33248/50000 [21:29:52<9:03:38,  1.95s/it]
Outer training loop:  66%|######6   | 33249/50000 [21:29:54<8:57:52,  1.93s/it]
Outer training loop:  66%|######6   | 33250/50000 [21:29:56<8:53:38,  1.91s/it]
Outer training loop:  67%|######6   | 33251/50000 [21:29:58<9:09:08,  1.97s/it]
Outer training loop:  67%|######6   | 33252/50000 [21:30:00<8:55:42,  1.92s/it]
Outer training loop:  67%|######6   | 33253/50000 [21:30:01<8:42:34,  1.87s/it]
Outer training loop:  67%|######6   | 33254/50000 [21:30:04<8:59:37,  1.93s/it]
Outer training loop:  67%|######6   | 33255/50000 [21:30:05<8:59:36,  1.93s/it]
Outer training loop:  67%|######6   | 33256/50000 [21:30:07<8:48:28,  1.89s/it]
Outer training loop:  67%|######6   | 33257/50000 [21:30:09<8:44:04,  1.88s/it]
Outer training loop:  67%|######6   | 33258/50000 [21:30:11<8:59:02,  1.93s/it]
Outer training loop:  67%|######6   | 33259/50000 [21:30:13<8:51:36,  1.91s/it]
Outer training loop:  67%|######6   | 33260/50000 [21:30:15<8:45:07,  1.88s/it]
Outer training loop:  67%|######6   | 33261/50000 [21:30:17<8:53:45,  1.91s/it]
Outer training loop:  67%|######6   | 33262/50000 [21:30:19<8:58:04,  1.93s/it]
Outer training loop:  67%|######6   | 33263/50000 [21:30:21<8:54:52,  1.92s/it]
Outer training loop:  67%|######6   | 33264/50000 [21:30:23<9:07:00,  1.96s/it]
Outer training loop:  67%|######6   | 33265/50000 [21:30:25<9:07:11,  1.96s/it]
Outer training loop:  67%|######6   | 33266/50000 [21:30:27<8:52:26,  1.91s/it]
Outer training loop:  67%|######6   | 33267/50000 [21:30:28<8:56:55,  1.93s/it]
Outer training loop:  67%|######6   | 33268/50000 [21:30:30<8:56:29,  1.92s/it]
Outer training loop:  67%|######6   | 33269/50000 [21:30:32<8:55:41,  1.92s/it]
Outer training loop:  67%|######6   | 33270/50000 [21:30:34<8:55:17,  1.92s/it]
Outer training loop:  67%|######6   | 33271/50000 [21:30:36<9:01:28,  1.94s/it]
Outer training loop:  67%|######6   | 33272/50000 [21:30:38<8:52:26,  1.91s/it]
Outer training loop:  67%|######6   | 33273/50000 [21:30:40<8:44:53,  1.88s/it]
Outer training loop:  67%|######6   | 33274/50000 [21:30:42<8:39:13,  1.86s/it]
Outer training loop:  67%|######6   | 33275/50000 [21:30:44<8:55:11,  1.92s/it]
Outer training loop:  67%|######6   | 33276/50000 [21:30:46<8:45:52,  1.89s/it]
Outer training loop:  67%|######6   | 33277/50000 [21:30:47<8:38:40,  1.86s/it]
Outer training loop:  67%|######6   | 33278/50000 [21:30:49<8:51:50,  1.91s/it]
Outer training loop:  67%|######6   | 33279/50000 [21:30:51<8:39:50,  1.87s/it]
Outer training loop:  67%|######6   | 33280/50000 [21:30:53<8:34:18,  1.85s/it]
Outer training loop:  67%|######6   | 33281/50000 [21:30:55<8:55:14,  1.92s/it]
Outer training loop:  67%|######6   | 33282/50000 [21:30:57<8:47:40,  1.89s/it]
Outer training loop:  67%|######6   | 33283/50000 [21:30:59<8:43:43,  1.88s/it]
Outer training loop:  67%|######6   | 33284/50000 [21:31:01<8:36:08,  1.85s/it]
Outer training loop:  67%|######6   | 33285/50000 [21:31:03<8:51:05,  1.91s/it]
Outer training loop:  67%|######6   | 33286/50000 [21:31:04<8:47:04,  1.89s/it]
Outer training loop:  67%|######6   | 33287/50000 [21:31:06<8:49:14,  1.90s/it]
Outer training loop:  67%|######6   | 33288/50000 [21:31:08<9:01:56,  1.95s/it]
Outer training loop:  67%|######6   | 33289/50000 [21:31:10<9:05:04,  1.96s/it]
Outer training loop:  67%|######6   | 33290/50000 [21:31:12<8:58:20,  1.93s/it]
Outer training loop:  67%|######6   | 33291/50000 [21:31:14<8:49:42,  1.90s/it]
Outer training loop:  67%|######6   | 33292/50000 [21:31:16<9:04:42,  1.96s/it]
Outer training loop:  67%|######6   | 33293/50000 [21:31:18<8:50:14,  1.90s/it]
Outer training loop:  67%|######6   | 33294/50000 [21:31:20<8:43:34,  1.88s/it]
Outer training loop:  67%|######6   | 33295/50000 [21:31:22<8:56:55,  1.93s/it]
Outer training loop:  67%|######6   | 33296/50000 [21:31:24<9:15:45,  2.00s/it]
Outer training loop:  67%|######6   | 33297/50000 [21:31:26<9:23:09,  2.02s/it]
Outer training loop:  67%|######6   | 33298/50000 [21:31:28<9:04:53,  1.96s/it]
Outer training loop:  67%|######6   | 33299/50000 [21:31:30<9:09:26,  1.97s/it]
Outer training loop:  67%|######6   | 33300/50000 [21:31:32<8:47:45,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.98s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]

Outer training loop:  67%|######6   | 33301/50000 [21:31:39<16:36:43,  3.58s/it]
Outer training loop:  67%|######6   | 33302/50000 [21:31:41<14:21:42,  3.10s/it]
Outer training loop:  67%|######6   | 33303/50000 [21:31:43<13:01:53,  2.81s/it]
Outer training loop:  67%|######6   | 33304/50000 [21:31:45<11:38:09,  2.51s/it]
Outer training loop:  67%|######6   | 33305/50000 [21:31:47<10:23:59,  2.24s/it]
Outer training loop:  67%|######6   | 33306/50000 [21:31:49<10:11:43,  2.20s/it]
Outer training loop:  67%|######6   | 33307/50000 [21:31:51<9:50:52,  2.12s/it] 
Outer training loop:  67%|######6   | 33308/50000 [21:31:52<9:19:14,  2.01s/it]
Outer training loop:  67%|######6   | 33309/50000 [21:31:54<9:23:41,  2.03s/it]
Outer training loop:  67%|######6   | 33310/50000 [21:31:56<9:03:27,  1.95s/it]
Outer training loop:  67%|######6   | 33311/50000 [21:31:58<8:56:25,  1.93s/it]
Outer training loop:  67%|######6   | 33312/50000 [21:32:00<9:03:18,  1.95s/it]
Outer training loop:  67%|######6   | 33313/50000 [21:32:02<8:53:45,  1.92s/it]
Outer training loop:  67%|######6   | 33314/50000 [21:32:04<8:48:48,  1.90s/it]
Outer training loop:  67%|######6   | 33315/50000 [21:32:06<8:41:20,  1.87s/it]
Outer training loop:  67%|######6   | 33316/50000 [21:32:08<8:45:43,  1.89s/it]
Outer training loop:  67%|######6   | 33317/50000 [21:32:09<8:33:46,  1.85s/it]
Outer training loop:  67%|######6   | 33318/50000 [21:32:11<8:20:32,  1.80s/it]
Outer training loop:  67%|######6   | 33319/50000 [21:32:13<8:31:14,  1.84s/it]
Outer training loop:  67%|######6   | 33320/50000 [21:32:15<8:32:37,  1.84s/it]
Outer training loop:  67%|######6   | 33321/50000 [21:32:17<8:35:33,  1.85s/it]
Outer training loop:  67%|######6   | 33322/50000 [21:32:19<8:51:40,  1.91s/it]
Outer training loop:  67%|######6   | 33323/50000 [21:32:21<8:46:48,  1.90s/it]
Outer training loop:  67%|######6   | 33324/50000 [21:32:22<8:39:47,  1.87s/it]
Outer training loop:  67%|######6   | 33325/50000 [21:32:24<8:57:21,  1.93s/it]
Outer training loop:  67%|######6   | 33326/50000 [21:32:26<8:58:19,  1.94s/it]
Outer training loop:  67%|######6   | 33327/50000 [21:32:28<8:53:17,  1.92s/it]
Outer training loop:  67%|######6   | 33328/50000 [21:32:30<8:57:35,  1.93s/it]
Outer training loop:  67%|######6   | 33329/50000 [21:32:32<8:43:09,  1.88s/it]
Outer training loop:  67%|######6   | 33330/50000 [21:32:34<8:37:26,  1.86s/it]
Outer training loop:  67%|######6   | 33331/50000 [21:32:36<8:41:19,  1.88s/it]
Outer training loop:  67%|######6   | 33332/50000 [21:32:38<9:00:32,  1.95s/it]
Outer training loop:  67%|######6   | 33333/50000 [21:32:40<8:56:29,  1.93s/it]
Outer training loop:  67%|######6   | 33334/50000 [21:32:42<8:50:31,  1.91s/it]
Outer training loop:  67%|######6   | 33335/50000 [21:32:44<9:07:43,  1.97s/it]
Outer training loop:  67%|######6   | 33336/50000 [21:32:46<9:01:51,  1.95s/it]
Outer training loop:  67%|######6   | 33337/50000 [21:32:47<8:55:08,  1.93s/it]
Outer training loop:  67%|######6   | 33338/50000 [21:32:49<8:50:55,  1.91s/it]
Outer training loop:  67%|######6   | 33339/50000 [21:32:51<8:59:34,  1.94s/it]
Outer training loop:  67%|######6   | 33340/50000 [21:32:53<8:46:15,  1.90s/it]
Outer training loop:  67%|######6   | 33341/50000 [21:32:55<8:41:13,  1.88s/it]
Outer training loop:  67%|######6   | 33342/50000 [21:32:57<8:52:25,  1.92s/it]
Outer training loop:  67%|######6   | 33343/50000 [21:32:59<8:58:04,  1.94s/it]
Outer training loop:  67%|######6   | 33344/50000 [21:33:01<8:46:38,  1.90s/it]
Outer training loop:  67%|######6   | 33345/50000 [21:33:03<8:48:43,  1.90s/it]
Outer training loop:  67%|######6   | 33346/50000 [21:33:05<8:44:56,  1.89s/it]
Outer training loop:  67%|######6   | 33347/50000 [21:33:07<8:50:16,  1.91s/it]
Outer training loop:  67%|######6   | 33348/50000 [21:33:09<9:09:46,  1.98s/it]
Outer training loop:  67%|######6   | 33349/50000 [21:33:11<9:09:24,  1.98s/it]
Outer training loop:  67%|######6   | 33350/50000 [21:33:13<9:00:20,  1.95s/it]
Outer training loop:  67%|######6   | 33351/50000 [21:33:14<8:52:36,  1.92s/it]
Outer training loop:  67%|######6   | 33352/50000 [21:33:16<9:01:45,  1.95s/it]
Outer training loop:  67%|######6   | 33353/50000 [21:33:18<8:44:47,  1.89s/it]
Outer training loop:  67%|######6   | 33354/50000 [21:33:20<8:55:48,  1.93s/it]
Outer training loop:  67%|######6   | 33355/50000 [21:33:22<9:14:02,  2.00s/it]
Outer training loop:  67%|######6   | 33356/50000 [21:33:24<9:09:07,  1.98s/it]
Outer training loop:  67%|######6   | 33357/50000 [21:33:26<9:03:22,  1.96s/it]
Outer training loop:  67%|######6   | 33358/50000 [21:33:28<8:51:03,  1.91s/it]
Outer training loop:  67%|######6   | 33359/50000 [21:33:30<8:58:44,  1.94s/it]
Outer training loop:  67%|######6   | 33360/50000 [21:33:32<8:43:57,  1.89s/it]
Outer training loop:  67%|######6   | 33361/50000 [21:33:34<8:40:02,  1.88s/it]
Outer training loop:  67%|######6   | 33362/50000 [21:33:36<8:49:27,  1.91s/it]
Outer training loop:  67%|######6   | 33363/50000 [21:33:37<8:37:43,  1.87s/it]
Outer training loop:  67%|######6   | 33364/50000 [21:33:39<8:32:26,  1.85s/it]
Outer training loop:  67%|######6   | 33365/50000 [21:33:41<8:33:52,  1.85s/it]
Outer training loop:  67%|######6   | 33366/50000 [21:33:43<8:55:13,  1.93s/it]
Outer training loop:  67%|######6   | 33367/50000 [21:33:45<8:53:28,  1.92s/it]
Outer training loop:  67%|######6   | 33368/50000 [21:33:47<8:45:09,  1.89s/it]
Outer training loop:  67%|######6   | 33369/50000 [21:33:49<8:50:01,  1.91s/it]
Outer training loop:  67%|######6   | 33370/50000 [21:33:51<8:44:10,  1.89s/it]
Outer training loop:  67%|######6   | 33371/50000 [21:33:53<8:43:42,  1.89s/it]
Outer training loop:  67%|######6   | 33372/50000 [21:33:55<9:02:12,  1.96s/it]
Outer training loop:  67%|######6   | 33373/50000 [21:33:57<8:58:44,  1.94s/it]
Outer training loop:  67%|######6   | 33374/50000 [21:33:59<8:56:28,  1.94s/it]
Outer training loop:  67%|######6   | 33375/50000 [21:34:01<9:06:43,  1.97s/it]
Outer training loop:  67%|######6   | 33376/50000 [21:34:02<8:52:52,  1.92s/it]
Outer training loop:  67%|######6   | 33377/50000 [21:34:04<8:44:13,  1.89s/it]
Outer training loop:  67%|######6   | 33378/50000 [21:34:06<8:51:10,  1.92s/it]
Outer training loop:  67%|######6   | 33379/50000 [21:34:08<8:56:19,  1.94s/it]
Outer training loop:  67%|######6   | 33380/50000 [21:34:10<8:51:44,  1.92s/it]
Outer training loop:  67%|######6   | 33381/50000 [21:34:12<8:41:46,  1.88s/it]
Outer training loop:  67%|######6   | 33382/50000 [21:34:14<8:50:40,  1.92s/it]
Outer training loop:  67%|######6   | 33383/50000 [21:34:16<8:38:09,  1.87s/it]
Outer training loop:  67%|######6   | 33384/50000 [21:34:17<8:33:49,  1.86s/it]
Outer training loop:  67%|######6   | 33385/50000 [21:34:19<8:24:41,  1.82s/it]
Outer training loop:  67%|######6   | 33386/50000 [21:34:21<8:44:07,  1.89s/it]
Outer training loop:  67%|######6   | 33387/50000 [21:34:23<8:46:49,  1.90s/it]
Outer training loop:  67%|######6   | 33388/50000 [21:34:25<8:40:19,  1.88s/it]
Outer training loop:  67%|######6   | 33389/50000 [21:34:27<9:17:40,  2.01s/it]
Outer training loop:  67%|######6   | 33390/50000 [21:34:29<8:57:15,  1.94s/it]
Outer training loop:  67%|######6   | 33391/50000 [21:34:31<8:55:12,  1.93s/it]
Outer training loop:  67%|######6   | 33392/50000 [21:34:33<8:48:58,  1.91s/it]
Outer training loop:  67%|######6   | 33393/50000 [21:34:35<9:17:32,  2.01s/it]
Outer training loop:  67%|######6   | 33394/50000 [21:34:37<8:58:02,  1.94s/it]
Outer training loop:  67%|######6   | 33395/50000 [21:34:39<8:47:07,  1.90s/it]
Outer training loop:  67%|######6   | 33396/50000 [21:34:41<8:58:43,  1.95s/it]
Outer training loop:  67%|######6   | 33397/50000 [21:34:43<8:56:02,  1.94s/it]
Outer training loop:  67%|######6   | 33398/50000 [21:34:45<8:49:17,  1.91s/it]
Outer training loop:  67%|######6   | 33399/50000 [21:34:47<9:01:23,  1.96s/it]
Outer training loop:  67%|######6   | 33400/50000 [21:34:48<8:55:23,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  67%|######6   | 33401/50000 [21:34:56<16:32:20,  3.59s/it]
Outer training loop:  67%|######6   | 33402/50000 [21:34:58<14:12:35,  3.08s/it]
Outer training loop:  67%|######6   | 33403/50000 [21:35:00<12:52:23,  2.79s/it]
Outer training loop:  67%|######6   | 33404/50000 [21:35:02<11:35:31,  2.51s/it]
Outer training loop:  67%|######6   | 33405/50000 [21:35:04<10:36:14,  2.30s/it]
Outer training loop:  67%|######6   | 33406/50000 [21:35:06<10:18:31,  2.24s/it]
Outer training loop:  67%|######6   | 33407/50000 [21:35:08<9:47:58,  2.13s/it] 
Outer training loop:  67%|######6   | 33408/50000 [21:35:09<9:17:02,  2.01s/it]
Outer training loop:  67%|######6   | 33409/50000 [21:35:11<8:56:32,  1.94s/it]
Outer training loop:  67%|######6   | 33410/50000 [21:35:13<9:01:41,  1.96s/it]
Outer training loop:  67%|######6   | 33411/50000 [21:35:15<8:52:30,  1.93s/it]
Outer training loop:  67%|######6   | 33412/50000 [21:35:17<8:30:33,  1.85s/it]
Outer training loop:  67%|######6   | 33413/50000 [21:35:19<8:39:35,  1.88s/it]
Outer training loop:  67%|######6   | 33414/50000 [21:35:20<8:46:25,  1.90s/it]
Outer training loop:  67%|######6   | 33415/50000 [21:35:22<8:39:20,  1.88s/it]
Outer training loop:  67%|######6   | 33416/50000 [21:35:24<8:38:12,  1.87s/it]
Outer training loop:  67%|######6   | 33417/50000 [21:35:26<8:48:10,  1.91s/it]
Outer training loop:  67%|######6   | 33418/50000 [21:35:28<8:37:47,  1.87s/it]
Outer training loop:  67%|######6   | 33419/50000 [21:35:30<8:44:40,  1.90s/it]
Outer training loop:  67%|######6   | 33420/50000 [21:35:32<8:53:53,  1.93s/it]
Outer training loop:  67%|######6   | 33421/50000 [21:35:34<8:50:27,  1.92s/it]
Outer training loop:  67%|######6   | 33422/50000 [21:35:36<8:47:31,  1.91s/it]
Outer training loop:  67%|######6   | 33423/50000 [21:35:37<8:32:22,  1.85s/it]
Outer training loop:  67%|######6   | 33424/50000 [21:35:39<8:46:43,  1.91s/it]
Outer training loop:  67%|######6   | 33425/50000 [21:35:41<8:43:31,  1.90s/it]
Outer training loop:  67%|######6   | 33426/50000 [21:35:43<8:40:17,  1.88s/it]
Outer training loop:  67%|######6   | 33427/50000 [21:35:45<8:53:18,  1.93s/it]
Outer training loop:  67%|######6   | 33428/50000 [21:35:47<8:44:18,  1.90s/it]
Outer training loop:  67%|######6   | 33429/50000 [21:35:49<8:57:31,  1.95s/it]
Outer training loop:  67%|######6   | 33430/50000 [21:35:51<8:49:05,  1.92s/it]
Outer training loop:  67%|######6   | 33431/50000 [21:35:53<8:55:45,  1.94s/it]
Outer training loop:  67%|######6   | 33432/50000 [21:35:55<8:55:55,  1.94s/it]
Outer training loop:  67%|######6   | 33433/50000 [21:35:57<8:38:07,  1.88s/it]
Outer training loop:  67%|######6   | 33434/50000 [21:35:59<8:41:10,  1.89s/it]
Outer training loop:  67%|######6   | 33435/50000 [21:36:00<8:29:48,  1.85s/it]
Outer training loop:  67%|######6   | 33436/50000 [21:36:02<8:26:14,  1.83s/it]
Outer training loop:  67%|######6   | 33437/50000 [21:36:04<8:39:52,  1.88s/it]
Outer training loop:  67%|######6   | 33438/50000 [21:36:06<8:25:27,  1.83s/it]
Outer training loop:  67%|######6   | 33439/50000 [21:36:08<8:26:02,  1.83s/it]
Outer training loop:  67%|######6   | 33440/50000 [21:36:10<8:46:49,  1.91s/it]
Outer training loop:  67%|######6   | 33441/50000 [21:36:12<8:48:19,  1.91s/it]
Outer training loop:  67%|######6   | 33442/50000 [21:36:13<8:41:31,  1.89s/it]
Outer training loop:  67%|######6   | 33443/50000 [21:36:15<8:29:33,  1.85s/it]
Outer training loop:  67%|######6   | 33444/50000 [21:36:17<8:34:22,  1.86s/it]
Outer training loop:  67%|######6   | 33445/50000 [21:36:19<8:45:13,  1.90s/it]
Outer training loop:  67%|######6   | 33446/50000 [21:36:21<8:46:31,  1.91s/it]
Outer training loop:  67%|######6   | 33447/50000 [21:36:23<8:34:28,  1.86s/it]
Outer training loop:  67%|######6   | 33448/50000 [21:36:25<8:45:45,  1.91s/it]
Outer training loop:  67%|######6   | 33449/50000 [21:36:27<8:39:24,  1.88s/it]
Outer training loop:  67%|######6   | 33450/50000 [21:36:29<8:38:58,  1.88s/it]
Outer training loop:  67%|######6   | 33451/50000 [21:36:30<8:44:53,  1.90s/it]
Outer training loop:  67%|######6   | 33452/50000 [21:36:32<8:48:36,  1.92s/it]
Outer training loop:  67%|######6   | 33453/50000 [21:36:34<8:40:25,  1.89s/it]
Outer training loop:  67%|######6   | 33454/50000 [21:36:37<9:17:17,  2.02s/it]
Outer training loop:  67%|######6   | 33455/50000 [21:36:38<9:09:30,  1.99s/it]
Outer training loop:  67%|######6   | 33456/50000 [21:36:40<8:58:25,  1.95s/it]
Outer training loop:  67%|######6   | 33457/50000 [21:36:42<8:56:33,  1.95s/it]
Outer training loop:  67%|######6   | 33458/50000 [21:36:44<8:51:22,  1.93s/it]
Outer training loop:  67%|######6   | 33459/50000 [21:36:46<8:42:42,  1.90s/it]
Outer training loop:  67%|######6   | 33460/50000 [21:36:48<8:38:59,  1.88s/it]
Outer training loop:  67%|######6   | 33461/50000 [21:36:50<8:59:21,  1.96s/it]
Outer training loop:  67%|######6   | 33462/50000 [21:36:52<8:49:34,  1.92s/it]
Outer training loop:  67%|######6   | 33463/50000 [21:36:54<8:45:30,  1.91s/it]
Outer training loop:  67%|######6   | 33464/50000 [21:36:56<8:52:57,  1.93s/it]
Outer training loop:  67%|######6   | 33465/50000 [21:36:58<8:48:30,  1.92s/it]
Outer training loop:  67%|######6   | 33466/50000 [21:36:59<8:44:27,  1.90s/it]
Outer training loop:  67%|######6   | 33467/50000 [21:37:01<8:42:59,  1.90s/it]
Outer training loop:  67%|######6   | 33468/50000 [21:37:03<8:53:31,  1.94s/it]
Outer training loop:  67%|######6   | 33469/50000 [21:37:05<8:53:08,  1.94s/it]
Outer training loop:  67%|######6   | 33470/50000 [21:37:07<8:41:12,  1.89s/it]
Outer training loop:  67%|######6   | 33471/50000 [21:37:09<8:50:23,  1.93s/it]
Outer training loop:  67%|######6   | 33472/50000 [21:37:11<8:50:12,  1.92s/it]
Outer training loop:  67%|######6   | 33473/50000 [21:37:13<8:41:36,  1.89s/it]
Outer training loop:  67%|######6   | 33474/50000 [21:37:15<8:54:13,  1.94s/it]
Outer training loop:  67%|######6   | 33475/50000 [21:37:17<8:37:20,  1.88s/it]
Outer training loop:  67%|######6   | 33476/50000 [21:37:18<8:30:51,  1.85s/it]
Outer training loop:  67%|######6   | 33477/50000 [21:37:21<8:55:29,  1.94s/it]
Outer training loop:  67%|######6   | 33478/50000 [21:37:22<8:40:26,  1.89s/it]
Outer training loop:  67%|######6   | 33479/50000 [21:37:24<8:43:39,  1.90s/it]
Outer training loop:  67%|######6   | 33480/50000 [21:37:26<8:37:00,  1.88s/it]
Outer training loop:  67%|######6   | 33481/50000 [21:37:28<8:58:45,  1.96s/it]
Outer training loop:  67%|######6   | 33482/50000 [21:37:30<8:46:27,  1.91s/it]
Outer training loop:  67%|######6   | 33483/50000 [21:37:32<8:34:13,  1.87s/it]
Outer training loop:  67%|######6   | 33484/50000 [21:37:34<8:22:25,  1.83s/it]
Outer training loop:  67%|######6   | 33485/50000 [21:37:36<8:45:36,  1.91s/it]
Outer training loop:  67%|######6   | 33486/50000 [21:37:37<8:29:58,  1.85s/it]
Outer training loop:  67%|######6   | 33487/50000 [21:37:39<8:26:10,  1.84s/it]
Outer training loop:  67%|######6   | 33488/50000 [21:37:41<8:39:59,  1.89s/it]
Outer training loop:  67%|######6   | 33489/50000 [21:37:43<8:41:04,  1.89s/it]
Outer training loop:  67%|######6   | 33490/50000 [21:37:45<8:43:22,  1.90s/it]
Outer training loop:  67%|######6   | 33491/50000 [21:37:47<9:01:17,  1.97s/it]
Outer training loop:  67%|######6   | 33492/50000 [21:37:49<8:53:01,  1.94s/it]
Outer training loop:  67%|######6   | 33493/50000 [21:37:51<8:41:03,  1.89s/it]
Outer training loop:  67%|######6   | 33494/50000 [21:37:53<8:33:48,  1.87s/it]
Outer training loop:  67%|######6   | 33495/50000 [21:37:55<8:50:18,  1.93s/it]
Outer training loop:  67%|######6   | 33496/50000 [21:37:56<8:37:16,  1.88s/it]
Outer training loop:  67%|######6   | 33497/50000 [21:37:58<8:33:56,  1.87s/it]
Outer training loop:  67%|######6   | 33498/50000 [21:38:00<8:41:47,  1.90s/it]
Outer training loop:  67%|######6   | 33499/50000 [21:38:02<8:33:00,  1.87s/it]
Outer training loop:  67%|######7   | 33500/50000 [21:38:04<8:31:03,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.71s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:41,  1.97s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:36,  1.93s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.90s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.86s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:20,  1.85s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.94s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:17,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.89s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:08,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:05,  1.85s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:00,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.86s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<02:00,  1.95s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:47,  1.82s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:45,  1.82s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:44,  1.86s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:42,  1.86s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.85s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:41,  1.96s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:26,  1.81s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:28,  1.89s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:30,  1.97s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:25,  1.90s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.85s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.87s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:16,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:16,  1.96s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:08,  1.86s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:01,  1.82s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.81s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:59,  1.91s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.85s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:49,  1.99s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:44,  1.92s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:42,  1.92s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.90s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:30,  1.80s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:30,  1.93s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:28,  1.92s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:21,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.87s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  2.00s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.93s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.90s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.92s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.89s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.84s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:36,  1.93s/it]

Evaluating policy:   4%|3         | 3/83 [00:06<02:35,  1.94s/it]

Evaluating policy:   5%|4         | 4/83 [00:08<02:33,  1.94s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.94s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.92s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.94s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:16,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:13,  1.88s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:08,  1.84s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:06,  1.95s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.90s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:57,  1.92s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:52,  1.88s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.85s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:46,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:38,  1.86s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:35,  1.83s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:32,  1.82s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:37,  1.95s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:33,  1.90s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:25,  1.93s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:21,  1.89s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.89s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.90s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.88s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:16,  2.01s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:11,  1.93s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:08,  1.90s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<01:01,  1.97s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:58,  1.96s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.90s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.88s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:45,  1.98s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:42,  1.94s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.92s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.88s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.88s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:29,  2.00s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:27,  1.97s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.91s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.87s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.79s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.81s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.84s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.84s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.96s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.93s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.86s/it]

Outer training loop:  67%|######7   | 33501/50000 [21:43:24<446:48:38, 97.49s/it]
Outer training loop:  67%|######7   | 33502/50000 [21:43:26<315:18:42, 68.80s/it]
Outer training loop:  67%|######7   | 33503/50000 [21:43:28<223:11:26, 48.70s/it]
Outer training loop:  67%|######7   | 33504/50000 [21:43:30<158:34:14, 34.61s/it]
Outer training loop:  67%|######7   | 33505/50000 [21:43:32<113:34:48, 24.79s/it]
Outer training loop:  67%|######7   | 33506/50000 [21:43:34<82:45:14, 18.06s/it] 
Outer training loop:  67%|######7   | 33507/50000 [21:43:36<60:30:26, 13.21s/it]
Outer training loop:  67%|######7   | 33508/50000 [21:43:38<44:56:01,  9.81s/it]
Outer training loop:  67%|######7   | 33509/50000 [21:43:40<34:13:28,  7.47s/it]
Outer training loop:  67%|######7   | 33510/50000 [21:43:42<26:23:23,  5.76s/it]
Outer training loop:  67%|######7   | 33511/50000 [21:43:44<21:07:41,  4.61s/it]
Outer training loop:  67%|######7   | 33512/50000 [21:43:45<17:16:59,  3.77s/it]
Outer training loop:  67%|######7   | 33513/50000 [21:43:47<14:44:19,  3.22s/it]
Outer training loop:  67%|######7   | 33514/50000 [21:43:49<12:48:52,  2.80s/it]
Outer training loop:  67%|######7   | 33515/50000 [21:43:51<11:35:20,  2.53s/it]
Outer training loop:  67%|######7   | 33516/50000 [21:43:53<10:54:02,  2.38s/it]
Outer training loop:  67%|######7   | 33517/50000 [21:43:55<10:18:14,  2.25s/it]
Outer training loop:  67%|######7   | 33518/50000 [21:43:57<9:38:28,  2.11s/it] 
Outer training loop:  67%|######7   | 33519/50000 [21:43:59<9:27:18,  2.07s/it]
Outer training loop:  67%|######7   | 33520/50000 [21:44:01<9:17:22,  2.03s/it]
Outer training loop:  67%|######7   | 33521/50000 [21:44:03<9:07:50,  1.99s/it]
Outer training loop:  67%|######7   | 33522/50000 [21:44:04<8:55:39,  1.95s/it]
Outer training loop:  67%|######7   | 33523/50000 [21:44:07<9:05:33,  1.99s/it]
Outer training loop:  67%|######7   | 33524/50000 [21:44:08<8:47:04,  1.92s/it]
Outer training loop:  67%|######7   | 33525/50000 [21:44:10<8:43:35,  1.91s/it]
Outer training loop:  67%|######7   | 33526/50000 [21:44:12<8:58:21,  1.96s/it]
Outer training loop:  67%|######7   | 33527/50000 [21:44:14<8:50:16,  1.93s/it]
Outer training loop:  67%|######7   | 33528/50000 [21:44:16<8:33:29,  1.87s/it]
Outer training loop:  67%|######7   | 33529/50000 [21:44:18<8:56:02,  1.95s/it]
Outer training loop:  67%|######7   | 33530/50000 [21:44:20<8:47:15,  1.92s/it]
Outer training loop:  67%|######7   | 33531/50000 [21:44:22<8:35:32,  1.88s/it]
Outer training loop:  67%|######7   | 33532/50000 [21:44:23<8:29:28,  1.86s/it]
Outer training loop:  67%|######7   | 33533/50000 [21:44:26<8:48:28,  1.93s/it]
Outer training loop:  67%|######7   | 33534/50000 [21:44:27<8:48:25,  1.93s/it]
Outer training loop:  67%|######7   | 33535/50000 [21:44:29<8:43:14,  1.91s/it]
Outer training loop:  67%|######7   | 33536/50000 [21:44:31<8:43:06,  1.91s/it]
Outer training loop:  67%|######7   | 33537/50000 [21:44:33<8:36:14,  1.88s/it]
Outer training loop:  67%|######7   | 33538/50000 [21:44:35<8:32:11,  1.87s/it]
Outer training loop:  67%|######7   | 33539/50000 [21:44:37<8:40:25,  1.90s/it]
Outer training loop:  67%|######7   | 33540/50000 [21:44:39<8:35:55,  1.88s/it]
Outer training loop:  67%|######7   | 33541/50000 [21:44:41<8:31:55,  1.87s/it]
Outer training loop:  67%|######7   | 33542/50000 [21:44:42<8:32:39,  1.87s/it]
Outer training loop:  67%|######7   | 33543/50000 [21:44:44<8:31:08,  1.86s/it]
Outer training loop:  67%|######7   | 33544/50000 [21:44:46<8:26:55,  1.85s/it]
Outer training loop:  67%|######7   | 33545/50000 [21:44:48<8:19:40,  1.82s/it]
Outer training loop:  67%|######7   | 33546/50000 [21:44:50<8:31:03,  1.86s/it]
Outer training loop:  67%|######7   | 33547/50000 [21:44:52<8:24:59,  1.84s/it]
Outer training loop:  67%|######7   | 33548/50000 [21:44:53<8:22:14,  1.83s/it]
Outer training loop:  67%|######7   | 33549/50000 [21:44:55<8:34:21,  1.88s/it]
Outer training loop:  67%|######7   | 33550/50000 [21:44:57<8:24:15,  1.84s/it]
Outer training loop:  67%|######7   | 33551/50000 [21:44:59<8:19:52,  1.82s/it]
Outer training loop:  67%|######7   | 33552/50000 [21:45:01<8:22:56,  1.83s/it]
Outer training loop:  67%|######7   | 33553/50000 [21:45:03<8:38:24,  1.89s/it]
Outer training loop:  67%|######7   | 33554/50000 [21:45:05<8:31:13,  1.87s/it]
Outer training loop:  67%|######7   | 33555/50000 [21:45:06<8:33:33,  1.87s/it]
Outer training loop:  67%|######7   | 33556/50000 [21:45:08<8:21:09,  1.83s/it]
Outer training loop:  67%|######7   | 33557/50000 [21:45:10<8:45:19,  1.92s/it]
Outer training loop:  67%|######7   | 33558/50000 [21:45:12<8:33:15,  1.87s/it]
Outer training loop:  67%|######7   | 33559/50000 [21:45:14<8:36:06,  1.88s/it]
Outer training loop:  67%|######7   | 33560/50000 [21:45:16<8:52:41,  1.94s/it]
Outer training loop:  67%|######7   | 33561/50000 [21:45:18<8:34:03,  1.88s/it]
Outer training loop:  67%|######7   | 33562/50000 [21:45:20<8:32:32,  1.87s/it]
Outer training loop:  67%|######7   | 33563/50000 [21:45:22<8:49:26,  1.93s/it]
Outer training loop:  67%|######7   | 33564/50000 [21:45:23<8:24:10,  1.84s/it]
Outer training loop:  67%|######7   | 33565/50000 [21:45:25<8:34:06,  1.88s/it]
Outer training loop:  67%|######7   | 33566/50000 [21:45:27<8:31:02,  1.87s/it]
Outer training loop:  67%|######7   | 33567/50000 [21:45:29<8:51:50,  1.94s/it]
Outer training loop:  67%|######7   | 33568/50000 [21:45:31<8:34:53,  1.88s/it]
Outer training loop:  67%|######7   | 33569/50000 [21:45:33<8:22:44,  1.84s/it]
Outer training loop:  67%|######7   | 33570/50000 [21:45:35<8:42:33,  1.91s/it]
Outer training loop:  67%|######7   | 33571/50000 [21:45:37<8:50:25,  1.94s/it]
Outer training loop:  67%|######7   | 33572/50000 [21:45:39<8:41:58,  1.91s/it]
Outer training loop:  67%|######7   | 33573/50000 [21:45:41<8:58:11,  1.97s/it]
Outer training loop:  67%|######7   | 33574/50000 [21:45:43<8:40:50,  1.90s/it]
Outer training loop:  67%|######7   | 33575/50000 [21:45:44<8:24:27,  1.84s/it]
Outer training loop:  67%|######7   | 33576/50000 [21:45:46<8:38:12,  1.89s/it]
Outer training loop:  67%|######7   | 33577/50000 [21:45:48<8:35:24,  1.88s/it]
Outer training loop:  67%|######7   | 33578/50000 [21:45:50<8:35:59,  1.89s/it]
Outer training loop:  67%|######7   | 33579/50000 [21:45:52<8:39:13,  1.90s/it]
Outer training loop:  67%|######7   | 33580/50000 [21:45:54<8:44:25,  1.92s/it]
Outer training loop:  67%|######7   | 33581/50000 [21:45:56<8:44:28,  1.92s/it]
Outer training loop:  67%|######7   | 33582/50000 [21:45:58<8:37:39,  1.89s/it]
Outer training loop:  67%|######7   | 33583/50000 [21:46:00<8:36:54,  1.89s/it]
Outer training loop:  67%|######7   | 33584/50000 [21:46:01<8:31:35,  1.87s/it]
Outer training loop:  67%|######7   | 33585/50000 [21:46:03<8:17:29,  1.82s/it]
Outer training loop:  67%|######7   | 33586/50000 [21:46:05<8:38:15,  1.89s/it]
Outer training loop:  67%|######7   | 33587/50000 [21:46:07<8:34:52,  1.88s/it]
Outer training loop:  67%|######7   | 33588/50000 [21:46:09<8:31:58,  1.87s/it]
Outer training loop:  67%|######7   | 33589/50000 [21:46:11<8:46:55,  1.93s/it]
Outer training loop:  67%|######7   | 33590/50000 [21:46:13<8:42:14,  1.91s/it]
Outer training loop:  67%|######7   | 33591/50000 [21:46:15<8:44:17,  1.92s/it]
Outer training loop:  67%|######7   | 33592/50000 [21:46:16<8:33:45,  1.88s/it]
Outer training loop:  67%|######7   | 33593/50000 [21:46:18<8:43:58,  1.92s/it]
Outer training loop:  67%|######7   | 33594/50000 [21:46:21<8:58:25,  1.97s/it]
Outer training loop:  67%|######7   | 33595/50000 [21:46:22<8:45:02,  1.92s/it]
Outer training loop:  67%|######7   | 33596/50000 [21:46:25<9:08:05,  2.00s/it]
Outer training loop:  67%|######7   | 33597/50000 [21:46:26<8:58:25,  1.97s/it]
Outer training loop:  67%|######7   | 33598/50000 [21:46:28<8:55:28,  1.96s/it]
Outer training loop:  67%|######7   | 33599/50000 [21:46:30<8:47:50,  1.93s/it]
Outer training loop:  67%|######7   | 33600/50000 [21:46:32<8:53:51,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.87s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  67%|######7   | 33601/50000 [21:46:40<16:32:59,  3.63s/it]
Outer training loop:  67%|######7   | 33602/50000 [21:46:42<14:05:14,  3.09s/it]
Outer training loop:  67%|######7   | 33603/50000 [21:46:43<12:23:38,  2.72s/it]
Outer training loop:  67%|######7   | 33604/50000 [21:46:46<11:33:05,  2.54s/it]
Outer training loop:  67%|######7   | 33605/50000 [21:46:48<10:43:30,  2.36s/it]
Outer training loop:  67%|######7   | 33606/50000 [21:46:49<10:09:54,  2.23s/it]
Outer training loop:  67%|######7   | 33607/50000 [21:46:52<10:02:21,  2.20s/it]
Outer training loop:  67%|######7   | 33608/50000 [21:46:53<9:33:12,  2.10s/it] 
Outer training loop:  67%|######7   | 33609/50000 [21:46:55<9:10:02,  2.01s/it]
Outer training loop:  67%|######7   | 33610/50000 [21:46:57<9:07:18,  2.00s/it]
Outer training loop:  67%|######7   | 33611/50000 [21:46:59<8:57:32,  1.97s/it]
Outer training loop:  67%|######7   | 33612/50000 [21:47:01<8:48:39,  1.94s/it]
Outer training loop:  67%|######7   | 33613/50000 [21:47:03<8:41:20,  1.91s/it]
Outer training loop:  67%|######7   | 33614/50000 [21:47:05<9:07:18,  2.00s/it]
Outer training loop:  67%|######7   | 33615/50000 [21:47:07<8:53:48,  1.95s/it]
Outer training loop:  67%|######7   | 33616/50000 [21:47:09<8:42:53,  1.91s/it]
Outer training loop:  67%|######7   | 33617/50000 [21:47:11<8:46:05,  1.93s/it]
Outer training loop:  67%|######7   | 33618/50000 [21:47:13<8:57:19,  1.97s/it]
Outer training loop:  67%|######7   | 33619/50000 [21:47:15<8:42:24,  1.91s/it]
Outer training loop:  67%|######7   | 33620/50000 [21:47:17<8:49:17,  1.94s/it]
Outer training loop:  67%|######7   | 33621/50000 [21:47:18<8:33:09,  1.88s/it]
Outer training loop:  67%|######7   | 33622/50000 [21:47:20<8:39:53,  1.90s/it]
Outer training loop:  67%|######7   | 33623/50000 [21:47:22<8:51:16,  1.95s/it]
Outer training loop:  67%|######7   | 33624/50000 [21:47:24<8:44:20,  1.92s/it]
Outer training loop:  67%|######7   | 33625/50000 [21:47:26<8:41:24,  1.91s/it]
Outer training loop:  67%|######7   | 33626/50000 [21:47:28<8:27:32,  1.86s/it]
Outer training loop:  67%|######7   | 33627/50000 [21:47:30<8:44:47,  1.92s/it]
Outer training loop:  67%|######7   | 33628/50000 [21:47:32<8:34:38,  1.89s/it]
Outer training loop:  67%|######7   | 33629/50000 [21:47:33<8:31:02,  1.87s/it]
Outer training loop:  67%|######7   | 33630/50000 [21:47:36<8:43:39,  1.92s/it]
Outer training loop:  67%|######7   | 33631/50000 [21:47:37<8:37:21,  1.90s/it]
Outer training loop:  67%|######7   | 33632/50000 [21:47:39<8:30:07,  1.87s/it]
Outer training loop:  67%|######7   | 33633/50000 [21:47:41<8:42:16,  1.91s/it]
Outer training loop:  67%|######7   | 33634/50000 [21:47:43<8:39:15,  1.90s/it]
Outer training loop:  67%|######7   | 33635/50000 [21:47:45<8:27:01,  1.86s/it]
Outer training loop:  67%|######7   | 33636/50000 [21:47:47<8:38:19,  1.90s/it]
Outer training loop:  67%|######7   | 33637/50000 [21:47:49<8:40:54,  1.91s/it]
Outer training loop:  67%|######7   | 33638/50000 [21:47:51<8:37:42,  1.90s/it]
Outer training loop:  67%|######7   | 33639/50000 [21:47:52<8:18:56,  1.83s/it]
Outer training loop:  67%|######7   | 33640/50000 [21:47:54<8:37:32,  1.90s/it]
Outer training loop:  67%|######7   | 33641/50000 [21:47:56<8:28:37,  1.87s/it]
Outer training loop:  67%|######7   | 33642/50000 [21:47:58<8:20:14,  1.83s/it]
Outer training loop:  67%|######7   | 33643/50000 [21:48:00<8:21:47,  1.84s/it]
Outer training loop:  67%|######7   | 33644/50000 [21:48:02<8:17:44,  1.83s/it]
Outer training loop:  67%|######7   | 33645/50000 [21:48:03<8:13:13,  1.81s/it]
Outer training loop:  67%|######7   | 33646/50000 [21:48:05<8:14:22,  1.81s/it]
Outer training loop:  67%|######7   | 33647/50000 [21:48:07<8:29:11,  1.87s/it]
Outer training loop:  67%|######7   | 33648/50000 [21:48:09<8:33:14,  1.88s/it]
Outer training loop:  67%|######7   | 33649/50000 [21:48:11<8:21:46,  1.84s/it]
Outer training loop:  67%|######7   | 33650/50000 [21:48:13<8:26:19,  1.86s/it]
Outer training loop:  67%|######7   | 33651/50000 [21:48:14<8:14:45,  1.82s/it]
Outer training loop:  67%|######7   | 33652/50000 [21:48:16<8:08:24,  1.79s/it]
Outer training loop:  67%|######7   | 33653/50000 [21:48:18<8:23:50,  1.85s/it]
Outer training loop:  67%|######7   | 33654/50000 [21:48:20<8:21:36,  1.84s/it]
Outer training loop:  67%|######7   | 33655/50000 [21:48:22<8:28:40,  1.87s/it]
Outer training loop:  67%|######7   | 33656/50000 [21:48:24<8:23:58,  1.85s/it]
Outer training loop:  67%|######7   | 33657/50000 [21:48:26<8:41:33,  1.91s/it]
Outer training loop:  67%|######7   | 33658/50000 [21:48:28<8:34:37,  1.89s/it]
Outer training loop:  67%|######7   | 33659/50000 [21:48:29<8:21:20,  1.84s/it]
Outer training loop:  67%|######7   | 33660/50000 [21:48:31<8:14:27,  1.82s/it]
Outer training loop:  67%|######7   | 33661/50000 [21:48:33<8:31:57,  1.88s/it]
Outer training loop:  67%|######7   | 33662/50000 [21:48:35<8:30:12,  1.87s/it]
Outer training loop:  67%|######7   | 33663/50000 [21:48:37<8:27:52,  1.87s/it]
Outer training loop:  67%|######7   | 33664/50000 [21:48:39<8:37:53,  1.90s/it]
Outer training loop:  67%|######7   | 33665/50000 [21:48:41<8:39:11,  1.91s/it]
Outer training loop:  67%|######7   | 33666/50000 [21:48:43<8:31:07,  1.88s/it]
Outer training loop:  67%|######7   | 33667/50000 [21:48:45<8:45:38,  1.93s/it]
Outer training loop:  67%|######7   | 33668/50000 [21:48:46<8:43:20,  1.92s/it]
Outer training loop:  67%|######7   | 33669/50000 [21:48:48<8:31:26,  1.88s/it]
Outer training loop:  67%|######7   | 33670/50000 [21:48:50<8:31:55,  1.88s/it]
Outer training loop:  67%|######7   | 33671/50000 [21:48:52<8:41:27,  1.92s/it]
Outer training loop:  67%|######7   | 33672/50000 [21:48:54<8:33:54,  1.89s/it]
Outer training loop:  67%|######7   | 33673/50000 [21:48:56<8:25:43,  1.86s/it]
Outer training loop:  67%|######7   | 33674/50000 [21:48:58<8:40:31,  1.91s/it]
Outer training loop:  67%|######7   | 33675/50000 [21:49:00<8:26:12,  1.86s/it]
Outer training loop:  67%|######7   | 33676/50000 [21:49:01<8:19:44,  1.84s/it]
Outer training loop:  67%|######7   | 33677/50000 [21:49:03<8:33:02,  1.89s/it]
Outer training loop:  67%|######7   | 33678/50000 [21:49:05<8:38:41,  1.91s/it]
Outer training loop:  67%|######7   | 33679/50000 [21:49:07<8:35:10,  1.89s/it]
Outer training loop:  67%|######7   | 33680/50000 [21:49:09<8:34:02,  1.89s/it]
Outer training loop:  67%|######7   | 33681/50000 [21:49:11<8:47:04,  1.94s/it]
Outer training loop:  67%|######7   | 33682/50000 [21:49:13<8:43:57,  1.93s/it]
Outer training loop:  67%|######7   | 33683/50000 [21:49:15<8:40:00,  1.91s/it]
Outer training loop:  67%|######7   | 33684/50000 [21:49:17<8:51:03,  1.95s/it]
Outer training loop:  67%|######7   | 33685/50000 [21:49:19<8:41:06,  1.92s/it]
Outer training loop:  67%|######7   | 33686/50000 [21:49:21<8:35:14,  1.89s/it]
Outer training loop:  67%|######7   | 33687/50000 [21:49:23<8:40:46,  1.92s/it]
Outer training loop:  67%|######7   | 33688/50000 [21:49:24<8:31:02,  1.88s/it]
Outer training loop:  67%|######7   | 33689/50000 [21:49:26<8:34:25,  1.89s/it]
Outer training loop:  67%|######7   | 33690/50000 [21:49:28<8:30:40,  1.88s/it]
Outer training loop:  67%|######7   | 33691/50000 [21:49:30<8:47:04,  1.94s/it]
Outer training loop:  67%|######7   | 33692/50000 [21:49:32<8:31:41,  1.88s/it]
Outer training loop:  67%|######7   | 33693/50000 [21:49:34<8:25:10,  1.86s/it]
Outer training loop:  67%|######7   | 33694/50000 [21:49:36<8:37:30,  1.90s/it]
Outer training loop:  67%|######7   | 33695/50000 [21:49:38<8:40:15,  1.91s/it]
Outer training loop:  67%|######7   | 33696/50000 [21:49:39<8:28:03,  1.87s/it]
Outer training loop:  67%|######7   | 33697/50000 [21:49:41<8:27:14,  1.87s/it]
Outer training loop:  67%|######7   | 33698/50000 [21:49:43<8:21:42,  1.85s/it]
Outer training loop:  67%|######7   | 33699/50000 [21:49:45<8:19:23,  1.84s/it]
Outer training loop:  67%|######7   | 33700/50000 [21:49:47<8:09:57,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  67%|######7   | 33701/50000 [21:49:54<15:45:25,  3.48s/it]
Outer training loop:  67%|######7   | 33702/50000 [21:49:56<13:39:43,  3.02s/it]
Outer training loop:  67%|######7   | 33703/50000 [21:49:58<12:10:06,  2.69s/it]
Outer training loop:  67%|######7   | 33704/50000 [21:50:00<11:01:05,  2.43s/it]
Outer training loop:  67%|######7   | 33705/50000 [21:50:02<10:26:18,  2.31s/it]
Outer training loop:  67%|######7   | 33706/50000 [21:50:03<9:37:58,  2.13s/it] 
Outer training loop:  67%|######7   | 33707/50000 [21:50:05<9:23:49,  2.08s/it]
Outer training loop:  67%|######7   | 33708/50000 [21:50:07<9:20:47,  2.07s/it]
Outer training loop:  67%|######7   | 33709/50000 [21:50:09<9:02:05,  2.00s/it]
Outer training loop:  67%|######7   | 33710/50000 [21:50:11<8:54:13,  1.97s/it]
Outer training loop:  67%|######7   | 33711/50000 [21:50:13<9:01:54,  2.00s/it]
Outer training loop:  67%|######7   | 33712/50000 [21:50:15<9:00:09,  1.99s/it]
Outer training loop:  67%|######7   | 33713/50000 [21:50:17<8:44:47,  1.93s/it]
Outer training loop:  67%|######7   | 33714/50000 [21:50:19<8:46:06,  1.94s/it]
Outer training loop:  67%|######7   | 33715/50000 [21:50:21<8:50:59,  1.96s/it]
Outer training loop:  67%|######7   | 33716/50000 [21:50:23<8:30:52,  1.88s/it]
Outer training loop:  67%|######7   | 33717/50000 [21:50:25<8:33:45,  1.89s/it]
Outer training loop:  67%|######7   | 33718/50000 [21:50:27<8:52:30,  1.96s/it]
Outer training loop:  67%|######7   | 33719/50000 [21:50:29<8:48:19,  1.95s/it]
Outer training loop:  67%|######7   | 33720/50000 [21:50:31<8:47:45,  1.95s/it]
Outer training loop:  67%|######7   | 33721/50000 [21:50:33<8:51:51,  1.96s/it]
Outer training loop:  67%|######7   | 33722/50000 [21:50:34<8:45:58,  1.94s/it]
Outer training loop:  67%|######7   | 33723/50000 [21:50:36<8:46:03,  1.94s/it]
Outer training loop:  67%|######7   | 33724/50000 [21:50:38<8:45:49,  1.94s/it]
Outer training loop:  67%|######7   | 33725/50000 [21:50:40<8:55:37,  1.97s/it]
Outer training loop:  67%|######7   | 33726/50000 [21:50:42<8:45:11,  1.94s/it]
Outer training loop:  67%|######7   | 33727/50000 [21:50:44<8:42:11,  1.93s/it]
Outer training loop:  67%|######7   | 33728/50000 [21:50:46<9:03:49,  2.01s/it]
Outer training loop:  67%|######7   | 33729/50000 [21:50:48<8:48:16,  1.95s/it]
Outer training loop:  67%|######7   | 33730/50000 [21:50:50<8:49:56,  1.95s/it]
Outer training loop:  67%|######7   | 33731/50000 [21:50:52<8:38:13,  1.91s/it]
Outer training loop:  67%|######7   | 33732/50000 [21:50:54<8:51:27,  1.96s/it]
Outer training loop:  67%|######7   | 33733/50000 [21:50:56<8:47:29,  1.95s/it]
Outer training loop:  67%|######7   | 33734/50000 [21:50:58<8:32:24,  1.89s/it]
Outer training loop:  67%|######7   | 33735/50000 [21:50:59<8:26:01,  1.87s/it]
Outer training loop:  67%|######7   | 33736/50000 [21:51:02<8:42:15,  1.93s/it]
Outer training loop:  67%|######7   | 33737/50000 [21:51:03<8:37:49,  1.91s/it]
Outer training loop:  67%|######7   | 33738/50000 [21:51:05<8:28:06,  1.87s/it]
Outer training loop:  67%|######7   | 33739/50000 [21:51:07<8:38:09,  1.91s/it]
Outer training loop:  67%|######7   | 33740/50000 [21:51:09<8:28:32,  1.88s/it]
Outer training loop:  67%|######7   | 33741/50000 [21:51:11<8:31:49,  1.89s/it]
Outer training loop:  67%|######7   | 33742/50000 [21:51:13<8:50:41,  1.96s/it]
Outer training loop:  67%|######7   | 33743/50000 [21:51:15<8:42:11,  1.93s/it]
Outer training loop:  67%|######7   | 33744/50000 [21:51:17<8:35:04,  1.90s/it]
Outer training loop:  67%|######7   | 33745/50000 [21:51:19<8:32:57,  1.89s/it]
Outer training loop:  67%|######7   | 33746/50000 [21:51:21<8:37:43,  1.91s/it]
Outer training loop:  67%|######7   | 33747/50000 [21:51:22<8:33:35,  1.90s/it]
Outer training loop:  67%|######7   | 33748/50000 [21:51:24<8:29:32,  1.88s/it]
Outer training loop:  67%|######7   | 33749/50000 [21:51:26<8:53:45,  1.97s/it]
Outer training loop:  68%|######7   | 33750/50000 [21:51:28<8:48:57,  1.95s/it]
Outer training loop:  68%|######7   | 33751/50000 [21:51:30<8:34:35,  1.90s/it]
Outer training loop:  68%|######7   | 33752/50000 [21:51:32<8:37:18,  1.91s/it]
Outer training loop:  68%|######7   | 33753/50000 [21:51:34<8:50:30,  1.96s/it]
Outer training loop:  68%|######7   | 33754/50000 [21:51:36<8:41:39,  1.93s/it]
Outer training loop:  68%|######7   | 33755/50000 [21:51:38<8:38:19,  1.91s/it]
Outer training loop:  68%|######7   | 33756/50000 [21:51:40<8:52:44,  1.97s/it]
Outer training loop:  68%|######7   | 33757/50000 [21:51:42<8:35:31,  1.90s/it]
Outer training loop:  68%|######7   | 33758/50000 [21:51:44<8:27:23,  1.87s/it]
Outer training loop:  68%|######7   | 33759/50000 [21:51:46<8:38:37,  1.92s/it]
Outer training loop:  68%|######7   | 33760/50000 [21:51:47<8:25:40,  1.87s/it]
Outer training loop:  68%|######7   | 33761/50000 [21:51:49<8:18:36,  1.84s/it]
Outer training loop:  68%|######7   | 33762/50000 [21:51:51<8:16:32,  1.83s/it]
Outer training loop:  68%|######7   | 33763/50000 [21:51:53<8:32:27,  1.89s/it]
Outer training loop:  68%|######7   | 33764/50000 [21:51:55<8:24:33,  1.86s/it]
Outer training loop:  68%|######7   | 33765/50000 [21:51:57<8:46:04,  1.94s/it]
Outer training loop:  68%|######7   | 33766/50000 [21:51:59<8:57:01,  1.98s/it]
Outer training loop:  68%|######7   | 33767/50000 [21:52:01<8:36:44,  1.91s/it]
Outer training loop:  68%|######7   | 33768/50000 [21:52:02<8:23:38,  1.86s/it]
Outer training loop:  68%|######7   | 33769/50000 [21:52:04<8:20:47,  1.85s/it]
Outer training loop:  68%|######7   | 33770/50000 [21:52:06<8:36:14,  1.91s/it]
Outer training loop:  68%|######7   | 33771/50000 [21:52:08<8:22:41,  1.86s/it]
Outer training loop:  68%|######7   | 33772/50000 [21:52:10<8:21:07,  1.85s/it]
Outer training loop:  68%|######7   | 33773/50000 [21:52:12<8:22:27,  1.86s/it]
Outer training loop:  68%|######7   | 33774/50000 [21:52:14<8:34:22,  1.90s/it]
Outer training loop:  68%|######7   | 33775/50000 [21:52:15<8:17:26,  1.84s/it]
Outer training loop:  68%|######7   | 33776/50000 [21:52:17<8:19:34,  1.85s/it]
Outer training loop:  68%|######7   | 33777/50000 [21:52:19<8:23:55,  1.86s/it]
Outer training loop:  68%|######7   | 33778/50000 [21:52:21<8:24:44,  1.87s/it]
Outer training loop:  68%|######7   | 33779/50000 [21:52:23<8:20:25,  1.85s/it]
Outer training loop:  68%|######7   | 33780/50000 [21:52:25<8:34:49,  1.90s/it]
Outer training loop:  68%|######7   | 33781/50000 [21:52:27<8:32:19,  1.90s/it]
Outer training loop:  68%|######7   | 33782/50000 [21:52:29<8:31:08,  1.89s/it]
Outer training loop:  68%|######7   | 33783/50000 [21:52:30<8:22:45,  1.86s/it]
Outer training loop:  68%|######7   | 33784/50000 [21:52:32<8:30:05,  1.89s/it]
Outer training loop:  68%|######7   | 33785/50000 [21:52:34<8:21:05,  1.85s/it]
Outer training loop:  68%|######7   | 33786/50000 [21:52:36<8:15:01,  1.83s/it]
Outer training loop:  68%|######7   | 33787/50000 [21:52:38<8:39:22,  1.92s/it]
Outer training loop:  68%|######7   | 33788/50000 [21:52:40<8:32:44,  1.90s/it]
Outer training loop:  68%|######7   | 33789/50000 [21:52:42<8:37:10,  1.91s/it]
Outer training loop:  68%|######7   | 33790/50000 [21:52:44<8:36:21,  1.91s/it]
Outer training loop:  68%|######7   | 33791/50000 [21:52:46<8:43:17,  1.94s/it]
Outer training loop:  68%|######7   | 33792/50000 [21:52:48<8:23:41,  1.86s/it]
Outer training loop:  68%|######7   | 33793/50000 [21:52:49<8:28:15,  1.88s/it]
Outer training loop:  68%|######7   | 33794/50000 [21:52:52<8:44:11,  1.94s/it]
Outer training loop:  68%|######7   | 33795/50000 [21:52:53<8:34:41,  1.91s/it]
Outer training loop:  68%|######7   | 33796/50000 [21:52:55<8:33:21,  1.90s/it]
Outer training loop:  68%|######7   | 33797/50000 [21:52:57<8:45:02,  1.94s/it]
Outer training loop:  68%|######7   | 33798/50000 [21:52:59<8:29:49,  1.89s/it]
Outer training loop:  68%|######7   | 33799/50000 [21:53:01<8:33:24,  1.90s/it]
Outer training loop:  68%|######7   | 33800/50000 [21:53:03<8:37:01,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  68%|######7   | 33801/50000 [21:53:10<16:07:35,  3.58s/it]
Outer training loop:  68%|######7   | 33802/50000 [21:53:12<14:00:00,  3.11s/it]
Outer training loop:  68%|######7   | 33803/50000 [21:53:14<12:19:54,  2.74s/it]
Outer training loop:  68%|######7   | 33804/50000 [21:53:16<10:55:23,  2.43s/it]
Outer training loop:  68%|######7   | 33805/50000 [21:53:18<10:15:45,  2.28s/it]
Outer training loop:  68%|######7   | 33806/50000 [21:53:20<9:36:36,  2.14s/it] 
Outer training loop:  68%|######7   | 33807/50000 [21:53:22<9:19:23,  2.07s/it]
Outer training loop:  68%|######7   | 33808/50000 [21:53:24<9:15:06,  2.06s/it]
Outer training loop:  68%|######7   | 33809/50000 [21:53:26<9:00:52,  2.00s/it]
Outer training loop:  68%|######7   | 33810/50000 [21:53:27<8:43:38,  1.94s/it]
Outer training loop:  68%|######7   | 33811/50000 [21:53:29<8:31:45,  1.90s/it]
Outer training loop:  68%|######7   | 33812/50000 [21:53:31<8:48:13,  1.96s/it]
Outer training loop:  68%|######7   | 33813/50000 [21:53:33<8:39:20,  1.93s/it]
Outer training loop:  68%|######7   | 33814/50000 [21:53:35<8:37:25,  1.92s/it]
Outer training loop:  68%|######7   | 33815/50000 [21:53:37<8:52:32,  1.97s/it]
Outer training loop:  68%|######7   | 33816/50000 [21:53:39<8:50:39,  1.97s/it]
Outer training loop:  68%|######7   | 33817/50000 [21:53:41<8:43:55,  1.94s/it]
Outer training loop:  68%|######7   | 33818/50000 [21:53:43<8:37:24,  1.92s/it]
Outer training loop:  68%|######7   | 33819/50000 [21:53:45<8:51:08,  1.97s/it]
Outer training loop:  68%|######7   | 33820/50000 [21:53:47<8:41:04,  1.93s/it]
Outer training loop:  68%|######7   | 33821/50000 [21:53:49<8:35:26,  1.91s/it]
Outer training loop:  68%|######7   | 33822/50000 [21:53:51<8:49:42,  1.96s/it]
Outer training loop:  68%|######7   | 33823/50000 [21:53:52<8:33:20,  1.90s/it]
Outer training loop:  68%|######7   | 33824/50000 [21:53:54<8:18:28,  1.85s/it]
Outer training loop:  68%|######7   | 33825/50000 [21:53:56<8:36:46,  1.92s/it]
Outer training loop:  68%|######7   | 33826/50000 [21:53:58<8:38:06,  1.92s/it]
Outer training loop:  68%|######7   | 33827/50000 [21:54:00<8:32:54,  1.90s/it]
Outer training loop:  68%|######7   | 33828/50000 [21:54:02<8:49:55,  1.97s/it]
Outer training loop:  68%|######7   | 33829/50000 [21:54:04<8:44:45,  1.95s/it]
Outer training loop:  68%|######7   | 33830/50000 [21:54:06<8:34:03,  1.91s/it]
Outer training loop:  68%|######7   | 33831/50000 [21:54:08<8:25:13,  1.87s/it]
Outer training loop:  68%|######7   | 33832/50000 [21:54:10<8:38:12,  1.92s/it]
Outer training loop:  68%|######7   | 33833/50000 [21:54:11<8:28:56,  1.89s/it]
Outer training loop:  68%|######7   | 33834/50000 [21:54:13<8:25:14,  1.88s/it]
Outer training loop:  68%|######7   | 33835/50000 [21:54:15<8:29:55,  1.89s/it]
Outer training loop:  68%|######7   | 33836/50000 [21:54:17<8:24:05,  1.87s/it]
Outer training loop:  68%|######7   | 33837/50000 [21:54:19<8:24:04,  1.87s/it]
Outer training loop:  68%|######7   | 33838/50000 [21:54:21<8:25:25,  1.88s/it]
Outer training loop:  68%|######7   | 33839/50000 [21:54:23<8:35:48,  1.92s/it]
Outer training loop:  68%|######7   | 33840/50000 [21:54:25<8:35:32,  1.91s/it]
Outer training loop:  68%|######7   | 33841/50000 [21:54:27<8:31:27,  1.90s/it]
Outer training loop:  68%|######7   | 33842/50000 [21:54:29<8:52:15,  1.98s/it]
Outer training loop:  68%|######7   | 33843/50000 [21:54:31<8:36:48,  1.92s/it]
Outer training loop:  68%|######7   | 33844/50000 [21:54:32<8:25:34,  1.88s/it]
Outer training loop:  68%|######7   | 33845/50000 [21:54:34<8:43:24,  1.94s/it]
Outer training loop:  68%|######7   | 33846/50000 [21:54:36<8:37:51,  1.92s/it]
Outer training loop:  68%|######7   | 33847/50000 [21:54:38<8:27:46,  1.89s/it]
Outer training loop:  68%|######7   | 33848/50000 [21:54:40<8:43:42,  1.95s/it]
Outer training loop:  68%|######7   | 33849/50000 [21:54:42<8:30:34,  1.90s/it]
Outer training loop:  68%|######7   | 33850/50000 [21:54:44<8:23:07,  1.87s/it]
Outer training loop:  68%|######7   | 33851/50000 [21:54:46<8:30:12,  1.90s/it]
Outer training loop:  68%|######7   | 33852/50000 [21:54:48<8:22:27,  1.87s/it]
Outer training loop:  68%|######7   | 33853/50000 [21:54:49<8:16:38,  1.85s/it]
Outer training loop:  68%|######7   | 33854/50000 [21:54:51<8:12:46,  1.83s/it]
Outer training loop:  68%|######7   | 33855/50000 [21:54:53<8:17:03,  1.85s/it]
Outer training loop:  68%|######7   | 33856/50000 [21:54:55<8:18:36,  1.85s/it]
Outer training loop:  68%|######7   | 33857/50000 [21:54:57<8:29:17,  1.89s/it]
Outer training loop:  68%|######7   | 33858/50000 [21:54:59<8:39:39,  1.93s/it]
Outer training loop:  68%|######7   | 33859/50000 [21:55:01<8:27:55,  1.89s/it]
Outer training loop:  68%|######7   | 33860/50000 [21:55:02<8:13:22,  1.83s/it]
Outer training loop:  68%|######7   | 33861/50000 [21:55:04<8:30:12,  1.90s/it]
Outer training loop:  68%|######7   | 33862/50000 [21:55:07<8:50:44,  1.97s/it]
Outer training loop:  68%|######7   | 33863/50000 [21:55:08<8:36:59,  1.92s/it]
Outer training loop:  68%|######7   | 33864/50000 [21:55:10<8:37:41,  1.92s/it]
Outer training loop:  68%|######7   | 33865/50000 [21:55:12<8:48:34,  1.97s/it]
Outer training loop:  68%|######7   | 33866/50000 [21:55:14<8:31:33,  1.90s/it]
Outer training loop:  68%|######7   | 33867/50000 [21:55:16<8:22:06,  1.87s/it]
Outer training loop:  68%|######7   | 33868/50000 [21:55:18<8:33:30,  1.91s/it]
Outer training loop:  68%|######7   | 33869/50000 [21:55:20<8:38:07,  1.93s/it]
Outer training loop:  68%|######7   | 33870/50000 [21:55:22<8:25:29,  1.88s/it]
Outer training loop:  68%|######7   | 33871/50000 [21:55:23<8:15:48,  1.84s/it]
Outer training loop:  68%|######7   | 33872/50000 [21:55:26<8:38:22,  1.93s/it]
Outer training loop:  68%|######7   | 33873/50000 [21:55:28<8:39:54,  1.93s/it]
Outer training loop:  68%|######7   | 33874/50000 [21:55:29<8:23:42,  1.87s/it]
Outer training loop:  68%|######7   | 33875/50000 [21:55:31<8:23:57,  1.88s/it]
Outer training loop:  68%|######7   | 33876/50000 [21:55:33<8:35:20,  1.92s/it]
Outer training loop:  68%|######7   | 33877/50000 [21:55:35<8:30:08,  1.90s/it]
Outer training loop:  68%|######7   | 33878/50000 [21:55:37<8:20:59,  1.86s/it]
Outer training loop:  68%|######7   | 33879/50000 [21:55:39<8:24:47,  1.88s/it]
Outer training loop:  68%|######7   | 33880/50000 [21:55:41<8:30:03,  1.90s/it]
Outer training loop:  68%|######7   | 33881/50000 [21:55:43<8:30:16,  1.90s/it]
Outer training loop:  68%|######7   | 33882/50000 [21:55:45<8:43:44,  1.95s/it]
Outer training loop:  68%|######7   | 33883/50000 [21:55:46<8:33:59,  1.91s/it]
Outer training loop:  68%|######7   | 33884/50000 [21:55:48<8:24:44,  1.88s/it]
Outer training loop:  68%|######7   | 33885/50000 [21:55:50<8:46:31,  1.96s/it]
Outer training loop:  68%|######7   | 33886/50000 [21:55:52<8:32:15,  1.91s/it]
Outer training loop:  68%|######7   | 33887/50000 [21:55:54<8:20:21,  1.86s/it]
Outer training loop:  68%|######7   | 33888/50000 [21:55:56<8:20:48,  1.86s/it]
Outer training loop:  68%|######7   | 33889/50000 [21:55:58<8:27:17,  1.89s/it]
Outer training loop:  68%|######7   | 33890/50000 [21:56:00<8:21:30,  1.87s/it]
Outer training loop:  68%|######7   | 33891/50000 [21:56:01<8:21:03,  1.87s/it]
Outer training loop:  68%|######7   | 33892/50000 [21:56:03<8:23:11,  1.87s/it]
Outer training loop:  68%|######7   | 33893/50000 [21:56:05<8:29:11,  1.90s/it]
Outer training loop:  68%|######7   | 33894/50000 [21:56:07<8:22:59,  1.87s/it]
Outer training loop:  68%|######7   | 33895/50000 [21:56:09<8:29:08,  1.90s/it]
Outer training loop:  68%|######7   | 33896/50000 [21:56:11<8:19:19,  1.86s/it]
Outer training loop:  68%|######7   | 33897/50000 [21:56:13<8:20:04,  1.86s/it]
Outer training loop:  68%|######7   | 33898/50000 [21:56:14<8:11:31,  1.83s/it]
Outer training loop:  68%|######7   | 33899/50000 [21:56:16<8:16:51,  1.85s/it]
Outer training loop:  68%|######7   | 33900/50000 [21:56:18<8:13:05,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.70s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  68%|######7   | 33901/50000 [21:56:25<15:35:43,  3.49s/it]
Outer training loop:  68%|######7   | 33902/50000 [21:56:27<13:13:08,  2.96s/it]
Outer training loop:  68%|######7   | 33903/50000 [21:56:29<12:08:06,  2.71s/it]
Outer training loop:  68%|######7   | 33904/50000 [21:56:31<11:01:39,  2.47s/it]
Outer training loop:  68%|######7   | 33905/50000 [21:56:33<10:12:40,  2.28s/it]
Outer training loop:  68%|######7   | 33906/50000 [21:56:35<9:38:34,  2.16s/it] 
Outer training loop:  68%|######7   | 33907/50000 [21:56:37<9:18:11,  2.08s/it]
Outer training loop:  68%|######7   | 33908/50000 [21:56:39<8:47:39,  1.97s/it]
Outer training loop:  68%|######7   | 33909/50000 [21:56:41<8:52:55,  1.99s/it]
Outer training loop:  68%|######7   | 33910/50000 [21:56:43<8:52:34,  1.99s/it]
Outer training loop:  68%|######7   | 33911/50000 [21:56:44<8:29:52,  1.90s/it]
Outer training loop:  68%|######7   | 33912/50000 [21:56:46<8:25:45,  1.89s/it]
Outer training loop:  68%|######7   | 33913/50000 [21:56:48<8:22:45,  1.88s/it]
Outer training loop:  68%|######7   | 33914/50000 [21:56:50<8:44:51,  1.96s/it]
Outer training loop:  68%|######7   | 33915/50000 [21:56:52<8:37:40,  1.93s/it]
Outer training loop:  68%|######7   | 33916/50000 [21:56:54<8:25:13,  1.88s/it]
Outer training loop:  68%|######7   | 33917/50000 [21:56:56<8:32:31,  1.91s/it]
Outer training loop:  68%|######7   | 33918/50000 [21:56:58<8:25:30,  1.89s/it]
Outer training loop:  68%|######7   | 33919/50000 [21:56:59<8:14:20,  1.84s/it]
Outer training loop:  68%|######7   | 33920/50000 [21:57:01<8:21:11,  1.87s/it]
Outer training loop:  68%|######7   | 33921/50000 [21:57:03<8:16:45,  1.85s/it]
Outer training loop:  68%|######7   | 33922/50000 [21:57:05<8:21:21,  1.87s/it]
Outer training loop:  68%|######7   | 33923/50000 [21:57:07<8:20:10,  1.87s/it]
Outer training loop:  68%|######7   | 33924/50000 [21:57:09<8:28:55,  1.90s/it]
Outer training loop:  68%|######7   | 33925/50000 [21:57:11<8:17:00,  1.86s/it]
Outer training loop:  68%|######7   | 33926/50000 [21:57:12<8:14:32,  1.85s/it]
Outer training loop:  68%|######7   | 33927/50000 [21:57:14<8:31:38,  1.91s/it]
Outer training loop:  68%|######7   | 33928/50000 [21:57:16<8:22:30,  1.88s/it]
Outer training loop:  68%|######7   | 33929/50000 [21:57:18<8:14:25,  1.85s/it]
Outer training loop:  68%|######7   | 33930/50000 [21:57:20<8:19:56,  1.87s/it]
Outer training loop:  68%|######7   | 33931/50000 [21:57:22<8:37:51,  1.93s/it]
Outer training loop:  68%|######7   | 33932/50000 [21:57:24<8:29:15,  1.90s/it]
Outer training loop:  68%|######7   | 33933/50000 [21:57:26<8:29:37,  1.90s/it]
Outer training loop:  68%|######7   | 33934/50000 [21:57:28<8:36:49,  1.93s/it]
Outer training loop:  68%|######7   | 33935/50000 [21:57:30<8:32:35,  1.91s/it]
Outer training loop:  68%|######7   | 33936/50000 [21:57:31<8:13:09,  1.84s/it]
Outer training loop:  68%|######7   | 33937/50000 [21:57:33<8:08:32,  1.82s/it]
Outer training loop:  68%|######7   | 33938/50000 [21:57:35<8:24:36,  1.88s/it]
Outer training loop:  68%|######7   | 33939/50000 [21:57:37<8:12:11,  1.84s/it]
Outer training loop:  68%|######7   | 33940/50000 [21:57:39<8:16:22,  1.85s/it]
Outer training loop:  68%|######7   | 33941/50000 [21:57:41<8:30:48,  1.91s/it]
Outer training loop:  68%|######7   | 33942/50000 [21:57:43<8:27:33,  1.90s/it]
Outer training loop:  68%|######7   | 33943/50000 [21:57:44<8:19:31,  1.87s/it]
Outer training loop:  68%|######7   | 33944/50000 [21:57:46<8:12:51,  1.84s/it]
Outer training loop:  68%|######7   | 33945/50000 [21:57:48<8:18:49,  1.86s/it]
Outer training loop:  68%|######7   | 33946/50000 [21:57:50<8:21:59,  1.88s/it]
Outer training loop:  68%|######7   | 33947/50000 [21:57:52<8:17:17,  1.86s/it]
Outer training loop:  68%|######7   | 33948/50000 [21:57:54<8:28:00,  1.90s/it]
Outer training loop:  68%|######7   | 33949/50000 [21:57:56<8:28:10,  1.90s/it]
Outer training loop:  68%|######7   | 33950/50000 [21:57:58<8:31:06,  1.91s/it]
Outer training loop:  68%|######7   | 33951/50000 [21:58:00<8:42:19,  1.95s/it]
Outer training loop:  68%|######7   | 33952/50000 [21:58:02<8:28:51,  1.90s/it]
Outer training loop:  68%|######7   | 33953/50000 [21:58:03<8:27:51,  1.90s/it]
Outer training loop:  68%|######7   | 33954/50000 [21:58:05<8:24:23,  1.89s/it]
Outer training loop:  68%|######7   | 33955/50000 [21:58:07<8:28:00,  1.90s/it]
Outer training loop:  68%|######7   | 33956/50000 [21:58:09<8:23:54,  1.88s/it]
Outer training loop:  68%|######7   | 33957/50000 [21:58:11<8:22:49,  1.88s/it]
Outer training loop:  68%|######7   | 33958/50000 [21:58:13<8:24:05,  1.89s/it]
Outer training loop:  68%|######7   | 33959/50000 [21:58:15<8:37:38,  1.94s/it]
Outer training loop:  68%|######7   | 33960/50000 [21:58:17<8:23:00,  1.88s/it]
Outer training loop:  68%|######7   | 33961/50000 [21:58:18<8:20:26,  1.87s/it]
Outer training loop:  68%|######7   | 33962/50000 [21:58:21<8:35:05,  1.93s/it]
Outer training loop:  68%|######7   | 33963/50000 [21:58:22<8:20:08,  1.87s/it]
Outer training loop:  68%|######7   | 33964/50000 [21:58:24<8:11:06,  1.84s/it]
Outer training loop:  68%|######7   | 33965/50000 [21:58:26<8:17:58,  1.86s/it]
Outer training loop:  68%|######7   | 33966/50000 [21:58:28<8:28:08,  1.90s/it]
Outer training loop:  68%|######7   | 33967/50000 [21:58:30<8:20:56,  1.87s/it]
Outer training loop:  68%|######7   | 33968/50000 [21:58:32<8:18:12,  1.86s/it]
Outer training loop:  68%|######7   | 33969/50000 [21:58:33<8:14:39,  1.85s/it]
Outer training loop:  68%|######7   | 33970/50000 [21:58:35<8:17:20,  1.86s/it]
Outer training loop:  68%|######7   | 33971/50000 [21:58:37<8:16:31,  1.86s/it]
Outer training loop:  68%|######7   | 33972/50000 [21:58:39<8:11:05,  1.84s/it]
Outer training loop:  68%|######7   | 33973/50000 [21:58:41<8:35:26,  1.93s/it]
Outer training loop:  68%|######7   | 33974/50000 [21:58:43<8:34:51,  1.93s/it]
Outer training loop:  68%|######7   | 33975/50000 [21:58:45<8:21:15,  1.88s/it]
Outer training loop:  68%|######7   | 33976/50000 [21:58:47<8:25:42,  1.89s/it]
Outer training loop:  68%|######7   | 33977/50000 [21:58:49<8:22:47,  1.88s/it]
Outer training loop:  68%|######7   | 33978/50000 [21:58:50<8:17:45,  1.86s/it]
Outer training loop:  68%|######7   | 33979/50000 [21:58:52<8:14:41,  1.85s/it]
Outer training loop:  68%|######7   | 33980/50000 [21:58:54<8:24:20,  1.89s/it]
Outer training loop:  68%|######7   | 33981/50000 [21:58:56<8:22:08,  1.88s/it]
Outer training loop:  68%|######7   | 33982/50000 [21:58:58<8:16:35,  1.86s/it]
Outer training loop:  68%|######7   | 33983/50000 [21:59:00<8:24:25,  1.89s/it]
Outer training loop:  68%|######7   | 33984/50000 [21:59:02<8:21:55,  1.88s/it]
Outer training loop:  68%|######7   | 33985/50000 [21:59:03<8:17:16,  1.86s/it]
Outer training loop:  68%|######7   | 33986/50000 [21:59:06<8:29:39,  1.91s/it]
Outer training loop:  68%|######7   | 33987/50000 [21:59:07<8:19:04,  1.87s/it]
Outer training loop:  68%|######7   | 33988/50000 [21:59:09<8:00:39,  1.80s/it]
Outer training loop:  68%|######7   | 33989/50000 [21:59:11<8:06:05,  1.82s/it]
Outer training loop:  68%|######7   | 33990/50000 [21:59:13<8:22:12,  1.88s/it]
Outer training loop:  68%|######7   | 33991/50000 [21:59:15<8:20:58,  1.88s/it]
Outer training loop:  68%|######7   | 33992/50000 [21:59:16<8:08:37,  1.83s/it]
Outer training loop:  68%|######7   | 33993/50000 [21:59:18<8:23:01,  1.89s/it]
Outer training loop:  68%|######7   | 33994/50000 [21:59:20<8:20:40,  1.88s/it]
Outer training loop:  68%|######7   | 33995/50000 [21:59:22<8:09:23,  1.83s/it]
Outer training loop:  68%|######7   | 33996/50000 [21:59:24<8:04:51,  1.82s/it]
Outer training loop:  68%|######7   | 33997/50000 [21:59:26<8:20:00,  1.87s/it]
Outer training loop:  68%|######7   | 33998/50000 [21:59:28<8:19:07,  1.87s/it]
Outer training loop:  68%|######7   | 33999/50000 [21:59:29<8:09:41,  1.84s/it]
Outer training loop:  68%|######8   | 34000/50000 [21:59:31<8:26:22,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:28,  1.81s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:26,  1.81s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:30,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:26,  1.86s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:29,  1.92s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.93s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:29,  1.97s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.93s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:25,  1.99s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:20,  1.95s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:12,  1.95s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.87s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.93s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:02,  1.94s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:52,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:55,  1.96s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:51,  1.92s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.89s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:21,  1.82s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:20,  1.82s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:20,  1.92s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.91s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:12,  1.87s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:09,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:07,  1.83s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:05,  1.82s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:07,  1.94s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:04,  1.89s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:02,  1.88s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:56,  2.02s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:53,  1.97s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:50,  1.94s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.90s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.87s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.98s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.90s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.85s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.84s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.81s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:21,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:21,  1.95s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.90s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.89s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.89s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.91s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:04,  2.04s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:02,  2.02s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.89s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.75s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:24,  1.80s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.81s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:20,  1.80s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:19,  1.81s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:20,  1.85s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:15,  1.81s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:25,  1.97s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:20,  1.93s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:19,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:15,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:15,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:03,  1.87s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:06,  1.95s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:56,  1.84s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:53,  1.83s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:56,  1.94s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:47,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:45,  1.85s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:41,  1.81s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:45,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:35,  1.84s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:31,  1.83s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:35,  1.94s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:32,  1.92s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:30,  1.93s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:28,  1.93s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:25,  1.91s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:23,  1.94s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:12,  1.91s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:13,  1.99s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:09,  1.94s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:05,  1.88s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:59,  1.99s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:55,  1.91s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:52,  1.88s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.83s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:43,  1.96s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.91s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.84s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.85s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:27,  2.00s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.96s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.91s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.90s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.86s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.84s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.82s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:10,  1.81s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:08,  1.80s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.94s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.88s/it]

Outer training loop:  68%|######8   | 34001/50000 [22:04:53<434:05:23, 97.68s/it]
Outer training loop:  68%|######8   | 34002/50000 [22:04:54<306:13:58, 68.91s/it]
Outer training loop:  68%|######8   | 34003/50000 [22:04:56<216:51:12, 48.80s/it]
Outer training loop:  68%|######8   | 34004/50000 [22:04:58<154:13:11, 34.71s/it]
Outer training loop:  68%|######8   | 34005/50000 [22:05:00<110:17:42, 24.82s/it]
Outer training loop:  68%|######8   | 34006/50000 [22:05:02<79:31:28, 17.90s/it] 
Outer training loop:  68%|######8   | 34007/50000 [22:05:04<58:55:41, 13.26s/it]
Outer training loop:  68%|######8   | 34008/50000 [22:05:06<43:45:37,  9.85s/it]
Outer training loop:  68%|######8   | 34009/50000 [22:05:08<33:02:23,  7.44s/it]
Outer training loop:  68%|######8   | 34010/50000 [22:05:10<25:53:45,  5.83s/it]
Outer training loop:  68%|######8   | 34011/50000 [22:05:12<20:26:13,  4.60s/it]
Outer training loop:  68%|######8   | 34012/50000 [22:05:13<16:49:23,  3.79s/it]
Outer training loop:  68%|######8   | 34013/50000 [22:05:15<14:23:59,  3.24s/it]
Outer training loop:  68%|######8   | 34014/50000 [22:05:17<12:44:35,  2.87s/it]
Outer training loop:  68%|######8   | 34015/50000 [22:05:19<11:32:18,  2.60s/it]
Outer training loop:  68%|######8   | 34016/50000 [22:05:21<10:34:15,  2.38s/it]
Outer training loop:  68%|######8   | 34017/50000 [22:05:23<9:57:28,  2.24s/it] 
Outer training loop:  68%|######8   | 34018/50000 [22:05:25<9:23:57,  2.12s/it]
Outer training loop:  68%|######8   | 34019/50000 [22:05:27<8:58:18,  2.02s/it]
Outer training loop:  68%|######8   | 34020/50000 [22:05:29<9:02:55,  2.04s/it]
Outer training loop:  68%|######8   | 34021/50000 [22:05:31<8:48:27,  1.98s/it]
Outer training loop:  68%|######8   | 34022/50000 [22:05:32<8:27:33,  1.91s/it]
Outer training loop:  68%|######8   | 34023/50000 [22:05:35<8:40:30,  1.95s/it]
Outer training loop:  68%|######8   | 34024/50000 [22:05:36<8:31:39,  1.92s/it]
Outer training loop:  68%|######8   | 34025/50000 [22:05:38<8:31:19,  1.92s/it]
Outer training loop:  68%|######8   | 34026/50000 [22:05:40<8:28:39,  1.91s/it]
Outer training loop:  68%|######8   | 34027/50000 [22:05:42<8:32:56,  1.93s/it]
Outer training loop:  68%|######8   | 34028/50000 [22:05:44<8:27:09,  1.91s/it]
Outer training loop:  68%|######8   | 34029/50000 [22:05:46<8:20:28,  1.88s/it]
Outer training loop:  68%|######8   | 34030/50000 [22:05:48<8:27:01,  1.90s/it]
Outer training loop:  68%|######8   | 34031/50000 [22:05:50<8:26:47,  1.90s/it]
Outer training loop:  68%|######8   | 34032/50000 [22:05:52<8:23:05,  1.89s/it]
Outer training loop:  68%|######8   | 34033/50000 [22:05:54<8:32:14,  1.92s/it]
Outer training loop:  68%|######8   | 34034/50000 [22:05:55<8:25:14,  1.90s/it]
Outer training loop:  68%|######8   | 34035/50000 [22:05:57<8:20:26,  1.88s/it]
Outer training loop:  68%|######8   | 34036/50000 [22:05:59<8:27:17,  1.91s/it]
Outer training loop:  68%|######8   | 34037/50000 [22:06:01<8:44:51,  1.97s/it]
Outer training loop:  68%|######8   | 34038/50000 [22:06:03<8:17:42,  1.87s/it]
Outer training loop:  68%|######8   | 34039/50000 [22:06:05<8:07:37,  1.83s/it]
Outer training loop:  68%|######8   | 34040/50000 [22:06:07<8:33:25,  1.93s/it]
Outer training loop:  68%|######8   | 34041/50000 [22:06:09<8:20:45,  1.88s/it]
Outer training loop:  68%|######8   | 34042/50000 [22:06:10<8:12:44,  1.85s/it]
Outer training loop:  68%|######8   | 34043/50000 [22:06:12<8:25:47,  1.90s/it]
Outer training loop:  68%|######8   | 34044/50000 [22:06:14<8:17:31,  1.87s/it]
Outer training loop:  68%|######8   | 34045/50000 [22:06:16<8:11:13,  1.85s/it]
Outer training loop:  68%|######8   | 34046/50000 [22:06:18<8:12:02,  1.85s/it]
Outer training loop:  68%|######8   | 34047/50000 [22:06:20<8:37:57,  1.95s/it]
Outer training loop:  68%|######8   | 34048/50000 [22:06:22<8:23:08,  1.89s/it]
Outer training loop:  68%|######8   | 34049/50000 [22:06:24<8:19:26,  1.88s/it]
Outer training loop:  68%|######8   | 34050/50000 [22:06:26<8:39:30,  1.95s/it]
Outer training loop:  68%|######8   | 34051/50000 [22:06:28<8:33:21,  1.93s/it]
Outer training loop:  68%|######8   | 34052/50000 [22:06:30<8:27:05,  1.91s/it]
Outer training loop:  68%|######8   | 34053/50000 [22:06:31<8:31:35,  1.92s/it]
Outer training loop:  68%|######8   | 34054/50000 [22:06:33<8:26:27,  1.91s/it]
Outer training loop:  68%|######8   | 34055/50000 [22:06:35<8:23:34,  1.89s/it]
Outer training loop:  68%|######8   | 34056/50000 [22:06:37<8:23:57,  1.90s/it]
Outer training loop:  68%|######8   | 34057/50000 [22:06:39<8:25:11,  1.90s/it]
Outer training loop:  68%|######8   | 34058/50000 [22:06:41<8:20:40,  1.88s/it]
Outer training loop:  68%|######8   | 34059/50000 [22:06:43<8:14:22,  1.86s/it]
Outer training loop:  68%|######8   | 34060/50000 [22:06:45<8:25:45,  1.90s/it]
Outer training loop:  68%|######8   | 34061/50000 [22:06:46<8:11:28,  1.85s/it]
Outer training loop:  68%|######8   | 34062/50000 [22:06:48<8:08:12,  1.84s/it]
Outer training loop:  68%|######8   | 34063/50000 [22:06:50<8:27:58,  1.91s/it]
Outer training loop:  68%|######8   | 34064/50000 [22:06:52<8:18:44,  1.88s/it]
Outer training loop:  68%|######8   | 34065/50000 [22:06:54<8:15:04,  1.86s/it]
Outer training loop:  68%|######8   | 34066/50000 [22:06:56<8:11:38,  1.85s/it]
Outer training loop:  68%|######8   | 34067/50000 [22:06:58<8:39:11,  1.96s/it]
Outer training loop:  68%|######8   | 34068/50000 [22:07:00<8:26:15,  1.91s/it]
Outer training loop:  68%|######8   | 34069/50000 [22:07:02<8:22:25,  1.89s/it]
Outer training loop:  68%|######8   | 34070/50000 [22:07:04<8:39:40,  1.96s/it]
Outer training loop:  68%|######8   | 34071/50000 [22:07:06<8:27:17,  1.91s/it]
Outer training loop:  68%|######8   | 34072/50000 [22:07:07<8:21:18,  1.89s/it]
Outer training loop:  68%|######8   | 34073/50000 [22:07:09<8:10:11,  1.85s/it]
Outer training loop:  68%|######8   | 34074/50000 [22:07:11<8:29:49,  1.92s/it]
Outer training loop:  68%|######8   | 34075/50000 [22:07:13<8:38:13,  1.95s/it]
Outer training loop:  68%|######8   | 34076/50000 [22:07:15<8:25:43,  1.91s/it]
Outer training loop:  68%|######8   | 34077/50000 [22:07:17<8:37:47,  1.95s/it]
Outer training loop:  68%|######8   | 34078/50000 [22:07:19<8:23:39,  1.90s/it]
Outer training loop:  68%|######8   | 34079/50000 [22:07:21<8:21:24,  1.89s/it]
Outer training loop:  68%|######8   | 34080/50000 [22:07:23<8:32:54,  1.93s/it]
Outer training loop:  68%|######8   | 34081/50000 [22:07:25<8:30:54,  1.93s/it]
Outer training loop:  68%|######8   | 34082/50000 [22:07:27<8:25:57,  1.91s/it]
Outer training loop:  68%|######8   | 34083/50000 [22:07:28<8:17:32,  1.88s/it]
Outer training loop:  68%|######8   | 34084/50000 [22:07:30<8:23:10,  1.90s/it]
Outer training loop:  68%|######8   | 34085/50000 [22:07:32<8:12:19,  1.86s/it]
Outer training loop:  68%|######8   | 34086/50000 [22:07:34<8:02:22,  1.82s/it]
Outer training loop:  68%|######8   | 34087/50000 [22:07:36<8:16:03,  1.87s/it]
Outer training loop:  68%|######8   | 34088/50000 [22:07:38<8:10:10,  1.85s/it]
Outer training loop:  68%|######8   | 34089/50000 [22:07:39<8:14:47,  1.87s/it]
Outer training loop:  68%|######8   | 34090/50000 [22:07:41<8:24:47,  1.90s/it]
Outer training loop:  68%|######8   | 34091/50000 [22:07:44<8:38:23,  1.96s/it]
Outer training loop:  68%|######8   | 34092/50000 [22:07:45<8:24:47,  1.90s/it]
Outer training loop:  68%|######8   | 34093/50000 [22:07:47<8:18:44,  1.88s/it]
Outer training loop:  68%|######8   | 34094/50000 [22:07:49<8:33:37,  1.94s/it]
Outer training loop:  68%|######8   | 34095/50000 [22:07:51<8:21:59,  1.89s/it]
Outer training loop:  68%|######8   | 34096/50000 [22:07:53<8:12:06,  1.86s/it]
Outer training loop:  68%|######8   | 34097/50000 [22:07:55<8:18:28,  1.88s/it]
Outer training loop:  68%|######8   | 34098/50000 [22:07:57<8:14:36,  1.87s/it]
Outer training loop:  68%|######8   | 34099/50000 [22:07:58<8:15:23,  1.87s/it]
Outer training loop:  68%|######8   | 34100/50000 [22:08:01<8:32:17,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.91s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.91s/it]

Outer training loop:  68%|######8   | 34101/50000 [22:08:08<16:00:57,  3.63s/it]
Outer training loop:  68%|######8   | 34102/50000 [22:08:10<13:33:53,  3.07s/it]
Outer training loop:  68%|######8   | 34103/50000 [22:08:12<12:01:19,  2.72s/it]
Outer training loop:  68%|######8   | 34104/50000 [22:08:14<10:49:28,  2.45s/it]
Outer training loop:  68%|######8   | 34105/50000 [22:08:16<10:10:47,  2.31s/it]
Outer training loop:  68%|######8   | 34106/50000 [22:08:17<9:26:36,  2.14s/it] 
Outer training loop:  68%|######8   | 34107/50000 [22:08:19<8:59:16,  2.04s/it]
Outer training loop:  68%|######8   | 34108/50000 [22:08:21<8:46:10,  1.99s/it]
Outer training loop:  68%|######8   | 34109/50000 [22:08:23<8:47:11,  1.99s/it]
Outer training loop:  68%|######8   | 34110/50000 [22:08:25<8:34:44,  1.94s/it]
Outer training loop:  68%|######8   | 34111/50000 [22:08:27<8:32:45,  1.94s/it]
Outer training loop:  68%|######8   | 34112/50000 [22:08:29<8:41:00,  1.97s/it]
Outer training loop:  68%|######8   | 34113/50000 [22:08:31<8:29:16,  1.92s/it]
Outer training loop:  68%|######8   | 34114/50000 [22:08:32<8:18:50,  1.88s/it]
Outer training loop:  68%|######8   | 34115/50000 [22:08:34<8:27:28,  1.92s/it]
Outer training loop:  68%|######8   | 34116/50000 [22:08:36<8:22:22,  1.90s/it]
Outer training loop:  68%|######8   | 34117/50000 [22:08:38<8:06:19,  1.84s/it]
Outer training loop:  68%|######8   | 34118/50000 [22:08:40<8:09:13,  1.85s/it]
Outer training loop:  68%|######8   | 34119/50000 [22:08:42<8:14:40,  1.87s/it]
Outer training loop:  68%|######8   | 34120/50000 [22:08:44<8:13:02,  1.86s/it]
Outer training loop:  68%|######8   | 34121/50000 [22:08:45<8:08:36,  1.85s/it]
Outer training loop:  68%|######8   | 34122/50000 [22:08:47<8:23:24,  1.90s/it]
Outer training loop:  68%|######8   | 34123/50000 [22:08:49<8:10:07,  1.85s/it]
Outer training loop:  68%|######8   | 34124/50000 [22:08:51<8:15:49,  1.87s/it]
Outer training loop:  68%|######8   | 34125/50000 [22:08:53<8:28:31,  1.92s/it]
Outer training loop:  68%|######8   | 34126/50000 [22:08:55<8:10:28,  1.85s/it]
Outer training loop:  68%|######8   | 34127/50000 [22:08:57<8:06:51,  1.84s/it]
Outer training loop:  68%|######8   | 34128/50000 [22:08:58<8:06:57,  1.84s/it]
Outer training loop:  68%|######8   | 34129/50000 [22:09:00<8:14:55,  1.87s/it]
Outer training loop:  68%|######8   | 34130/50000 [22:09:02<8:10:24,  1.85s/it]
Outer training loop:  68%|######8   | 34131/50000 [22:09:04<8:01:21,  1.82s/it]
Outer training loop:  68%|######8   | 34132/50000 [22:09:06<8:17:20,  1.88s/it]
Outer training loop:  68%|######8   | 34133/50000 [22:09:08<8:10:54,  1.86s/it]
Outer training loop:  68%|######8   | 34134/50000 [22:09:10<8:03:38,  1.83s/it]
Outer training loop:  68%|######8   | 34135/50000 [22:09:11<8:14:25,  1.87s/it]
Outer training loop:  68%|######8   | 34136/50000 [22:09:13<8:25:08,  1.91s/it]
Outer training loop:  68%|######8   | 34137/50000 [22:09:15<8:28:30,  1.92s/it]
Outer training loop:  68%|######8   | 34138/50000 [22:09:17<8:22:38,  1.90s/it]
Outer training loop:  68%|######8   | 34139/50000 [22:09:19<8:31:54,  1.94s/it]
Outer training loop:  68%|######8   | 34140/50000 [22:09:21<8:25:50,  1.91s/it]
Outer training loop:  68%|######8   | 34141/50000 [22:09:23<8:19:50,  1.89s/it]
Outer training loop:  68%|######8   | 34142/50000 [22:09:25<8:40:01,  1.97s/it]
Outer training loop:  68%|######8   | 34143/50000 [22:09:27<8:23:09,  1.90s/it]
Outer training loop:  68%|######8   | 34144/50000 [22:09:29<8:16:54,  1.88s/it]
Outer training loop:  68%|######8   | 34145/50000 [22:09:31<8:30:28,  1.93s/it]
Outer training loop:  68%|######8   | 34146/50000 [22:09:33<8:28:27,  1.92s/it]
Outer training loop:  68%|######8   | 34147/50000 [22:09:35<8:18:50,  1.89s/it]
Outer training loop:  68%|######8   | 34148/50000 [22:09:36<8:09:19,  1.85s/it]
Outer training loop:  68%|######8   | 34149/50000 [22:09:38<8:25:31,  1.91s/it]
Outer training loop:  68%|######8   | 34150/50000 [22:09:40<8:31:32,  1.94s/it]
Outer training loop:  68%|######8   | 34151/50000 [22:09:42<8:25:56,  1.92s/it]
Outer training loop:  68%|######8   | 34152/50000 [22:09:44<8:38:27,  1.96s/it]
Outer training loop:  68%|######8   | 34153/50000 [22:09:46<8:33:46,  1.95s/it]
Outer training loop:  68%|######8   | 34154/50000 [22:09:48<8:24:38,  1.91s/it]
Outer training loop:  68%|######8   | 34155/50000 [22:09:50<8:35:50,  1.95s/it]
Outer training loop:  68%|######8   | 34156/50000 [22:09:52<8:22:02,  1.90s/it]
Outer training loop:  68%|######8   | 34157/50000 [22:09:54<8:07:17,  1.85s/it]
Outer training loop:  68%|######8   | 34158/50000 [22:09:55<7:56:31,  1.80s/it]
Outer training loop:  68%|######8   | 34159/50000 [22:09:57<8:20:06,  1.89s/it]
Outer training loop:  68%|######8   | 34160/50000 [22:09:59<8:16:00,  1.88s/it]
Outer training loop:  68%|######8   | 34161/50000 [22:10:01<8:02:38,  1.83s/it]
Outer training loop:  68%|######8   | 34162/50000 [22:10:03<8:14:46,  1.87s/it]
Outer training loop:  68%|######8   | 34163/50000 [22:10:05<8:12:36,  1.87s/it]
Outer training loop:  68%|######8   | 34164/50000 [22:10:07<8:13:24,  1.87s/it]
Outer training loop:  68%|######8   | 34165/50000 [22:10:08<8:07:20,  1.85s/it]
Outer training loop:  68%|######8   | 34166/50000 [22:10:10<8:24:11,  1.91s/it]
Outer training loop:  68%|######8   | 34167/50000 [22:10:12<8:13:43,  1.87s/it]
Outer training loop:  68%|######8   | 34168/50000 [22:10:14<8:22:44,  1.91s/it]
Outer training loop:  68%|######8   | 34169/50000 [22:10:16<8:36:01,  1.96s/it]
Outer training loop:  68%|######8   | 34170/50000 [22:10:18<8:27:16,  1.92s/it]
Outer training loop:  68%|######8   | 34171/50000 [22:10:20<8:28:57,  1.93s/it]
Outer training loop:  68%|######8   | 34172/50000 [22:10:22<8:41:29,  1.98s/it]
Outer training loop:  68%|######8   | 34173/50000 [22:10:24<8:38:17,  1.96s/it]
Outer training loop:  68%|######8   | 34174/50000 [22:10:26<8:32:33,  1.94s/it]
Outer training loop:  68%|######8   | 34175/50000 [22:10:28<8:28:00,  1.93s/it]
Outer training loop:  68%|######8   | 34176/50000 [22:10:30<8:36:25,  1.96s/it]
Outer training loop:  68%|######8   | 34177/50000 [22:10:32<8:27:38,  1.92s/it]
Outer training loop:  68%|######8   | 34178/50000 [22:10:34<8:17:53,  1.89s/it]
Outer training loop:  68%|######8   | 34179/50000 [22:10:36<8:29:07,  1.93s/it]
Outer training loop:  68%|######8   | 34180/50000 [22:10:38<8:26:31,  1.92s/it]
Outer training loop:  68%|######8   | 34181/50000 [22:10:39<8:20:01,  1.90s/it]
Outer training loop:  68%|######8   | 34182/50000 [22:10:41<8:23:47,  1.91s/it]
Outer training loop:  68%|######8   | 34183/50000 [22:10:43<8:16:30,  1.88s/it]
Outer training loop:  68%|######8   | 34184/50000 [22:10:45<8:21:48,  1.90s/it]
Outer training loop:  68%|######8   | 34185/50000 [22:10:47<8:15:28,  1.88s/it]
Outer training loop:  68%|######8   | 34186/50000 [22:10:49<8:23:45,  1.91s/it]
Outer training loop:  68%|######8   | 34187/50000 [22:10:51<8:40:26,  1.97s/it]
Outer training loop:  68%|######8   | 34188/50000 [22:10:53<8:31:19,  1.94s/it]
Outer training loop:  68%|######8   | 34189/50000 [22:10:55<8:35:16,  1.96s/it]
Outer training loop:  68%|######8   | 34190/50000 [22:10:57<8:23:14,  1.91s/it]
Outer training loop:  68%|######8   | 34191/50000 [22:10:59<8:32:25,  1.94s/it]
Outer training loop:  68%|######8   | 34192/50000 [22:11:01<8:40:29,  1.98s/it]
Outer training loop:  68%|######8   | 34193/50000 [22:11:03<8:29:56,  1.94s/it]
Outer training loop:  68%|######8   | 34194/50000 [22:11:04<8:26:07,  1.92s/it]
Outer training loop:  68%|######8   | 34195/50000 [22:11:06<8:22:23,  1.91s/it]
Outer training loop:  68%|######8   | 34196/50000 [22:11:08<8:30:46,  1.94s/it]
Outer training loop:  68%|######8   | 34197/50000 [22:11:10<8:22:43,  1.91s/it]
Outer training loop:  68%|######8   | 34198/50000 [22:11:12<8:17:10,  1.89s/it]
Outer training loop:  68%|######8   | 34199/50000 [22:11:14<8:24:20,  1.92s/it]
Outer training loop:  68%|######8   | 34200/50000 [22:11:16<8:20:52,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  68%|######8   | 34201/50000 [22:11:23<15:41:30,  3.58s/it]
Outer training loop:  68%|######8   | 34202/50000 [22:11:25<13:31:16,  3.08s/it]
Outer training loop:  68%|######8   | 34203/50000 [22:11:27<11:57:46,  2.73s/it]
Outer training loop:  68%|######8   | 34204/50000 [22:11:29<10:57:06,  2.50s/it]
Outer training loop:  68%|######8   | 34205/50000 [22:11:31<10:01:14,  2.28s/it]
Outer training loop:  68%|######8   | 34206/50000 [22:11:33<9:25:43,  2.15s/it] 
Outer training loop:  68%|######8   | 34207/50000 [22:11:35<9:20:15,  2.13s/it]
Outer training loop:  68%|######8   | 34208/50000 [22:11:37<8:54:25,  2.03s/it]
Outer training loop:  68%|######8   | 34209/50000 [22:11:38<8:38:21,  1.97s/it]
Outer training loop:  68%|######8   | 34210/50000 [22:11:40<8:42:36,  1.99s/it]
Outer training loop:  68%|######8   | 34211/50000 [22:11:42<8:42:42,  1.99s/it]
Outer training loop:  68%|######8   | 34212/50000 [22:11:44<8:36:34,  1.96s/it]
Outer training loop:  68%|######8   | 34213/50000 [22:11:46<8:39:41,  1.98s/it]
Outer training loop:  68%|######8   | 34214/50000 [22:11:48<8:30:01,  1.94s/it]
Outer training loop:  68%|######8   | 34215/50000 [22:11:50<8:20:08,  1.90s/it]
Outer training loop:  68%|######8   | 34216/50000 [22:11:52<8:20:39,  1.90s/it]
Outer training loop:  68%|######8   | 34217/50000 [22:11:54<8:42:32,  1.99s/it]
Outer training loop:  68%|######8   | 34218/50000 [22:11:56<8:37:43,  1.97s/it]
Outer training loop:  68%|######8   | 34219/50000 [22:11:58<8:25:30,  1.92s/it]
Outer training loop:  68%|######8   | 34220/50000 [22:12:00<8:39:48,  1.98s/it]
Outer training loop:  68%|######8   | 34221/50000 [22:12:02<8:31:55,  1.95s/it]
Outer training loop:  68%|######8   | 34222/50000 [22:12:04<8:28:39,  1.93s/it]
Outer training loop:  68%|######8   | 34223/50000 [22:12:06<8:34:31,  1.96s/it]
Outer training loop:  68%|######8   | 34224/50000 [22:12:08<8:29:23,  1.94s/it]
Outer training loop:  68%|######8   | 34225/50000 [22:12:10<8:20:37,  1.90s/it]
Outer training loop:  68%|######8   | 34226/50000 [22:12:11<8:15:31,  1.88s/it]
Outer training loop:  68%|######8   | 34227/50000 [22:12:13<8:27:52,  1.93s/it]
Outer training loop:  68%|######8   | 34228/50000 [22:12:15<8:15:16,  1.88s/it]
Outer training loop:  68%|######8   | 34229/50000 [22:12:17<8:09:56,  1.86s/it]
Outer training loop:  68%|######8   | 34230/50000 [22:12:19<8:17:47,  1.89s/it]
Outer training loop:  68%|######8   | 34231/50000 [22:12:21<8:06:10,  1.85s/it]
Outer training loop:  68%|######8   | 34232/50000 [22:12:23<8:07:10,  1.85s/it]
Outer training loop:  68%|######8   | 34233/50000 [22:12:25<8:23:17,  1.92s/it]
Outer training loop:  68%|######8   | 34234/50000 [22:12:27<8:30:03,  1.94s/it]
Outer training loop:  68%|######8   | 34235/50000 [22:12:28<8:25:14,  1.92s/it]
Outer training loop:  68%|######8   | 34236/50000 [22:12:30<8:24:49,  1.92s/it]
Outer training loop:  68%|######8   | 34237/50000 [22:12:33<8:42:52,  1.99s/it]
Outer training loop:  68%|######8   | 34238/50000 [22:12:34<8:31:49,  1.95s/it]
Outer training loop:  68%|######8   | 34239/50000 [22:12:36<8:24:54,  1.92s/it]
Outer training loop:  68%|######8   | 34240/50000 [22:12:38<8:38:51,  1.98s/it]
Outer training loop:  68%|######8   | 34241/50000 [22:12:40<8:21:55,  1.91s/it]
Outer training loop:  68%|######8   | 34242/50000 [22:12:42<8:16:02,  1.89s/it]
Outer training loop:  68%|######8   | 34243/50000 [22:12:44<8:32:46,  1.95s/it]
Outer training loop:  68%|######8   | 34244/50000 [22:12:46<8:10:41,  1.87s/it]
Outer training loop:  68%|######8   | 34245/50000 [22:12:48<8:02:35,  1.84s/it]
Outer training loop:  68%|######8   | 34246/50000 [22:12:49<7:59:35,  1.83s/it]
Outer training loop:  68%|######8   | 34247/50000 [22:12:51<8:26:30,  1.93s/it]
Outer training loop:  68%|######8   | 34248/50000 [22:12:53<8:21:12,  1.91s/it]
Outer training loop:  68%|######8   | 34249/50000 [22:12:55<8:12:53,  1.88s/it]
Outer training loop:  68%|######8   | 34250/50000 [22:12:57<8:27:15,  1.93s/it]
Outer training loop:  69%|######8   | 34251/50000 [22:12:59<8:13:28,  1.88s/it]
Outer training loop:  69%|######8   | 34252/50000 [22:13:01<8:09:22,  1.86s/it]
Outer training loop:  69%|######8   | 34253/50000 [22:13:03<8:08:39,  1.86s/it]
Outer training loop:  69%|######8   | 34254/50000 [22:13:05<8:33:18,  1.96s/it]
Outer training loop:  69%|######8   | 34255/50000 [22:13:07<8:17:24,  1.90s/it]
Outer training loop:  69%|######8   | 34256/50000 [22:13:08<8:13:48,  1.88s/it]
Outer training loop:  69%|######8   | 34257/50000 [22:13:10<8:20:16,  1.91s/it]
Outer training loop:  69%|######8   | 34258/50000 [22:13:12<8:08:08,  1.86s/it]
Outer training loop:  69%|######8   | 34259/50000 [22:13:14<8:03:32,  1.84s/it]
Outer training loop:  69%|######8   | 34260/50000 [22:13:16<7:52:23,  1.80s/it]
Outer training loop:  69%|######8   | 34261/50000 [22:13:18<8:15:00,  1.89s/it]
Outer training loop:  69%|######8   | 34262/50000 [22:13:20<8:18:01,  1.90s/it]
Outer training loop:  69%|######8   | 34263/50000 [22:13:22<8:14:29,  1.89s/it]
Outer training loop:  69%|######8   | 34264/50000 [22:13:24<8:28:27,  1.94s/it]
Outer training loop:  69%|######8   | 34265/50000 [22:13:25<8:25:07,  1.93s/it]
Outer training loop:  69%|######8   | 34266/50000 [22:13:27<8:19:57,  1.91s/it]
Outer training loop:  69%|######8   | 34267/50000 [22:13:29<8:22:07,  1.91s/it]
Outer training loop:  69%|######8   | 34268/50000 [22:13:31<8:17:10,  1.90s/it]
Outer training loop:  69%|######8   | 34269/50000 [22:13:33<8:14:47,  1.89s/it]
Outer training loop:  69%|######8   | 34270/50000 [22:13:35<8:30:00,  1.95s/it]
Outer training loop:  69%|######8   | 34271/50000 [22:13:37<8:23:21,  1.92s/it]
Outer training loop:  69%|######8   | 34272/50000 [22:13:39<8:11:08,  1.87s/it]
Outer training loop:  69%|######8   | 34273/50000 [22:13:41<8:06:46,  1.86s/it]
Outer training loop:  69%|######8   | 34274/50000 [22:13:43<8:23:55,  1.92s/it]
Outer training loop:  69%|######8   | 34275/50000 [22:13:44<8:12:37,  1.88s/it]
Outer training loop:  69%|######8   | 34276/50000 [22:13:46<8:19:11,  1.90s/it]
Outer training loop:  69%|######8   | 34277/50000 [22:13:48<8:18:43,  1.90s/it]
Outer training loop:  69%|######8   | 34278/50000 [22:13:50<8:23:09,  1.92s/it]
Outer training loop:  69%|######8   | 34279/50000 [22:13:52<8:00:35,  1.83s/it]
Outer training loop:  69%|######8   | 34280/50000 [22:13:54<8:16:40,  1.90s/it]
Outer training loop:  69%|######8   | 34281/50000 [22:13:56<8:16:24,  1.89s/it]
Outer training loop:  69%|######8   | 34282/50000 [22:13:58<8:11:41,  1.88s/it]
Outer training loop:  69%|######8   | 34283/50000 [22:13:59<8:08:11,  1.86s/it]
Outer training loop:  69%|######8   | 34284/50000 [22:14:02<8:45:23,  2.01s/it]
Outer training loop:  69%|######8   | 34285/50000 [22:14:04<8:31:40,  1.95s/it]
Outer training loop:  69%|######8   | 34286/50000 [22:14:05<8:15:47,  1.89s/it]
Outer training loop:  69%|######8   | 34287/50000 [22:14:07<8:31:27,  1.95s/it]
Outer training loop:  69%|######8   | 34288/50000 [22:14:09<8:16:58,  1.90s/it]
Outer training loop:  69%|######8   | 34289/50000 [22:14:11<8:04:47,  1.85s/it]
Outer training loop:  69%|######8   | 34290/50000 [22:14:13<8:19:28,  1.91s/it]
Outer training loop:  69%|######8   | 34291/50000 [22:14:15<8:09:16,  1.87s/it]
Outer training loop:  69%|######8   | 34292/50000 [22:14:17<8:08:37,  1.87s/it]
Outer training loop:  69%|######8   | 34293/50000 [22:14:18<8:02:53,  1.84s/it]
Outer training loop:  69%|######8   | 34294/50000 [22:14:20<8:16:36,  1.90s/it]
Outer training loop:  69%|######8   | 34295/50000 [22:14:22<8:06:32,  1.86s/it]
Outer training loop:  69%|######8   | 34296/50000 [22:14:24<8:03:58,  1.85s/it]
Outer training loop:  69%|######8   | 34297/50000 [22:14:26<8:18:25,  1.90s/it]
Outer training loop:  69%|######8   | 34298/50000 [22:14:28<8:16:50,  1.90s/it]
Outer training loop:  69%|######8   | 34299/50000 [22:14:30<8:09:16,  1.87s/it]
Outer training loop:  69%|######8   | 34300/50000 [22:14:32<8:21:49,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  69%|######8   | 34301/50000 [22:14:39<15:21:38,  3.52s/it]
Outer training loop:  69%|######8   | 34302/50000 [22:14:41<13:10:39,  3.02s/it]
Outer training loop:  69%|######8   | 34303/50000 [22:14:43<11:35:16,  2.66s/it]
Outer training loop:  69%|######8   | 34304/50000 [22:14:45<10:51:21,  2.49s/it]
Outer training loop:  69%|######8   | 34305/50000 [22:14:47<9:58:57,  2.29s/it] 
Outer training loop:  69%|######8   | 34306/50000 [22:14:48<9:21:46,  2.15s/it]
Outer training loop:  69%|######8   | 34307/50000 [22:14:50<8:59:45,  2.06s/it]
Outer training loop:  69%|######8   | 34308/50000 [22:14:52<8:56:58,  2.05s/it]
Outer training loop:  69%|######8   | 34309/50000 [22:14:54<8:33:56,  1.97s/it]
Outer training loop:  69%|######8   | 34310/50000 [22:14:56<8:20:38,  1.91s/it]
Outer training loop:  69%|######8   | 34311/50000 [22:14:58<8:40:05,  1.99s/it]
Outer training loop:  69%|######8   | 34312/50000 [22:15:00<8:27:11,  1.94s/it]
Outer training loop:  69%|######8   | 34313/50000 [22:15:02<8:19:31,  1.91s/it]
Outer training loop:  69%|######8   | 34314/50000 [22:15:03<8:05:22,  1.86s/it]
Outer training loop:  69%|######8   | 34315/50000 [22:15:06<8:26:23,  1.94s/it]
Outer training loop:  69%|######8   | 34316/50000 [22:15:07<8:09:22,  1.87s/it]
Outer training loop:  69%|######8   | 34317/50000 [22:15:09<8:03:25,  1.85s/it]
Outer training loop:  69%|######8   | 34318/50000 [22:15:11<8:13:35,  1.89s/it]
Outer training loop:  69%|######8   | 34319/50000 [22:15:13<8:07:35,  1.87s/it]
Outer training loop:  69%|######8   | 34320/50000 [22:15:15<8:02:23,  1.85s/it]
Outer training loop:  69%|######8   | 34321/50000 [22:15:17<8:16:41,  1.90s/it]
Outer training loop:  69%|######8   | 34322/50000 [22:15:18<8:01:08,  1.84s/it]
Outer training loop:  69%|######8   | 34323/50000 [22:15:20<7:58:52,  1.83s/it]
Outer training loop:  69%|######8   | 34324/50000 [22:15:22<8:06:30,  1.86s/it]
Outer training loop:  69%|######8   | 34325/50000 [22:15:24<8:22:05,  1.92s/it]
Outer training loop:  69%|######8   | 34326/50000 [22:15:26<8:23:21,  1.93s/it]
Outer training loop:  69%|######8   | 34327/50000 [22:15:28<8:18:42,  1.91s/it]
Outer training loop:  69%|######8   | 34328/50000 [22:15:30<8:31:40,  1.96s/it]
Outer training loop:  69%|######8   | 34329/50000 [22:15:32<8:26:16,  1.94s/it]
Outer training loop:  69%|######8   | 34330/50000 [22:15:34<8:13:05,  1.89s/it]
Outer training loop:  69%|######8   | 34331/50000 [22:15:36<8:38:36,  1.99s/it]
Outer training loop:  69%|######8   | 34332/50000 [22:15:38<8:27:52,  1.94s/it]
Outer training loop:  69%|######8   | 34333/50000 [22:15:40<8:19:11,  1.91s/it]
Outer training loop:  69%|######8   | 34334/50000 [22:15:41<8:09:12,  1.87s/it]
Outer training loop:  69%|######8   | 34335/50000 [22:15:44<8:22:29,  1.92s/it]
Outer training loop:  69%|######8   | 34336/50000 [22:15:45<8:14:03,  1.89s/it]
Outer training loop:  69%|######8   | 34337/50000 [22:15:47<8:02:32,  1.85s/it]
Outer training loop:  69%|######8   | 34338/50000 [22:15:49<8:00:20,  1.84s/it]
Outer training loop:  69%|######8   | 34339/50000 [22:15:51<8:10:24,  1.88s/it]
Outer training loop:  69%|######8   | 34340/50000 [22:15:53<8:03:23,  1.85s/it]
Outer training loop:  69%|######8   | 34341/50000 [22:15:54<7:58:03,  1.83s/it]
Outer training loop:  69%|######8   | 34342/50000 [22:15:57<8:24:46,  1.93s/it]
Outer training loop:  69%|######8   | 34343/50000 [22:15:58<8:16:43,  1.90s/it]
Outer training loop:  69%|######8   | 34344/50000 [22:16:00<8:06:00,  1.86s/it]
Outer training loop:  69%|######8   | 34345/50000 [22:16:02<8:06:34,  1.86s/it]
Outer training loop:  69%|######8   | 34346/50000 [22:16:04<8:16:25,  1.90s/it]
Outer training loop:  69%|######8   | 34347/50000 [22:16:06<8:09:29,  1.88s/it]
Outer training loop:  69%|######8   | 34348/50000 [22:16:08<7:57:31,  1.83s/it]
Outer training loop:  69%|######8   | 34349/50000 [22:16:10<8:18:46,  1.91s/it]
Outer training loop:  69%|######8   | 34350/50000 [22:16:11<8:05:08,  1.86s/it]
Outer training loop:  69%|######8   | 34351/50000 [22:16:13<8:02:42,  1.85s/it]
Outer training loop:  69%|######8   | 34352/50000 [22:16:15<8:06:03,  1.86s/it]
Outer training loop:  69%|######8   | 34353/50000 [22:16:17<8:21:29,  1.92s/it]
Outer training loop:  69%|######8   | 34354/50000 [22:16:19<8:22:05,  1.93s/it]
Outer training loop:  69%|######8   | 34355/50000 [22:16:21<8:25:11,  1.94s/it]
Outer training loop:  69%|######8   | 34356/50000 [22:16:23<8:29:15,  1.95s/it]
Outer training loop:  69%|######8   | 34357/50000 [22:16:25<8:31:07,  1.96s/it]
Outer training loop:  69%|######8   | 34358/50000 [22:16:27<8:17:21,  1.91s/it]
Outer training loop:  69%|######8   | 34359/50000 [22:16:29<8:34:24,  1.97s/it]
Outer training loop:  69%|######8   | 34360/50000 [22:16:31<8:18:15,  1.91s/it]
Outer training loop:  69%|######8   | 34361/50000 [22:16:33<8:09:38,  1.88s/it]
Outer training loop:  69%|######8   | 34362/50000 [22:16:34<8:11:28,  1.89s/it]
Outer training loop:  69%|######8   | 34363/50000 [22:16:36<8:21:33,  1.92s/it]
Outer training loop:  69%|######8   | 34364/50000 [22:16:38<8:05:21,  1.86s/it]
Outer training loop:  69%|######8   | 34365/50000 [22:16:40<8:14:20,  1.90s/it]
Outer training loop:  69%|######8   | 34366/50000 [22:16:42<8:35:11,  1.98s/it]
Outer training loop:  69%|######8   | 34367/50000 [22:16:44<8:26:35,  1.94s/it]
Outer training loop:  69%|######8   | 34368/50000 [22:16:46<8:27:57,  1.95s/it]
Outer training loop:  69%|######8   | 34369/50000 [22:16:48<8:31:07,  1.96s/it]
Outer training loop:  69%|######8   | 34370/50000 [22:16:50<8:32:04,  1.97s/it]
Outer training loop:  69%|######8   | 34371/50000 [22:16:52<8:14:52,  1.90s/it]
Outer training loop:  69%|######8   | 34372/50000 [22:16:54<8:14:57,  1.90s/it]
Outer training loop:  69%|######8   | 34373/50000 [22:16:56<8:09:15,  1.88s/it]
Outer training loop:  69%|######8   | 34374/50000 [22:16:58<8:09:45,  1.88s/it]
Outer training loop:  69%|######8   | 34375/50000 [22:17:00<8:21:10,  1.92s/it]
Outer training loop:  69%|######8   | 34376/50000 [22:17:01<8:17:24,  1.91s/it]
Outer training loop:  69%|######8   | 34377/50000 [22:17:03<8:09:54,  1.88s/it]
Outer training loop:  69%|######8   | 34378/50000 [22:17:05<8:08:46,  1.88s/it]
Outer training loop:  69%|######8   | 34379/50000 [22:17:07<8:17:20,  1.91s/it]
Outer training loop:  69%|######8   | 34380/50000 [22:17:09<8:22:41,  1.93s/it]
Outer training loop:  69%|######8   | 34381/50000 [22:17:11<8:33:45,  1.97s/it]
Outer training loop:  69%|######8   | 34382/50000 [22:17:13<8:35:48,  1.98s/it]
Outer training loop:  69%|######8   | 34383/50000 [22:17:15<8:23:45,  1.94s/it]
Outer training loop:  69%|######8   | 34384/50000 [22:17:17<8:04:46,  1.86s/it]
Outer training loop:  69%|######8   | 34385/50000 [22:17:19<8:12:48,  1.89s/it]
Outer training loop:  69%|######8   | 34386/50000 [22:17:20<8:01:50,  1.85s/it]
Outer training loop:  69%|######8   | 34387/50000 [22:17:22<8:02:25,  1.85s/it]
Outer training loop:  69%|######8   | 34388/50000 [22:17:24<8:14:21,  1.90s/it]
Outer training loop:  69%|######8   | 34389/50000 [22:17:26<8:19:18,  1.92s/it]
Outer training loop:  69%|######8   | 34390/50000 [22:17:28<8:23:13,  1.93s/it]
Outer training loop:  69%|######8   | 34391/50000 [22:17:30<8:28:24,  1.95s/it]
Outer training loop:  69%|######8   | 34392/50000 [22:17:32<8:16:46,  1.91s/it]
Outer training loop:  69%|######8   | 34393/50000 [22:17:34<8:16:37,  1.91s/it]
Outer training loop:  69%|######8   | 34394/50000 [22:17:36<8:14:25,  1.90s/it]
Outer training loop:  69%|######8   | 34395/50000 [22:17:38<8:27:33,  1.95s/it]
Outer training loop:  69%|######8   | 34396/50000 [22:17:40<8:25:02,  1.94s/it]
Outer training loop:  69%|######8   | 34397/50000 [22:17:42<8:10:21,  1.89s/it]
Outer training loop:  69%|######8   | 34398/50000 [22:17:44<8:24:47,  1.94s/it]
Outer training loop:  69%|######8   | 34399/50000 [22:17:45<8:20:28,  1.92s/it]
Outer training loop:  69%|######8   | 34400/50000 [22:17:47<8:17:02,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  69%|######8   | 34401/50000 [22:17:55<15:26:42,  3.56s/it]
Outer training loop:  69%|######8   | 34402/50000 [22:17:57<13:25:58,  3.10s/it]
Outer training loop:  69%|######8   | 34403/50000 [22:17:59<11:47:58,  2.72s/it]
Outer training loop:  69%|######8   | 34404/50000 [22:18:00<10:34:32,  2.44s/it]
Outer training loop:  69%|######8   | 34405/50000 [22:18:02<10:00:08,  2.31s/it]
Outer training loop:  69%|######8   | 34406/50000 [22:18:04<9:20:46,  2.16s/it] 
Outer training loop:  69%|######8   | 34407/50000 [22:18:06<8:58:08,  2.07s/it]
Outer training loop:  69%|######8   | 34408/50000 [22:18:08<8:41:01,  2.00s/it]
Outer training loop:  69%|######8   | 34409/50000 [22:18:10<8:55:36,  2.06s/it]
Outer training loop:  69%|######8   | 34410/50000 [22:18:12<8:38:52,  2.00s/it]
Outer training loop:  69%|######8   | 34411/50000 [22:18:14<8:25:49,  1.95s/it]
Outer training loop:  69%|######8   | 34412/50000 [22:18:16<8:29:49,  1.96s/it]
Outer training loop:  69%|######8   | 34413/50000 [22:18:18<8:10:34,  1.89s/it]
Outer training loop:  69%|######8   | 34414/50000 [22:18:19<8:10:46,  1.89s/it]
Outer training loop:  69%|######8   | 34415/50000 [22:18:21<8:12:36,  1.90s/it]
Outer training loop:  69%|######8   | 34416/50000 [22:18:23<8:25:11,  1.95s/it]
Outer training loop:  69%|######8   | 34417/50000 [22:18:25<8:22:12,  1.93s/it]
Outer training loop:  69%|######8   | 34418/50000 [22:18:27<8:01:30,  1.85s/it]
Outer training loop:  69%|######8   | 34419/50000 [22:18:29<8:12:49,  1.90s/it]
Outer training loop:  69%|######8   | 34420/50000 [22:18:31<7:58:34,  1.84s/it]
Outer training loop:  69%|######8   | 34421/50000 [22:18:32<7:53:27,  1.82s/it]
Outer training loop:  69%|######8   | 34422/50000 [22:18:35<8:11:04,  1.89s/it]
Outer training loop:  69%|######8   | 34423/50000 [22:18:36<8:05:25,  1.87s/it]
Outer training loop:  69%|######8   | 34424/50000 [22:18:38<8:03:30,  1.86s/it]
Outer training loop:  69%|######8   | 34425/50000 [22:18:40<8:05:46,  1.87s/it]
Outer training loop:  69%|######8   | 34426/50000 [22:18:42<8:08:43,  1.88s/it]
Outer training loop:  69%|######8   | 34427/50000 [22:18:44<8:04:29,  1.87s/it]
Outer training loop:  69%|######8   | 34428/50000 [22:18:46<8:17:26,  1.92s/it]
Outer training loop:  69%|######8   | 34429/50000 [22:18:48<8:23:52,  1.94s/it]
Outer training loop:  69%|######8   | 34430/50000 [22:18:50<8:11:38,  1.89s/it]
Outer training loop:  69%|######8   | 34431/50000 [22:18:51<7:54:49,  1.83s/it]
Outer training loop:  69%|######8   | 34432/50000 [22:18:53<8:09:18,  1.89s/it]
Outer training loop:  69%|######8   | 34433/50000 [22:18:55<8:01:07,  1.85s/it]
Outer training loop:  69%|######8   | 34434/50000 [22:18:57<8:00:24,  1.85s/it]
Outer training loop:  69%|######8   | 34435/50000 [22:18:59<8:03:00,  1.86s/it]
Outer training loop:  69%|######8   | 34436/50000 [22:19:01<8:13:29,  1.90s/it]
Outer training loop:  69%|######8   | 34437/50000 [22:19:03<8:09:08,  1.89s/it]
Outer training loop:  69%|######8   | 34438/50000 [22:19:04<8:00:53,  1.85s/it]
Outer training loop:  69%|######8   | 34439/50000 [22:19:06<8:09:49,  1.89s/it]
Outer training loop:  69%|######8   | 34440/50000 [22:19:08<8:06:12,  1.87s/it]
Outer training loop:  69%|######8   | 34441/50000 [22:19:10<8:02:55,  1.86s/it]
Outer training loop:  69%|######8   | 34442/50000 [22:19:12<7:53:19,  1.83s/it]
Outer training loop:  69%|######8   | 34443/50000 [22:19:14<8:10:56,  1.89s/it]
Outer training loop:  69%|######8   | 34444/50000 [22:19:16<7:58:44,  1.85s/it]
Outer training loop:  69%|######8   | 34445/50000 [22:19:17<7:52:18,  1.82s/it]
Outer training loop:  69%|######8   | 34446/50000 [22:19:19<8:06:13,  1.88s/it]
Outer training loop:  69%|######8   | 34447/50000 [22:19:21<7:57:12,  1.84s/it]
Outer training loop:  69%|######8   | 34448/50000 [22:19:23<7:53:04,  1.83s/it]
Outer training loop:  69%|######8   | 34449/50000 [22:19:25<7:53:10,  1.83s/it]
Outer training loop:  69%|######8   | 34450/50000 [22:19:27<8:09:45,  1.89s/it]
Outer training loop:  69%|######8   | 34451/50000 [22:19:29<8:18:09,  1.92s/it]
Outer training loop:  69%|######8   | 34452/50000 [22:19:31<8:09:35,  1.89s/it]
Outer training loop:  69%|######8   | 34453/50000 [22:19:33<8:17:34,  1.92s/it]
Outer training loop:  69%|######8   | 34454/50000 [22:19:34<8:13:49,  1.91s/it]
Outer training loop:  69%|######8   | 34455/50000 [22:19:36<8:09:21,  1.89s/it]
Outer training loop:  69%|######8   | 34456/50000 [22:19:38<8:06:31,  1.88s/it]
Outer training loop:  69%|######8   | 34457/50000 [22:19:40<8:19:22,  1.93s/it]
Outer training loop:  69%|######8   | 34458/50000 [22:19:42<8:10:34,  1.89s/it]
Outer training loop:  69%|######8   | 34459/50000 [22:19:44<7:59:25,  1.85s/it]
Outer training loop:  69%|######8   | 34460/50000 [22:19:46<8:02:58,  1.86s/it]
Outer training loop:  69%|######8   | 34461/50000 [22:19:47<7:58:47,  1.85s/it]
Outer training loop:  69%|######8   | 34462/50000 [22:19:49<8:02:51,  1.86s/it]
Outer training loop:  69%|######8   | 34463/50000 [22:19:51<8:14:40,  1.91s/it]
Outer training loop:  69%|######8   | 34464/50000 [22:19:53<8:08:28,  1.89s/it]
Outer training loop:  69%|######8   | 34465/50000 [22:19:55<7:58:22,  1.85s/it]
Outer training loop:  69%|######8   | 34466/50000 [22:19:57<7:57:30,  1.84s/it]
Outer training loop:  69%|######8   | 34467/50000 [22:19:59<8:10:44,  1.90s/it]
Outer training loop:  69%|######8   | 34468/50000 [22:20:01<8:02:42,  1.86s/it]
Outer training loop:  69%|######8   | 34469/50000 [22:20:02<8:00:47,  1.86s/it]
Outer training loop:  69%|######8   | 34470/50000 [22:20:04<8:11:46,  1.90s/it]
Outer training loop:  69%|######8   | 34471/50000 [22:20:06<8:14:01,  1.91s/it]
Outer training loop:  69%|######8   | 34472/50000 [22:20:08<8:13:18,  1.91s/it]
Outer training loop:  69%|######8   | 34473/50000 [22:20:10<8:10:36,  1.90s/it]
Outer training loop:  69%|######8   | 34474/50000 [22:20:12<8:29:16,  1.97s/it]
Outer training loop:  69%|######8   | 34475/50000 [22:20:14<8:22:55,  1.94s/it]
Outer training loop:  69%|######8   | 34476/50000 [22:20:16<8:13:50,  1.91s/it]
Outer training loop:  69%|######8   | 34477/50000 [22:20:18<8:09:51,  1.89s/it]
Outer training loop:  69%|######8   | 34478/50000 [22:20:20<8:19:58,  1.93s/it]
Outer training loop:  69%|######8   | 34479/50000 [22:20:22<8:12:46,  1.90s/it]
Outer training loop:  69%|######8   | 34480/50000 [22:20:24<8:08:47,  1.89s/it]
Outer training loop:  69%|######8   | 34481/50000 [22:20:26<8:32:24,  1.98s/it]
Outer training loop:  69%|######8   | 34482/50000 [22:20:28<8:27:10,  1.96s/it]
Outer training loop:  69%|######8   | 34483/50000 [22:20:30<8:17:04,  1.92s/it]
Outer training loop:  69%|######8   | 34484/50000 [22:20:32<8:21:06,  1.94s/it]
Outer training loop:  69%|######8   | 34485/50000 [22:20:33<8:10:29,  1.90s/it]
Outer training loop:  69%|######8   | 34486/50000 [22:20:35<8:10:03,  1.90s/it]
Outer training loop:  69%|######8   | 34487/50000 [22:20:37<8:18:40,  1.93s/it]
Outer training loop:  69%|######8   | 34488/50000 [22:20:39<8:07:45,  1.89s/it]
Outer training loop:  69%|######8   | 34489/50000 [22:20:41<8:06:32,  1.88s/it]
Outer training loop:  69%|######8   | 34490/50000 [22:20:43<8:08:22,  1.89s/it]
Outer training loop:  69%|######8   | 34491/50000 [22:20:45<8:05:42,  1.88s/it]
Outer training loop:  69%|######8   | 34492/50000 [22:20:46<7:47:53,  1.81s/it]
Outer training loop:  69%|######8   | 34493/50000 [22:20:48<7:47:12,  1.81s/it]
Outer training loop:  69%|######8   | 34494/50000 [22:20:50<8:07:43,  1.89s/it]
Outer training loop:  69%|######8   | 34495/50000 [22:20:52<8:09:58,  1.90s/it]
Outer training loop:  69%|######8   | 34496/50000 [22:20:54<8:06:03,  1.88s/it]
Outer training loop:  69%|######8   | 34497/50000 [22:20:56<8:23:25,  1.95s/it]
Outer training loop:  69%|######8   | 34498/50000 [22:20:58<8:14:51,  1.92s/it]
Outer training loop:  69%|######8   | 34499/50000 [22:21:00<8:06:32,  1.88s/it]
Outer training loop:  69%|######9   | 34500/50000 [22:21:01<7:59:48,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.87s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:39,  1.95s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:36,  1.93s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:32,  1.91s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.90s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:25,  1.94s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.96s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.88s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.88s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:06,  1.86s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:59,  1.83s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:56,  1.82s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:56,  1.84s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.88s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:54,  1.97s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.88s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:41,  1.85s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:36,  1.79s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:40,  1.94s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:26,  1.81s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:25,  1.81s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.87s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:22,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.81s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:16,  1.82s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:16,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:13,  1.84s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:10,  1.86s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:09,  1.87s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:29<01:03,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:01,  1.82s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:02,  1.95s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:42<00:50,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:47,  1.82s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:45,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:46,  1.95s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:44,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.90s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:32,  1.80s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:32,  1.94s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:28,  1.87s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.82s/it]

Evaluating policy:  84%|########4 | 70/83 [02:10<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:19,  1.80s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.90s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:14,  2.03s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.97s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.97s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.96s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.92s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.88s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:20,  1.72s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:19,  1.73s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:34,  1.93s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.92s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.90s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:28,  1.96s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.90s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:18,  1.88s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.88s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.94s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:59,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.90s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:51,  1.83s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:50,  1.85s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:46,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:42,  1.84s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:40,  1.82s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:38,  1.81s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:41,  1.92s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.85s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:32,  1.85s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:31,  1.94s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:28,  1.93s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.86s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:20,  1.82s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.82s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:20,  1.98s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:17,  1.93s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:14,  1.90s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:13,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:09,  1.87s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:07,  1.86s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:06,  1.97s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.92s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.89s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:53,  1.98s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:49,  1.92s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.90s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.84s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:37,  1.95s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:34,  1.92s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:32,  1.90s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.90s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.88s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.86s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:21,  1.95s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.90s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.81s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:10,  1.81s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:08,  1.76s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.82s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.82s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.78s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.94s/it]

Outer training loop:  69%|######9   | 34501/50000 [22:26:21<418:42:46, 97.26s/it]
Outer training loop:  69%|######9   | 34502/50000 [22:26:23<295:22:45, 68.61s/it]
Outer training loop:  69%|######9   | 34503/50000 [22:26:25<209:04:00, 48.57s/it]
Outer training loop:  69%|######9   | 34504/50000 [22:26:27<148:42:55, 34.55s/it]
Outer training loop:  69%|######9   | 34505/50000 [22:26:28<106:20:41, 24.71s/it]
Outer training loop:  69%|######9   | 34506/50000 [22:26:30<76:41:12, 17.82s/it] 
Outer training loop:  69%|######9   | 34507/50000 [22:26:32<55:56:05, 13.00s/it]
Outer training loop:  69%|######9   | 34508/50000 [22:26:34<41:32:28,  9.65s/it]
Outer training loop:  69%|######9   | 34509/50000 [22:26:36<32:24:42,  7.53s/it]
Outer training loop:  69%|######9   | 34510/50000 [22:26:38<24:57:33,  5.80s/it]
Outer training loop:  69%|######9   | 34511/50000 [22:26:40<19:41:29,  4.58s/it]
Outer training loop:  69%|######9   | 34512/50000 [22:26:42<16:06:36,  3.74s/it]
Outer training loop:  69%|######9   | 34513/50000 [22:26:44<13:56:01,  3.24s/it]
Outer training loop:  69%|######9   | 34514/50000 [22:26:46<12:02:50,  2.80s/it]
Outer training loop:  69%|######9   | 34515/50000 [22:26:47<10:49:18,  2.52s/it]
Outer training loop:  69%|######9   | 34516/50000 [22:26:49<9:59:27,  2.32s/it] 
Outer training loop:  69%|######9   | 34517/50000 [22:26:51<9:20:36,  2.17s/it]
Outer training loop:  69%|######9   | 34518/50000 [22:26:53<8:46:49,  2.04s/it]
Outer training loop:  69%|######9   | 34519/50000 [22:26:55<8:41:19,  2.02s/it]
Outer training loop:  69%|######9   | 34520/50000 [22:26:56<8:11:23,  1.90s/it]
Outer training loop:  69%|######9   | 34521/50000 [22:26:58<8:02:56,  1.87s/it]
Outer training loop:  69%|######9   | 34522/50000 [22:27:00<8:00:34,  1.86s/it]
Outer training loop:  69%|######9   | 34523/50000 [22:27:02<8:12:38,  1.91s/it]
Outer training loop:  69%|######9   | 34524/50000 [22:27:04<7:59:33,  1.86s/it]
Outer training loop:  69%|######9   | 34525/50000 [22:27:06<7:47:41,  1.81s/it]
Outer training loop:  69%|######9   | 34526/50000 [22:27:08<8:11:03,  1.90s/it]
Outer training loop:  69%|######9   | 34527/50000 [22:27:09<8:05:17,  1.88s/it]
Outer training loop:  69%|######9   | 34528/50000 [22:27:11<7:57:56,  1.85s/it]
Outer training loop:  69%|######9   | 34529/50000 [22:27:13<8:10:24,  1.90s/it]
Outer training loop:  69%|######9   | 34530/50000 [22:27:15<7:58:20,  1.86s/it]
Outer training loop:  69%|######9   | 34531/50000 [22:27:17<7:49:19,  1.82s/it]
Outer training loop:  69%|######9   | 34532/50000 [22:27:19<7:58:24,  1.86s/it]
Outer training loop:  69%|######9   | 34533/50000 [22:27:21<8:08:07,  1.89s/it]
Outer training loop:  69%|######9   | 34534/50000 [22:27:22<7:58:17,  1.86s/it]
Outer training loop:  69%|######9   | 34535/50000 [22:27:24<7:52:08,  1.83s/it]
Outer training loop:  69%|######9   | 34536/50000 [22:27:26<8:02:55,  1.87s/it]
Outer training loop:  69%|######9   | 34537/50000 [22:27:28<7:58:31,  1.86s/it]
Outer training loop:  69%|######9   | 34538/50000 [22:27:30<7:50:04,  1.82s/it]
Outer training loop:  69%|######9   | 34539/50000 [22:27:32<7:45:48,  1.81s/it]
Outer training loop:  69%|######9   | 34540/50000 [22:27:34<8:01:26,  1.87s/it]
Outer training loop:  69%|######9   | 34541/50000 [22:27:35<7:57:48,  1.85s/it]
Outer training loop:  69%|######9   | 34542/50000 [22:27:37<8:09:06,  1.90s/it]
Outer training loop:  69%|######9   | 34543/50000 [22:27:39<8:13:50,  1.92s/it]
Outer training loop:  69%|######9   | 34544/50000 [22:27:41<8:04:48,  1.88s/it]
Outer training loop:  69%|######9   | 34545/50000 [22:27:43<7:55:45,  1.85s/it]
Outer training loop:  69%|######9   | 34546/50000 [22:27:45<7:55:54,  1.85s/it]
Outer training loop:  69%|######9   | 34547/50000 [22:27:47<8:05:21,  1.88s/it]
Outer training loop:  69%|######9   | 34548/50000 [22:27:48<7:53:46,  1.84s/it]
Outer training loop:  69%|######9   | 34549/50000 [22:27:50<8:01:13,  1.87s/it]
Outer training loop:  69%|######9   | 34550/50000 [22:27:52<8:07:16,  1.89s/it]
Outer training loop:  69%|######9   | 34551/50000 [22:27:54<8:01:02,  1.87s/it]
Outer training loop:  69%|######9   | 34552/50000 [22:27:56<7:45:01,  1.81s/it]
Outer training loop:  69%|######9   | 34553/50000 [22:27:58<7:42:24,  1.80s/it]
Outer training loop:  69%|######9   | 34554/50000 [22:28:00<7:56:33,  1.85s/it]
Outer training loop:  69%|######9   | 34555/50000 [22:28:01<7:57:34,  1.86s/it]
Outer training loop:  69%|######9   | 34556/50000 [22:28:03<7:49:08,  1.82s/it]
Outer training loop:  69%|######9   | 34557/50000 [22:28:05<7:56:20,  1.85s/it]
Outer training loop:  69%|######9   | 34558/50000 [22:28:07<7:43:25,  1.80s/it]
Outer training loop:  69%|######9   | 34559/50000 [22:28:09<7:57:46,  1.86s/it]
Outer training loop:  69%|######9   | 34560/50000 [22:28:10<7:48:10,  1.82s/it]
Outer training loop:  69%|######9   | 34561/50000 [22:28:13<8:04:32,  1.88s/it]
Outer training loop:  69%|######9   | 34562/50000 [22:28:14<7:57:39,  1.86s/it]
Outer training loop:  69%|######9   | 34563/50000 [22:28:16<7:52:50,  1.84s/it]
Outer training loop:  69%|######9   | 34564/50000 [22:28:18<7:59:30,  1.86s/it]
Outer training loop:  69%|######9   | 34565/50000 [22:28:20<8:04:40,  1.88s/it]
Outer training loop:  69%|######9   | 34566/50000 [22:28:22<8:01:10,  1.87s/it]
Outer training loop:  69%|######9   | 34567/50000 [22:28:24<8:11:18,  1.91s/it]
Outer training loop:  69%|######9   | 34568/50000 [22:28:26<7:59:02,  1.86s/it]
Outer training loop:  69%|######9   | 34569/50000 [22:28:27<7:57:48,  1.86s/it]
Outer training loop:  69%|######9   | 34570/50000 [22:28:29<8:02:49,  1.88s/it]
Outer training loop:  69%|######9   | 34571/50000 [22:28:31<8:00:53,  1.87s/it]
Outer training loop:  69%|######9   | 34572/50000 [22:28:33<7:50:05,  1.83s/it]
Outer training loop:  69%|######9   | 34573/50000 [22:28:35<7:50:18,  1.83s/it]
Outer training loop:  69%|######9   | 34574/50000 [22:28:37<7:54:53,  1.85s/it]
Outer training loop:  69%|######9   | 34575/50000 [22:28:38<7:51:43,  1.83s/it]
Outer training loop:  69%|######9   | 34576/50000 [22:28:40<7:51:09,  1.83s/it]
Outer training loop:  69%|######9   | 34577/50000 [22:28:42<7:58:11,  1.86s/it]
Outer training loop:  69%|######9   | 34578/50000 [22:28:44<8:11:45,  1.91s/it]
Outer training loop:  69%|######9   | 34579/50000 [22:28:46<7:53:20,  1.84s/it]
Outer training loop:  69%|######9   | 34580/50000 [22:28:48<7:51:42,  1.84s/it]
Outer training loop:  69%|######9   | 34581/50000 [22:28:50<8:02:57,  1.88s/it]
Outer training loop:  69%|######9   | 34582/50000 [22:28:51<7:53:46,  1.84s/it]
Outer training loop:  69%|######9   | 34583/50000 [22:28:53<7:50:28,  1.83s/it]
Outer training loop:  69%|######9   | 34584/50000 [22:28:55<8:02:59,  1.88s/it]
Outer training loop:  69%|######9   | 34585/50000 [22:28:57<8:00:15,  1.87s/it]
Outer training loop:  69%|######9   | 34586/50000 [22:28:59<7:40:22,  1.79s/it]
Outer training loop:  69%|######9   | 34587/50000 [22:29:01<8:01:13,  1.87s/it]
Outer training loop:  69%|######9   | 34588/50000 [22:29:02<7:48:43,  1.82s/it]
Outer training loop:  69%|######9   | 34589/50000 [22:29:04<7:50:09,  1.83s/it]
Outer training loop:  69%|######9   | 34590/50000 [22:29:06<7:49:54,  1.83s/it]
Outer training loop:  69%|######9   | 34591/50000 [22:29:08<8:00:48,  1.87s/it]
Outer training loop:  69%|######9   | 34592/50000 [22:29:10<7:57:47,  1.86s/it]
Outer training loop:  69%|######9   | 34593/50000 [22:29:12<7:50:21,  1.83s/it]
Outer training loop:  69%|######9   | 34594/50000 [22:29:14<8:02:48,  1.88s/it]
Outer training loop:  69%|######9   | 34595/50000 [22:29:16<8:00:02,  1.87s/it]
Outer training loop:  69%|######9   | 34596/50000 [22:29:17<7:56:33,  1.86s/it]
Outer training loop:  69%|######9   | 34597/50000 [22:29:19<8:03:14,  1.88s/it]
Outer training loop:  69%|######9   | 34598/50000 [22:29:21<8:03:43,  1.88s/it]
Outer training loop:  69%|######9   | 34599/50000 [22:29:23<7:52:20,  1.84s/it]
Outer training loop:  69%|######9   | 34600/50000 [22:29:25<7:54:35,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  69%|######9   | 34601/50000 [22:29:32<15:12:10,  3.55s/it]
Outer training loop:  69%|######9   | 34602/50000 [22:29:35<13:24:27,  3.13s/it]
Outer training loop:  69%|######9   | 34603/50000 [22:29:36<11:37:21,  2.72s/it]
Outer training loop:  69%|######9   | 34604/50000 [22:29:38<10:35:35,  2.48s/it]
Outer training loop:  69%|######9   | 34605/50000 [22:29:40<10:01:30,  2.34s/it]
Outer training loop:  69%|######9   | 34606/50000 [22:29:42<9:41:30,  2.27s/it] 
Outer training loop:  69%|######9   | 34607/50000 [22:29:44<9:00:51,  2.11s/it]
Outer training loop:  69%|######9   | 34608/50000 [22:29:46<8:55:23,  2.09s/it]
Outer training loop:  69%|######9   | 34609/50000 [22:29:48<8:35:34,  2.01s/it]
Outer training loop:  69%|######9   | 34610/50000 [22:29:50<8:31:21,  1.99s/it]
Outer training loop:  69%|######9   | 34611/50000 [22:29:52<8:19:03,  1.95s/it]
Outer training loop:  69%|######9   | 34612/50000 [22:29:54<8:22:29,  1.96s/it]
Outer training loop:  69%|######9   | 34613/50000 [22:29:56<8:14:12,  1.93s/it]
Outer training loop:  69%|######9   | 34614/50000 [22:29:57<8:00:50,  1.88s/it]
Outer training loop:  69%|######9   | 34615/50000 [22:29:59<8:08:37,  1.91s/it]
Outer training loop:  69%|######9   | 34616/50000 [22:30:01<8:11:43,  1.92s/it]
Outer training loop:  69%|######9   | 34617/50000 [22:30:03<8:14:24,  1.93s/it]
Outer training loop:  69%|######9   | 34618/50000 [22:30:05<8:18:18,  1.94s/it]
Outer training loop:  69%|######9   | 34619/50000 [22:30:07<8:25:51,  1.97s/it]
Outer training loop:  69%|######9   | 34620/50000 [22:30:09<8:09:24,  1.91s/it]
Outer training loop:  69%|######9   | 34621/50000 [22:30:11<8:24:48,  1.97s/it]
Outer training loop:  69%|######9   | 34622/50000 [22:30:13<8:13:12,  1.92s/it]
Outer training loop:  69%|######9   | 34623/50000 [22:30:15<8:08:01,  1.90s/it]
Outer training loop:  69%|######9   | 34624/50000 [22:30:17<8:00:20,  1.87s/it]
Outer training loop:  69%|######9   | 34625/50000 [22:30:19<8:06:44,  1.90s/it]
Outer training loop:  69%|######9   | 34626/50000 [22:30:20<8:01:18,  1.88s/it]
Outer training loop:  69%|######9   | 34627/50000 [22:30:22<7:49:10,  1.83s/it]
Outer training loop:  69%|######9   | 34628/50000 [22:30:24<7:55:45,  1.86s/it]
Outer training loop:  69%|######9   | 34629/50000 [22:30:26<8:01:56,  1.88s/it]
Outer training loop:  69%|######9   | 34630/50000 [22:30:28<8:01:47,  1.88s/it]
Outer training loop:  69%|######9   | 34631/50000 [22:30:30<8:00:02,  1.87s/it]
Outer training loop:  69%|######9   | 34632/50000 [22:30:31<7:57:52,  1.87s/it]
Outer training loop:  69%|######9   | 34633/50000 [22:30:33<7:54:08,  1.85s/it]
Outer training loop:  69%|######9   | 34634/50000 [22:30:35<8:02:54,  1.89s/it]
Outer training loop:  69%|######9   | 34635/50000 [22:30:37<8:12:08,  1.92s/it]
Outer training loop:  69%|######9   | 34636/50000 [22:30:39<8:04:58,  1.89s/it]
Outer training loop:  69%|######9   | 34637/50000 [22:30:41<8:02:05,  1.88s/it]
Outer training loop:  69%|######9   | 34638/50000 [22:30:43<8:13:33,  1.93s/it]
Outer training loop:  69%|######9   | 34639/50000 [22:30:45<8:08:50,  1.91s/it]
Outer training loop:  69%|######9   | 34640/50000 [22:30:47<8:05:34,  1.90s/it]
Outer training loop:  69%|######9   | 34641/50000 [22:30:49<8:22:53,  1.96s/it]
Outer training loop:  69%|######9   | 34642/50000 [22:30:51<8:10:25,  1.92s/it]
Outer training loop:  69%|######9   | 34643/50000 [22:30:52<7:56:33,  1.86s/it]
Outer training loop:  69%|######9   | 34644/50000 [22:30:54<7:49:17,  1.83s/it]
Outer training loop:  69%|######9   | 34645/50000 [22:30:56<7:57:48,  1.87s/it]
Outer training loop:  69%|######9   | 34646/50000 [22:30:58<7:59:20,  1.87s/it]
Outer training loop:  69%|######9   | 34647/50000 [22:31:00<7:59:20,  1.87s/it]
Outer training loop:  69%|######9   | 34648/50000 [22:31:02<7:59:48,  1.88s/it]
Outer training loop:  69%|######9   | 34649/50000 [22:31:04<7:52:04,  1.85s/it]
Outer training loop:  69%|######9   | 34650/50000 [22:31:05<7:56:47,  1.86s/it]
Outer training loop:  69%|######9   | 34651/50000 [22:31:07<7:50:35,  1.84s/it]
Outer training loop:  69%|######9   | 34652/50000 [22:31:09<8:06:10,  1.90s/it]
Outer training loop:  69%|######9   | 34653/50000 [22:31:11<8:03:29,  1.89s/it]
Outer training loop:  69%|######9   | 34654/50000 [22:31:13<8:02:38,  1.89s/it]
Outer training loop:  69%|######9   | 34655/50000 [22:31:15<8:12:49,  1.93s/it]
Outer training loop:  69%|######9   | 34656/50000 [22:31:17<8:16:25,  1.94s/it]
Outer training loop:  69%|######9   | 34657/50000 [22:31:19<8:09:18,  1.91s/it]
Outer training loop:  69%|######9   | 34658/50000 [22:31:21<8:32:06,  2.00s/it]
Outer training loop:  69%|######9   | 34659/50000 [22:31:23<8:16:18,  1.94s/it]
Outer training loop:  69%|######9   | 34660/50000 [22:31:25<8:22:32,  1.97s/it]
Outer training loop:  69%|######9   | 34661/50000 [22:31:27<8:13:33,  1.93s/it]
Outer training loop:  69%|######9   | 34662/50000 [22:31:29<8:23:34,  1.97s/it]
Outer training loop:  69%|######9   | 34663/50000 [22:31:31<8:12:25,  1.93s/it]
Outer training loop:  69%|######9   | 34664/50000 [22:31:32<8:04:02,  1.89s/it]
Outer training loop:  69%|######9   | 34665/50000 [22:31:34<8:14:22,  1.93s/it]
Outer training loop:  69%|######9   | 34666/50000 [22:31:36<8:08:10,  1.91s/it]
Outer training loop:  69%|######9   | 34667/50000 [22:31:38<8:03:49,  1.89s/it]
Outer training loop:  69%|######9   | 34668/50000 [22:31:40<8:08:12,  1.91s/it]
Outer training loop:  69%|######9   | 34669/50000 [22:31:42<8:16:46,  1.94s/it]
Outer training loop:  69%|######9   | 34670/50000 [22:31:44<8:06:57,  1.91s/it]
Outer training loop:  69%|######9   | 34671/50000 [22:31:46<8:03:10,  1.89s/it]
Outer training loop:  69%|######9   | 34672/50000 [22:31:48<8:16:39,  1.94s/it]
Outer training loop:  69%|######9   | 34673/50000 [22:31:50<8:19:34,  1.96s/it]
Outer training loop:  69%|######9   | 34674/50000 [22:31:52<8:06:34,  1.90s/it]
Outer training loop:  69%|######9   | 34675/50000 [22:31:53<7:54:30,  1.86s/it]
Outer training loop:  69%|######9   | 34676/50000 [22:31:56<8:14:22,  1.94s/it]
Outer training loop:  69%|######9   | 34677/50000 [22:31:57<8:08:08,  1.91s/it]
Outer training loop:  69%|######9   | 34678/50000 [22:31:59<8:08:52,  1.91s/it]
Outer training loop:  69%|######9   | 34679/50000 [22:32:01<8:13:03,  1.93s/it]
Outer training loop:  69%|######9   | 34680/50000 [22:32:03<7:58:04,  1.87s/it]
Outer training loop:  69%|######9   | 34681/50000 [22:32:05<8:04:17,  1.90s/it]
Outer training loop:  69%|######9   | 34682/50000 [22:32:07<8:05:26,  1.90s/it]
Outer training loop:  69%|######9   | 34683/50000 [22:32:09<7:58:03,  1.87s/it]
Outer training loop:  69%|######9   | 34684/50000 [22:32:11<7:55:55,  1.86s/it]
Outer training loop:  69%|######9   | 34685/50000 [22:32:13<8:11:56,  1.93s/it]
Outer training loop:  69%|######9   | 34686/50000 [22:32:14<7:59:32,  1.88s/it]
Outer training loop:  69%|######9   | 34687/50000 [22:32:16<7:52:37,  1.85s/it]
Outer training loop:  69%|######9   | 34688/50000 [22:32:18<8:05:03,  1.90s/it]
Outer training loop:  69%|######9   | 34689/50000 [22:32:20<8:12:25,  1.93s/it]
Outer training loop:  69%|######9   | 34690/50000 [22:32:22<7:57:46,  1.87s/it]
Outer training loop:  69%|######9   | 34691/50000 [22:32:24<7:57:00,  1.87s/it]
Outer training loop:  69%|######9   | 34692/50000 [22:32:26<8:09:16,  1.92s/it]
Outer training loop:  69%|######9   | 34693/50000 [22:32:28<8:15:04,  1.94s/it]
Outer training loop:  69%|######9   | 34694/50000 [22:32:29<7:55:29,  1.86s/it]
Outer training loop:  69%|######9   | 34695/50000 [22:32:32<8:11:22,  1.93s/it]
Outer training loop:  69%|######9   | 34696/50000 [22:32:33<7:54:36,  1.86s/it]
Outer training loop:  69%|######9   | 34697/50000 [22:32:35<8:01:19,  1.89s/it]
Outer training loop:  69%|######9   | 34698/50000 [22:32:37<8:07:27,  1.91s/it]
Outer training loop:  69%|######9   | 34699/50000 [22:32:39<7:55:02,  1.86s/it]
Outer training loop:  69%|######9   | 34700/50000 [22:32:41<7:53:03,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.90s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  69%|######9   | 34701/50000 [22:32:48<14:45:42,  3.47s/it]
Outer training loop:  69%|######9   | 34702/50000 [22:32:50<12:54:35,  3.04s/it]
Outer training loop:  69%|######9   | 34703/50000 [22:32:52<11:39:47,  2.74s/it]
Outer training loop:  69%|######9   | 34704/50000 [22:32:54<10:32:14,  2.48s/it]
Outer training loop:  69%|######9   | 34705/50000 [22:32:56<9:47:01,  2.30s/it] 
Outer training loop:  69%|######9   | 34706/50000 [22:32:58<9:25:05,  2.22s/it]
Outer training loop:  69%|######9   | 34707/50000 [22:33:00<8:52:50,  2.09s/it]
Outer training loop:  69%|######9   | 34708/50000 [22:33:02<8:36:10,  2.03s/it]
Outer training loop:  69%|######9   | 34709/50000 [22:33:03<8:31:48,  2.01s/it]
Outer training loop:  69%|######9   | 34710/50000 [22:33:05<8:23:31,  1.98s/it]
Outer training loop:  69%|######9   | 34711/50000 [22:33:07<8:11:02,  1.93s/it]
Outer training loop:  69%|######9   | 34712/50000 [22:33:09<8:12:49,  1.93s/it]
Outer training loop:  69%|######9   | 34713/50000 [22:33:11<8:09:53,  1.92s/it]
Outer training loop:  69%|######9   | 34714/50000 [22:33:13<8:06:32,  1.91s/it]
Outer training loop:  69%|######9   | 34715/50000 [22:33:15<7:58:52,  1.88s/it]
Outer training loop:  69%|######9   | 34716/50000 [22:33:17<7:59:38,  1.88s/it]
Outer training loop:  69%|######9   | 34717/50000 [22:33:18<7:55:51,  1.87s/it]
Outer training loop:  69%|######9   | 34718/50000 [22:33:20<7:58:08,  1.88s/it]
Outer training loop:  69%|######9   | 34719/50000 [22:33:22<7:52:30,  1.86s/it]
Outer training loop:  69%|######9   | 34720/50000 [22:33:24<7:55:42,  1.87s/it]
Outer training loop:  69%|######9   | 34721/50000 [22:33:26<7:58:32,  1.88s/it]
Outer training loop:  69%|######9   | 34722/50000 [22:33:28<7:52:12,  1.85s/it]
Outer training loop:  69%|######9   | 34723/50000 [22:33:30<8:10:35,  1.93s/it]
Outer training loop:  69%|######9   | 34724/50000 [22:33:32<8:06:13,  1.91s/it]
Outer training loop:  69%|######9   | 34725/50000 [22:33:34<8:10:22,  1.93s/it]
Outer training loop:  69%|######9   | 34726/50000 [22:33:36<8:20:32,  1.97s/it]
Outer training loop:  69%|######9   | 34727/50000 [22:33:37<7:59:59,  1.89s/it]
Outer training loop:  69%|######9   | 34728/50000 [22:33:39<7:45:19,  1.83s/it]
Outer training loop:  69%|######9   | 34729/50000 [22:33:41<7:43:29,  1.82s/it]
Outer training loop:  69%|######9   | 34730/50000 [22:33:43<8:00:01,  1.89s/it]
Outer training loop:  69%|######9   | 34731/50000 [22:33:45<7:58:44,  1.88s/it]
Outer training loop:  69%|######9   | 34732/50000 [22:33:47<7:50:16,  1.85s/it]
Outer training loop:  69%|######9   | 34733/50000 [22:33:49<7:56:45,  1.87s/it]
Outer training loop:  69%|######9   | 34734/50000 [22:33:50<7:47:15,  1.84s/it]
Outer training loop:  69%|######9   | 34735/50000 [22:33:52<7:44:59,  1.83s/it]
Outer training loop:  69%|######9   | 34736/50000 [22:33:54<7:41:20,  1.81s/it]
Outer training loop:  69%|######9   | 34737/50000 [22:33:56<8:00:44,  1.89s/it]
Outer training loop:  69%|######9   | 34738/50000 [22:33:58<7:47:50,  1.84s/it]
Outer training loop:  69%|######9   | 34739/50000 [22:33:59<7:36:34,  1.80s/it]
Outer training loop:  69%|######9   | 34740/50000 [22:34:01<7:32:33,  1.78s/it]
Outer training loop:  69%|######9   | 34741/50000 [22:34:03<7:51:49,  1.86s/it]
Outer training loop:  69%|######9   | 34742/50000 [22:34:05<7:51:16,  1.85s/it]
Outer training loop:  69%|######9   | 34743/50000 [22:34:07<7:50:13,  1.85s/it]
Outer training loop:  69%|######9   | 34744/50000 [22:34:09<8:01:26,  1.89s/it]
Outer training loop:  69%|######9   | 34745/50000 [22:34:11<7:50:50,  1.85s/it]
Outer training loop:  69%|######9   | 34746/50000 [22:34:12<7:44:54,  1.83s/it]
Outer training loop:  69%|######9   | 34747/50000 [22:34:14<7:55:38,  1.87s/it]
Outer training loop:  69%|######9   | 34748/50000 [22:34:16<7:54:38,  1.87s/it]
Outer training loop:  69%|######9   | 34749/50000 [22:34:18<7:50:20,  1.85s/it]
Outer training loop:  70%|######9   | 34750/50000 [22:34:20<7:47:09,  1.84s/it]
Outer training loop:  70%|######9   | 34751/50000 [22:34:22<8:02:37,  1.90s/it]
Outer training loop:  70%|######9   | 34752/50000 [22:34:24<7:57:04,  1.88s/it]
Outer training loop:  70%|######9   | 34753/50000 [22:34:26<8:14:37,  1.95s/it]
Outer training loop:  70%|######9   | 34754/50000 [22:34:28<8:22:18,  1.98s/it]
Outer training loop:  70%|######9   | 34755/50000 [22:34:30<8:15:56,  1.95s/it]
Outer training loop:  70%|######9   | 34756/50000 [22:34:32<8:05:35,  1.91s/it]
Outer training loop:  70%|######9   | 34757/50000 [22:34:33<8:00:07,  1.89s/it]
Outer training loop:  70%|######9   | 34758/50000 [22:34:35<8:08:03,  1.92s/it]
Outer training loop:  70%|######9   | 34759/50000 [22:34:37<8:04:22,  1.91s/it]
Outer training loop:  70%|######9   | 34760/50000 [22:34:39<7:54:45,  1.87s/it]
Outer training loop:  70%|######9   | 34761/50000 [22:34:41<8:10:58,  1.93s/it]
Outer training loop:  70%|######9   | 34762/50000 [22:34:43<8:03:33,  1.90s/it]
Outer training loop:  70%|######9   | 34763/50000 [22:34:45<8:05:09,  1.91s/it]
Outer training loop:  70%|######9   | 34764/50000 [22:34:47<7:57:37,  1.88s/it]
Outer training loop:  70%|######9   | 34765/50000 [22:34:49<8:12:11,  1.94s/it]
Outer training loop:  70%|######9   | 34766/50000 [22:34:51<8:06:40,  1.92s/it]
Outer training loop:  70%|######9   | 34767/50000 [22:34:53<8:22:33,  1.98s/it]
Outer training loop:  70%|######9   | 34768/50000 [22:34:55<8:28:54,  2.00s/it]
Outer training loop:  70%|######9   | 34769/50000 [22:34:57<8:18:23,  1.96s/it]
Outer training loop:  70%|######9   | 34770/50000 [22:34:59<8:06:54,  1.92s/it]
Outer training loop:  70%|######9   | 34771/50000 [22:35:00<7:54:46,  1.87s/it]
Outer training loop:  70%|######9   | 34772/50000 [22:35:02<8:00:39,  1.89s/it]
Outer training loop:  70%|######9   | 34773/50000 [22:35:04<7:53:45,  1.87s/it]
Outer training loop:  70%|######9   | 34774/50000 [22:35:06<8:01:33,  1.90s/it]
Outer training loop:  70%|######9   | 34775/50000 [22:35:08<8:05:17,  1.91s/it]
Outer training loop:  70%|######9   | 34776/50000 [22:35:10<8:04:44,  1.91s/it]
Outer training loop:  70%|######9   | 34777/50000 [22:35:12<7:55:45,  1.88s/it]
Outer training loop:  70%|######9   | 34778/50000 [22:35:14<8:20:57,  1.97s/it]
Outer training loop:  70%|######9   | 34779/50000 [22:35:16<8:02:30,  1.90s/it]
Outer training loop:  70%|######9   | 34780/50000 [22:35:18<8:12:01,  1.94s/it]
Outer training loop:  70%|######9   | 34781/50000 [22:35:19<8:02:59,  1.90s/it]
Outer training loop:  70%|######9   | 34782/50000 [22:35:21<8:13:03,  1.94s/it]
Outer training loop:  70%|######9   | 34783/50000 [22:35:23<8:13:50,  1.95s/it]
Outer training loop:  70%|######9   | 34784/50000 [22:35:25<8:14:15,  1.95s/it]
Outer training loop:  70%|######9   | 34785/50000 [22:35:27<8:16:59,  1.96s/it]
Outer training loop:  70%|######9   | 34786/50000 [22:35:29<8:10:29,  1.93s/it]
Outer training loop:  70%|######9   | 34787/50000 [22:35:31<7:54:58,  1.87s/it]
Outer training loop:  70%|######9   | 34788/50000 [22:35:33<8:03:02,  1.91s/it]
Outer training loop:  70%|######9   | 34789/50000 [22:35:35<7:59:08,  1.89s/it]
Outer training loop:  70%|######9   | 34790/50000 [22:35:37<7:54:21,  1.87s/it]
Outer training loop:  70%|######9   | 34791/50000 [22:35:38<7:48:52,  1.85s/it]
Outer training loop:  70%|######9   | 34792/50000 [22:35:41<8:08:32,  1.93s/it]
Outer training loop:  70%|######9   | 34793/50000 [22:35:42<8:09:49,  1.93s/it]
Outer training loop:  70%|######9   | 34794/50000 [22:35:44<8:03:08,  1.91s/it]
Outer training loop:  70%|######9   | 34795/50000 [22:35:46<8:05:52,  1.92s/it]
Outer training loop:  70%|######9   | 34796/50000 [22:35:48<7:51:11,  1.86s/it]
Outer training loop:  70%|######9   | 34797/50000 [22:35:50<7:38:20,  1.81s/it]
Outer training loop:  70%|######9   | 34798/50000 [22:35:52<7:56:19,  1.88s/it]
Outer training loop:  70%|######9   | 34799/50000 [22:35:54<7:48:54,  1.85s/it]
Outer training loop:  70%|######9   | 34800/50000 [22:35:55<7:39:05,  1.81s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.96s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.99s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  70%|######9   | 34801/50000 [22:36:03<14:56:27,  3.54s/it]
Outer training loop:  70%|######9   | 34802/50000 [22:36:05<13:05:01,  3.10s/it]
Outer training loop:  70%|######9   | 34803/50000 [22:36:07<11:24:29,  2.70s/it]
Outer training loop:  70%|######9   | 34804/50000 [22:36:08<10:12:03,  2.42s/it]
Outer training loop:  70%|######9   | 34805/50000 [22:36:10<9:30:36,  2.25s/it] 
Outer training loop:  70%|######9   | 34806/50000 [22:36:12<9:09:31,  2.17s/it]
Outer training loop:  70%|######9   | 34807/50000 [22:36:14<8:42:43,  2.06s/it]
Outer training loop:  70%|######9   | 34808/50000 [22:36:16<8:26:16,  2.00s/it]
Outer training loop:  70%|######9   | 34809/50000 [22:36:18<8:30:13,  2.02s/it]
Outer training loop:  70%|######9   | 34810/50000 [22:36:20<8:31:39,  2.02s/it]
Outer training loop:  70%|######9   | 34811/50000 [22:36:22<8:20:55,  1.98s/it]
Outer training loop:  70%|######9   | 34812/50000 [22:36:24<8:12:55,  1.95s/it]
Outer training loop:  70%|######9   | 34813/50000 [22:36:26<8:18:42,  1.97s/it]
Outer training loop:  70%|######9   | 34814/50000 [22:36:28<8:08:26,  1.93s/it]
Outer training loop:  70%|######9   | 34815/50000 [22:36:30<8:06:30,  1.92s/it]
Outer training loop:  70%|######9   | 34816/50000 [22:36:31<8:10:53,  1.94s/it]
Outer training loop:  70%|######9   | 34817/50000 [22:36:33<7:56:22,  1.88s/it]
Outer training loop:  70%|######9   | 34818/50000 [22:36:35<7:54:17,  1.87s/it]
Outer training loop:  70%|######9   | 34819/50000 [22:36:37<7:44:10,  1.83s/it]
Outer training loop:  70%|######9   | 34820/50000 [22:36:39<8:00:02,  1.90s/it]
Outer training loop:  70%|######9   | 34821/50000 [22:36:41<7:56:33,  1.88s/it]
Outer training loop:  70%|######9   | 34822/50000 [22:36:43<7:52:32,  1.87s/it]
Outer training loop:  70%|######9   | 34823/50000 [22:36:45<8:01:52,  1.90s/it]
Outer training loop:  70%|######9   | 34824/50000 [22:36:46<7:46:41,  1.85s/it]
Outer training loop:  70%|######9   | 34825/50000 [22:36:48<7:44:10,  1.84s/it]
Outer training loop:  70%|######9   | 34826/50000 [22:36:50<8:03:27,  1.91s/it]
Outer training loop:  70%|######9   | 34827/50000 [22:36:52<7:58:48,  1.89s/it]
Outer training loop:  70%|######9   | 34828/50000 [22:36:54<7:50:38,  1.86s/it]
Outer training loop:  70%|######9   | 34829/50000 [22:36:56<7:46:05,  1.84s/it]
Outer training loop:  70%|######9   | 34830/50000 [22:36:58<8:00:37,  1.90s/it]
Outer training loop:  70%|######9   | 34831/50000 [22:37:00<8:00:35,  1.90s/it]
Outer training loop:  70%|######9   | 34832/50000 [22:37:01<7:56:50,  1.89s/it]
Outer training loop:  70%|######9   | 34833/50000 [22:37:03<7:59:39,  1.90s/it]
Outer training loop:  70%|######9   | 34834/50000 [22:37:05<8:06:42,  1.93s/it]
Outer training loop:  70%|######9   | 34835/50000 [22:37:07<8:00:29,  1.90s/it]
Outer training loop:  70%|######9   | 34836/50000 [22:37:09<8:07:44,  1.93s/it]
Outer training loop:  70%|######9   | 34837/50000 [22:37:11<7:52:39,  1.87s/it]
Outer training loop:  70%|######9   | 34838/50000 [22:37:13<7:48:35,  1.85s/it]
Outer training loop:  70%|######9   | 34839/50000 [22:37:15<8:01:56,  1.91s/it]
Outer training loop:  70%|######9   | 34840/50000 [22:37:17<7:56:00,  1.88s/it]
Outer training loop:  70%|######9   | 34841/50000 [22:37:18<7:46:19,  1.85s/it]
Outer training loop:  70%|######9   | 34842/50000 [22:37:20<7:52:21,  1.87s/it]
Outer training loop:  70%|######9   | 34843/50000 [22:37:22<8:05:44,  1.92s/it]
Outer training loop:  70%|######9   | 34844/50000 [22:37:24<7:53:44,  1.88s/it]
Outer training loop:  70%|######9   | 34845/50000 [22:37:26<7:55:11,  1.88s/it]
Outer training loop:  70%|######9   | 34846/50000 [22:37:28<8:03:04,  1.91s/it]
Outer training loop:  70%|######9   | 34847/50000 [22:37:30<7:49:51,  1.86s/it]
Outer training loop:  70%|######9   | 34848/50000 [22:37:31<7:43:46,  1.84s/it]
Outer training loop:  70%|######9   | 34849/50000 [22:37:33<7:57:17,  1.89s/it]
Outer training loop:  70%|######9   | 34850/50000 [22:37:36<8:17:53,  1.97s/it]
Outer training loop:  70%|######9   | 34851/50000 [22:37:37<8:08:26,  1.93s/it]
Outer training loop:  70%|######9   | 34852/50000 [22:37:39<8:05:23,  1.92s/it]
Outer training loop:  70%|######9   | 34853/50000 [22:37:41<8:09:27,  1.94s/it]
Outer training loop:  70%|######9   | 34854/50000 [22:37:43<7:58:28,  1.90s/it]
Outer training loop:  70%|######9   | 34855/50000 [22:37:45<7:56:40,  1.89s/it]
Outer training loop:  70%|######9   | 34856/50000 [22:37:47<8:01:57,  1.91s/it]
Outer training loop:  70%|######9   | 34857/50000 [22:37:49<8:01:24,  1.91s/it]
Outer training loop:  70%|######9   | 34858/50000 [22:37:51<8:02:14,  1.91s/it]
Outer training loop:  70%|######9   | 34859/50000 [22:37:53<7:51:44,  1.87s/it]
Outer training loop:  70%|######9   | 34860/50000 [22:37:54<7:55:17,  1.88s/it]
Outer training loop:  70%|######9   | 34861/50000 [22:37:56<7:52:29,  1.87s/it]
Outer training loop:  70%|######9   | 34862/50000 [22:37:58<7:57:16,  1.89s/it]
Outer training loop:  70%|######9   | 34863/50000 [22:38:00<8:02:58,  1.91s/it]
Outer training loop:  70%|######9   | 34864/50000 [22:38:02<7:47:29,  1.85s/it]
Outer training loop:  70%|######9   | 34865/50000 [22:38:04<7:47:02,  1.85s/it]
Outer training loop:  70%|######9   | 34866/50000 [22:38:06<7:38:30,  1.82s/it]
Outer training loop:  70%|######9   | 34867/50000 [22:38:07<7:44:44,  1.84s/it]
Outer training loop:  70%|######9   | 34868/50000 [22:38:09<7:37:34,  1.81s/it]
Outer training loop:  70%|######9   | 34869/50000 [22:38:11<7:44:52,  1.84s/it]
Outer training loop:  70%|######9   | 34870/50000 [22:38:13<7:43:07,  1.84s/it]
Outer training loop:  70%|######9   | 34871/50000 [22:38:15<7:52:17,  1.87s/it]
Outer training loop:  70%|######9   | 34872/50000 [22:38:17<7:42:54,  1.84s/it]
Outer training loop:  70%|######9   | 34873/50000 [22:38:18<7:40:07,  1.83s/it]
Outer training loop:  70%|######9   | 34874/50000 [22:38:20<7:54:31,  1.88s/it]
Outer training loop:  70%|######9   | 34875/50000 [22:38:22<7:53:02,  1.88s/it]
Outer training loop:  70%|######9   | 34876/50000 [22:38:24<7:45:34,  1.85s/it]
Outer training loop:  70%|######9   | 34877/50000 [22:38:26<7:44:37,  1.84s/it]
Outer training loop:  70%|######9   | 34878/50000 [22:38:28<7:51:55,  1.87s/it]
Outer training loop:  70%|######9   | 34879/50000 [22:38:30<7:42:51,  1.84s/it]
Outer training loop:  70%|######9   | 34880/50000 [22:38:31<7:39:56,  1.83s/it]
Outer training loop:  70%|######9   | 34881/50000 [22:38:33<7:50:12,  1.87s/it]
Outer training loop:  70%|######9   | 34882/50000 [22:38:35<7:49:27,  1.86s/it]
Outer training loop:  70%|######9   | 34883/50000 [22:38:37<7:43:50,  1.84s/it]
Outer training loop:  70%|######9   | 34884/50000 [22:38:39<7:52:47,  1.88s/it]
Outer training loop:  70%|######9   | 34885/50000 [22:38:41<7:48:49,  1.86s/it]
Outer training loop:  70%|######9   | 34886/50000 [22:38:43<7:44:58,  1.85s/it]
Outer training loop:  70%|######9   | 34887/50000 [22:38:44<7:37:01,  1.81s/it]
Outer training loop:  70%|######9   | 34888/50000 [22:38:46<7:52:05,  1.87s/it]
Outer training loop:  70%|######9   | 34889/50000 [22:38:48<7:47:43,  1.86s/it]
Outer training loop:  70%|######9   | 34890/50000 [22:38:50<7:55:18,  1.89s/it]
Outer training loop:  70%|######9   | 34891/50000 [22:38:52<8:01:23,  1.91s/it]
Outer training loop:  70%|######9   | 34892/50000 [22:38:54<7:50:44,  1.87s/it]
Outer training loop:  70%|######9   | 34893/50000 [22:38:56<7:53:37,  1.88s/it]
Outer training loop:  70%|######9   | 34894/50000 [22:38:58<7:47:27,  1.86s/it]
Outer training loop:  70%|######9   | 34895/50000 [22:39:00<8:03:25,  1.92s/it]
Outer training loop:  70%|######9   | 34896/50000 [22:39:02<8:00:39,  1.91s/it]
Outer training loop:  70%|######9   | 34897/50000 [22:39:03<7:53:29,  1.88s/it]
Outer training loop:  70%|######9   | 34898/50000 [22:39:05<8:11:14,  1.95s/it]
Outer training loop:  70%|######9   | 34899/50000 [22:39:07<8:05:52,  1.93s/it]
Outer training loop:  70%|######9   | 34900/50000 [22:39:09<8:14:27,  1.96s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  70%|######9   | 34901/50000 [22:39:17<15:06:57,  3.60s/it]
Outer training loop:  70%|######9   | 34902/50000 [22:39:19<13:10:27,  3.14s/it]
Outer training loop:  70%|######9   | 34903/50000 [22:39:21<11:34:38,  2.76s/it]
Outer training loop:  70%|######9   | 34904/50000 [22:39:23<10:23:48,  2.48s/it]
Outer training loop:  70%|######9   | 34905/50000 [22:39:25<9:56:36,  2.37s/it] 
Outer training loop:  70%|######9   | 34906/50000 [22:39:27<9:15:22,  2.21s/it]
Outer training loop:  70%|######9   | 34907/50000 [22:39:28<8:57:46,  2.14s/it]
Outer training loop:  70%|######9   | 34908/50000 [22:39:30<8:28:03,  2.02s/it]
Outer training loop:  70%|######9   | 34909/50000 [22:39:32<8:26:56,  2.02s/it]
Outer training loop:  70%|######9   | 34910/50000 [22:39:34<8:06:47,  1.94s/it]
Outer training loop:  70%|######9   | 34911/50000 [22:39:36<8:05:18,  1.93s/it]
Outer training loop:  70%|######9   | 34912/50000 [22:39:38<8:26:19,  2.01s/it]
Outer training loop:  70%|######9   | 34913/50000 [22:39:40<8:10:54,  1.95s/it]
Outer training loop:  70%|######9   | 34914/50000 [22:39:42<8:05:17,  1.93s/it]
Outer training loop:  70%|######9   | 34915/50000 [22:39:44<8:16:20,  1.97s/it]
Outer training loop:  70%|######9   | 34916/50000 [22:39:46<8:07:07,  1.94s/it]
Outer training loop:  70%|######9   | 34917/50000 [22:39:47<7:53:02,  1.88s/it]
Outer training loop:  70%|######9   | 34918/50000 [22:39:50<8:07:02,  1.94s/it]
Outer training loop:  70%|######9   | 34919/50000 [22:39:51<7:54:08,  1.89s/it]
Outer training loop:  70%|######9   | 34920/50000 [22:39:53<7:47:55,  1.86s/it]
Outer training loop:  70%|######9   | 34921/50000 [22:39:55<7:45:12,  1.85s/it]
Outer training loop:  70%|######9   | 34922/50000 [22:39:57<7:54:05,  1.89s/it]
Outer training loop:  70%|######9   | 34923/50000 [22:39:59<7:48:46,  1.87s/it]
Outer training loop:  70%|######9   | 34924/50000 [22:40:01<7:42:23,  1.84s/it]
Outer training loop:  70%|######9   | 34925/50000 [22:40:03<7:58:16,  1.90s/it]
Outer training loop:  70%|######9   | 34926/50000 [22:40:04<7:51:47,  1.88s/it]
Outer training loop:  70%|######9   | 34927/50000 [22:40:06<7:45:20,  1.85s/it]
Outer training loop:  70%|######9   | 34928/50000 [22:40:08<7:38:50,  1.83s/it]
Outer training loop:  70%|######9   | 34929/50000 [22:40:10<7:54:16,  1.89s/it]
Outer training loop:  70%|######9   | 34930/50000 [22:40:12<7:56:20,  1.90s/it]
Outer training loop:  70%|######9   | 34931/50000 [22:40:14<8:07:50,  1.94s/it]
Outer training loop:  70%|######9   | 34932/50000 [22:40:16<8:16:10,  1.98s/it]
Outer training loop:  70%|######9   | 34933/50000 [22:40:18<7:58:22,  1.90s/it]
Outer training loop:  70%|######9   | 34934/50000 [22:40:20<8:02:33,  1.92s/it]
Outer training loop:  70%|######9   | 34935/50000 [22:40:22<8:06:47,  1.94s/it]
Outer training loop:  70%|######9   | 34936/50000 [22:40:24<8:00:43,  1.91s/it]
Outer training loop:  70%|######9   | 34937/50000 [22:40:25<8:00:47,  1.92s/it]
Outer training loop:  70%|######9   | 34938/50000 [22:40:27<8:00:38,  1.91s/it]
Outer training loop:  70%|######9   | 34939/50000 [22:40:29<8:00:50,  1.92s/it]
Outer training loop:  70%|######9   | 34940/50000 [22:40:31<7:56:00,  1.90s/it]
Outer training loop:  70%|######9   | 34941/50000 [22:40:33<7:47:29,  1.86s/it]
Outer training loop:  70%|######9   | 34942/50000 [22:40:35<8:04:47,  1.93s/it]
Outer training loop:  70%|######9   | 34943/50000 [22:40:37<7:58:46,  1.91s/it]
Outer training loop:  70%|######9   | 34944/50000 [22:40:39<7:55:31,  1.90s/it]
Outer training loop:  70%|######9   | 34945/50000 [22:40:41<7:52:51,  1.88s/it]
Outer training loop:  70%|######9   | 34946/50000 [22:40:43<8:13:02,  1.97s/it]
Outer training loop:  70%|######9   | 34947/50000 [22:40:45<8:14:23,  1.97s/it]
Outer training loop:  70%|######9   | 34948/50000 [22:40:47<8:03:38,  1.93s/it]
Outer training loop:  70%|######9   | 34949/50000 [22:40:48<8:01:41,  1.92s/it]
Outer training loop:  70%|######9   | 34950/50000 [22:40:50<7:52:23,  1.88s/it]
Outer training loop:  70%|######9   | 34951/50000 [22:40:52<7:46:33,  1.86s/it]
Outer training loop:  70%|######9   | 34952/50000 [22:40:54<7:55:44,  1.90s/it]
Outer training loop:  70%|######9   | 34953/50000 [22:40:56<8:06:16,  1.94s/it]
Outer training loop:  70%|######9   | 34954/50000 [22:40:58<8:01:51,  1.92s/it]
Outer training loop:  70%|######9   | 34955/50000 [22:41:00<7:51:59,  1.88s/it]
Outer training loop:  70%|######9   | 34956/50000 [22:41:02<7:53:49,  1.89s/it]
Outer training loop:  70%|######9   | 34957/50000 [22:41:04<7:57:31,  1.90s/it]
Outer training loop:  70%|######9   | 34958/50000 [22:41:05<7:53:04,  1.89s/it]
Outer training loop:  70%|######9   | 34959/50000 [22:41:07<7:52:40,  1.89s/it]
Outer training loop:  70%|######9   | 34960/50000 [22:41:09<7:59:18,  1.91s/it]
Outer training loop:  70%|######9   | 34961/50000 [22:41:11<7:51:29,  1.88s/it]
Outer training loop:  70%|######9   | 34962/50000 [22:41:13<7:45:15,  1.86s/it]
Outer training loop:  70%|######9   | 34963/50000 [22:41:15<7:51:35,  1.88s/it]
Outer training loop:  70%|######9   | 34964/50000 [22:41:17<7:43:35,  1.85s/it]
Outer training loop:  70%|######9   | 34965/50000 [22:41:19<7:45:41,  1.86s/it]
Outer training loop:  70%|######9   | 34966/50000 [22:41:21<8:05:10,  1.94s/it]
Outer training loop:  70%|######9   | 34967/50000 [22:41:22<7:49:04,  1.87s/it]
Outer training loop:  70%|######9   | 34968/50000 [22:41:24<7:42:02,  1.84s/it]
Outer training loop:  70%|######9   | 34969/50000 [22:41:26<7:49:05,  1.87s/it]
Outer training loop:  70%|######9   | 34970/50000 [22:41:28<8:06:30,  1.94s/it]
Outer training loop:  70%|######9   | 34971/50000 [22:41:30<7:53:58,  1.89s/it]
Outer training loop:  70%|######9   | 34972/50000 [22:41:32<7:40:04,  1.84s/it]
Outer training loop:  70%|######9   | 34973/50000 [22:41:34<8:01:53,  1.92s/it]
Outer training loop:  70%|######9   | 34974/50000 [22:41:36<8:01:05,  1.92s/it]
Outer training loop:  70%|######9   | 34975/50000 [22:41:38<7:54:33,  1.90s/it]
Outer training loop:  70%|######9   | 34976/50000 [22:41:40<8:00:44,  1.92s/it]
Outer training loop:  70%|######9   | 34977/50000 [22:41:41<7:47:44,  1.87s/it]
Outer training loop:  70%|######9   | 34978/50000 [22:41:43<7:44:21,  1.85s/it]
Outer training loop:  70%|######9   | 34979/50000 [22:41:45<7:52:45,  1.89s/it]
Outer training loop:  70%|######9   | 34980/50000 [22:41:47<7:49:06,  1.87s/it]
Outer training loop:  70%|######9   | 34981/50000 [22:41:49<7:48:54,  1.87s/it]
Outer training loop:  70%|######9   | 34982/50000 [22:41:51<7:45:15,  1.86s/it]
Outer training loop:  70%|######9   | 34983/50000 [22:41:52<7:47:38,  1.87s/it]
Outer training loop:  70%|######9   | 34984/50000 [22:41:54<7:38:32,  1.83s/it]
Outer training loop:  70%|######9   | 34985/50000 [22:41:56<7:38:49,  1.83s/it]
Outer training loop:  70%|######9   | 34986/50000 [22:41:58<7:49:39,  1.88s/it]
Outer training loop:  70%|######9   | 34987/50000 [22:42:00<7:44:31,  1.86s/it]
Outer training loop:  70%|######9   | 34988/50000 [22:42:02<7:32:40,  1.81s/it]
Outer training loop:  70%|######9   | 34989/50000 [22:42:03<7:31:02,  1.80s/it]
Outer training loop:  70%|######9   | 34990/50000 [22:42:05<7:47:59,  1.87s/it]
Outer training loop:  70%|######9   | 34991/50000 [22:42:07<7:43:11,  1.85s/it]
Outer training loop:  70%|######9   | 34992/50000 [22:42:09<7:33:19,  1.81s/it]
Outer training loop:  70%|######9   | 34993/50000 [22:42:11<7:53:32,  1.89s/it]
Outer training loop:  70%|######9   | 34994/50000 [22:42:13<7:49:10,  1.88s/it]
Outer training loop:  70%|######9   | 34995/50000 [22:42:15<7:47:25,  1.87s/it]
Outer training loop:  70%|######9   | 34996/50000 [22:42:17<7:54:27,  1.90s/it]
Outer training loop:  70%|######9   | 34997/50000 [22:42:19<7:53:36,  1.89s/it]
Outer training loop:  70%|######9   | 34998/50000 [22:42:20<7:49:28,  1.88s/it]
Outer training loop:  70%|######9   | 34999/50000 [22:42:22<7:58:14,  1.91s/it]
Outer training loop:  70%|#######   | 35000/50000 [22:42:24<7:47:43,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.77s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.82s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.86s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.88s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:20,  1.90s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:23,  1.96s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:13,  1.96s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.90s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.87s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:04,  1.95s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:55,  1.90s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.88s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:54,  1.94s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.92s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.93s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:44,  1.94s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:41,  1.91s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:38,  1.90s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:35,  1.87s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:29,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:21,  1.82s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:17,  1.80s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:15,  1.80s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:19,  1.93s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:17,  1.94s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:08,  1.86s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:06,  1.95s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.92s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.89s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:53,  2.05s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:49,  1.97s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.90s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.90s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.90s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:36,  2.01s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:33,  1.95s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:30,  1.89s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.91s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.90s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.90s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:19,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.99s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.96s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.95s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.92s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.89s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.92s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.97s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:24,  1.78s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.78s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:24,  1.83s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:22,  1.83s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:21,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:19,  1.88s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:34,  2.12s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:28,  2.06s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:23,  2.02s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:17,  1.96s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:19,  2.02s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:14,  1.97s/it]

Evaluating policy:  19%|#9        | 16/83 [00:31<02:11,  1.96s/it]

Evaluating policy:  20%|##        | 17/83 [00:33<02:08,  1.95s/it]

Evaluating policy:  22%|##1       | 18/83 [00:35<02:09,  2.00s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:04,  1.94s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:58,  1.95s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:55,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:50,  1.91s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:46,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:45,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:41,  1.89s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:37,  1.87s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:35,  1.87s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:38,  1.97s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:33,  1.91s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:32,  1.92s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:30,  1.93s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:29,  1.94s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:25,  1.90s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:22,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:17<01:26,  2.00s/it]

Evaluating policy:  49%|####9     | 41/83 [01:19<01:22,  1.95s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:19,  1.93s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:14,  1.91s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:14,  2.01s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:11,  2.00s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:06,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:04,  1.90s/it]

Evaluating policy:  60%|######    | 50/83 [01:36<01:01,  1.85s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:42<00:59,  1.99s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:55,  1.91s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:52,  1.88s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:50,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:47,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:44,  1.93s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:44,  2.03s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:41,  1.97s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:01<00:38,  1.94s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:03<00:36,  1.92s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:33,  1.89s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:06<00:32,  1.89s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:28,  1.87s/it]

Evaluating policy:  83%|########3 | 69/83 [02:12<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:26,  2.00s/it]

Evaluating policy:  86%|########5 | 71/83 [02:16<00:23,  1.98s/it]

Evaluating policy:  87%|########6 | 72/83 [02:18<00:21,  1.91s/it]

Evaluating policy:  88%|########7 | 73/83 [02:20<00:19,  1.90s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:16,  1.86s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:14,  1.86s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:13,  1.86s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:11,  1.84s/it]

Evaluating policy:  94%|#########3| 78/83 [02:29<00:08,  1.80s/it]

Evaluating policy:  95%|#########5| 79/83 [02:31<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:33<00:05,  1.95s/it]

Evaluating policy:  98%|#########7| 81/83 [02:35<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.87s/it]

Outer training loop:  70%|#######   | 35001/50000 [22:47:48<410:49:13, 98.60s/it]
Outer training loop:  70%|#######   | 35002/50000 [22:47:50<289:56:57, 69.60s/it]
Outer training loop:  70%|#######   | 35003/50000 [22:47:52<205:14:27, 49.27s/it]
Outer training loop:  70%|#######   | 35004/50000 [22:47:54<145:59:31, 35.05s/it]
Outer training loop:  70%|#######   | 35005/50000 [22:47:56<104:33:28, 25.10s/it]
Outer training loop:  70%|#######   | 35006/50000 [22:47:58<75:27:39, 18.12s/it] 
Outer training loop:  70%|#######   | 35007/50000 [22:48:00<55:55:44, 13.43s/it]
Outer training loop:  70%|#######   | 35008/50000 [22:48:02<41:25:30,  9.95s/it]
Outer training loop:  70%|#######   | 35009/50000 [22:48:04<31:22:28,  7.53s/it]
Outer training loop:  70%|#######   | 35010/50000 [22:48:06<24:38:50,  5.92s/it]
Outer training loop:  70%|#######   | 35011/50000 [22:48:08<19:27:13,  4.67s/it]
Outer training loop:  70%|#######   | 35012/50000 [22:48:10<15:56:24,  3.83s/it]
Outer training loop:  70%|#######   | 35013/50000 [22:48:11<13:19:11,  3.20s/it]
Outer training loop:  70%|#######   | 35014/50000 [22:48:13<11:49:18,  2.84s/it]
Outer training loop:  70%|#######   | 35015/50000 [22:48:15<10:29:12,  2.52s/it]
Outer training loop:  70%|#######   | 35016/50000 [22:48:17<9:27:48,  2.27s/it] 
Outer training loop:  70%|#######   | 35017/50000 [22:48:19<9:09:23,  2.20s/it]
Outer training loop:  70%|#######   | 35018/50000 [22:48:21<8:45:57,  2.11s/it]
Outer training loop:  70%|#######   | 35019/50000 [22:48:23<8:21:31,  2.01s/it]
Outer training loop:  70%|#######   | 35020/50000 [22:48:24<8:06:20,  1.95s/it]
Outer training loop:  70%|#######   | 35021/50000 [22:48:26<8:09:03,  1.96s/it]
Outer training loop:  70%|#######   | 35022/50000 [22:48:28<7:57:26,  1.91s/it]
Outer training loop:  70%|#######   | 35023/50000 [22:48:30<7:55:32,  1.91s/it]
Outer training loop:  70%|#######   | 35024/50000 [22:48:32<7:56:36,  1.91s/it]
Outer training loop:  70%|#######   | 35025/50000 [22:48:34<7:50:34,  1.89s/it]
Outer training loop:  70%|#######   | 35026/50000 [22:48:36<7:42:06,  1.85s/it]
Outer training loop:  70%|#######   | 35027/50000 [22:48:37<7:39:32,  1.84s/it]
Outer training loop:  70%|#######   | 35028/50000 [22:48:40<8:08:34,  1.96s/it]
Outer training loop:  70%|#######   | 35029/50000 [22:48:42<7:58:02,  1.92s/it]
Outer training loop:  70%|#######   | 35030/50000 [22:48:43<7:56:45,  1.91s/it]
Outer training loop:  70%|#######   | 35031/50000 [22:48:45<8:03:29,  1.94s/it]
Outer training loop:  70%|#######   | 35032/50000 [22:48:47<7:55:38,  1.91s/it]
Outer training loop:  70%|#######   | 35033/50000 [22:48:49<7:53:52,  1.90s/it]
Outer training loop:  70%|#######   | 35034/50000 [22:48:51<7:51:27,  1.89s/it]
Outer training loop:  70%|#######   | 35035/50000 [22:48:53<7:59:59,  1.92s/it]
Outer training loop:  70%|#######   | 35036/50000 [22:48:55<7:50:42,  1.89s/it]
Outer training loop:  70%|#######   | 35037/50000 [22:48:57<7:44:27,  1.86s/it]
Outer training loop:  70%|#######   | 35038/50000 [22:48:59<7:54:41,  1.90s/it]
Outer training loop:  70%|#######   | 35039/50000 [22:49:01<7:55:39,  1.91s/it]
Outer training loop:  70%|#######   | 35040/50000 [22:49:02<7:46:34,  1.87s/it]
Outer training loop:  70%|#######   | 35041/50000 [22:49:04<7:51:48,  1.89s/it]
Outer training loop:  70%|#######   | 35042/50000 [22:49:06<8:08:44,  1.96s/it]
Outer training loop:  70%|#######   | 35043/50000 [22:49:08<8:07:17,  1.95s/it]
Outer training loop:  70%|#######   | 35044/50000 [22:49:10<7:45:03,  1.87s/it]
Outer training loop:  70%|#######   | 35045/50000 [22:49:12<8:03:05,  1.94s/it]
Outer training loop:  70%|#######   | 35046/50000 [22:49:14<7:55:02,  1.91s/it]
Outer training loop:  70%|#######   | 35047/50000 [22:49:16<7:50:35,  1.89s/it]
Outer training loop:  70%|#######   | 35048/50000 [22:49:18<7:44:09,  1.86s/it]
Outer training loop:  70%|#######   | 35049/50000 [22:49:20<8:00:35,  1.93s/it]
Outer training loop:  70%|#######   | 35050/50000 [22:49:22<8:02:26,  1.94s/it]
Outer training loop:  70%|#######   | 35051/50000 [22:49:24<8:00:08,  1.93s/it]
Outer training loop:  70%|#######   | 35052/50000 [22:49:26<8:11:34,  1.97s/it]
Outer training loop:  70%|#######   | 35053/50000 [22:49:27<8:05:06,  1.95s/it]
Outer training loop:  70%|#######   | 35054/50000 [22:49:29<7:49:10,  1.88s/it]
Outer training loop:  70%|#######   | 35055/50000 [22:49:31<7:57:47,  1.92s/it]
Outer training loop:  70%|#######   | 35056/50000 [22:49:33<7:54:27,  1.90s/it]
Outer training loop:  70%|#######   | 35057/50000 [22:49:35<7:57:56,  1.92s/it]
Outer training loop:  70%|#######   | 35058/50000 [22:49:37<7:43:02,  1.86s/it]
Outer training loop:  70%|#######   | 35059/50000 [22:49:39<7:59:54,  1.93s/it]
Outer training loop:  70%|#######   | 35060/50000 [22:49:41<7:57:24,  1.92s/it]
Outer training loop:  70%|#######   | 35061/50000 [22:49:43<7:55:05,  1.91s/it]
Outer training loop:  70%|#######   | 35062/50000 [22:49:45<8:03:23,  1.94s/it]
Outer training loop:  70%|#######   | 35063/50000 [22:49:46<7:54:36,  1.91s/it]
Outer training loop:  70%|#######   | 35064/50000 [22:49:48<7:46:07,  1.87s/it]
Outer training loop:  70%|#######   | 35065/50000 [22:49:50<7:46:02,  1.87s/it]
Outer training loop:  70%|#######   | 35066/50000 [22:49:52<7:59:22,  1.93s/it]
Outer training loop:  70%|#######   | 35067/50000 [22:49:54<7:55:38,  1.91s/it]
Outer training loop:  70%|#######   | 35068/50000 [22:49:56<7:49:06,  1.88s/it]
Outer training loop:  70%|#######   | 35069/50000 [22:49:58<8:00:01,  1.93s/it]
Outer training loop:  70%|#######   | 35070/50000 [22:50:00<7:53:22,  1.90s/it]
Outer training loop:  70%|#######   | 35071/50000 [22:50:02<7:52:38,  1.90s/it]
Outer training loop:  70%|#######   | 35072/50000 [22:50:04<8:04:06,  1.95s/it]
Outer training loop:  70%|#######   | 35073/50000 [22:50:06<8:02:23,  1.94s/it]
Outer training loop:  70%|#######   | 35074/50000 [22:50:07<7:48:41,  1.88s/it]
Outer training loop:  70%|#######   | 35075/50000 [22:50:09<7:54:17,  1.91s/it]
Outer training loop:  70%|#######   | 35076/50000 [22:50:11<7:52:57,  1.90s/it]
Outer training loop:  70%|#######   | 35077/50000 [22:50:13<7:58:20,  1.92s/it]
Outer training loop:  70%|#######   | 35078/50000 [22:50:15<8:13:19,  1.98s/it]
Outer training loop:  70%|#######   | 35079/50000 [22:50:17<8:09:43,  1.97s/it]
Outer training loop:  70%|#######   | 35080/50000 [22:50:19<8:04:14,  1.95s/it]
Outer training loop:  70%|#######   | 35081/50000 [22:50:21<7:58:51,  1.93s/it]
Outer training loop:  70%|#######   | 35082/50000 [22:50:23<8:06:56,  1.96s/it]
Outer training loop:  70%|#######   | 35083/50000 [22:50:25<7:57:36,  1.92s/it]
Outer training loop:  70%|#######   | 35084/50000 [22:50:27<7:50:25,  1.89s/it]
Outer training loop:  70%|#######   | 35085/50000 [22:50:28<7:40:31,  1.85s/it]
Outer training loop:  70%|#######   | 35086/50000 [22:50:31<7:56:01,  1.92s/it]
Outer training loop:  70%|#######   | 35087/50000 [22:50:32<7:50:13,  1.89s/it]
Outer training loop:  70%|#######   | 35088/50000 [22:50:34<7:52:39,  1.90s/it]
Outer training loop:  70%|#######   | 35089/50000 [22:50:36<7:57:34,  1.92s/it]
Outer training loop:  70%|#######   | 35090/50000 [22:50:38<7:48:48,  1.89s/it]
Outer training loop:  70%|#######   | 35091/50000 [22:50:40<7:44:23,  1.87s/it]
Outer training loop:  70%|#######   | 35092/50000 [22:50:42<7:39:21,  1.85s/it]
Outer training loop:  70%|#######   | 35093/50000 [22:50:44<7:51:55,  1.90s/it]
Outer training loop:  70%|#######   | 35094/50000 [22:50:46<7:44:46,  1.87s/it]
Outer training loop:  70%|#######   | 35095/50000 [22:50:47<7:38:07,  1.84s/it]
Outer training loop:  70%|#######   | 35096/50000 [22:50:49<7:42:38,  1.86s/it]
Outer training loop:  70%|#######   | 35097/50000 [22:50:51<7:53:19,  1.91s/it]
Outer training loop:  70%|#######   | 35098/50000 [22:50:53<7:53:09,  1.91s/it]
Outer training loop:  70%|#######   | 35099/50000 [22:50:55<7:53:21,  1.91s/it]
Outer training loop:  70%|#######   | 35100/50000 [22:50:57<7:53:58,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  70%|#######   | 35101/50000 [22:51:05<15:08:03,  3.66s/it]
Outer training loop:  70%|#######   | 35102/50000 [22:51:07<13:01:28,  3.15s/it]
Outer training loop:  70%|#######   | 35103/50000 [22:51:09<11:25:57,  2.76s/it]
Outer training loop:  70%|#######   | 35104/50000 [22:51:11<10:36:56,  2.57s/it]
Outer training loop:  70%|#######   | 35105/50000 [22:51:12<9:40:59,  2.34s/it] 
Outer training loop:  70%|#######   | 35106/50000 [22:51:14<8:55:15,  2.16s/it]
Outer training loop:  70%|#######   | 35107/50000 [22:51:16<8:34:56,  2.07s/it]
Outer training loop:  70%|#######   | 35108/50000 [22:51:18<8:18:26,  2.01s/it]
Outer training loop:  70%|#######   | 35109/50000 [22:51:20<8:06:36,  1.96s/it]
Outer training loop:  70%|#######   | 35110/50000 [22:51:22<8:08:15,  1.97s/it]
Outer training loop:  70%|#######   | 35111/50000 [22:51:23<7:49:57,  1.89s/it]
Outer training loop:  70%|#######   | 35112/50000 [22:51:25<7:54:03,  1.91s/it]
Outer training loop:  70%|#######   | 35113/50000 [22:51:27<7:50:24,  1.90s/it]
Outer training loop:  70%|#######   | 35114/50000 [22:51:29<8:01:41,  1.94s/it]
Outer training loop:  70%|#######   | 35115/50000 [22:51:31<7:49:29,  1.89s/it]
Outer training loop:  70%|#######   | 35116/50000 [22:51:33<7:41:33,  1.86s/it]
Outer training loop:  70%|#######   | 35117/50000 [22:51:35<7:55:29,  1.92s/it]
Outer training loop:  70%|#######   | 35118/50000 [22:51:37<7:48:20,  1.89s/it]
Outer training loop:  70%|#######   | 35119/50000 [22:51:38<7:33:18,  1.83s/it]
Outer training loop:  70%|#######   | 35120/50000 [22:51:40<7:35:55,  1.84s/it]
Outer training loop:  70%|#######   | 35121/50000 [22:51:42<7:44:45,  1.87s/it]
Outer training loop:  70%|#######   | 35122/50000 [22:51:44<7:43:45,  1.87s/it]
Outer training loop:  70%|#######   | 35123/50000 [22:51:46<7:39:22,  1.85s/it]
Outer training loop:  70%|#######   | 35124/50000 [22:51:48<7:46:15,  1.88s/it]
Outer training loop:  70%|#######   | 35125/50000 [22:51:50<8:03:15,  1.95s/it]
Outer training loop:  70%|#######   | 35126/50000 [22:51:52<7:53:28,  1.91s/it]
Outer training loop:  70%|#######   | 35127/50000 [22:51:54<7:50:51,  1.90s/it]
Outer training loop:  70%|#######   | 35128/50000 [22:51:56<8:05:38,  1.96s/it]
Outer training loop:  70%|#######   | 35129/50000 [22:51:58<7:58:58,  1.93s/it]
Outer training loop:  70%|#######   | 35130/50000 [22:51:59<7:48:15,  1.89s/it]
Outer training loop:  70%|#######   | 35131/50000 [22:52:02<8:02:53,  1.95s/it]
Outer training loop:  70%|#######   | 35132/50000 [22:52:03<7:54:40,  1.92s/it]
Outer training loop:  70%|#######   | 35133/50000 [22:52:05<7:47:35,  1.89s/it]
Outer training loop:  70%|#######   | 35134/50000 [22:52:07<8:07:24,  1.97s/it]
Outer training loop:  70%|#######   | 35135/50000 [22:52:09<7:48:48,  1.89s/it]
Outer training loop:  70%|#######   | 35136/50000 [22:52:11<7:39:48,  1.86s/it]
Outer training loop:  70%|#######   | 35137/50000 [22:52:13<7:37:24,  1.85s/it]
Outer training loop:  70%|#######   | 35138/50000 [22:52:15<7:58:18,  1.93s/it]
Outer training loop:  70%|#######   | 35139/50000 [22:52:17<7:43:53,  1.87s/it]
Outer training loop:  70%|#######   | 35140/50000 [22:52:18<7:44:59,  1.88s/it]
Outer training loop:  70%|#######   | 35141/50000 [22:52:20<7:54:48,  1.92s/it]
Outer training loop:  70%|#######   | 35142/50000 [22:52:22<7:43:48,  1.87s/it]
Outer training loop:  70%|#######   | 35143/50000 [22:52:24<7:39:26,  1.86s/it]
Outer training loop:  70%|#######   | 35144/50000 [22:52:26<7:29:10,  1.81s/it]
Outer training loop:  70%|#######   | 35145/50000 [22:52:28<7:39:09,  1.85s/it]
Outer training loop:  70%|#######   | 35146/50000 [22:52:29<7:35:49,  1.84s/it]
Outer training loop:  70%|#######   | 35147/50000 [22:52:31<7:35:43,  1.84s/it]
Outer training loop:  70%|#######   | 35148/50000 [22:52:33<7:42:09,  1.87s/it]
Outer training loop:  70%|#######   | 35149/50000 [22:52:35<7:44:24,  1.88s/it]
Outer training loop:  70%|#######   | 35150/50000 [22:52:37<7:37:12,  1.85s/it]
Outer training loop:  70%|#######   | 35151/50000 [22:52:39<7:51:28,  1.91s/it]
Outer training loop:  70%|#######   | 35152/50000 [22:52:41<7:44:44,  1.88s/it]
Outer training loop:  70%|#######   | 35153/50000 [22:52:43<7:39:43,  1.86s/it]
Outer training loop:  70%|#######   | 35154/50000 [22:52:45<7:50:16,  1.90s/it]
Outer training loop:  70%|#######   | 35155/50000 [22:52:46<7:42:55,  1.87s/it]
Outer training loop:  70%|#######   | 35156/50000 [22:52:48<7:34:45,  1.84s/it]
Outer training loop:  70%|#######   | 35157/50000 [22:52:50<7:38:31,  1.85s/it]
Outer training loop:  70%|#######   | 35158/50000 [22:52:52<7:55:34,  1.92s/it]
Outer training loop:  70%|#######   | 35159/50000 [22:52:54<7:54:52,  1.92s/it]
Outer training loop:  70%|#######   | 35160/50000 [22:52:56<7:48:27,  1.89s/it]
Outer training loop:  70%|#######   | 35161/50000 [22:52:58<7:51:34,  1.91s/it]
Outer training loop:  70%|#######   | 35162/50000 [22:53:00<7:42:48,  1.87s/it]
Outer training loop:  70%|#######   | 35163/50000 [22:53:01<7:42:55,  1.87s/it]
Outer training loop:  70%|#######   | 35164/50000 [22:53:03<7:32:54,  1.83s/it]
Outer training loop:  70%|#######   | 35165/50000 [22:53:05<7:53:09,  1.91s/it]
Outer training loop:  70%|#######   | 35166/50000 [22:53:07<7:45:02,  1.88s/it]
Outer training loop:  70%|#######   | 35167/50000 [22:53:09<7:32:59,  1.83s/it]
Outer training loop:  70%|#######   | 35168/50000 [22:53:11<7:56:36,  1.93s/it]
Outer training loop:  70%|#######   | 35169/50000 [22:53:13<7:53:00,  1.91s/it]
Outer training loop:  70%|#######   | 35170/50000 [22:53:15<7:43:42,  1.88s/it]
Outer training loop:  70%|#######   | 35171/50000 [22:53:17<7:49:52,  1.90s/it]
Outer training loop:  70%|#######   | 35172/50000 [22:53:19<8:00:42,  1.95s/it]
Outer training loop:  70%|#######   | 35173/50000 [22:53:20<7:46:45,  1.89s/it]
Outer training loop:  70%|#######   | 35174/50000 [22:53:22<7:44:00,  1.88s/it]
Outer training loop:  70%|#######   | 35175/50000 [22:53:25<8:18:25,  2.02s/it]
Outer training loop:  70%|#######   | 35176/50000 [22:53:26<8:05:52,  1.97s/it]
Outer training loop:  70%|#######   | 35177/50000 [22:53:28<7:51:53,  1.91s/it]
Outer training loop:  70%|#######   | 35178/50000 [22:53:30<7:51:23,  1.91s/it]
Outer training loop:  70%|#######   | 35179/50000 [22:53:32<7:46:27,  1.89s/it]
Outer training loop:  70%|#######   | 35180/50000 [22:53:34<7:37:24,  1.85s/it]
Outer training loop:  70%|#######   | 35181/50000 [22:53:36<7:38:43,  1.86s/it]
Outer training loop:  70%|#######   | 35182/50000 [22:53:38<7:52:56,  1.92s/it]
Outer training loop:  70%|#######   | 35183/50000 [22:53:39<7:42:34,  1.87s/it]
Outer training loop:  70%|#######   | 35184/50000 [22:53:41<7:29:32,  1.82s/it]
Outer training loop:  70%|#######   | 35185/50000 [22:53:43<7:44:37,  1.88s/it]
Outer training loop:  70%|#######   | 35186/50000 [22:53:45<7:42:41,  1.87s/it]
Outer training loop:  70%|#######   | 35187/50000 [22:53:47<7:36:09,  1.85s/it]
Outer training loop:  70%|#######   | 35188/50000 [22:53:49<7:30:17,  1.82s/it]
Outer training loop:  70%|#######   | 35189/50000 [22:53:51<7:45:34,  1.89s/it]
Outer training loop:  70%|#######   | 35190/50000 [22:53:52<7:37:16,  1.85s/it]
Outer training loop:  70%|#######   | 35191/50000 [22:53:54<7:36:43,  1.85s/it]
Outer training loop:  70%|#######   | 35192/50000 [22:53:56<7:53:56,  1.92s/it]
Outer training loop:  70%|#######   | 35193/50000 [22:53:58<8:01:56,  1.95s/it]
Outer training loop:  70%|#######   | 35194/50000 [22:54:00<7:46:55,  1.89s/it]
Outer training loop:  70%|#######   | 35195/50000 [22:54:02<7:51:30,  1.91s/it]
Outer training loop:  70%|#######   | 35196/50000 [22:54:04<7:53:43,  1.92s/it]
Outer training loop:  70%|#######   | 35197/50000 [22:54:06<7:47:47,  1.90s/it]
Outer training loop:  70%|#######   | 35198/50000 [22:54:08<7:48:50,  1.90s/it]
Outer training loop:  70%|#######   | 35199/50000 [22:54:10<7:47:27,  1.89s/it]
Outer training loop:  70%|#######   | 35200/50000 [22:54:11<7:39:46,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.99s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.98s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]

Outer training loop:  70%|#######   | 35201/50000 [22:54:19<14:31:40,  3.53s/it]
Outer training loop:  70%|#######   | 35202/50000 [22:54:21<12:40:14,  3.08s/it]
Outer training loop:  70%|#######   | 35203/50000 [22:54:23<11:05:01,  2.70s/it]
Outer training loop:  70%|#######   | 35204/50000 [22:54:25<10:03:31,  2.45s/it]
Outer training loop:  70%|#######   | 35205/50000 [22:54:27<9:29:33,  2.31s/it] 
Outer training loop:  70%|#######   | 35206/50000 [22:54:28<8:59:41,  2.19s/it]
Outer training loop:  70%|#######   | 35207/50000 [22:54:30<8:25:04,  2.05s/it]
Outer training loop:  70%|#######   | 35208/50000 [22:54:32<8:14:35,  2.01s/it]
Outer training loop:  70%|#######   | 35209/50000 [22:54:34<7:53:51,  1.92s/it]
Outer training loop:  70%|#######   | 35210/50000 [22:54:36<7:49:04,  1.90s/it]
Outer training loop:  70%|#######   | 35211/50000 [22:54:37<7:41:18,  1.87s/it]
Outer training loop:  70%|#######   | 35212/50000 [22:54:39<7:50:40,  1.91s/it]
Outer training loop:  70%|#######   | 35213/50000 [22:54:41<7:41:01,  1.87s/it]
Outer training loop:  70%|#######   | 35214/50000 [22:54:43<7:41:42,  1.87s/it]
Outer training loop:  70%|#######   | 35215/50000 [22:54:45<7:51:36,  1.91s/it]
Outer training loop:  70%|#######   | 35216/50000 [22:54:47<7:41:18,  1.87s/it]
Outer training loop:  70%|#######   | 35217/50000 [22:54:49<7:31:48,  1.83s/it]
Outer training loop:  70%|#######   | 35218/50000 [22:54:51<7:35:02,  1.85s/it]
Outer training loop:  70%|#######   | 35219/50000 [22:54:53<7:51:02,  1.91s/it]
Outer training loop:  70%|#######   | 35220/50000 [22:54:54<7:51:42,  1.91s/it]
Outer training loop:  70%|#######   | 35221/50000 [22:54:56<7:47:13,  1.90s/it]
Outer training loop:  70%|#######   | 35222/50000 [22:54:59<8:14:30,  2.01s/it]
Outer training loop:  70%|#######   | 35223/50000 [22:55:00<8:03:06,  1.96s/it]
Outer training loop:  70%|#######   | 35224/50000 [22:55:02<7:53:51,  1.92s/it]
Outer training loop:  70%|#######   | 35225/50000 [22:55:04<8:04:43,  1.97s/it]
Outer training loop:  70%|#######   | 35226/50000 [22:55:06<7:56:18,  1.93s/it]
Outer training loop:  70%|#######   | 35227/50000 [22:55:08<7:46:15,  1.89s/it]
Outer training loop:  70%|#######   | 35228/50000 [22:55:10<8:06:38,  1.98s/it]
Outer training loop:  70%|#######   | 35229/50000 [22:55:12<7:57:14,  1.94s/it]
Outer training loop:  70%|#######   | 35230/50000 [22:55:14<7:47:00,  1.90s/it]
Outer training loop:  70%|#######   | 35231/50000 [22:55:16<7:35:29,  1.85s/it]
Outer training loop:  70%|#######   | 35232/50000 [22:55:18<7:44:40,  1.89s/it]
Outer training loop:  70%|#######   | 35233/50000 [22:55:19<7:44:02,  1.89s/it]
Outer training loop:  70%|#######   | 35234/50000 [22:55:21<7:49:48,  1.91s/it]
Outer training loop:  70%|#######   | 35235/50000 [22:55:23<8:00:56,  1.95s/it]
Outer training loop:  70%|#######   | 35236/50000 [22:55:25<7:54:54,  1.93s/it]
Outer training loop:  70%|#######   | 35237/50000 [22:55:27<7:51:36,  1.92s/it]
Outer training loop:  70%|#######   | 35238/50000 [22:55:29<7:45:03,  1.89s/it]
Outer training loop:  70%|#######   | 35239/50000 [22:55:31<7:47:24,  1.90s/it]
Outer training loop:  70%|#######   | 35240/50000 [22:55:33<7:45:28,  1.89s/it]
Outer training loop:  70%|#######   | 35241/50000 [22:55:35<7:36:58,  1.86s/it]
Outer training loop:  70%|#######   | 35242/50000 [22:55:37<7:50:32,  1.91s/it]
Outer training loop:  70%|#######   | 35243/50000 [22:55:38<7:38:00,  1.86s/it]
Outer training loop:  70%|#######   | 35244/50000 [22:55:40<7:48:30,  1.91s/it]
Outer training loop:  70%|#######   | 35245/50000 [22:55:42<7:41:41,  1.88s/it]
Outer training loop:  70%|#######   | 35246/50000 [22:55:44<7:48:01,  1.90s/it]
Outer training loop:  70%|#######   | 35247/50000 [22:55:46<7:44:47,  1.89s/it]
Outer training loop:  70%|#######   | 35248/50000 [22:55:48<7:40:27,  1.87s/it]
Outer training loop:  70%|#######   | 35249/50000 [22:55:50<7:56:40,  1.94s/it]
Outer training loop:  70%|#######   | 35250/50000 [22:55:52<7:49:53,  1.91s/it]
Outer training loop:  71%|#######   | 35251/50000 [22:55:54<7:41:33,  1.88s/it]
Outer training loop:  71%|#######   | 35252/50000 [22:55:56<7:44:44,  1.89s/it]
Outer training loop:  71%|#######   | 35253/50000 [22:55:57<7:33:47,  1.85s/it]
Outer training loop:  71%|#######   | 35254/50000 [22:55:59<7:23:08,  1.80s/it]
Outer training loop:  71%|#######   | 35255/50000 [22:56:01<7:17:28,  1.78s/it]
Outer training loop:  71%|#######   | 35256/50000 [22:56:03<7:36:27,  1.86s/it]
Outer training loop:  71%|#######   | 35257/50000 [22:56:05<7:33:37,  1.85s/it]
Outer training loop:  71%|#######   | 35258/50000 [22:56:06<7:32:27,  1.84s/it]
Outer training loop:  71%|#######   | 35259/50000 [22:56:08<7:48:52,  1.91s/it]
Outer training loop:  71%|#######   | 35260/50000 [22:56:10<7:48:25,  1.91s/it]
Outer training loop:  71%|#######   | 35261/50000 [22:56:12<7:50:13,  1.91s/it]
Outer training loop:  71%|#######   | 35262/50000 [22:56:14<7:38:34,  1.87s/it]
Outer training loop:  71%|#######   | 35263/50000 [22:56:16<7:42:32,  1.88s/it]
Outer training loop:  71%|#######   | 35264/50000 [22:56:18<7:45:13,  1.89s/it]
Outer training loop:  71%|#######   | 35265/50000 [22:56:20<7:40:25,  1.87s/it]
Outer training loop:  71%|#######   | 35266/50000 [22:56:22<7:48:03,  1.91s/it]
Outer training loop:  71%|#######   | 35267/50000 [22:56:24<7:40:37,  1.88s/it]
Outer training loop:  71%|#######   | 35268/50000 [22:56:25<7:42:22,  1.88s/it]
Outer training loop:  71%|#######   | 35269/50000 [22:56:27<7:50:08,  1.91s/it]
Outer training loop:  71%|#######   | 35270/50000 [22:56:29<7:47:35,  1.90s/it]
Outer training loop:  71%|#######   | 35271/50000 [22:56:31<7:36:28,  1.86s/it]
Outer training loop:  71%|#######   | 35272/50000 [22:56:33<7:57:42,  1.95s/it]
Outer training loop:  71%|#######   | 35273/50000 [22:56:35<7:44:03,  1.89s/it]
Outer training loop:  71%|#######   | 35274/50000 [22:56:37<7:41:18,  1.88s/it]
Outer training loop:  71%|#######   | 35275/50000 [22:56:39<7:34:23,  1.85s/it]
Outer training loop:  71%|#######   | 35276/50000 [22:56:41<7:52:08,  1.92s/it]
Outer training loop:  71%|#######   | 35277/50000 [22:56:43<7:48:56,  1.91s/it]
Outer training loop:  71%|#######   | 35278/50000 [22:56:44<7:45:13,  1.90s/it]
Outer training loop:  71%|#######   | 35279/50000 [22:56:47<7:58:36,  1.95s/it]
Outer training loop:  71%|#######   | 35280/50000 [22:56:48<7:45:34,  1.90s/it]
Outer training loop:  71%|#######   | 35281/50000 [22:56:50<7:50:44,  1.92s/it]
Outer training loop:  71%|#######   | 35282/50000 [22:56:52<8:12:01,  2.01s/it]
Outer training loop:  71%|#######   | 35283/50000 [22:56:54<8:04:23,  1.97s/it]
Outer training loop:  71%|#######   | 35284/50000 [22:56:56<7:57:24,  1.95s/it]
Outer training loop:  71%|#######   | 35285/50000 [22:56:58<8:01:49,  1.96s/it]
Outer training loop:  71%|#######   | 35286/50000 [22:57:00<8:17:53,  2.03s/it]
Outer training loop:  71%|#######   | 35287/50000 [22:57:02<8:06:53,  1.99s/it]
Outer training loop:  71%|#######   | 35288/50000 [22:57:04<7:53:45,  1.93s/it]
Outer training loop:  71%|#######   | 35289/50000 [22:57:06<8:02:48,  1.97s/it]
Outer training loop:  71%|#######   | 35290/50000 [22:57:08<7:57:49,  1.95s/it]
Outer training loop:  71%|#######   | 35291/50000 [22:57:10<7:56:43,  1.94s/it]
Outer training loop:  71%|#######   | 35292/50000 [22:57:12<7:47:44,  1.91s/it]
Outer training loop:  71%|#######   | 35293/50000 [22:57:14<8:06:25,  1.98s/it]
Outer training loop:  71%|#######   | 35294/50000 [22:57:16<8:04:13,  1.98s/it]
Outer training loop:  71%|#######   | 35295/50000 [22:57:18<7:53:18,  1.93s/it]
Outer training loop:  71%|#######   | 35296/50000 [22:57:20<8:12:21,  2.01s/it]
Outer training loop:  71%|#######   | 35297/50000 [22:57:22<7:58:09,  1.95s/it]
Outer training loop:  71%|#######   | 35298/50000 [22:57:24<7:52:27,  1.93s/it]
Outer training loop:  71%|#######   | 35299/50000 [22:57:26<8:00:56,  1.96s/it]
Outer training loop:  71%|#######   | 35300/50000 [22:57:28<7:55:00,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  71%|#######   | 35301/50000 [22:57:35<14:37:42,  3.58s/it]
Outer training loop:  71%|#######   | 35302/50000 [22:57:37<12:37:15,  3.09s/it]
Outer training loop:  71%|#######   | 35303/50000 [22:57:39<11:16:19,  2.76s/it]
Outer training loop:  71%|#######   | 35304/50000 [22:57:41<10:18:06,  2.52s/it]
Outer training loop:  71%|#######   | 35305/50000 [22:57:43<9:25:51,  2.31s/it] 
Outer training loop:  71%|#######   | 35306/50000 [22:57:45<8:47:41,  2.15s/it]
Outer training loop:  71%|#######   | 35307/50000 [22:57:47<8:47:11,  2.15s/it]
Outer training loop:  71%|#######   | 35308/50000 [22:57:49<8:27:13,  2.07s/it]
Outer training loop:  71%|#######   | 35309/50000 [22:57:50<8:17:38,  2.03s/it]
Outer training loop:  71%|#######   | 35310/50000 [22:57:52<8:07:12,  1.99s/it]
Outer training loop:  71%|#######   | 35311/50000 [22:57:54<8:13:10,  2.01s/it]
Outer training loop:  71%|#######   | 35312/50000 [22:57:56<8:05:37,  1.98s/it]
Outer training loop:  71%|#######   | 35313/50000 [22:57:58<8:01:42,  1.97s/it]
Outer training loop:  71%|#######   | 35314/50000 [22:58:00<7:53:16,  1.93s/it]
Outer training loop:  71%|#######   | 35315/50000 [22:58:02<7:46:50,  1.91s/it]
Outer training loop:  71%|#######   | 35316/50000 [22:58:04<7:45:23,  1.90s/it]
Outer training loop:  71%|#######   | 35317/50000 [22:58:06<7:56:09,  1.95s/it]
Outer training loop:  71%|#######   | 35318/50000 [22:58:08<7:45:04,  1.90s/it]
Outer training loop:  71%|#######   | 35319/50000 [22:58:10<8:02:54,  1.97s/it]
Outer training loop:  71%|#######   | 35320/50000 [22:58:12<8:11:21,  2.01s/it]
Outer training loop:  71%|#######   | 35321/50000 [22:58:14<7:58:06,  1.95s/it]
Outer training loop:  71%|#######   | 35322/50000 [22:58:16<7:55:15,  1.94s/it]
Outer training loop:  71%|#######   | 35323/50000 [22:58:18<7:56:23,  1.95s/it]
Outer training loop:  71%|#######   | 35324/50000 [22:58:20<8:16:36,  2.03s/it]
Outer training loop:  71%|#######   | 35325/50000 [22:58:22<8:12:49,  2.01s/it]
Outer training loop:  71%|#######   | 35326/50000 [22:58:24<7:56:36,  1.95s/it]
Outer training loop:  71%|#######   | 35327/50000 [22:58:26<8:11:18,  2.01s/it]
Outer training loop:  71%|#######   | 35328/50000 [22:58:28<8:03:18,  1.98s/it]
Outer training loop:  71%|#######   | 35329/50000 [22:58:30<8:00:22,  1.96s/it]
Outer training loop:  71%|#######   | 35330/50000 [22:58:32<8:03:21,  1.98s/it]
Outer training loop:  71%|#######   | 35331/50000 [22:58:33<7:48:36,  1.92s/it]
Outer training loop:  71%|#######   | 35332/50000 [22:58:35<7:49:15,  1.92s/it]
Outer training loop:  71%|#######   | 35333/50000 [22:58:37<7:49:17,  1.92s/it]
Outer training loop:  71%|#######   | 35334/50000 [22:58:39<8:05:30,  1.99s/it]
Outer training loop:  71%|#######   | 35335/50000 [22:58:41<7:45:41,  1.91s/it]
Outer training loop:  71%|#######   | 35336/50000 [22:58:43<7:37:20,  1.87s/it]
Outer training loop:  71%|#######   | 35337/50000 [22:58:45<7:24:56,  1.82s/it]
Outer training loop:  71%|#######   | 35338/50000 [22:58:47<7:39:16,  1.88s/it]
Outer training loop:  71%|#######   | 35339/50000 [22:58:48<7:34:09,  1.86s/it]
Outer training loop:  71%|#######   | 35340/50000 [22:58:50<7:36:26,  1.87s/it]
Outer training loop:  71%|#######   | 35341/50000 [22:58:53<8:01:36,  1.97s/it]
Outer training loop:  71%|#######   | 35342/50000 [22:58:55<8:01:10,  1.97s/it]
Outer training loop:  71%|#######   | 35343/50000 [22:58:56<7:55:39,  1.95s/it]
Outer training loop:  71%|#######   | 35344/50000 [22:58:58<7:58:42,  1.96s/it]
Outer training loop:  71%|#######   | 35345/50000 [22:59:00<7:46:56,  1.91s/it]
Outer training loop:  71%|#######   | 35346/50000 [22:59:02<7:38:14,  1.88s/it]
Outer training loop:  71%|#######   | 35347/50000 [22:59:04<8:00:45,  1.97s/it]
Outer training loop:  71%|#######   | 35348/50000 [22:59:06<7:55:50,  1.95s/it]
Outer training loop:  71%|#######   | 35349/50000 [22:59:08<7:43:46,  1.90s/it]
Outer training loop:  71%|#######   | 35350/50000 [22:59:10<7:36:32,  1.87s/it]
Outer training loop:  71%|#######   | 35351/50000 [22:59:12<7:51:35,  1.93s/it]
Outer training loop:  71%|#######   | 35352/50000 [22:59:14<7:42:09,  1.89s/it]
Outer training loop:  71%|#######   | 35353/50000 [22:59:15<7:46:08,  1.91s/it]
Outer training loop:  71%|#######   | 35354/50000 [22:59:18<7:55:42,  1.95s/it]
Outer training loop:  71%|#######   | 35355/50000 [22:59:20<7:59:30,  1.96s/it]
Outer training loop:  71%|#######   | 35356/50000 [22:59:21<7:42:27,  1.89s/it]
Outer training loop:  71%|#######   | 35357/50000 [22:59:23<7:44:09,  1.90s/it]
Outer training loop:  71%|#######   | 35358/50000 [22:59:25<7:35:12,  1.87s/it]
Outer training loop:  71%|#######   | 35359/50000 [22:59:27<7:36:53,  1.87s/it]
Outer training loop:  71%|#######   | 35360/50000 [22:59:29<7:37:29,  1.87s/it]
Outer training loop:  71%|#######   | 35361/50000 [22:59:31<7:53:15,  1.94s/it]
Outer training loop:  71%|#######   | 35362/50000 [22:59:33<7:34:01,  1.86s/it]
Outer training loop:  71%|#######   | 35363/50000 [22:59:34<7:29:56,  1.84s/it]
Outer training loop:  71%|#######   | 35364/50000 [22:59:36<7:39:58,  1.89s/it]
Outer training loop:  71%|#######   | 35365/50000 [22:59:38<7:40:14,  1.89s/it]
Outer training loop:  71%|#######   | 35366/50000 [22:59:40<7:44:15,  1.90s/it]
Outer training loop:  71%|#######   | 35367/50000 [22:59:42<7:51:56,  1.94s/it]
Outer training loop:  71%|#######   | 35368/50000 [22:59:44<7:33:33,  1.86s/it]
Outer training loop:  71%|#######   | 35369/50000 [22:59:46<7:42:40,  1.90s/it]
Outer training loop:  71%|#######   | 35370/50000 [22:59:48<7:34:03,  1.86s/it]
Outer training loop:  71%|#######   | 35371/50000 [22:59:50<7:53:48,  1.94s/it]
Outer training loop:  71%|#######   | 35372/50000 [22:59:52<7:42:57,  1.90s/it]
Outer training loop:  71%|#######   | 35373/50000 [22:59:53<7:44:55,  1.91s/it]
Outer training loop:  71%|#######   | 35374/50000 [22:59:55<7:52:30,  1.94s/it]
Outer training loop:  71%|#######   | 35375/50000 [22:59:57<7:41:02,  1.89s/it]
Outer training loop:  71%|#######   | 35376/50000 [22:59:59<7:45:56,  1.91s/it]
Outer training loop:  71%|#######   | 35377/50000 [23:00:01<7:40:24,  1.89s/it]
Outer training loop:  71%|#######   | 35378/50000 [23:00:03<7:54:23,  1.95s/it]
Outer training loop:  71%|#######   | 35379/50000 [23:00:05<8:03:57,  1.99s/it]
Outer training loop:  71%|#######   | 35380/50000 [23:00:07<7:48:36,  1.92s/it]
Outer training loop:  71%|#######   | 35381/50000 [23:00:09<7:52:48,  1.94s/it]
Outer training loop:  71%|#######   | 35382/50000 [23:00:11<7:54:41,  1.95s/it]
Outer training loop:  71%|#######   | 35383/50000 [23:00:13<7:47:48,  1.92s/it]
Outer training loop:  71%|#######   | 35384/50000 [23:00:15<8:05:39,  1.99s/it]
Outer training loop:  71%|#######   | 35385/50000 [23:00:17<7:54:26,  1.95s/it]
Outer training loop:  71%|#######   | 35386/50000 [23:00:19<7:38:52,  1.88s/it]
Outer training loop:  71%|#######   | 35387/50000 [23:00:20<7:37:45,  1.88s/it]
Outer training loop:  71%|#######   | 35388/50000 [23:00:22<7:44:57,  1.91s/it]
Outer training loop:  71%|#######   | 35389/50000 [23:00:24<7:36:11,  1.87s/it]
Outer training loop:  71%|#######   | 35390/50000 [23:00:26<7:37:09,  1.88s/it]
Outer training loop:  71%|#######   | 35391/50000 [23:00:28<7:44:42,  1.91s/it]
Outer training loop:  71%|#######   | 35392/50000 [23:00:30<7:44:58,  1.91s/it]
Outer training loop:  71%|#######   | 35393/50000 [23:00:32<7:37:27,  1.88s/it]
Outer training loop:  71%|#######   | 35394/50000 [23:00:34<7:37:44,  1.88s/it]
Outer training loop:  71%|#######   | 35395/50000 [23:00:36<7:57:43,  1.96s/it]
Outer training loop:  71%|#######   | 35396/50000 [23:00:38<7:42:20,  1.90s/it]
Outer training loop:  71%|#######   | 35397/50000 [23:00:39<7:38:44,  1.88s/it]
Outer training loop:  71%|#######   | 35398/50000 [23:00:42<7:56:57,  1.96s/it]
Outer training loop:  71%|#######   | 35399/50000 [23:00:43<7:47:02,  1.92s/it]
Outer training loop:  71%|#######   | 35400/50000 [23:00:45<7:40:54,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  71%|#######   | 35401/50000 [23:00:53<14:27:57,  3.57s/it]
Outer training loop:  71%|#######   | 35402/50000 [23:00:55<12:45:31,  3.15s/it]
Outer training loop:  71%|#######   | 35403/50000 [23:00:57<11:09:53,  2.75s/it]
Outer training loop:  71%|#######   | 35404/50000 [23:00:58<10:03:59,  2.48s/it]
Outer training loop:  71%|#######   | 35405/50000 [23:01:00<9:26:23,  2.33s/it] 
Outer training loop:  71%|#######   | 35406/50000 [23:01:02<8:46:45,  2.17s/it]
Outer training loop:  71%|#######   | 35407/50000 [23:01:04<8:24:07,  2.07s/it]
Outer training loop:  71%|#######   | 35408/50000 [23:01:06<8:10:01,  2.01s/it]
Outer training loop:  71%|#######   | 35409/50000 [23:01:08<8:12:06,  2.02s/it]
Outer training loop:  71%|#######   | 35410/50000 [23:01:10<8:04:08,  1.99s/it]
Outer training loop:  71%|#######   | 35411/50000 [23:01:12<7:42:34,  1.90s/it]
Outer training loop:  71%|#######   | 35412/50000 [23:01:14<7:54:29,  1.95s/it]
Outer training loop:  71%|#######   | 35413/50000 [23:01:15<7:40:39,  1.89s/it]
Outer training loop:  71%|#######   | 35414/50000 [23:01:17<7:35:20,  1.87s/it]
Outer training loop:  71%|#######   | 35415/50000 [23:01:19<7:31:36,  1.86s/it]
Outer training loop:  71%|#######   | 35416/50000 [23:01:21<8:04:57,  2.00s/it]
Outer training loop:  71%|#######   | 35417/50000 [23:01:23<7:43:24,  1.91s/it]
Outer training loop:  71%|#######   | 35418/50000 [23:01:25<7:54:52,  1.95s/it]
Outer training loop:  71%|#######   | 35419/50000 [23:01:27<7:57:20,  1.96s/it]
Outer training loop:  71%|#######   | 35420/50000 [23:01:29<7:50:01,  1.93s/it]
Outer training loop:  71%|#######   | 35421/50000 [23:01:31<7:42:33,  1.90s/it]
Outer training loop:  71%|#######   | 35422/50000 [23:01:33<7:37:26,  1.88s/it]
Outer training loop:  71%|#######   | 35423/50000 [23:01:35<7:52:25,  1.94s/it]
Outer training loop:  71%|#######   | 35424/50000 [23:01:37<7:36:39,  1.88s/it]
Outer training loop:  71%|#######   | 35425/50000 [23:01:38<7:31:35,  1.86s/it]
Outer training loop:  71%|#######   | 35426/50000 [23:01:40<7:46:17,  1.92s/it]
Outer training loop:  71%|#######   | 35427/50000 [23:01:42<7:44:40,  1.91s/it]
Outer training loop:  71%|#######   | 35428/50000 [23:01:44<7:28:53,  1.85s/it]
Outer training loop:  71%|#######   | 35429/50000 [23:01:46<7:38:37,  1.89s/it]
Outer training loop:  71%|#######   | 35430/50000 [23:01:48<7:31:13,  1.86s/it]
Outer training loop:  71%|#######   | 35431/50000 [23:01:50<7:25:20,  1.83s/it]
Outer training loop:  71%|#######   | 35432/50000 [23:01:51<7:27:35,  1.84s/it]
Outer training loop:  71%|#######   | 35433/50000 [23:01:53<7:45:30,  1.92s/it]
Outer training loop:  71%|#######   | 35434/50000 [23:01:55<7:32:04,  1.86s/it]
Outer training loop:  71%|#######   | 35435/50000 [23:01:57<7:27:38,  1.84s/it]
Outer training loop:  71%|#######   | 35436/50000 [23:01:59<7:27:31,  1.84s/it]
Outer training loop:  71%|#######   | 35437/50000 [23:02:01<7:33:01,  1.87s/it]
Outer training loop:  71%|#######   | 35438/50000 [23:02:03<7:22:25,  1.82s/it]
Outer training loop:  71%|#######   | 35439/50000 [23:02:04<7:31:57,  1.86s/it]
Outer training loop:  71%|#######   | 35440/50000 [23:02:07<7:45:18,  1.92s/it]
Outer training loop:  71%|#######   | 35441/50000 [23:02:08<7:33:55,  1.87s/it]
Outer training loop:  71%|#######   | 35442/50000 [23:02:10<7:26:56,  1.84s/it]
Outer training loop:  71%|#######   | 35443/50000 [23:02:12<7:18:19,  1.81s/it]
Outer training loop:  71%|#######   | 35444/50000 [23:02:14<7:30:41,  1.86s/it]
Outer training loop:  71%|#######   | 35445/50000 [23:02:16<7:26:28,  1.84s/it]
Outer training loop:  71%|#######   | 35446/50000 [23:02:17<7:24:48,  1.83s/it]
Outer training loop:  71%|#######   | 35447/50000 [23:02:20<7:55:51,  1.96s/it]
Outer training loop:  71%|#######   | 35448/50000 [23:02:21<7:37:53,  1.89s/it]
Outer training loop:  71%|#######   | 35449/50000 [23:02:23<7:32:54,  1.87s/it]
Outer training loop:  71%|#######   | 35450/50000 [23:02:25<7:47:08,  1.93s/it]
Outer training loop:  71%|#######   | 35451/50000 [23:02:27<7:49:24,  1.94s/it]
Outer training loop:  71%|#######   | 35452/50000 [23:02:29<7:44:36,  1.92s/it]
Outer training loop:  71%|#######   | 35453/50000 [23:02:31<7:40:27,  1.90s/it]
Outer training loop:  71%|#######   | 35454/50000 [23:02:33<7:53:01,  1.95s/it]
Outer training loop:  71%|#######   | 35455/50000 [23:02:35<7:45:24,  1.92s/it]
Outer training loop:  71%|#######   | 35456/50000 [23:02:37<7:38:39,  1.89s/it]
Outer training loop:  71%|#######   | 35457/50000 [23:02:39<7:51:21,  1.94s/it]
Outer training loop:  71%|#######   | 35458/50000 [23:02:41<7:41:03,  1.90s/it]
Outer training loop:  71%|#######   | 35459/50000 [23:02:42<7:30:12,  1.86s/it]
Outer training loop:  71%|#######   | 35460/50000 [23:02:44<7:41:54,  1.91s/it]
Outer training loop:  71%|#######   | 35461/50000 [23:02:46<7:34:02,  1.87s/it]
Outer training loop:  71%|#######   | 35462/50000 [23:02:48<7:28:40,  1.85s/it]
Outer training loop:  71%|#######   | 35463/50000 [23:02:50<7:20:08,  1.82s/it]
Outer training loop:  71%|#######   | 35464/50000 [23:02:52<7:33:48,  1.87s/it]
Outer training loop:  71%|#######   | 35465/50000 [23:02:53<7:30:44,  1.86s/it]
Outer training loop:  71%|#######   | 35466/50000 [23:02:56<7:42:38,  1.91s/it]
Outer training loop:  71%|#######   | 35467/50000 [23:02:58<7:51:57,  1.95s/it]
Outer training loop:  71%|#######   | 35468/50000 [23:02:59<7:46:12,  1.92s/it]
Outer training loop:  71%|#######   | 35469/50000 [23:03:01<7:40:35,  1.90s/it]
Outer training loop:  71%|#######   | 35470/50000 [23:03:03<7:35:13,  1.88s/it]
Outer training loop:  71%|#######   | 35471/50000 [23:03:05<7:51:48,  1.95s/it]
Outer training loop:  71%|#######   | 35472/50000 [23:03:07<7:47:39,  1.93s/it]
Outer training loop:  71%|#######   | 35473/50000 [23:03:09<7:49:32,  1.94s/it]
Outer training loop:  71%|#######   | 35474/50000 [23:03:11<7:57:28,  1.97s/it]
Outer training loop:  71%|#######   | 35475/50000 [23:03:13<7:42:40,  1.91s/it]
Outer training loop:  71%|#######   | 35476/50000 [23:03:15<7:44:38,  1.92s/it]
Outer training loop:  71%|#######   | 35477/50000 [23:03:17<7:36:29,  1.89s/it]
Outer training loop:  71%|#######   | 35478/50000 [23:03:19<8:01:25,  1.99s/it]
Outer training loop:  71%|#######   | 35479/50000 [23:03:21<7:52:18,  1.95s/it]
Outer training loop:  71%|#######   | 35480/50000 [23:03:23<7:45:41,  1.92s/it]
Outer training loop:  71%|#######   | 35481/50000 [23:03:25<7:57:24,  1.97s/it]
Outer training loop:  71%|#######   | 35482/50000 [23:03:26<7:47:15,  1.93s/it]
Outer training loop:  71%|#######   | 35483/50000 [23:03:28<7:30:32,  1.86s/it]
Outer training loop:  71%|#######   | 35484/50000 [23:03:30<7:40:47,  1.90s/it]
Outer training loop:  71%|#######   | 35485/50000 [23:03:32<7:37:49,  1.89s/it]
Outer training loop:  71%|#######   | 35486/50000 [23:03:34<7:27:16,  1.85s/it]
Outer training loop:  71%|#######   | 35487/50000 [23:03:36<7:24:47,  1.84s/it]
Outer training loop:  71%|#######   | 35488/50000 [23:03:38<7:28:06,  1.85s/it]
Outer training loop:  71%|#######   | 35489/50000 [23:03:39<7:27:54,  1.85s/it]
Outer training loop:  71%|#######   | 35490/50000 [23:03:41<7:26:50,  1.85s/it]
Outer training loop:  71%|#######   | 35491/50000 [23:03:43<7:46:06,  1.93s/it]
Outer training loop:  71%|#######   | 35492/50000 [23:03:45<7:38:45,  1.90s/it]
Outer training loop:  71%|#######   | 35493/50000 [23:03:47<7:29:14,  1.86s/it]
Outer training loop:  71%|#######   | 35494/50000 [23:03:49<7:28:23,  1.85s/it]
Outer training loop:  71%|#######   | 35495/50000 [23:03:51<7:53:48,  1.96s/it]
Outer training loop:  71%|#######   | 35496/50000 [23:03:53<7:30:59,  1.87s/it]
Outer training loop:  71%|#######   | 35497/50000 [23:03:54<7:22:03,  1.83s/it]
Outer training loop:  71%|#######   | 35498/50000 [23:03:56<7:22:22,  1.83s/it]
Outer training loop:  71%|#######   | 35499/50000 [23:03:58<7:28:13,  1.85s/it]
Outer training loop:  71%|#######1  | 35500/50000 [23:04:00<7:18:27,  1.81s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:26,  1.78s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:30,  1.86s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:29,  1.87s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.81s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.87s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.86s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:19,  1.86s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:16,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:18,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:17,  1.96s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:15,  1.96s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.94s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.90s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:01,  1.95s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.90s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.92s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:45,  1.85s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.93s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:43,  1.88s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:38,  1.83s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.87s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:39,  1.95s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:27,  1.85s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:23,  1.82s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:25,  1.90s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:22,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:10,  1.82s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:08,  1.85s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:01,  1.80s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<01:00,  1.94s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:57,  1.90s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:47,  1.96s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:44,  1.93s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:40,  1.94s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.90s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:31,  1.99s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:29,  1.96s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.91s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.90s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:14,  2.02s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.97s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.93s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.91s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.81s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:21,  1.73s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:35,  1.92s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.90s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:24,  1.86s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:18,  1.85s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:14,  1.82s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:17,  1.88s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:16,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:09,  1.85s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:05,  1.93s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:52,  1.82s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:49,  1.80s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:47,  1.85s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:45,  1.84s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:44,  1.93s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:35,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:32,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:33,  1.95s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:32,  1.96s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:28,  1.92s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.89s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:26,  1.96s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:23,  1.94s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.89s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.88s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.87s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.87s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:12,  1.97s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:10,  1.95s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.90s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:59,  1.97s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:55,  1.92s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.89s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:45,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:43,  1.97s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.92s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.88s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.82s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.85s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:27,  1.98s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.92s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.84s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.80s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:10,  1.79s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.94s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.93s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.88s/it]

Outer training loop:  71%|#######1  | 35501/50000 [23:09:20<392:16:52, 97.40s/it]
Outer training loop:  71%|#######1  | 35502/50000 [23:09:22<276:51:09, 68.75s/it]
Outer training loop:  71%|#######1  | 35503/50000 [23:09:24<196:04:18, 48.69s/it]
Outer training loop:  71%|#######1  | 35504/50000 [23:09:26<139:31:18, 34.65s/it]
Outer training loop:  71%|#######1  | 35505/50000 [23:09:28<99:57:00, 24.82s/it] 
Outer training loop:  71%|#######1  | 35506/50000 [23:09:30<72:12:28, 17.93s/it]
Outer training loop:  71%|#######1  | 35507/50000 [23:09:32<53:34:59, 13.31s/it]
Outer training loop:  71%|#######1  | 35508/50000 [23:09:34<39:50:54,  9.90s/it]
Outer training loop:  71%|#######1  | 35509/50000 [23:09:36<30:13:30,  7.51s/it]
Outer training loop:  71%|#######1  | 35510/50000 [23:09:38<23:35:00,  5.86s/it]
Outer training loop:  71%|#######1  | 35511/50000 [23:09:40<18:39:19,  4.64s/it]
Outer training loop:  71%|#######1  | 35512/50000 [23:09:42<15:13:14,  3.78s/it]
Outer training loop:  71%|#######1  | 35513/50000 [23:09:44<13:03:58,  3.25s/it]
Outer training loop:  71%|#######1  | 35514/50000 [23:09:45<11:20:35,  2.82s/it]
Outer training loop:  71%|#######1  | 35515/50000 [23:09:47<10:08:43,  2.52s/it]
Outer training loop:  71%|#######1  | 35516/50000 [23:09:49<9:16:35,  2.31s/it] 
Outer training loop:  71%|#######1  | 35517/50000 [23:09:51<8:53:44,  2.21s/it]
Outer training loop:  71%|#######1  | 35518/50000 [23:09:53<8:22:49,  2.08s/it]
Outer training loop:  71%|#######1  | 35519/50000 [23:09:55<8:06:42,  2.02s/it]
Outer training loop:  71%|#######1  | 35520/50000 [23:09:57<8:04:43,  2.01s/it]
Outer training loop:  71%|#######1  | 35521/50000 [23:09:59<7:52:03,  1.96s/it]
Outer training loop:  71%|#######1  | 35522/50000 [23:10:00<7:44:27,  1.92s/it]
Outer training loop:  71%|#######1  | 35523/50000 [23:10:02<7:41:35,  1.91s/it]
Outer training loop:  71%|#######1  | 35524/50000 [23:10:04<7:49:59,  1.95s/it]
Outer training loop:  71%|#######1  | 35525/50000 [23:10:06<7:42:55,  1.92s/it]
Outer training loop:  71%|#######1  | 35526/50000 [23:10:08<7:38:44,  1.90s/it]
Outer training loop:  71%|#######1  | 35527/50000 [23:10:10<7:47:10,  1.94s/it]
Outer training loop:  71%|#######1  | 35528/50000 [23:10:12<7:44:00,  1.92s/it]
Outer training loop:  71%|#######1  | 35529/50000 [23:10:14<7:36:19,  1.89s/it]
Outer training loop:  71%|#######1  | 35530/50000 [23:10:16<7:33:51,  1.88s/it]
Outer training loop:  71%|#######1  | 35531/50000 [23:10:18<7:41:10,  1.91s/it]
Outer training loop:  71%|#######1  | 35532/50000 [23:10:19<7:30:55,  1.87s/it]
Outer training loop:  71%|#######1  | 35533/50000 [23:10:21<7:27:27,  1.86s/it]
Outer training loop:  71%|#######1  | 35534/50000 [23:10:23<7:35:52,  1.89s/it]
Outer training loop:  71%|#######1  | 35535/50000 [23:10:25<7:33:53,  1.88s/it]
Outer training loop:  71%|#######1  | 35536/50000 [23:10:27<7:39:46,  1.91s/it]
Outer training loop:  71%|#######1  | 35537/50000 [23:10:29<7:56:44,  1.98s/it]
Outer training loop:  71%|#######1  | 35538/50000 [23:10:31<7:45:31,  1.93s/it]
Outer training loop:  71%|#######1  | 35539/50000 [23:10:33<7:35:04,  1.89s/it]
Outer training loop:  71%|#######1  | 35540/50000 [23:10:35<7:31:55,  1.88s/it]
Outer training loop:  71%|#######1  | 35541/50000 [23:10:37<7:44:17,  1.93s/it]
Outer training loop:  71%|#######1  | 35542/50000 [23:10:38<7:26:36,  1.85s/it]
Outer training loop:  71%|#######1  | 35543/50000 [23:10:40<7:29:47,  1.87s/it]
Outer training loop:  71%|#######1  | 35544/50000 [23:10:42<7:30:11,  1.87s/it]
Outer training loop:  71%|#######1  | 35545/50000 [23:10:44<7:45:10,  1.93s/it]
Outer training loop:  71%|#######1  | 35546/50000 [23:10:46<7:42:29,  1.92s/it]
Outer training loop:  71%|#######1  | 35547/50000 [23:10:48<7:45:35,  1.93s/it]
Outer training loop:  71%|#######1  | 35548/50000 [23:10:50<8:06:17,  2.02s/it]
Outer training loop:  71%|#######1  | 35549/50000 [23:10:52<7:49:54,  1.95s/it]
Outer training loop:  71%|#######1  | 35550/50000 [23:10:54<7:41:04,  1.91s/it]
Outer training loop:  71%|#######1  | 35551/50000 [23:10:56<7:41:56,  1.92s/it]
Outer training loop:  71%|#######1  | 35552/50000 [23:10:58<8:01:29,  2.00s/it]
Outer training loop:  71%|#######1  | 35553/50000 [23:11:00<7:47:35,  1.94s/it]
Outer training loop:  71%|#######1  | 35554/50000 [23:11:02<7:52:13,  1.96s/it]
Outer training loop:  71%|#######1  | 35555/50000 [23:11:04<7:45:17,  1.93s/it]
Outer training loop:  71%|#######1  | 35556/50000 [23:11:05<7:35:10,  1.89s/it]
Outer training loop:  71%|#######1  | 35557/50000 [23:11:07<7:29:59,  1.87s/it]
Outer training loop:  71%|#######1  | 35558/50000 [23:11:09<7:41:54,  1.92s/it]
Outer training loop:  71%|#######1  | 35559/50000 [23:11:11<7:40:50,  1.91s/it]
Outer training loop:  71%|#######1  | 35560/50000 [23:11:13<7:33:28,  1.88s/it]
Outer training loop:  71%|#######1  | 35561/50000 [23:11:15<7:36:52,  1.90s/it]
Outer training loop:  71%|#######1  | 35562/50000 [23:11:17<7:25:26,  1.85s/it]
Outer training loop:  71%|#######1  | 35563/50000 [23:11:18<7:18:34,  1.82s/it]
Outer training loop:  71%|#######1  | 35564/50000 [23:11:20<7:29:08,  1.87s/it]
Outer training loop:  71%|#######1  | 35565/50000 [23:11:22<7:43:59,  1.93s/it]
Outer training loop:  71%|#######1  | 35566/50000 [23:11:24<7:32:50,  1.88s/it]
Outer training loop:  71%|#######1  | 35567/50000 [23:11:26<7:37:39,  1.90s/it]
Outer training loop:  71%|#######1  | 35568/50000 [23:11:28<7:39:04,  1.91s/it]
Outer training loop:  71%|#######1  | 35569/50000 [23:11:30<7:27:51,  1.86s/it]
Outer training loop:  71%|#######1  | 35570/50000 [23:11:32<7:27:31,  1.86s/it]
Outer training loop:  71%|#######1  | 35571/50000 [23:11:34<7:20:04,  1.83s/it]
Outer training loop:  71%|#######1  | 35572/50000 [23:11:36<7:44:08,  1.93s/it]
Outer training loop:  71%|#######1  | 35573/50000 [23:11:37<7:35:23,  1.89s/it]
Outer training loop:  71%|#######1  | 35574/50000 [23:11:39<7:35:07,  1.89s/it]
Outer training loop:  71%|#######1  | 35575/50000 [23:11:41<7:37:09,  1.90s/it]
Outer training loop:  71%|#######1  | 35576/50000 [23:11:43<7:36:08,  1.90s/it]
Outer training loop:  71%|#######1  | 35577/50000 [23:11:45<7:30:12,  1.87s/it]
Outer training loop:  71%|#######1  | 35578/50000 [23:11:47<7:33:10,  1.89s/it]
Outer training loop:  71%|#######1  | 35579/50000 [23:11:49<7:28:25,  1.87s/it]
Outer training loop:  71%|#######1  | 35580/50000 [23:11:50<7:21:17,  1.84s/it]
Outer training loop:  71%|#######1  | 35581/50000 [23:11:52<7:16:01,  1.81s/it]
Outer training loop:  71%|#######1  | 35582/50000 [23:11:54<7:44:44,  1.93s/it]
Outer training loop:  71%|#######1  | 35583/50000 [23:11:56<7:27:36,  1.86s/it]
Outer training loop:  71%|#######1  | 35584/50000 [23:11:58<7:29:45,  1.87s/it]
Outer training loop:  71%|#######1  | 35585/50000 [23:12:00<7:48:46,  1.95s/it]
Outer training loop:  71%|#######1  | 35586/50000 [23:12:02<7:40:23,  1.92s/it]
Outer training loop:  71%|#######1  | 35587/50000 [23:12:04<7:26:24,  1.86s/it]
Outer training loop:  71%|#######1  | 35588/50000 [23:12:06<7:52:58,  1.97s/it]
Outer training loop:  71%|#######1  | 35589/50000 [23:12:08<7:42:23,  1.93s/it]
Outer training loop:  71%|#######1  | 35590/50000 [23:12:10<7:37:53,  1.91s/it]
Outer training loop:  71%|#######1  | 35591/50000 [23:12:12<7:33:07,  1.89s/it]
Outer training loop:  71%|#######1  | 35592/50000 [23:12:14<7:45:08,  1.94s/it]
Outer training loop:  71%|#######1  | 35593/50000 [23:12:15<7:33:59,  1.89s/it]
Outer training loop:  71%|#######1  | 35594/50000 [23:12:17<7:23:31,  1.85s/it]
Outer training loop:  71%|#######1  | 35595/50000 [23:12:19<7:32:37,  1.89s/it]
Outer training loop:  71%|#######1  | 35596/50000 [23:12:21<7:33:19,  1.89s/it]
Outer training loop:  71%|#######1  | 35597/50000 [23:12:23<7:45:29,  1.94s/it]
Outer training loop:  71%|#######1  | 35598/50000 [23:12:25<7:59:07,  2.00s/it]
Outer training loop:  71%|#######1  | 35599/50000 [23:12:27<7:47:56,  1.95s/it]
Outer training loop:  71%|#######1  | 35600/50000 [23:12:29<7:45:01,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.09s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.04s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.93s/it]

Outer training loop:  71%|#######1  | 35601/50000 [23:12:37<14:34:08,  3.64s/it]
Outer training loop:  71%|#######1  | 35602/50000 [23:12:38<12:28:34,  3.12s/it]
Outer training loop:  71%|#######1  | 35603/50000 [23:12:40<10:59:32,  2.75s/it]
Outer training loop:  71%|#######1  | 35604/50000 [23:12:42<9:48:42,  2.45s/it] 
Outer training loop:  71%|#######1  | 35605/50000 [23:12:44<9:08:19,  2.29s/it]
Outer training loop:  71%|#######1  | 35606/50000 [23:12:46<8:56:57,  2.24s/it]
Outer training loop:  71%|#######1  | 35607/50000 [23:12:48<8:42:15,  2.18s/it]
Outer training loop:  71%|#######1  | 35608/50000 [23:12:50<8:22:02,  2.09s/it]
Outer training loop:  71%|#######1  | 35609/50000 [23:12:52<8:10:43,  2.05s/it]
Outer training loop:  71%|#######1  | 35610/50000 [23:12:54<8:03:05,  2.01s/it]
Outer training loop:  71%|#######1  | 35611/50000 [23:12:56<7:50:00,  1.96s/it]
Outer training loop:  71%|#######1  | 35612/50000 [23:12:58<8:07:34,  2.03s/it]
Outer training loop:  71%|#######1  | 35613/50000 [23:13:00<7:53:15,  1.97s/it]
Outer training loop:  71%|#######1  | 35614/50000 [23:13:02<7:40:01,  1.92s/it]
Outer training loop:  71%|#######1  | 35615/50000 [23:13:03<7:32:40,  1.89s/it]
Outer training loop:  71%|#######1  | 35616/50000 [23:13:05<7:44:35,  1.94s/it]
Outer training loop:  71%|#######1  | 35617/50000 [23:13:07<7:32:32,  1.89s/it]
Outer training loop:  71%|#######1  | 35618/50000 [23:13:09<7:24:06,  1.85s/it]
Outer training loop:  71%|#######1  | 35619/50000 [23:13:11<7:23:33,  1.85s/it]
Outer training loop:  71%|#######1  | 35620/50000 [23:13:13<7:42:39,  1.93s/it]
Outer training loop:  71%|#######1  | 35621/50000 [23:13:15<7:44:05,  1.94s/it]
Outer training loop:  71%|#######1  | 35622/50000 [23:13:17<7:30:50,  1.88s/it]
Outer training loop:  71%|#######1  | 35623/50000 [23:13:19<7:37:25,  1.91s/it]
Outer training loop:  71%|#######1  | 35624/50000 [23:13:20<7:35:11,  1.90s/it]
Outer training loop:  71%|#######1  | 35625/50000 [23:13:22<7:31:33,  1.88s/it]
Outer training loop:  71%|#######1  | 35626/50000 [23:13:24<7:38:52,  1.92s/it]
Outer training loop:  71%|#######1  | 35627/50000 [23:13:26<7:26:40,  1.86s/it]
Outer training loop:  71%|#######1  | 35628/50000 [23:13:28<7:29:03,  1.87s/it]
Outer training loop:  71%|#######1  | 35629/50000 [23:13:30<7:19:41,  1.84s/it]
Outer training loop:  71%|#######1  | 35630/50000 [23:13:32<7:28:39,  1.87s/it]
Outer training loop:  71%|#######1  | 35631/50000 [23:13:34<7:33:54,  1.90s/it]
Outer training loop:  71%|#######1  | 35632/50000 [23:13:35<7:28:30,  1.87s/it]
Outer training loop:  71%|#######1  | 35633/50000 [23:13:38<7:42:57,  1.93s/it]
Outer training loop:  71%|#######1  | 35634/50000 [23:13:39<7:43:22,  1.94s/it]
Outer training loop:  71%|#######1  | 35635/50000 [23:13:41<7:43:10,  1.93s/it]
Outer training loop:  71%|#######1  | 35636/50000 [23:13:44<7:59:25,  2.00s/it]
Outer training loop:  71%|#######1  | 35637/50000 [23:13:45<7:47:50,  1.95s/it]
Outer training loop:  71%|#######1  | 35638/50000 [23:13:47<7:33:32,  1.89s/it]
Outer training loop:  71%|#######1  | 35639/50000 [23:13:49<7:29:34,  1.88s/it]
Outer training loop:  71%|#######1  | 35640/50000 [23:13:51<7:32:38,  1.89s/it]
Outer training loop:  71%|#######1  | 35641/50000 [23:13:53<7:16:31,  1.82s/it]
Outer training loop:  71%|#######1  | 35642/50000 [23:13:55<7:26:05,  1.86s/it]
Outer training loop:  71%|#######1  | 35643/50000 [23:13:57<7:35:26,  1.90s/it]
Outer training loop:  71%|#######1  | 35644/50000 [23:13:58<7:38:06,  1.91s/it]
Outer training loop:  71%|#######1  | 35645/50000 [23:14:00<7:32:43,  1.89s/it]
Outer training loop:  71%|#######1  | 35646/50000 [23:14:02<7:25:18,  1.86s/it]
Outer training loop:  71%|#######1  | 35647/50000 [23:14:04<7:40:12,  1.92s/it]
Outer training loop:  71%|#######1  | 35648/50000 [23:14:06<7:26:48,  1.87s/it]
Outer training loop:  71%|#######1  | 35649/50000 [23:14:08<7:19:26,  1.84s/it]
Outer training loop:  71%|#######1  | 35650/50000 [23:14:10<7:34:12,  1.90s/it]
Outer training loop:  71%|#######1  | 35651/50000 [23:14:11<7:24:01,  1.86s/it]
Outer training loop:  71%|#######1  | 35652/50000 [23:14:13<7:23:43,  1.86s/it]
Outer training loop:  71%|#######1  | 35653/50000 [23:14:15<7:33:55,  1.90s/it]
Outer training loop:  71%|#######1  | 35654/50000 [23:14:17<7:26:46,  1.87s/it]
Outer training loop:  71%|#######1  | 35655/50000 [23:14:19<7:27:11,  1.87s/it]
Outer training loop:  71%|#######1  | 35656/50000 [23:14:21<7:30:54,  1.89s/it]
Outer training loop:  71%|#######1  | 35657/50000 [23:14:23<7:48:10,  1.96s/it]
Outer training loop:  71%|#######1  | 35658/50000 [23:14:25<7:34:40,  1.90s/it]
Outer training loop:  71%|#######1  | 35659/50000 [23:14:27<7:33:48,  1.90s/it]
Outer training loop:  71%|#######1  | 35660/50000 [23:14:29<7:41:10,  1.93s/it]
Outer training loop:  71%|#######1  | 35661/50000 [23:14:31<7:30:49,  1.89s/it]
Outer training loop:  71%|#######1  | 35662/50000 [23:14:32<7:22:55,  1.85s/it]
Outer training loop:  71%|#######1  | 35663/50000 [23:14:34<7:38:33,  1.92s/it]
Outer training loop:  71%|#######1  | 35664/50000 [23:14:36<7:25:44,  1.87s/it]
Outer training loop:  71%|#######1  | 35665/50000 [23:14:38<7:13:24,  1.81s/it]
Outer training loop:  71%|#######1  | 35666/50000 [23:14:40<7:07:38,  1.79s/it]
Outer training loop:  71%|#######1  | 35667/50000 [23:14:42<7:26:04,  1.87s/it]
Outer training loop:  71%|#######1  | 35668/50000 [23:14:43<7:18:06,  1.83s/it]
Outer training loop:  71%|#######1  | 35669/50000 [23:14:45<7:13:03,  1.81s/it]
Outer training loop:  71%|#######1  | 35670/50000 [23:14:47<7:27:50,  1.88s/it]
Outer training loop:  71%|#######1  | 35671/50000 [23:14:49<7:16:10,  1.83s/it]
Outer training loop:  71%|#######1  | 35672/50000 [23:14:51<7:16:13,  1.83s/it]
Outer training loop:  71%|#######1  | 35673/50000 [23:14:52<7:15:11,  1.82s/it]
Outer training loop:  71%|#######1  | 35674/50000 [23:14:54<7:26:31,  1.87s/it]
Outer training loop:  71%|#######1  | 35675/50000 [23:14:56<7:20:54,  1.85s/it]
Outer training loop:  71%|#######1  | 35676/50000 [23:14:58<7:14:03,  1.82s/it]
Outer training loop:  71%|#######1  | 35677/50000 [23:15:00<7:28:49,  1.88s/it]
Outer training loop:  71%|#######1  | 35678/50000 [23:15:02<7:22:26,  1.85s/it]
Outer training loop:  71%|#######1  | 35679/50000 [23:15:04<7:16:56,  1.83s/it]
Outer training loop:  71%|#######1  | 35680/50000 [23:15:05<7:19:57,  1.84s/it]
Outer training loop:  71%|#######1  | 35681/50000 [23:15:07<7:30:34,  1.89s/it]
Outer training loop:  71%|#######1  | 35682/50000 [23:15:09<7:36:43,  1.91s/it]
Outer training loop:  71%|#######1  | 35683/50000 [23:15:11<7:28:56,  1.88s/it]
Outer training loop:  71%|#######1  | 35684/50000 [23:15:13<7:42:15,  1.94s/it]
Outer training loop:  71%|#######1  | 35685/50000 [23:15:15<7:27:21,  1.88s/it]
Outer training loop:  71%|#######1  | 35686/50000 [23:15:17<7:25:33,  1.87s/it]
Outer training loop:  71%|#######1  | 35687/50000 [23:15:19<7:23:59,  1.86s/it]
Outer training loop:  71%|#######1  | 35688/50000 [23:15:21<7:37:33,  1.92s/it]
Outer training loop:  71%|#######1  | 35689/50000 [23:15:23<7:33:16,  1.90s/it]
Outer training loop:  71%|#######1  | 35690/50000 [23:15:24<7:27:49,  1.88s/it]
Outer training loop:  71%|#######1  | 35691/50000 [23:15:26<7:35:30,  1.91s/it]
Outer training loop:  71%|#######1  | 35692/50000 [23:15:28<7:27:01,  1.87s/it]
Outer training loop:  71%|#######1  | 35693/50000 [23:15:30<7:29:45,  1.89s/it]
Outer training loop:  71%|#######1  | 35694/50000 [23:15:32<7:49:33,  1.97s/it]
Outer training loop:  71%|#######1  | 35695/50000 [23:15:34<7:38:07,  1.92s/it]
Outer training loop:  71%|#######1  | 35696/50000 [23:15:36<7:44:13,  1.95s/it]
Outer training loop:  71%|#######1  | 35697/50000 [23:15:38<7:44:05,  1.95s/it]
Outer training loop:  71%|#######1  | 35698/50000 [23:15:40<7:37:40,  1.92s/it]
Outer training loop:  71%|#######1  | 35699/50000 [23:15:42<7:24:03,  1.86s/it]
Outer training loop:  71%|#######1  | 35700/50000 [23:15:44<7:37:54,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  71%|#######1  | 35701/50000 [23:15:51<14:20:38,  3.61s/it]
Outer training loop:  71%|#######1  | 35702/50000 [23:15:53<12:12:14,  3.07s/it]
Outer training loop:  71%|#######1  | 35703/50000 [23:15:55<10:46:11,  2.71s/it]
Outer training loop:  71%|#######1  | 35704/50000 [23:15:57<9:46:21,  2.46s/it] 
Outer training loop:  71%|#######1  | 35705/50000 [23:15:59<9:07:52,  2.30s/it]
Outer training loop:  71%|#######1  | 35706/50000 [23:16:01<8:35:08,  2.16s/it]
Outer training loop:  71%|#######1  | 35707/50000 [23:16:02<7:59:36,  2.01s/it]
Outer training loop:  71%|#######1  | 35708/50000 [23:16:04<7:58:50,  2.01s/it]
Outer training loop:  71%|#######1  | 35709/50000 [23:16:06<7:47:46,  1.96s/it]
Outer training loop:  71%|#######1  | 35710/50000 [23:16:08<7:43:25,  1.95s/it]
Outer training loop:  71%|#######1  | 35711/50000 [23:16:10<7:58:29,  2.01s/it]
Outer training loop:  71%|#######1  | 35712/50000 [23:16:12<7:47:25,  1.96s/it]
Outer training loop:  71%|#######1  | 35713/50000 [23:16:14<7:34:48,  1.91s/it]
Outer training loop:  71%|#######1  | 35714/50000 [23:16:16<7:26:15,  1.87s/it]
Outer training loop:  71%|#######1  | 35715/50000 [23:16:18<7:31:12,  1.90s/it]
Outer training loop:  71%|#######1  | 35716/50000 [23:16:19<7:28:15,  1.88s/it]
Outer training loop:  71%|#######1  | 35717/50000 [23:16:21<7:26:35,  1.88s/it]
Outer training loop:  71%|#######1  | 35718/50000 [23:16:23<7:44:51,  1.95s/it]
Outer training loop:  71%|#######1  | 35719/50000 [23:16:25<7:54:16,  1.99s/it]
Outer training loop:  71%|#######1  | 35720/50000 [23:16:27<7:45:03,  1.95s/it]
Outer training loop:  71%|#######1  | 35721/50000 [23:16:29<7:38:27,  1.93s/it]
Outer training loop:  71%|#######1  | 35722/50000 [23:16:31<7:38:39,  1.93s/it]
Outer training loop:  71%|#######1  | 35723/50000 [23:16:33<7:42:20,  1.94s/it]
Outer training loop:  71%|#######1  | 35724/50000 [23:16:35<7:44:48,  1.95s/it]
Outer training loop:  71%|#######1  | 35725/50000 [23:16:37<7:46:45,  1.96s/it]
Outer training loop:  71%|#######1  | 35726/50000 [23:16:39<7:37:14,  1.92s/it]
Outer training loop:  71%|#######1  | 35727/50000 [23:16:41<7:24:25,  1.87s/it]
Outer training loop:  71%|#######1  | 35728/50000 [23:16:43<7:35:05,  1.91s/it]
Outer training loop:  71%|#######1  | 35729/50000 [23:16:45<7:32:22,  1.90s/it]
Outer training loop:  71%|#######1  | 35730/50000 [23:16:46<7:22:08,  1.86s/it]
Outer training loop:  71%|#######1  | 35731/50000 [23:16:48<7:38:22,  1.93s/it]
Outer training loop:  71%|#######1  | 35732/50000 [23:16:50<7:37:18,  1.92s/it]
Outer training loop:  71%|#######1  | 35733/50000 [23:16:52<7:27:44,  1.88s/it]
Outer training loop:  71%|#######1  | 35734/50000 [23:16:54<7:20:16,  1.85s/it]
Outer training loop:  71%|#######1  | 35735/50000 [23:16:56<7:27:22,  1.88s/it]
Outer training loop:  71%|#######1  | 35736/50000 [23:16:58<7:20:39,  1.85s/it]
Outer training loop:  71%|#######1  | 35737/50000 [23:16:59<7:21:26,  1.86s/it]
Outer training loop:  71%|#######1  | 35738/50000 [23:17:01<7:16:06,  1.83s/it]
Outer training loop:  71%|#######1  | 35739/50000 [23:17:03<7:18:04,  1.84s/it]
Outer training loop:  71%|#######1  | 35740/50000 [23:17:05<7:22:12,  1.86s/it]
Outer training loop:  71%|#######1  | 35741/50000 [23:17:07<7:28:36,  1.89s/it]
Outer training loop:  71%|#######1  | 35742/50000 [23:17:09<7:29:20,  1.89s/it]
Outer training loop:  71%|#######1  | 35743/50000 [23:17:11<7:30:43,  1.90s/it]
Outer training loop:  71%|#######1  | 35744/50000 [23:17:13<7:25:48,  1.88s/it]
Outer training loop:  71%|#######1  | 35745/50000 [23:17:15<7:40:11,  1.94s/it]
Outer training loop:  71%|#######1  | 35746/50000 [23:17:16<7:28:26,  1.89s/it]
Outer training loop:  71%|#######1  | 35747/50000 [23:17:18<7:17:28,  1.84s/it]
Outer training loop:  71%|#######1  | 35748/50000 [23:17:20<7:22:39,  1.86s/it]
Outer training loop:  71%|#######1  | 35749/50000 [23:17:22<7:27:07,  1.88s/it]
Outer training loop:  72%|#######1  | 35750/50000 [23:17:24<7:22:23,  1.86s/it]
Outer training loop:  72%|#######1  | 35751/50000 [23:17:26<7:26:42,  1.88s/it]
Outer training loop:  72%|#######1  | 35752/50000 [23:17:28<7:28:40,  1.89s/it]
Outer training loop:  72%|#######1  | 35753/50000 [23:17:30<7:25:34,  1.88s/it]
Outer training loop:  72%|#######1  | 35754/50000 [23:17:31<7:19:50,  1.85s/it]
Outer training loop:  72%|#######1  | 35755/50000 [23:17:33<7:31:08,  1.90s/it]
Outer training loop:  72%|#######1  | 35756/50000 [23:17:35<7:29:46,  1.89s/it]
Outer training loop:  72%|#######1  | 35757/50000 [23:17:37<7:29:16,  1.89s/it]
Outer training loop:  72%|#######1  | 35758/50000 [23:17:39<7:28:53,  1.89s/it]
Outer training loop:  72%|#######1  | 35759/50000 [23:17:41<7:36:56,  1.93s/it]
Outer training loop:  72%|#######1  | 35760/50000 [23:17:43<7:43:05,  1.95s/it]
Outer training loop:  72%|#######1  | 35761/50000 [23:17:45<7:25:42,  1.88s/it]
Outer training loop:  72%|#######1  | 35762/50000 [23:17:47<7:36:36,  1.92s/it]
Outer training loop:  72%|#######1  | 35763/50000 [23:17:49<7:33:51,  1.91s/it]
Outer training loop:  72%|#######1  | 35764/50000 [23:17:51<7:41:54,  1.95s/it]
Outer training loop:  72%|#######1  | 35765/50000 [23:17:53<7:46:32,  1.97s/it]
Outer training loop:  72%|#######1  | 35766/50000 [23:17:55<7:38:01,  1.93s/it]
Outer training loop:  72%|#######1  | 35767/50000 [23:17:56<7:38:31,  1.93s/it]
Outer training loop:  72%|#######1  | 35768/50000 [23:17:58<7:33:29,  1.91s/it]
Outer training loop:  72%|#######1  | 35769/50000 [23:18:00<7:41:36,  1.95s/it]
Outer training loop:  72%|#######1  | 35770/50000 [23:18:02<7:25:21,  1.88s/it]
Outer training loop:  72%|#######1  | 35771/50000 [23:18:04<7:26:10,  1.88s/it]
Outer training loop:  72%|#######1  | 35772/50000 [23:18:06<7:36:55,  1.93s/it]
Outer training loop:  72%|#######1  | 35773/50000 [23:18:08<7:26:52,  1.88s/it]
Outer training loop:  72%|#######1  | 35774/50000 [23:18:10<7:26:24,  1.88s/it]
Outer training loop:  72%|#######1  | 35775/50000 [23:18:12<7:25:02,  1.88s/it]
Outer training loop:  72%|#######1  | 35776/50000 [23:18:14<7:41:27,  1.95s/it]
Outer training loop:  72%|#######1  | 35777/50000 [23:18:15<7:22:11,  1.87s/it]
Outer training loop:  72%|#######1  | 35778/50000 [23:18:17<7:08:44,  1.81s/it]
Outer training loop:  72%|#######1  | 35779/50000 [23:18:19<7:22:57,  1.87s/it]
Outer training loop:  72%|#######1  | 35780/50000 [23:18:21<7:17:24,  1.85s/it]
Outer training loop:  72%|#######1  | 35781/50000 [23:18:22<7:06:08,  1.80s/it]
Outer training loop:  72%|#######1  | 35782/50000 [23:18:24<7:08:35,  1.81s/it]
Outer training loop:  72%|#######1  | 35783/50000 [23:18:26<7:32:55,  1.91s/it]
Outer training loop:  72%|#######1  | 35784/50000 [23:18:28<7:36:56,  1.93s/it]
Outer training loop:  72%|#######1  | 35785/50000 [23:18:30<7:27:00,  1.89s/it]
Outer training loop:  72%|#######1  | 35786/50000 [23:18:32<7:32:42,  1.91s/it]
Outer training loop:  72%|#######1  | 35787/50000 [23:18:34<7:23:32,  1.87s/it]
Outer training loop:  72%|#######1  | 35788/50000 [23:18:36<7:18:11,  1.85s/it]
Outer training loop:  72%|#######1  | 35789/50000 [23:18:38<7:28:41,  1.89s/it]
Outer training loop:  72%|#######1  | 35790/50000 [23:18:40<7:23:05,  1.87s/it]
Outer training loop:  72%|#######1  | 35791/50000 [23:18:42<7:37:41,  1.93s/it]
Outer training loop:  72%|#######1  | 35792/50000 [23:18:43<7:26:30,  1.89s/it]
Outer training loop:  72%|#######1  | 35793/50000 [23:18:45<7:34:54,  1.92s/it]
Outer training loop:  72%|#######1  | 35794/50000 [23:18:47<7:26:42,  1.89s/it]
Outer training loop:  72%|#######1  | 35795/50000 [23:18:49<7:27:35,  1.89s/it]
Outer training loop:  72%|#######1  | 35796/50000 [23:18:51<7:36:53,  1.93s/it]
Outer training loop:  72%|#######1  | 35797/50000 [23:18:53<7:27:00,  1.89s/it]
Outer training loop:  72%|#######1  | 35798/50000 [23:18:55<7:19:55,  1.86s/it]
Outer training loop:  72%|#######1  | 35799/50000 [23:18:57<7:26:56,  1.89s/it]
Outer training loop:  72%|#######1  | 35800/50000 [23:18:59<7:26:57,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  72%|#######1  | 35801/50000 [23:19:06<13:48:23,  3.50s/it]
Outer training loop:  72%|#######1  | 35802/50000 [23:19:08<11:50:56,  3.00s/it]
Outer training loop:  72%|#######1  | 35803/50000 [23:19:10<10:37:26,  2.69s/it]
Outer training loop:  72%|#######1  | 35804/50000 [23:19:11<9:32:13,  2.42s/it] 
Outer training loop:  72%|#######1  | 35805/50000 [23:19:13<8:55:00,  2.26s/it]
Outer training loop:  72%|#######1  | 35806/50000 [23:19:15<8:36:52,  2.18s/it]
Outer training loop:  72%|#######1  | 35807/50000 [23:19:17<8:08:47,  2.07s/it]
Outer training loop:  72%|#######1  | 35808/50000 [23:19:19<7:49:49,  1.99s/it]
Outer training loop:  72%|#######1  | 35809/50000 [23:19:21<7:39:44,  1.94s/it]
Outer training loop:  72%|#######1  | 35810/50000 [23:19:23<7:48:44,  1.98s/it]
Outer training loop:  72%|#######1  | 35811/50000 [23:19:25<7:48:01,  1.98s/it]
Outer training loop:  72%|#######1  | 35812/50000 [23:19:27<7:28:37,  1.90s/it]
Outer training loop:  72%|#######1  | 35813/50000 [23:19:29<7:43:11,  1.96s/it]
Outer training loop:  72%|#######1  | 35814/50000 [23:19:30<7:29:41,  1.90s/it]
Outer training loop:  72%|#######1  | 35815/50000 [23:19:32<7:26:32,  1.89s/it]
Outer training loop:  72%|#######1  | 35816/50000 [23:19:34<7:19:37,  1.86s/it]
Outer training loop:  72%|#######1  | 35817/50000 [23:19:36<7:28:38,  1.90s/it]
Outer training loop:  72%|#######1  | 35818/50000 [23:19:38<7:21:01,  1.87s/it]
Outer training loop:  72%|#######1  | 35819/50000 [23:19:40<7:13:32,  1.83s/it]
Outer training loop:  72%|#######1  | 35820/50000 [23:19:42<7:23:32,  1.88s/it]
Outer training loop:  72%|#######1  | 35821/50000 [23:19:43<7:20:09,  1.86s/it]
Outer training loop:  72%|#######1  | 35822/50000 [23:19:45<7:13:16,  1.83s/it]
Outer training loop:  72%|#######1  | 35823/50000 [23:19:47<7:18:55,  1.86s/it]
Outer training loop:  72%|#######1  | 35824/50000 [23:19:49<7:23:08,  1.88s/it]
Outer training loop:  72%|#######1  | 35825/50000 [23:19:51<7:22:02,  1.87s/it]
Outer training loop:  72%|#######1  | 35826/50000 [23:19:53<7:17:09,  1.85s/it]
Outer training loop:  72%|#######1  | 35827/50000 [23:19:55<7:23:19,  1.88s/it]
Outer training loop:  72%|#######1  | 35828/50000 [23:19:57<7:26:25,  1.89s/it]
Outer training loop:  72%|#######1  | 35829/50000 [23:19:58<7:18:42,  1.86s/it]
Outer training loop:  72%|#######1  | 35830/50000 [23:20:00<7:18:18,  1.86s/it]
Outer training loop:  72%|#######1  | 35831/50000 [23:20:02<7:28:17,  1.90s/it]
Outer training loop:  72%|#######1  | 35832/50000 [23:20:04<7:20:37,  1.87s/it]
Outer training loop:  72%|#######1  | 35833/50000 [23:20:06<7:18:04,  1.86s/it]
Outer training loop:  72%|#######1  | 35834/50000 [23:20:08<7:28:02,  1.90s/it]
Outer training loop:  72%|#######1  | 35835/50000 [23:20:10<7:32:59,  1.92s/it]
Outer training loop:  72%|#######1  | 35836/50000 [23:20:11<7:21:34,  1.87s/it]
Outer training loop:  72%|#######1  | 35837/50000 [23:20:13<7:20:12,  1.86s/it]
Outer training loop:  72%|#######1  | 35838/50000 [23:20:16<7:41:06,  1.95s/it]
Outer training loop:  72%|#######1  | 35839/50000 [23:20:17<7:22:19,  1.87s/it]
Outer training loop:  72%|#######1  | 35840/50000 [23:20:19<7:15:32,  1.85s/it]
Outer training loop:  72%|#######1  | 35841/50000 [23:20:21<7:31:27,  1.91s/it]
Outer training loop:  72%|#######1  | 35842/50000 [23:20:23<7:19:36,  1.86s/it]
Outer training loop:  72%|#######1  | 35843/50000 [23:20:25<7:18:37,  1.86s/it]
Outer training loop:  72%|#######1  | 35844/50000 [23:20:27<7:26:16,  1.89s/it]
Outer training loop:  72%|#######1  | 35845/50000 [23:20:28<7:20:52,  1.87s/it]
Outer training loop:  72%|#######1  | 35846/50000 [23:20:30<7:16:06,  1.85s/it]
Outer training loop:  72%|#######1  | 35847/50000 [23:20:32<7:03:27,  1.80s/it]
Outer training loop:  72%|#######1  | 35848/50000 [23:20:34<7:13:46,  1.84s/it]
Outer training loop:  72%|#######1  | 35849/50000 [23:20:36<7:08:01,  1.81s/it]
Outer training loop:  72%|#######1  | 35850/50000 [23:20:37<7:05:27,  1.80s/it]
Outer training loop:  72%|#######1  | 35851/50000 [23:20:39<7:15:49,  1.85s/it]
Outer training loop:  72%|#######1  | 35852/50000 [23:20:41<7:10:55,  1.83s/it]
Outer training loop:  72%|#######1  | 35853/50000 [23:20:43<7:12:46,  1.84s/it]
Outer training loop:  72%|#######1  | 35854/50000 [23:20:45<7:18:21,  1.86s/it]
Outer training loop:  72%|#######1  | 35855/50000 [23:20:47<7:14:41,  1.84s/it]
Outer training loop:  72%|#######1  | 35856/50000 [23:20:48<7:05:58,  1.81s/it]
Outer training loop:  72%|#######1  | 35857/50000 [23:20:50<7:17:09,  1.85s/it]
Outer training loop:  72%|#######1  | 35858/50000 [23:20:52<7:27:07,  1.90s/it]
Outer training loop:  72%|#######1  | 35859/50000 [23:20:54<7:22:04,  1.88s/it]
Outer training loop:  72%|#######1  | 35860/50000 [23:20:56<7:22:27,  1.88s/it]
Outer training loop:  72%|#######1  | 35861/50000 [23:20:58<7:31:32,  1.92s/it]
Outer training loop:  72%|#######1  | 35862/50000 [23:21:00<7:22:21,  1.88s/it]
Outer training loop:  72%|#######1  | 35863/50000 [23:21:02<7:15:49,  1.85s/it]
Outer training loop:  72%|#######1  | 35864/50000 [23:21:03<7:06:41,  1.81s/it]
Outer training loop:  72%|#######1  | 35865/50000 [23:21:05<7:21:45,  1.88s/it]
Outer training loop:  72%|#######1  | 35866/50000 [23:21:07<7:13:50,  1.84s/it]
Outer training loop:  72%|#######1  | 35867/50000 [23:21:09<7:15:00,  1.85s/it]
Outer training loop:  72%|#######1  | 35868/50000 [23:21:11<7:22:20,  1.88s/it]
Outer training loop:  72%|#######1  | 35869/50000 [23:21:13<7:24:40,  1.89s/it]
Outer training loop:  72%|#######1  | 35870/50000 [23:21:15<7:18:51,  1.86s/it]
Outer training loop:  72%|#######1  | 35871/50000 [23:21:16<7:10:43,  1.83s/it]
Outer training loop:  72%|#######1  | 35872/50000 [23:21:18<7:23:26,  1.88s/it]
Outer training loop:  72%|#######1  | 35873/50000 [23:21:20<7:29:32,  1.91s/it]
Outer training loop:  72%|#######1  | 35874/50000 [23:21:22<7:17:10,  1.86s/it]
Outer training loop:  72%|#######1  | 35875/50000 [23:21:24<7:20:05,  1.87s/it]
Outer training loop:  72%|#######1  | 35876/50000 [23:21:26<7:24:43,  1.89s/it]
Outer training loop:  72%|#######1  | 35877/50000 [23:21:28<7:21:15,  1.87s/it]
Outer training loop:  72%|#######1  | 35878/50000 [23:21:30<7:36:05,  1.94s/it]
Outer training loop:  72%|#######1  | 35879/50000 [23:21:32<7:23:02,  1.88s/it]
Outer training loop:  72%|#######1  | 35880/50000 [23:21:34<7:21:56,  1.88s/it]
Outer training loop:  72%|#######1  | 35881/50000 [23:21:36<7:32:03,  1.92s/it]
Outer training loop:  72%|#######1  | 35882/50000 [23:21:37<7:20:23,  1.87s/it]
Outer training loop:  72%|#######1  | 35883/50000 [23:21:39<7:15:25,  1.85s/it]
Outer training loop:  72%|#######1  | 35884/50000 [23:21:41<7:26:43,  1.90s/it]
Outer training loop:  72%|#######1  | 35885/50000 [23:21:43<7:18:42,  1.86s/it]
Outer training loop:  72%|#######1  | 35886/50000 [23:21:45<7:14:42,  1.85s/it]
Outer training loop:  72%|#######1  | 35887/50000 [23:21:47<7:15:00,  1.85s/it]
Outer training loop:  72%|#######1  | 35888/50000 [23:21:49<7:26:18,  1.90s/it]
Outer training loop:  72%|#######1  | 35889/50000 [23:21:50<7:18:01,  1.86s/it]
Outer training loop:  72%|#######1  | 35890/50000 [23:21:52<7:11:16,  1.83s/it]
Outer training loop:  72%|#######1  | 35891/50000 [23:21:54<7:18:22,  1.86s/it]
Outer training loop:  72%|#######1  | 35892/50000 [23:21:56<7:20:42,  1.87s/it]
Outer training loop:  72%|#######1  | 35893/50000 [23:21:58<7:22:12,  1.88s/it]
Outer training loop:  72%|#######1  | 35894/50000 [23:22:00<7:29:42,  1.91s/it]
Outer training loop:  72%|#######1  | 35895/50000 [23:22:02<7:18:28,  1.87s/it]
Outer training loop:  72%|#######1  | 35896/50000 [23:22:04<7:24:33,  1.89s/it]
Outer training loop:  72%|#######1  | 35897/50000 [23:22:05<7:26:05,  1.90s/it]
Outer training loop:  72%|#######1  | 35898/50000 [23:22:07<7:34:36,  1.93s/it]
Outer training loop:  72%|#######1  | 35899/50000 [23:22:09<7:26:26,  1.90s/it]
Outer training loop:  72%|#######1  | 35900/50000 [23:22:11<7:23:58,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.97s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  72%|#######1  | 35901/50000 [23:22:18<13:44:47,  3.51s/it]
Outer training loop:  72%|#######1  | 35902/50000 [23:22:21<12:05:12,  3.09s/it]
Outer training loop:  72%|#######1  | 35903/50000 [23:22:22<10:31:45,  2.69s/it]
Outer training loop:  72%|#######1  | 35904/50000 [23:22:24<9:30:46,  2.43s/it] 
Outer training loop:  72%|#######1  | 35905/50000 [23:22:26<9:04:28,  2.32s/it]
Outer training loop:  72%|#######1  | 35906/50000 [23:22:28<8:28:13,  2.16s/it]
Outer training loop:  72%|#######1  | 35907/50000 [23:22:30<8:03:02,  2.06s/it]
Outer training loop:  72%|#######1  | 35908/50000 [23:22:32<7:45:34,  1.98s/it]
Outer training loop:  72%|#######1  | 35909/50000 [23:22:34<7:50:37,  2.00s/it]
Outer training loop:  72%|#######1  | 35910/50000 [23:22:36<7:45:25,  1.98s/it]
Outer training loop:  72%|#######1  | 35911/50000 [23:22:37<7:32:41,  1.93s/it]
Outer training loop:  72%|#######1  | 35912/50000 [23:22:39<7:44:19,  1.98s/it]
Outer training loop:  72%|#######1  | 35913/50000 [23:22:41<7:28:10,  1.91s/it]
Outer training loop:  72%|#######1  | 35914/50000 [23:22:43<7:27:16,  1.91s/it]
Outer training loop:  72%|#######1  | 35915/50000 [23:22:45<7:18:15,  1.87s/it]
Outer training loop:  72%|#######1  | 35916/50000 [23:22:47<7:25:00,  1.90s/it]
Outer training loop:  72%|#######1  | 35917/50000 [23:22:49<7:20:40,  1.88s/it]
Outer training loop:  72%|#######1  | 35918/50000 [23:22:51<7:16:43,  1.86s/it]
Outer training loop:  72%|#######1  | 35919/50000 [23:22:53<7:30:33,  1.92s/it]
Outer training loop:  72%|#######1  | 35920/50000 [23:22:54<7:13:45,  1.85s/it]
Outer training loop:  72%|#######1  | 35921/50000 [23:22:56<7:05:18,  1.81s/it]
Outer training loop:  72%|#######1  | 35922/50000 [23:22:58<7:27:46,  1.91s/it]
Outer training loop:  72%|#######1  | 35923/50000 [23:23:00<7:17:57,  1.87s/it]
Outer training loop:  72%|#######1  | 35924/50000 [23:23:02<7:08:07,  1.82s/it]
Outer training loop:  72%|#######1  | 35925/50000 [23:23:03<7:08:51,  1.83s/it]
Outer training loop:  72%|#######1  | 35926/50000 [23:23:06<7:32:18,  1.93s/it]
Outer training loop:  72%|#######1  | 35927/50000 [23:23:07<7:19:00,  1.87s/it]
Outer training loop:  72%|#######1  | 35928/50000 [23:23:09<7:14:24,  1.85s/it]
Outer training loop:  72%|#######1  | 35929/50000 [23:23:11<7:24:43,  1.90s/it]
Outer training loop:  72%|#######1  | 35930/50000 [23:23:13<7:18:16,  1.87s/it]
Outer training loop:  72%|#######1  | 35931/50000 [23:23:15<7:13:05,  1.85s/it]
Outer training loop:  72%|#######1  | 35932/50000 [23:23:17<7:24:40,  1.90s/it]
Outer training loop:  72%|#######1  | 35933/50000 [23:23:19<7:20:09,  1.88s/it]
Outer training loop:  72%|#######1  | 35934/50000 [23:23:21<7:26:28,  1.90s/it]
Outer training loop:  72%|#######1  | 35935/50000 [23:23:23<7:38:30,  1.96s/it]
Outer training loop:  72%|#######1  | 35936/50000 [23:23:25<8:01:10,  2.05s/it]
Outer training loop:  72%|#######1  | 35937/50000 [23:23:27<7:39:59,  1.96s/it]
Outer training loop:  72%|#######1  | 35938/50000 [23:23:29<7:29:39,  1.92s/it]
Outer training loop:  72%|#######1  | 35939/50000 [23:23:31<7:40:37,  1.97s/it]
Outer training loop:  72%|#######1  | 35940/50000 [23:23:32<7:27:45,  1.91s/it]
Outer training loop:  72%|#######1  | 35941/50000 [23:23:34<7:34:04,  1.94s/it]
Outer training loop:  72%|#######1  | 35942/50000 [23:23:36<7:25:26,  1.90s/it]
Outer training loop:  72%|#######1  | 35943/50000 [23:23:38<7:39:25,  1.96s/it]
Outer training loop:  72%|#######1  | 35944/50000 [23:23:40<7:33:12,  1.93s/it]
Outer training loop:  72%|#######1  | 35945/50000 [23:23:42<7:24:24,  1.90s/it]
Outer training loop:  72%|#######1  | 35946/50000 [23:23:44<7:19:51,  1.88s/it]
Outer training loop:  72%|#######1  | 35947/50000 [23:23:46<7:28:48,  1.92s/it]
Outer training loop:  72%|#######1  | 35948/50000 [23:23:48<7:23:58,  1.90s/it]
Outer training loop:  72%|#######1  | 35949/50000 [23:23:49<7:19:08,  1.88s/it]
Outer training loop:  72%|#######1  | 35950/50000 [23:23:51<7:22:41,  1.89s/it]
Outer training loop:  72%|#######1  | 35951/50000 [23:23:53<7:14:22,  1.86s/it]
Outer training loop:  72%|#######1  | 35952/50000 [23:23:55<7:20:08,  1.88s/it]
Outer training loop:  72%|#######1  | 35953/50000 [23:23:57<7:25:44,  1.90s/it]
Outer training loop:  72%|#######1  | 35954/50000 [23:23:59<7:21:13,  1.88s/it]
Outer training loop:  72%|#######1  | 35955/50000 [23:24:01<7:18:30,  1.87s/it]
Outer training loop:  72%|#######1  | 35956/50000 [23:24:02<7:08:02,  1.83s/it]
Outer training loop:  72%|#######1  | 35957/50000 [23:24:05<7:26:03,  1.91s/it]
Outer training loop:  72%|#######1  | 35958/50000 [23:24:06<7:18:47,  1.87s/it]
Outer training loop:  72%|#######1  | 35959/50000 [23:24:08<7:13:11,  1.85s/it]
Outer training loop:  72%|#######1  | 35960/50000 [23:24:10<7:14:15,  1.86s/it]
Outer training loop:  72%|#######1  | 35961/50000 [23:24:12<7:34:16,  1.94s/it]
Outer training loop:  72%|#######1  | 35962/50000 [23:24:14<7:21:38,  1.89s/it]
Outer training loop:  72%|#######1  | 35963/50000 [23:24:16<7:18:34,  1.87s/it]
Outer training loop:  72%|#######1  | 35964/50000 [23:24:18<7:36:02,  1.95s/it]
Outer training loop:  72%|#######1  | 35965/50000 [23:24:20<7:33:18,  1.94s/it]
Outer training loop:  72%|#######1  | 35966/50000 [23:24:22<7:30:21,  1.93s/it]
Outer training loop:  72%|#######1  | 35967/50000 [23:24:24<7:32:14,  1.93s/it]
Outer training loop:  72%|#######1  | 35968/50000 [23:24:26<7:36:09,  1.95s/it]
Outer training loop:  72%|#######1  | 35969/50000 [23:24:28<7:29:10,  1.92s/it]
Outer training loop:  72%|#######1  | 35970/50000 [23:24:29<7:18:36,  1.88s/it]
Outer training loop:  72%|#######1  | 35971/50000 [23:24:31<7:24:29,  1.90s/it]
Outer training loop:  72%|#######1  | 35972/50000 [23:24:33<7:17:00,  1.87s/it]
Outer training loop:  72%|#######1  | 35973/50000 [23:24:35<7:11:43,  1.85s/it]
Outer training loop:  72%|#######1  | 35974/50000 [23:24:37<7:16:01,  1.87s/it]
Outer training loop:  72%|#######1  | 35975/50000 [23:24:39<7:22:22,  1.89s/it]
Outer training loop:  72%|#######1  | 35976/50000 [23:24:41<7:22:24,  1.89s/it]
Outer training loop:  72%|#######1  | 35977/50000 [23:24:43<7:24:16,  1.90s/it]
Outer training loop:  72%|#######1  | 35978/50000 [23:24:45<7:34:31,  1.94s/it]
Outer training loop:  72%|#######1  | 35979/50000 [23:24:46<7:33:10,  1.94s/it]
Outer training loop:  72%|#######1  | 35980/50000 [23:24:48<7:27:43,  1.92s/it]
Outer training loop:  72%|#######1  | 35981/50000 [23:24:51<7:46:29,  2.00s/it]
Outer training loop:  72%|#######1  | 35982/50000 [23:24:52<7:29:29,  1.92s/it]
Outer training loop:  72%|#######1  | 35983/50000 [23:24:54<7:25:55,  1.91s/it]
Outer training loop:  72%|#######1  | 35984/50000 [23:24:56<7:16:19,  1.87s/it]
Outer training loop:  72%|#######1  | 35985/50000 [23:24:58<7:44:26,  1.99s/it]
Outer training loop:  72%|#######1  | 35986/50000 [23:25:00<7:27:28,  1.92s/it]
Outer training loop:  72%|#######1  | 35987/50000 [23:25:02<7:18:41,  1.88s/it]
Outer training loop:  72%|#######1  | 35988/50000 [23:25:04<7:22:55,  1.90s/it]
Outer training loop:  72%|#######1  | 35989/50000 [23:25:06<7:26:17,  1.91s/it]
Outer training loop:  72%|#######1  | 35990/50000 [23:25:08<7:24:54,  1.91s/it]
Outer training loop:  72%|#######1  | 35991/50000 [23:25:10<7:33:38,  1.94s/it]
Outer training loop:  72%|#######1  | 35992/50000 [23:25:11<7:18:55,  1.88s/it]
Outer training loop:  72%|#######1  | 35993/50000 [23:25:13<7:13:26,  1.86s/it]
Outer training loop:  72%|#######1  | 35994/50000 [23:25:15<7:24:10,  1.90s/it]
Outer training loop:  72%|#######1  | 35995/50000 [23:25:17<7:13:23,  1.86s/it]
Outer training loop:  72%|#######1  | 35996/50000 [23:25:19<7:06:52,  1.83s/it]
Outer training loop:  72%|#######1  | 35997/50000 [23:25:20<7:03:26,  1.81s/it]
Outer training loop:  72%|#######1  | 35998/50000 [23:25:22<7:20:28,  1.89s/it]
Outer training loop:  72%|#######1  | 35999/50000 [23:25:24<7:09:52,  1.84s/it]
Outer training loop:  72%|#######2  | 36000/50000 [23:25:26<7:17:08,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.75s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:33,  1.87s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:30,  1.85s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:27,  1.85s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.91s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.87s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:17,  1.88s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:16,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:14,  1.92s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:06,  1.86s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:59,  1.84s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:57,  1.83s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:55,  1.84s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.86s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:50,  1.85s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:46,  1.81s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:45,  1.83s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:45,  1.89s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:43,  1.88s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:36,  1.82s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:43,  1.98s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:38,  1.94s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:36,  1.93s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:28,  1.88s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:25,  1.86s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:28,  1.96s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:24,  1.93s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:21,  1.89s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.88s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:15,  1.88s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:17,  1.98s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:01,  1.82s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.79s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.92s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:59,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.89s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:49,  1.98s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:32,  1.94s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.89s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:28,  1.90s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.80s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  2.00s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.94s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.92s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.92s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.84s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:24,  1.76s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:38,  1.95s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.90s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.90s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.87s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.90s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:19,  1.86s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:16,  1.84s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:20,  1.93s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:14,  1.95s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:03,  1.87s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:07,  1.95s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.90s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.90s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.88s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:45,  1.85s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:41,  1.81s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.91s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.83s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:33,  1.79s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:31,  1.79s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.89s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.83s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:27,  1.85s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:24,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:21,  1.82s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:18,  1.78s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:22,  1.93s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.89s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:12,  1.85s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:10,  1.87s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:13,  1.97s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:09,  1.92s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:07,  1.94s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:05,  1.92s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.88s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:57,  1.84s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:58,  1.96s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:43,  1.98s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.90s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.82s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.81s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.84s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:25,  1.96s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.91s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:14,  1.83s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:12,  1.81s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:10,  1.81s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:08,  1.78s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.96s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.85s/it]

Outer training loop:  72%|#######2  | 36001/50000 [23:30:45<377:32:42, 97.09s/it]
Outer training loop:  72%|#######2  | 36002/50000 [23:30:47<266:23:37, 68.51s/it]
Outer training loop:  72%|#######2  | 36003/50000 [23:30:49<188:30:03, 48.48s/it]
Outer training loop:  72%|#######2  | 36004/50000 [23:30:51<134:01:47, 34.47s/it]
Outer training loop:  72%|#######2  | 36005/50000 [23:30:53<95:53:12, 24.67s/it] 
Outer training loop:  72%|#######2  | 36006/50000 [23:30:54<69:18:56, 17.83s/it]
Outer training loop:  72%|#######2  | 36007/50000 [23:30:57<51:15:04, 13.19s/it]
Outer training loop:  72%|#######2  | 36008/50000 [23:30:59<38:07:12,  9.81s/it]
Outer training loop:  72%|#######2  | 36009/50000 [23:31:01<28:48:55,  7.41s/it]
Outer training loop:  72%|#######2  | 36010/50000 [23:31:03<22:28:30,  5.78s/it]
Outer training loop:  72%|#######2  | 36011/50000 [23:31:04<17:45:31,  4.57s/it]
Outer training loop:  72%|#######2  | 36012/50000 [23:31:06<14:37:51,  3.77s/it]
Outer training loop:  72%|#######2  | 36013/50000 [23:31:08<12:30:13,  3.22s/it]
Outer training loop:  72%|#######2  | 36014/50000 [23:31:10<10:50:01,  2.79s/it]
Outer training loop:  72%|#######2  | 36015/50000 [23:31:12<9:41:08,  2.49s/it] 
Outer training loop:  72%|#######2  | 36016/50000 [23:31:14<9:13:06,  2.37s/it]
Outer training loop:  72%|#######2  | 36017/50000 [23:31:16<8:44:49,  2.25s/it]
Outer training loop:  72%|#######2  | 36018/50000 [23:31:18<8:12:43,  2.11s/it]
Outer training loop:  72%|#######2  | 36019/50000 [23:31:19<8:01:14,  2.07s/it]
Outer training loop:  72%|#######2  | 36020/50000 [23:31:21<7:54:49,  2.04s/it]
Outer training loop:  72%|#######2  | 36021/50000 [23:31:23<7:39:28,  1.97s/it]
Outer training loop:  72%|#######2  | 36022/50000 [23:31:25<7:24:40,  1.91s/it]
Outer training loop:  72%|#######2  | 36023/50000 [23:31:27<7:15:56,  1.87s/it]
Outer training loop:  72%|#######2  | 36024/50000 [23:31:29<7:23:36,  1.90s/it]
Outer training loop:  72%|#######2  | 36025/50000 [23:31:31<7:19:24,  1.89s/it]
Outer training loop:  72%|#######2  | 36026/50000 [23:31:32<7:10:30,  1.85s/it]
Outer training loop:  72%|#######2  | 36027/50000 [23:31:34<7:06:04,  1.83s/it]
Outer training loop:  72%|#######2  | 36028/50000 [23:31:36<7:27:17,  1.92s/it]
Outer training loop:  72%|#######2  | 36029/50000 [23:31:38<7:23:20,  1.90s/it]
Outer training loop:  72%|#######2  | 36030/50000 [23:31:40<7:26:40,  1.92s/it]
Outer training loop:  72%|#######2  | 36031/50000 [23:31:42<7:34:15,  1.95s/it]
Outer training loop:  72%|#######2  | 36032/50000 [23:31:44<7:27:16,  1.92s/it]
Outer training loop:  72%|#######2  | 36033/50000 [23:31:46<7:16:54,  1.88s/it]
Outer training loop:  72%|#######2  | 36034/50000 [23:31:48<7:08:10,  1.84s/it]
Outer training loop:  72%|#######2  | 36035/50000 [23:31:50<7:19:06,  1.89s/it]
Outer training loop:  72%|#######2  | 36036/50000 [23:31:51<7:16:17,  1.87s/it]
Outer training loop:  72%|#######2  | 36037/50000 [23:31:53<7:07:58,  1.84s/it]
Outer training loop:  72%|#######2  | 36038/50000 [23:31:55<7:19:39,  1.89s/it]
Outer training loop:  72%|#######2  | 36039/50000 [23:31:57<7:12:25,  1.86s/it]
Outer training loop:  72%|#######2  | 36040/50000 [23:31:59<7:08:40,  1.84s/it]
Outer training loop:  72%|#######2  | 36041/50000 [23:32:01<7:18:04,  1.88s/it]
Outer training loop:  72%|#######2  | 36042/50000 [23:32:03<7:11:50,  1.86s/it]
Outer training loop:  72%|#######2  | 36043/50000 [23:32:04<7:09:08,  1.84s/it]
Outer training loop:  72%|#######2  | 36044/50000 [23:32:06<7:28:48,  1.93s/it]
Outer training loop:  72%|#######2  | 36045/50000 [23:32:08<7:23:46,  1.91s/it]
Outer training loop:  72%|#######2  | 36046/50000 [23:32:10<7:17:24,  1.88s/it]
Outer training loop:  72%|#######2  | 36047/50000 [23:32:12<7:11:52,  1.86s/it]
Outer training loop:  72%|#######2  | 36048/50000 [23:32:14<7:22:34,  1.90s/it]
Outer training loop:  72%|#######2  | 36049/50000 [23:32:16<7:09:53,  1.85s/it]
Outer training loop:  72%|#######2  | 36050/50000 [23:32:17<7:07:51,  1.84s/it]
Outer training loop:  72%|#######2  | 36051/50000 [23:32:20<7:24:15,  1.91s/it]
Outer training loop:  72%|#######2  | 36052/50000 [23:32:21<7:15:14,  1.87s/it]
Outer training loop:  72%|#######2  | 36053/50000 [23:32:23<7:16:32,  1.88s/it]
Outer training loop:  72%|#######2  | 36054/50000 [23:32:25<7:11:44,  1.86s/it]
Outer training loop:  72%|#######2  | 36055/50000 [23:32:27<7:27:26,  1.93s/it]
Outer training loop:  72%|#######2  | 36056/50000 [23:32:29<7:23:39,  1.91s/it]
Outer training loop:  72%|#######2  | 36057/50000 [23:32:31<7:19:09,  1.89s/it]
Outer training loop:  72%|#######2  | 36058/50000 [23:32:33<7:24:48,  1.91s/it]
Outer training loop:  72%|#######2  | 36059/50000 [23:32:35<7:15:19,  1.87s/it]
Outer training loop:  72%|#######2  | 36060/50000 [23:32:36<7:10:24,  1.85s/it]
Outer training loop:  72%|#######2  | 36061/50000 [23:32:38<7:01:29,  1.81s/it]
Outer training loop:  72%|#######2  | 36062/50000 [23:32:40<7:14:11,  1.87s/it]
Outer training loop:  72%|#######2  | 36063/50000 [23:32:42<7:19:58,  1.89s/it]
Outer training loop:  72%|#######2  | 36064/50000 [23:32:44<7:13:36,  1.87s/it]
Outer training loop:  72%|#######2  | 36065/50000 [23:32:46<7:10:18,  1.85s/it]
Outer training loop:  72%|#######2  | 36066/50000 [23:32:48<7:28:54,  1.93s/it]
Outer training loop:  72%|#######2  | 36067/50000 [23:32:50<7:16:18,  1.88s/it]
Outer training loop:  72%|#######2  | 36068/50000 [23:32:51<7:11:40,  1.86s/it]
Outer training loop:  72%|#######2  | 36069/50000 [23:32:53<7:05:06,  1.83s/it]
Outer training loop:  72%|#######2  | 36070/50000 [23:32:55<7:22:41,  1.91s/it]
Outer training loop:  72%|#######2  | 36071/50000 [23:32:57<7:18:39,  1.89s/it]
Outer training loop:  72%|#######2  | 36072/50000 [23:32:59<7:12:43,  1.86s/it]
Outer training loop:  72%|#######2  | 36073/50000 [23:33:01<7:23:00,  1.91s/it]
Outer training loop:  72%|#######2  | 36074/50000 [23:33:03<7:15:08,  1.87s/it]
Outer training loop:  72%|#######2  | 36075/50000 [23:33:05<7:14:19,  1.87s/it]
Outer training loop:  72%|#######2  | 36076/50000 [23:33:06<7:17:36,  1.89s/it]
Outer training loop:  72%|#######2  | 36077/50000 [23:33:08<7:23:30,  1.91s/it]
Outer training loop:  72%|#######2  | 36078/50000 [23:33:10<7:12:35,  1.86s/it]
Outer training loop:  72%|#######2  | 36079/50000 [23:33:12<7:03:58,  1.83s/it]
Outer training loop:  72%|#######2  | 36080/50000 [23:33:14<7:25:48,  1.92s/it]
Outer training loop:  72%|#######2  | 36081/50000 [23:33:16<7:16:43,  1.88s/it]
Outer training loop:  72%|#######2  | 36082/50000 [23:33:18<7:10:20,  1.86s/it]
Outer training loop:  72%|#######2  | 36083/50000 [23:33:19<7:02:53,  1.82s/it]
Outer training loop:  72%|#######2  | 36084/50000 [23:33:21<7:12:59,  1.87s/it]
Outer training loop:  72%|#######2  | 36085/50000 [23:33:23<7:10:25,  1.86s/it]
Outer training loop:  72%|#######2  | 36086/50000 [23:33:25<7:12:25,  1.86s/it]
Outer training loop:  72%|#######2  | 36087/50000 [23:33:27<7:29:19,  1.94s/it]
Outer training loop:  72%|#######2  | 36088/50000 [23:33:29<7:29:02,  1.94s/it]
Outer training loop:  72%|#######2  | 36089/50000 [23:33:31<7:24:24,  1.92s/it]
Outer training loop:  72%|#######2  | 36090/50000 [23:33:33<7:33:07,  1.95s/it]
Outer training loop:  72%|#######2  | 36091/50000 [23:33:35<7:38:30,  1.98s/it]
Outer training loop:  72%|#######2  | 36092/50000 [23:33:37<7:26:28,  1.93s/it]
Outer training loop:  72%|#######2  | 36093/50000 [23:33:39<7:33:24,  1.96s/it]
Outer training loop:  72%|#######2  | 36094/50000 [23:33:41<7:28:10,  1.93s/it]
Outer training loop:  72%|#######2  | 36095/50000 [23:33:43<7:18:19,  1.89s/it]
Outer training loop:  72%|#######2  | 36096/50000 [23:33:44<7:11:01,  1.86s/it]
Outer training loop:  72%|#######2  | 36097/50000 [23:33:46<7:26:16,  1.93s/it]
Outer training loop:  72%|#######2  | 36098/50000 [23:33:48<7:24:54,  1.92s/it]
Outer training loop:  72%|#######2  | 36099/50000 [23:33:50<7:28:18,  1.94s/it]
Outer training loop:  72%|#######2  | 36100/50000 [23:33:52<7:37:01,  1.97s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  72%|#######2  | 36101/50000 [23:34:00<14:09:53,  3.67s/it]
Outer training loop:  72%|#######2  | 36102/50000 [23:34:02<12:07:14,  3.14s/it]
Outer training loop:  72%|#######2  | 36103/50000 [23:34:04<10:32:04,  2.73s/it]
Outer training loop:  72%|#######2  | 36104/50000 [23:34:06<9:56:06,  2.57s/it] 
Outer training loop:  72%|#######2  | 36105/50000 [23:34:08<9:08:52,  2.37s/it]
Outer training loop:  72%|#######2  | 36106/50000 [23:34:10<8:35:41,  2.23s/it]
Outer training loop:  72%|#######2  | 36107/50000 [23:34:11<7:57:59,  2.06s/it]
Outer training loop:  72%|#######2  | 36108/50000 [23:34:13<7:52:37,  2.04s/it]
Outer training loop:  72%|#######2  | 36109/50000 [23:34:15<7:47:56,  2.02s/it]
Outer training loop:  72%|#######2  | 36110/50000 [23:34:17<7:38:48,  1.98s/it]
Outer training loop:  72%|#######2  | 36111/50000 [23:34:19<7:40:24,  1.99s/it]
Outer training loop:  72%|#######2  | 36112/50000 [23:34:21<7:36:06,  1.97s/it]
Outer training loop:  72%|#######2  | 36113/50000 [23:34:23<7:35:31,  1.97s/it]
Outer training loop:  72%|#######2  | 36114/50000 [23:34:25<7:39:42,  1.99s/it]
Outer training loop:  72%|#######2  | 36115/50000 [23:34:27<7:19:47,  1.90s/it]
Outer training loop:  72%|#######2  | 36116/50000 [23:34:29<7:18:52,  1.90s/it]
Outer training loop:  72%|#######2  | 36117/50000 [23:34:30<7:08:48,  1.85s/it]
Outer training loop:  72%|#######2  | 36118/50000 [23:34:32<7:15:07,  1.88s/it]
Outer training loop:  72%|#######2  | 36119/50000 [23:34:34<7:15:38,  1.88s/it]
Outer training loop:  72%|#######2  | 36120/50000 [23:34:36<7:20:46,  1.91s/it]
Outer training loop:  72%|#######2  | 36121/50000 [23:34:38<7:18:12,  1.89s/it]
Outer training loop:  72%|#######2  | 36122/50000 [23:34:40<7:20:14,  1.90s/it]
Outer training loop:  72%|#######2  | 36123/50000 [23:34:42<7:13:43,  1.88s/it]
Outer training loop:  72%|#######2  | 36124/50000 [23:34:44<7:21:40,  1.91s/it]
Outer training loop:  72%|#######2  | 36125/50000 [23:34:46<7:15:52,  1.88s/it]
Outer training loop:  72%|#######2  | 36126/50000 [23:34:48<7:17:56,  1.89s/it]
Outer training loop:  72%|#######2  | 36127/50000 [23:34:50<7:32:15,  1.96s/it]
Outer training loop:  72%|#######2  | 36128/50000 [23:34:52<7:20:57,  1.91s/it]
Outer training loop:  72%|#######2  | 36129/50000 [23:34:53<7:17:07,  1.89s/it]
Outer training loop:  72%|#######2  | 36130/50000 [23:34:55<7:11:13,  1.87s/it]
Outer training loop:  72%|#######2  | 36131/50000 [23:34:57<7:25:36,  1.93s/it]
Outer training loop:  72%|#######2  | 36132/50000 [23:34:59<7:14:18,  1.88s/it]
Outer training loop:  72%|#######2  | 36133/50000 [23:35:01<7:12:44,  1.87s/it]
Outer training loop:  72%|#######2  | 36134/50000 [23:35:03<7:25:33,  1.93s/it]
Outer training loop:  72%|#######2  | 36135/50000 [23:35:05<7:20:09,  1.90s/it]
Outer training loop:  72%|#######2  | 36136/50000 [23:35:07<7:11:34,  1.87s/it]
Outer training loop:  72%|#######2  | 36137/50000 [23:35:08<7:07:01,  1.85s/it]
Outer training loop:  72%|#######2  | 36138/50000 [23:35:10<7:18:58,  1.90s/it]
Outer training loop:  72%|#######2  | 36139/50000 [23:35:12<7:15:48,  1.89s/it]
Outer training loop:  72%|#######2  | 36140/50000 [23:35:14<7:12:18,  1.87s/it]
Outer training loop:  72%|#######2  | 36141/50000 [23:35:16<7:19:21,  1.90s/it]
Outer training loop:  72%|#######2  | 36142/50000 [23:35:18<7:13:48,  1.88s/it]
Outer training loop:  72%|#######2  | 36143/50000 [23:35:20<7:11:48,  1.87s/it]
Outer training loop:  72%|#######2  | 36144/50000 [23:35:22<7:29:14,  1.95s/it]
Outer training loop:  72%|#######2  | 36145/50000 [23:35:24<7:21:51,  1.91s/it]
Outer training loop:  72%|#######2  | 36146/50000 [23:35:25<7:14:56,  1.88s/it]
Outer training loop:  72%|#######2  | 36147/50000 [23:35:27<7:15:42,  1.89s/it]
Outer training loop:  72%|#######2  | 36148/50000 [23:35:29<7:24:39,  1.93s/it]
Outer training loop:  72%|#######2  | 36149/50000 [23:35:31<7:18:09,  1.90s/it]
Outer training loop:  72%|#######2  | 36150/50000 [23:35:33<7:15:59,  1.89s/it]
Outer training loop:  72%|#######2  | 36151/50000 [23:35:35<7:26:19,  1.93s/it]
Outer training loop:  72%|#######2  | 36152/50000 [23:35:37<7:18:13,  1.90s/it]
Outer training loop:  72%|#######2  | 36153/50000 [23:35:39<7:04:23,  1.84s/it]
Outer training loop:  72%|#######2  | 36154/50000 [23:35:40<7:04:27,  1.84s/it]
Outer training loop:  72%|#######2  | 36155/50000 [23:35:42<7:12:55,  1.88s/it]
Outer training loop:  72%|#######2  | 36156/50000 [23:35:44<7:06:39,  1.85s/it]
Outer training loop:  72%|#######2  | 36157/50000 [23:35:46<7:02:00,  1.83s/it]
Outer training loop:  72%|#######2  | 36158/50000 [23:35:48<7:11:59,  1.87s/it]
Outer training loop:  72%|#######2  | 36159/50000 [23:35:50<7:11:19,  1.87s/it]
Outer training loop:  72%|#######2  | 36160/50000 [23:35:52<6:59:11,  1.82s/it]
Outer training loop:  72%|#######2  | 36161/50000 [23:35:53<6:55:46,  1.80s/it]
Outer training loop:  72%|#######2  | 36162/50000 [23:35:55<7:10:19,  1.87s/it]
Outer training loop:  72%|#######2  | 36163/50000 [23:35:57<7:18:51,  1.90s/it]
Outer training loop:  72%|#######2  | 36164/50000 [23:35:59<7:11:20,  1.87s/it]
Outer training loop:  72%|#######2  | 36165/50000 [23:36:01<7:12:29,  1.88s/it]
Outer training loop:  72%|#######2  | 36166/50000 [23:36:03<7:08:09,  1.86s/it]
Outer training loop:  72%|#######2  | 36167/50000 [23:36:05<7:20:06,  1.91s/it]
Outer training loop:  72%|#######2  | 36168/50000 [23:36:07<7:24:22,  1.93s/it]
Outer training loop:  72%|#######2  | 36169/50000 [23:36:09<7:26:27,  1.94s/it]
Outer training loop:  72%|#######2  | 36170/50000 [23:36:11<7:22:39,  1.92s/it]
Outer training loop:  72%|#######2  | 36171/50000 [23:36:13<7:42:55,  2.01s/it]
Outer training loop:  72%|#######2  | 36172/50000 [23:36:15<7:31:33,  1.96s/it]
Outer training loop:  72%|#######2  | 36173/50000 [23:36:17<7:24:13,  1.93s/it]
Outer training loop:  72%|#######2  | 36174/50000 [23:36:18<7:18:01,  1.90s/it]
Outer training loop:  72%|#######2  | 36175/50000 [23:36:20<7:27:36,  1.94s/it]
Outer training loop:  72%|#######2  | 36176/50000 [23:36:22<7:21:31,  1.92s/it]
Outer training loop:  72%|#######2  | 36177/50000 [23:36:24<7:09:38,  1.86s/it]
Outer training loop:  72%|#######2  | 36178/50000 [23:36:26<7:25:47,  1.94s/it]
Outer training loop:  72%|#######2  | 36179/50000 [23:36:28<7:18:00,  1.90s/it]
Outer training loop:  72%|#######2  | 36180/50000 [23:36:30<6:56:54,  1.81s/it]
Outer training loop:  72%|#######2  | 36181/50000 [23:36:32<7:07:31,  1.86s/it]
Outer training loop:  72%|#######2  | 36182/50000 [23:36:33<7:00:31,  1.83s/it]
Outer training loop:  72%|#######2  | 36183/50000 [23:36:35<6:58:15,  1.82s/it]
Outer training loop:  72%|#######2  | 36184/50000 [23:36:37<7:01:53,  1.83s/it]
Outer training loop:  72%|#######2  | 36185/50000 [23:36:39<7:14:17,  1.89s/it]
Outer training loop:  72%|#######2  | 36186/50000 [23:36:41<7:06:55,  1.85s/it]
Outer training loop:  72%|#######2  | 36187/50000 [23:36:42<6:56:36,  1.81s/it]
Outer training loop:  72%|#######2  | 36188/50000 [23:36:44<7:11:51,  1.88s/it]
Outer training loop:  72%|#######2  | 36189/50000 [23:36:46<7:18:36,  1.91s/it]
Outer training loop:  72%|#######2  | 36190/50000 [23:36:48<7:20:30,  1.91s/it]
Outer training loop:  72%|#######2  | 36191/50000 [23:36:50<7:21:13,  1.92s/it]
Outer training loop:  72%|#######2  | 36192/50000 [23:36:52<7:16:44,  1.90s/it]
Outer training loop:  72%|#######2  | 36193/50000 [23:36:54<7:08:45,  1.86s/it]
Outer training loop:  72%|#######2  | 36194/50000 [23:36:56<7:17:21,  1.90s/it]
Outer training loop:  72%|#######2  | 36195/50000 [23:36:58<7:10:09,  1.87s/it]
Outer training loop:  72%|#######2  | 36196/50000 [23:37:00<7:04:09,  1.84s/it]
Outer training loop:  72%|#######2  | 36197/50000 [23:37:02<7:20:34,  1.92s/it]
Outer training loop:  72%|#######2  | 36198/50000 [23:37:03<7:15:26,  1.89s/it]
Outer training loop:  72%|#######2  | 36199/50000 [23:37:05<7:10:26,  1.87s/it]
Outer training loop:  72%|#######2  | 36200/50000 [23:37:07<7:13:10,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  72%|#######2  | 36201/50000 [23:37:14<13:26:11,  3.51s/it]
Outer training loop:  72%|#######2  | 36202/50000 [23:37:16<11:40:34,  3.05s/it]
Outer training loop:  72%|#######2  | 36203/50000 [23:37:18<10:16:01,  2.68s/it]
Outer training loop:  72%|#######2  | 36204/50000 [23:37:20<9:20:59,  2.44s/it] 
Outer training loop:  72%|#######2  | 36205/50000 [23:37:22<8:39:06,  2.26s/it]
Outer training loop:  72%|#######2  | 36206/50000 [23:37:24<8:16:33,  2.16s/it]
Outer training loop:  72%|#######2  | 36207/50000 [23:37:26<8:04:57,  2.11s/it]
Outer training loop:  72%|#######2  | 36208/50000 [23:37:28<7:46:24,  2.03s/it]
Outer training loop:  72%|#######2  | 36209/50000 [23:37:30<7:46:30,  2.03s/it]
Outer training loop:  72%|#######2  | 36210/50000 [23:37:32<7:44:07,  2.02s/it]
Outer training loop:  72%|#######2  | 36211/50000 [23:37:34<7:24:26,  1.93s/it]
Outer training loop:  72%|#######2  | 36212/50000 [23:37:35<7:28:27,  1.95s/it]
Outer training loop:  72%|#######2  | 36213/50000 [23:37:37<7:26:18,  1.94s/it]
Outer training loop:  72%|#######2  | 36214/50000 [23:37:39<7:12:16,  1.88s/it]
Outer training loop:  72%|#######2  | 36215/50000 [23:37:41<7:22:25,  1.93s/it]
Outer training loop:  72%|#######2  | 36216/50000 [23:37:43<7:16:18,  1.90s/it]
Outer training loop:  72%|#######2  | 36217/50000 [23:37:45<7:09:34,  1.87s/it]
Outer training loop:  72%|#######2  | 36218/50000 [23:37:47<7:00:17,  1.83s/it]
Outer training loop:  72%|#######2  | 36219/50000 [23:37:49<7:13:24,  1.89s/it]
Outer training loop:  72%|#######2  | 36220/50000 [23:37:50<7:13:53,  1.89s/it]
Outer training loop:  72%|#######2  | 36221/50000 [23:37:52<7:08:16,  1.86s/it]
Outer training loop:  72%|#######2  | 36222/50000 [23:37:54<7:19:51,  1.92s/it]
Outer training loop:  72%|#######2  | 36223/50000 [23:37:56<7:21:21,  1.92s/it]
Outer training loop:  72%|#######2  | 36224/50000 [23:37:58<7:30:43,  1.96s/it]
Outer training loop:  72%|#######2  | 36225/50000 [23:38:00<7:19:59,  1.92s/it]
Outer training loop:  72%|#######2  | 36226/50000 [23:38:02<7:19:53,  1.92s/it]
Outer training loop:  72%|#######2  | 36227/50000 [23:38:04<7:16:55,  1.90s/it]
Outer training loop:  72%|#######2  | 36228/50000 [23:38:06<7:22:38,  1.93s/it]
Outer training loop:  72%|#######2  | 36229/50000 [23:38:08<7:28:35,  1.95s/it]
Outer training loop:  72%|#######2  | 36230/50000 [23:38:10<7:24:29,  1.94s/it]
Outer training loop:  72%|#######2  | 36231/50000 [23:38:12<7:13:00,  1.89s/it]
Outer training loop:  72%|#######2  | 36232/50000 [23:38:14<7:22:55,  1.93s/it]
Outer training loop:  72%|#######2  | 36233/50000 [23:38:15<7:14:51,  1.90s/it]
Outer training loop:  72%|#######2  | 36234/50000 [23:38:17<7:07:30,  1.86s/it]
Outer training loop:  72%|#######2  | 36235/50000 [23:38:19<7:04:37,  1.85s/it]
Outer training loop:  72%|#######2  | 36236/50000 [23:38:21<7:17:44,  1.91s/it]
Outer training loop:  72%|#######2  | 36237/50000 [23:38:23<7:07:08,  1.86s/it]
Outer training loop:  72%|#######2  | 36238/50000 [23:38:25<7:07:55,  1.87s/it]
Outer training loop:  72%|#######2  | 36239/50000 [23:38:27<7:16:55,  1.91s/it]
Outer training loop:  72%|#######2  | 36240/50000 [23:38:29<7:13:43,  1.89s/it]
Outer training loop:  72%|#######2  | 36241/50000 [23:38:30<7:05:41,  1.86s/it]
Outer training loop:  72%|#######2  | 36242/50000 [23:38:32<7:02:45,  1.84s/it]
Outer training loop:  72%|#######2  | 36243/50000 [23:38:34<7:11:40,  1.88s/it]
Outer training loop:  72%|#######2  | 36244/50000 [23:38:36<7:08:54,  1.87s/it]
Outer training loop:  72%|#######2  | 36245/50000 [23:38:38<7:03:22,  1.85s/it]
Outer training loop:  72%|#######2  | 36246/50000 [23:38:40<7:18:53,  1.91s/it]
Outer training loop:  72%|#######2  | 36247/50000 [23:38:42<7:13:18,  1.89s/it]
Outer training loop:  72%|#######2  | 36248/50000 [23:38:44<7:19:01,  1.92s/it]
Outer training loop:  72%|#######2  | 36249/50000 [23:38:45<7:12:47,  1.89s/it]
Outer training loop:  72%|#######2  | 36250/50000 [23:38:47<7:15:19,  1.90s/it]
Outer training loop:  73%|#######2  | 36251/50000 [23:38:49<7:09:59,  1.88s/it]
Outer training loop:  73%|#######2  | 36252/50000 [23:38:51<7:04:34,  1.85s/it]
Outer training loop:  73%|#######2  | 36253/50000 [23:38:53<7:12:01,  1.89s/it]
Outer training loop:  73%|#######2  | 36254/50000 [23:38:55<7:16:15,  1.90s/it]
Outer training loop:  73%|#######2  | 36255/50000 [23:38:57<7:16:24,  1.91s/it]
Outer training loop:  73%|#######2  | 36256/50000 [23:38:59<7:21:15,  1.93s/it]
Outer training loop:  73%|#######2  | 36257/50000 [23:39:01<7:11:13,  1.88s/it]
Outer training loop:  73%|#######2  | 36258/50000 [23:39:02<7:12:49,  1.89s/it]
Outer training loop:  73%|#######2  | 36259/50000 [23:39:04<7:05:21,  1.86s/it]
Outer training loop:  73%|#######2  | 36260/50000 [23:39:07<7:32:18,  1.98s/it]
Outer training loop:  73%|#######2  | 36261/50000 [23:39:08<7:21:19,  1.93s/it]
Outer training loop:  73%|#######2  | 36262/50000 [23:39:10<7:11:37,  1.89s/it]
Outer training loop:  73%|#######2  | 36263/50000 [23:39:12<7:13:49,  1.89s/it]
Outer training loop:  73%|#######2  | 36264/50000 [23:39:14<7:09:10,  1.87s/it]
Outer training loop:  73%|#######2  | 36265/50000 [23:39:16<6:55:04,  1.81s/it]
Outer training loop:  73%|#######2  | 36266/50000 [23:39:18<7:12:07,  1.89s/it]
Outer training loop:  73%|#######2  | 36267/50000 [23:39:19<7:02:11,  1.84s/it]
Outer training loop:  73%|#######2  | 36268/50000 [23:39:21<7:03:02,  1.85s/it]
Outer training loop:  73%|#######2  | 36269/50000 [23:39:23<7:21:09,  1.93s/it]
Outer training loop:  73%|#######2  | 36270/50000 [23:39:25<7:22:20,  1.93s/it]
Outer training loop:  73%|#######2  | 36271/50000 [23:39:27<7:11:27,  1.89s/it]
Outer training loop:  73%|#######2  | 36272/50000 [23:39:29<7:02:45,  1.85s/it]
Outer training loop:  73%|#######2  | 36273/50000 [23:39:31<7:12:01,  1.89s/it]
Outer training loop:  73%|#######2  | 36274/50000 [23:39:33<7:02:10,  1.85s/it]
Outer training loop:  73%|#######2  | 36275/50000 [23:39:34<7:01:22,  1.84s/it]
Outer training loop:  73%|#######2  | 36276/50000 [23:39:36<7:13:02,  1.89s/it]
Outer training loop:  73%|#######2  | 36277/50000 [23:39:38<7:06:12,  1.86s/it]
Outer training loop:  73%|#######2  | 36278/50000 [23:39:40<7:04:13,  1.85s/it]
Outer training loop:  73%|#######2  | 36279/50000 [23:39:42<7:04:49,  1.86s/it]
Outer training loop:  73%|#######2  | 36280/50000 [23:39:44<7:18:50,  1.92s/it]
Outer training loop:  73%|#######2  | 36281/50000 [23:39:46<7:12:12,  1.89s/it]
Outer training loop:  73%|#######2  | 36282/50000 [23:39:48<7:10:08,  1.88s/it]
Outer training loop:  73%|#######2  | 36283/50000 [23:39:50<7:27:32,  1.96s/it]
Outer training loop:  73%|#######2  | 36284/50000 [23:39:52<7:19:21,  1.92s/it]
Outer training loop:  73%|#######2  | 36285/50000 [23:39:53<7:11:40,  1.89s/it]
Outer training loop:  73%|#######2  | 36286/50000 [23:39:55<7:18:52,  1.92s/it]
Outer training loop:  73%|#######2  | 36287/50000 [23:39:57<7:20:12,  1.93s/it]
Outer training loop:  73%|#######2  | 36288/50000 [23:39:59<7:17:38,  1.91s/it]
Outer training loop:  73%|#######2  | 36289/50000 [23:40:01<7:15:48,  1.91s/it]
Outer training loop:  73%|#######2  | 36290/50000 [23:40:03<7:33:48,  1.99s/it]
Outer training loop:  73%|#######2  | 36291/50000 [23:40:05<7:26:49,  1.96s/it]
Outer training loop:  73%|#######2  | 36292/50000 [23:40:07<7:27:21,  1.96s/it]
Outer training loop:  73%|#######2  | 36293/50000 [23:40:09<7:31:16,  1.98s/it]
Outer training loop:  73%|#######2  | 36294/50000 [23:40:11<7:21:21,  1.93s/it]
Outer training loop:  73%|#######2  | 36295/50000 [23:40:13<7:21:21,  1.93s/it]
Outer training loop:  73%|#######2  | 36296/50000 [23:40:15<7:24:35,  1.95s/it]
Outer training loop:  73%|#######2  | 36297/50000 [23:40:17<7:15:29,  1.91s/it]
Outer training loop:  73%|#######2  | 36298/50000 [23:40:19<7:21:35,  1.93s/it]
Outer training loop:  73%|#######2  | 36299/50000 [23:40:20<7:06:03,  1.87s/it]
Outer training loop:  73%|#######2  | 36300/50000 [23:40:22<7:10:22,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  73%|#######2  | 36301/50000 [23:40:30<13:28:39,  3.54s/it]
Outer training loop:  73%|#######2  | 36302/50000 [23:40:31<11:26:16,  3.01s/it]
Outer training loop:  73%|#######2  | 36303/50000 [23:40:33<10:11:57,  2.68s/it]
Outer training loop:  73%|#######2  | 36304/50000 [23:40:35<9:28:17,  2.49s/it] 
Outer training loop:  73%|#######2  | 36305/50000 [23:40:37<8:44:11,  2.30s/it]
Outer training loop:  73%|#######2  | 36306/50000 [23:40:39<8:10:00,  2.15s/it]
Outer training loop:  73%|#######2  | 36307/50000 [23:40:41<8:26:44,  2.22s/it]
Outer training loop:  73%|#######2  | 36308/50000 [23:40:43<7:57:16,  2.09s/it]
Outer training loop:  73%|#######2  | 36309/50000 [23:40:45<7:30:03,  1.97s/it]
Outer training loop:  73%|#######2  | 36310/50000 [23:40:47<7:25:23,  1.95s/it]
Outer training loop:  73%|#######2  | 36311/50000 [23:40:49<7:24:00,  1.95s/it]
Outer training loop:  73%|#######2  | 36312/50000 [23:40:51<7:16:34,  1.91s/it]
Outer training loop:  73%|#######2  | 36313/50000 [23:40:52<7:09:57,  1.88s/it]
Outer training loop:  73%|#######2  | 36314/50000 [23:40:55<7:26:03,  1.96s/it]
Outer training loop:  73%|#######2  | 36315/50000 [23:40:56<7:12:55,  1.90s/it]
Outer training loop:  73%|#######2  | 36316/50000 [23:40:58<7:22:50,  1.94s/it]
Outer training loop:  73%|#######2  | 36317/50000 [23:41:00<7:15:03,  1.91s/it]
Outer training loop:  73%|#######2  | 36318/50000 [23:41:02<7:19:23,  1.93s/it]
Outer training loop:  73%|#######2  | 36319/50000 [23:41:04<7:10:57,  1.89s/it]
Outer training loop:  73%|#######2  | 36320/50000 [23:41:06<7:05:50,  1.87s/it]
Outer training loop:  73%|#######2  | 36321/50000 [23:41:08<7:18:56,  1.93s/it]
Outer training loop:  73%|#######2  | 36322/50000 [23:41:10<7:12:46,  1.90s/it]
Outer training loop:  73%|#######2  | 36323/50000 [23:41:12<7:07:53,  1.88s/it]
Outer training loop:  73%|#######2  | 36324/50000 [23:41:13<7:02:57,  1.86s/it]
Outer training loop:  73%|#######2  | 36325/50000 [23:41:15<7:15:02,  1.91s/it]
Outer training loop:  73%|#######2  | 36326/50000 [23:41:17<7:13:55,  1.90s/it]
Outer training loop:  73%|#######2  | 36327/50000 [23:41:19<7:10:39,  1.89s/it]
Outer training loop:  73%|#######2  | 36328/50000 [23:41:21<7:13:44,  1.90s/it]
Outer training loop:  73%|#######2  | 36329/50000 [23:41:23<7:07:32,  1.88s/it]
Outer training loop:  73%|#######2  | 36330/50000 [23:41:25<7:08:47,  1.88s/it]
Outer training loop:  73%|#######2  | 36331/50000 [23:41:27<7:03:16,  1.86s/it]
Outer training loop:  73%|#######2  | 36332/50000 [23:41:29<7:12:26,  1.90s/it]
Outer training loop:  73%|#######2  | 36333/50000 [23:41:30<7:06:06,  1.87s/it]
Outer training loop:  73%|#######2  | 36334/50000 [23:41:32<7:04:03,  1.86s/it]
Outer training loop:  73%|#######2  | 36335/50000 [23:41:34<7:19:58,  1.93s/it]
Outer training loop:  73%|#######2  | 36336/50000 [23:41:36<7:12:27,  1.90s/it]
Outer training loop:  73%|#######2  | 36337/50000 [23:41:38<7:06:19,  1.87s/it]
Outer training loop:  73%|#######2  | 36338/50000 [23:41:40<7:13:30,  1.90s/it]
Outer training loop:  73%|#######2  | 36339/50000 [23:41:42<7:06:43,  1.87s/it]
Outer training loop:  73%|#######2  | 36340/50000 [23:41:44<7:02:35,  1.86s/it]
Outer training loop:  73%|#######2  | 36341/50000 [23:41:46<7:10:29,  1.89s/it]
Outer training loop:  73%|#######2  | 36342/50000 [23:41:47<7:08:04,  1.88s/it]
Outer training loop:  73%|#######2  | 36343/50000 [23:41:49<6:59:08,  1.84s/it]
Outer training loop:  73%|#######2  | 36344/50000 [23:41:51<7:06:08,  1.87s/it]
Outer training loop:  73%|#######2  | 36345/50000 [23:41:53<7:21:04,  1.94s/it]
Outer training loop:  73%|#######2  | 36346/50000 [23:41:55<7:06:53,  1.88s/it]
Outer training loop:  73%|#######2  | 36347/50000 [23:41:57<6:56:59,  1.83s/it]
Outer training loop:  73%|#######2  | 36348/50000 [23:41:59<7:08:59,  1.89s/it]
Outer training loop:  73%|#######2  | 36349/50000 [23:42:00<7:05:29,  1.87s/it]
Outer training loop:  73%|#######2  | 36350/50000 [23:42:02<7:02:09,  1.86s/it]
Outer training loop:  73%|#######2  | 36351/50000 [23:42:04<7:02:49,  1.86s/it]
Outer training loop:  73%|#######2  | 36352/50000 [23:42:06<7:13:40,  1.91s/it]
Outer training loop:  73%|#######2  | 36353/50000 [23:42:08<7:06:51,  1.88s/it]
Outer training loop:  73%|#######2  | 36354/50000 [23:42:10<7:00:12,  1.85s/it]
Outer training loop:  73%|#######2  | 36355/50000 [23:42:12<7:15:28,  1.91s/it]
Outer training loop:  73%|#######2  | 36356/50000 [23:42:14<7:16:53,  1.92s/it]
Outer training loop:  73%|#######2  | 36357/50000 [23:42:16<7:18:15,  1.93s/it]
Outer training loop:  73%|#######2  | 36358/50000 [23:42:18<7:20:02,  1.94s/it]
Outer training loop:  73%|#######2  | 36359/50000 [23:42:20<7:15:35,  1.92s/it]
Outer training loop:  73%|#######2  | 36360/50000 [23:42:21<7:09:37,  1.89s/it]
Outer training loop:  73%|#######2  | 36361/50000 [23:42:23<7:17:03,  1.92s/it]
Outer training loop:  73%|#######2  | 36362/50000 [23:42:25<7:20:15,  1.94s/it]
Outer training loop:  73%|#######2  | 36363/50000 [23:42:27<7:09:32,  1.89s/it]
Outer training loop:  73%|#######2  | 36364/50000 [23:42:29<7:13:58,  1.91s/it]
Outer training loop:  73%|#######2  | 36365/50000 [23:42:31<6:58:13,  1.84s/it]
Outer training loop:  73%|#######2  | 36366/50000 [23:42:33<7:00:40,  1.85s/it]
Outer training loop:  73%|#######2  | 36367/50000 [23:42:35<7:03:43,  1.86s/it]
Outer training loop:  73%|#######2  | 36368/50000 [23:42:36<7:06:01,  1.88s/it]
Outer training loop:  73%|#######2  | 36369/50000 [23:42:38<7:00:53,  1.85s/it]
Outer training loop:  73%|#######2  | 36370/50000 [23:42:40<7:04:30,  1.87s/it]
Outer training loop:  73%|#######2  | 36371/50000 [23:42:42<7:17:03,  1.92s/it]
Outer training loop:  73%|#######2  | 36372/50000 [23:42:44<7:06:45,  1.88s/it]
Outer training loop:  73%|#######2  | 36373/50000 [23:42:46<6:56:42,  1.83s/it]
Outer training loop:  73%|#######2  | 36374/50000 [23:42:48<7:08:23,  1.89s/it]
Outer training loop:  73%|#######2  | 36375/50000 [23:42:50<7:04:10,  1.87s/it]
Outer training loop:  73%|#######2  | 36376/50000 [23:42:51<6:58:46,  1.84s/it]
Outer training loop:  73%|#######2  | 36377/50000 [23:42:53<6:52:15,  1.82s/it]
Outer training loop:  73%|#######2  | 36378/50000 [23:42:55<7:06:47,  1.88s/it]
Outer training loop:  73%|#######2  | 36379/50000 [23:42:57<7:03:54,  1.87s/it]
Outer training loop:  73%|#######2  | 36380/50000 [23:42:59<6:58:47,  1.84s/it]
Outer training loop:  73%|#######2  | 36381/50000 [23:43:01<7:09:21,  1.89s/it]
Outer training loop:  73%|#######2  | 36382/50000 [23:43:02<6:57:59,  1.84s/it]
Outer training loop:  73%|#######2  | 36383/50000 [23:43:04<6:56:13,  1.83s/it]
Outer training loop:  73%|#######2  | 36384/50000 [23:43:06<6:59:15,  1.85s/it]
Outer training loop:  73%|#######2  | 36385/50000 [23:43:08<7:05:23,  1.87s/it]
Outer training loop:  73%|#######2  | 36386/50000 [23:43:10<7:02:15,  1.86s/it]
Outer training loop:  73%|#######2  | 36387/50000 [23:43:12<6:59:20,  1.85s/it]
Outer training loop:  73%|#######2  | 36388/50000 [23:43:14<7:07:27,  1.88s/it]
Outer training loop:  73%|#######2  | 36389/50000 [23:43:15<6:56:07,  1.83s/it]
Outer training loop:  73%|#######2  | 36390/50000 [23:43:17<6:53:21,  1.82s/it]
Outer training loop:  73%|#######2  | 36391/50000 [23:43:19<6:52:51,  1.82s/it]
Outer training loop:  73%|#######2  | 36392/50000 [23:43:21<7:08:44,  1.89s/it]
Outer training loop:  73%|#######2  | 36393/50000 [23:43:23<7:08:37,  1.89s/it]
Outer training loop:  73%|#######2  | 36394/50000 [23:43:25<7:07:55,  1.89s/it]
Outer training loop:  73%|#######2  | 36395/50000 [23:43:27<7:20:43,  1.94s/it]
Outer training loop:  73%|#######2  | 36396/50000 [23:43:29<7:23:40,  1.96s/it]
Outer training loop:  73%|#######2  | 36397/50000 [23:43:31<7:25:38,  1.97s/it]
Outer training loop:  73%|#######2  | 36398/50000 [23:43:33<7:22:42,  1.95s/it]
Outer training loop:  73%|#######2  | 36399/50000 [23:43:35<7:31:05,  1.99s/it]
Outer training loop:  73%|#######2  | 36400/50000 [23:43:37<7:29:29,  1.98s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.99s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.99s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  73%|#######2  | 36401/50000 [23:43:44<13:41:00,  3.62s/it]
Outer training loop:  73%|#######2  | 36402/50000 [23:43:46<11:48:53,  3.13s/it]
Outer training loop:  73%|#######2  | 36403/50000 [23:43:48<10:26:43,  2.77s/it]
Outer training loop:  73%|#######2  | 36404/50000 [23:43:50<9:45:38,  2.58s/it] 
Outer training loop:  73%|#######2  | 36405/50000 [23:43:52<8:59:33,  2.38s/it]
Outer training loop:  73%|#######2  | 36406/50000 [23:43:54<8:33:16,  2.27s/it]
Outer training loop:  73%|#######2  | 36407/50000 [23:43:56<8:04:27,  2.14s/it]
Outer training loop:  73%|#######2  | 36408/50000 [23:43:58<7:41:47,  2.04s/it]
Outer training loop:  73%|#######2  | 36409/50000 [23:44:00<7:45:43,  2.06s/it]
Outer training loop:  73%|#######2  | 36410/50000 [23:44:02<7:28:45,  1.98s/it]
Outer training loop:  73%|#######2  | 36411/50000 [23:44:04<7:24:51,  1.96s/it]
Outer training loop:  73%|#######2  | 36412/50000 [23:44:06<7:30:53,  1.99s/it]
Outer training loop:  73%|#######2  | 36413/50000 [23:44:08<7:12:41,  1.91s/it]
Outer training loop:  73%|#######2  | 36414/50000 [23:44:09<7:04:26,  1.87s/it]
Outer training loop:  73%|#######2  | 36415/50000 [23:44:11<7:09:37,  1.90s/it]
Outer training loop:  73%|#######2  | 36416/50000 [23:44:13<7:20:34,  1.95s/it]
Outer training loop:  73%|#######2  | 36417/50000 [23:44:15<7:20:35,  1.95s/it]
Outer training loop:  73%|#######2  | 36418/50000 [23:44:17<7:07:02,  1.89s/it]
Outer training loop:  73%|#######2  | 36419/50000 [23:44:19<7:12:18,  1.91s/it]
Outer training loop:  73%|#######2  | 36420/50000 [23:44:21<7:07:20,  1.89s/it]
Outer training loop:  73%|#######2  | 36421/50000 [23:44:23<7:04:04,  1.87s/it]
Outer training loop:  73%|#######2  | 36422/50000 [23:44:24<6:58:13,  1.85s/it]
Outer training loop:  73%|#######2  | 36423/50000 [23:44:26<7:07:04,  1.89s/it]
Outer training loop:  73%|#######2  | 36424/50000 [23:44:28<7:08:31,  1.89s/it]
Outer training loop:  73%|#######2  | 36425/50000 [23:44:30<7:02:27,  1.87s/it]
Outer training loop:  73%|#######2  | 36426/50000 [23:44:32<7:21:47,  1.95s/it]
Outer training loop:  73%|#######2  | 36427/50000 [23:44:34<7:08:44,  1.90s/it]
Outer training loop:  73%|#######2  | 36428/50000 [23:44:36<7:03:28,  1.87s/it]
Outer training loop:  73%|#######2  | 36429/50000 [23:44:38<7:03:11,  1.87s/it]
Outer training loop:  73%|#######2  | 36430/50000 [23:44:40<7:02:32,  1.87s/it]
Outer training loop:  73%|#######2  | 36431/50000 [23:44:42<7:05:51,  1.88s/it]
Outer training loop:  73%|#######2  | 36432/50000 [23:44:43<7:03:34,  1.87s/it]
Outer training loop:  73%|#######2  | 36433/50000 [23:44:45<7:19:59,  1.95s/it]
Outer training loop:  73%|#######2  | 36434/50000 [23:44:47<7:11:40,  1.91s/it]
Outer training loop:  73%|#######2  | 36435/50000 [23:44:49<7:12:41,  1.91s/it]
Outer training loop:  73%|#######2  | 36436/50000 [23:44:51<7:06:11,  1.89s/it]
Outer training loop:  73%|#######2  | 36437/50000 [23:44:53<7:09:47,  1.90s/it]
Outer training loop:  73%|#######2  | 36438/50000 [23:44:55<7:00:13,  1.86s/it]
Outer training loop:  73%|#######2  | 36439/50000 [23:44:57<6:54:09,  1.83s/it]
Outer training loop:  73%|#######2  | 36440/50000 [23:44:59<7:09:26,  1.90s/it]
Outer training loop:  73%|#######2  | 36441/50000 [23:45:00<7:02:47,  1.87s/it]
Outer training loop:  73%|#######2  | 36442/50000 [23:45:02<6:59:05,  1.85s/it]
Outer training loop:  73%|#######2  | 36443/50000 [23:45:04<7:08:10,  1.89s/it]
Outer training loop:  73%|#######2  | 36444/50000 [23:45:06<7:10:17,  1.90s/it]
Outer training loop:  73%|#######2  | 36445/50000 [23:45:08<7:03:19,  1.87s/it]
Outer training loop:  73%|#######2  | 36446/50000 [23:45:10<7:19:11,  1.94s/it]
Outer training loop:  73%|#######2  | 36447/50000 [23:45:12<7:09:34,  1.90s/it]
Outer training loop:  73%|#######2  | 36448/50000 [23:45:14<7:05:15,  1.88s/it]
Outer training loop:  73%|#######2  | 36449/50000 [23:45:15<6:54:10,  1.83s/it]
Outer training loop:  73%|#######2  | 36450/50000 [23:45:17<7:08:12,  1.90s/it]
Outer training loop:  73%|#######2  | 36451/50000 [23:45:19<7:03:07,  1.87s/it]
Outer training loop:  73%|#######2  | 36452/50000 [23:45:21<6:55:02,  1.84s/it]
Outer training loop:  73%|#######2  | 36453/50000 [23:45:23<7:02:56,  1.87s/it]
Outer training loop:  73%|#######2  | 36454/50000 [23:45:25<7:14:30,  1.92s/it]
Outer training loop:  73%|#######2  | 36455/50000 [23:45:27<7:09:53,  1.90s/it]
Outer training loop:  73%|#######2  | 36456/50000 [23:45:29<6:55:14,  1.84s/it]
Outer training loop:  73%|#######2  | 36457/50000 [23:45:31<7:06:55,  1.89s/it]
Outer training loop:  73%|#######2  | 36458/50000 [23:45:32<6:59:47,  1.86s/it]
Outer training loop:  73%|#######2  | 36459/50000 [23:45:34<6:55:59,  1.84s/it]
Outer training loop:  73%|#######2  | 36460/50000 [23:45:36<7:06:39,  1.89s/it]
Outer training loop:  73%|#######2  | 36461/50000 [23:45:38<7:06:00,  1.89s/it]
Outer training loop:  73%|#######2  | 36462/50000 [23:45:40<7:01:04,  1.87s/it]
Outer training loop:  73%|#######2  | 36463/50000 [23:45:42<7:09:41,  1.90s/it]
Outer training loop:  73%|#######2  | 36464/50000 [23:45:44<7:09:11,  1.90s/it]
Outer training loop:  73%|#######2  | 36465/50000 [23:45:45<6:57:26,  1.85s/it]
Outer training loop:  73%|#######2  | 36466/50000 [23:45:47<6:52:23,  1.83s/it]
Outer training loop:  73%|#######2  | 36467/50000 [23:45:49<7:11:05,  1.91s/it]
Outer training loop:  73%|#######2  | 36468/50000 [23:45:51<7:07:56,  1.90s/it]
Outer training loop:  73%|#######2  | 36469/50000 [23:45:53<7:07:48,  1.90s/it]
Outer training loop:  73%|#######2  | 36470/50000 [23:45:55<7:18:18,  1.94s/it]
Outer training loop:  73%|#######2  | 36471/50000 [23:45:57<7:07:13,  1.89s/it]
Outer training loop:  73%|#######2  | 36472/50000 [23:45:59<6:57:47,  1.85s/it]
Outer training loop:  73%|#######2  | 36473/50000 [23:46:01<6:59:25,  1.86s/it]
Outer training loop:  73%|#######2  | 36474/50000 [23:46:03<7:11:45,  1.92s/it]
Outer training loop:  73%|#######2  | 36475/50000 [23:46:04<7:07:15,  1.90s/it]
Outer training loop:  73%|#######2  | 36476/50000 [23:46:06<7:05:52,  1.89s/it]
Outer training loop:  73%|#######2  | 36477/50000 [23:46:08<7:08:15,  1.90s/it]
Outer training loop:  73%|#######2  | 36478/50000 [23:46:10<7:03:01,  1.88s/it]
Outer training loop:  73%|#######2  | 36479/50000 [23:46:12<6:59:22,  1.86s/it]
Outer training loop:  73%|#######2  | 36480/50000 [23:46:14<7:13:13,  1.92s/it]
Outer training loop:  73%|#######2  | 36481/50000 [23:46:16<7:04:00,  1.88s/it]
Outer training loop:  73%|#######2  | 36482/50000 [23:46:18<6:58:24,  1.86s/it]
Outer training loop:  73%|#######2  | 36483/50000 [23:46:20<7:12:33,  1.92s/it]
Outer training loop:  73%|#######2  | 36484/50000 [23:46:21<7:02:53,  1.88s/it]
Outer training loop:  73%|#######2  | 36485/50000 [23:46:23<6:49:36,  1.82s/it]
Outer training loop:  73%|#######2  | 36486/50000 [23:46:25<7:00:28,  1.87s/it]
Outer training loop:  73%|#######2  | 36487/50000 [23:46:27<7:08:34,  1.90s/it]
Outer training loop:  73%|#######2  | 36488/50000 [23:46:29<7:09:37,  1.91s/it]
Outer training loop:  73%|#######2  | 36489/50000 [23:46:31<6:58:31,  1.86s/it]
Outer training loop:  73%|#######2  | 36490/50000 [23:46:33<7:07:42,  1.90s/it]
Outer training loop:  73%|#######2  | 36491/50000 [23:46:35<7:03:47,  1.88s/it]
Outer training loop:  73%|#######2  | 36492/50000 [23:46:36<7:01:15,  1.87s/it]
Outer training loop:  73%|#######2  | 36493/50000 [23:46:38<6:55:57,  1.85s/it]
Outer training loop:  73%|#######2  | 36494/50000 [23:46:40<7:15:28,  1.93s/it]
Outer training loop:  73%|#######2  | 36495/50000 [23:46:42<7:08:48,  1.91s/it]
Outer training loop:  73%|#######2  | 36496/50000 [23:46:44<7:06:44,  1.90s/it]
Outer training loop:  73%|#######2  | 36497/50000 [23:46:46<7:11:20,  1.92s/it]
Outer training loop:  73%|#######2  | 36498/50000 [23:46:48<7:02:07,  1.88s/it]
Outer training loop:  73%|#######2  | 36499/50000 [23:46:50<7:05:54,  1.89s/it]
Outer training loop:  73%|#######3  | 36500/50000 [23:46:52<7:15:23,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:31,  1.85s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:29,  1.84s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:35,  1.94s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.90s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.90s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.90s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:28,  1.96s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:17,  1.88s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:24,  2.01s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:18,  1.96s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:15,  1.94s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:13,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:14,  1.97s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.93s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:05,  1.96s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.93s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:57,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:53,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:51,  1.86s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:53,  1.93s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.91s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.94s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:48,  2.00s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:45,  2.00s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:44,  2.01s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:41,  1.99s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:37,  1.95s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:33,  1.90s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:36,  2.01s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:32,  1.97s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:28,  1.92s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:25,  1.91s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:19<01:21,  1.95s/it]

Evaluating policy:  51%|#####     | 42/83 [01:21<01:20,  1.97s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:23<01:17,  1.93s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:14,  1.90s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:11,  1.88s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:07,  1.87s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:08,  1.97s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:04,  1.91s/it]

Evaluating policy:  60%|######    | 50/83 [01:36<01:03,  1.93s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:56,  1.88s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:53,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:55,  1.98s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:52,  1.95s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:50,  1.95s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:48,  1.94s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:45,  1.90s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:43,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:42,  1.92s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:40,  1.92s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:01<00:40,  2.04s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:03<00:37,  1.98s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:05<00:34,  1.94s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:07<00:32,  1.92s/it]

Evaluating policy:  81%|########  | 67/83 [02:09<00:30,  1.89s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:28,  1.89s/it]

Evaluating policy:  83%|########3 | 69/83 [02:12<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:16<00:23,  1.99s/it]

Evaluating policy:  87%|########6 | 72/83 [02:18<00:21,  1.92s/it]

Evaluating policy:  88%|########7 | 73/83 [02:20<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:22<00:17,  1.90s/it]

Evaluating policy:  90%|######### | 75/83 [02:24<00:15,  1.92s/it]

Evaluating policy:  92%|#########1| 76/83 [02:26<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:28<00:11,  1.93s/it]

Evaluating policy:  94%|#########3| 78/83 [02:30<00:09,  1.92s/it]

Evaluating policy:  95%|#########5| 79/83 [02:32<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:34<00:06,  2.11s/it]

Evaluating policy:  98%|#########7| 81/83 [02:36<00:04,  2.04s/it]

Evaluating policy:  99%|#########8| 82/83 [02:38<00:02,  2.01s/it]

Evaluating policy: 100%|##########| 83/83 [02:40<00:00,  1.91s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.78s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:24,  1.79s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.80s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:24,  1.83s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:19,  1.79s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:22,  1.85s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:20,  1.85s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:29,  2.00s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.94s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:18,  1.96s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.91s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:11,  1.96s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:10,  1.98s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.86s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:00,  1.94s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:51,  1.95s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.89s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:42,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.86s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:39,  1.92s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:35,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:31,  1.99s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:26,  1.93s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:22,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:21,  1.91s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.91s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:16,  1.97s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:14,  1.97s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.92s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:08,  1.89s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.84s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:03,  1.93s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.91s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.89s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.89s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:52,  2.00s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:48,  1.95s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:43,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:35,  1.97s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:32,  1.91s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:31,  1.96s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.92s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.92s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.83s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  2.00s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.99s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.93s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.90s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.86s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  2.06s/it]

Outer training loop:  73%|#######3  | 36501/50000 [23:52:18<371:44:55, 99.14s/it]
Outer training loop:  73%|#######3  | 36502/50000 [23:52:20<262:32:50, 70.02s/it]
Outer training loop:  73%|#######3  | 36503/50000 [23:52:22<185:50:37, 49.57s/it]
Outer training loop:  73%|#######3  | 36504/50000 [23:52:23<132:05:34, 35.24s/it]
Outer training loop:  73%|#######3  | 36505/50000 [23:52:25<94:39:05, 25.25s/it] 
Outer training loop:  73%|#######3  | 36506/50000 [23:52:27<68:21:26, 18.24s/it]
Outer training loop:  73%|#######3  | 36507/50000 [23:52:29<49:53:54, 13.31s/it]
Outer training loop:  73%|#######3  | 36508/50000 [23:52:31<37:05:53,  9.90s/it]
Outer training loop:  73%|#######3  | 36509/50000 [23:52:33<27:57:17,  7.46s/it]
Outer training loop:  73%|#######3  | 36510/50000 [23:52:35<21:46:38,  5.81s/it]
Outer training loop:  73%|#######3  | 36511/50000 [23:52:37<17:59:51,  4.80s/it]
Outer training loop:  73%|#######3  | 36512/50000 [23:52:39<14:46:56,  3.95s/it]
Outer training loop:  73%|#######3  | 36513/50000 [23:52:41<12:29:35,  3.33s/it]
Outer training loop:  73%|#######3  | 36514/50000 [23:52:43<10:47:45,  2.88s/it]
Outer training loop:  73%|#######3  | 36515/50000 [23:52:45<9:48:24,  2.62s/it] 
Outer training loop:  73%|#######3  | 36516/50000 [23:52:47<9:02:53,  2.42s/it]
Outer training loop:  73%|#######3  | 36517/50000 [23:52:49<8:24:00,  2.24s/it]
Outer training loop:  73%|#######3  | 36518/50000 [23:52:51<8:15:35,  2.21s/it]
Outer training loop:  73%|#######3  | 36519/50000 [23:52:53<7:47:35,  2.08s/it]
Outer training loop:  73%|#######3  | 36520/50000 [23:52:54<7:31:35,  2.01s/it]
Outer training loop:  73%|#######3  | 36521/50000 [23:52:56<7:36:45,  2.03s/it]
Outer training loop:  73%|#######3  | 36522/50000 [23:52:58<7:29:40,  2.00s/it]
Outer training loop:  73%|#######3  | 36523/50000 [23:53:00<7:25:26,  1.98s/it]
Outer training loop:  73%|#######3  | 36524/50000 [23:53:02<7:18:05,  1.95s/it]
Outer training loop:  73%|#######3  | 36525/50000 [23:53:04<7:20:04,  1.96s/it]
Outer training loop:  73%|#######3  | 36526/50000 [23:53:06<7:10:21,  1.92s/it]
Outer training loop:  73%|#######3  | 36527/50000 [23:53:08<6:51:25,  1.83s/it]
Outer training loop:  73%|#######3  | 36528/50000 [23:53:10<6:56:26,  1.85s/it]
Outer training loop:  73%|#######3  | 36529/50000 [23:53:12<7:08:10,  1.91s/it]
Outer training loop:  73%|#######3  | 36530/50000 [23:53:13<7:03:23,  1.89s/it]
Outer training loop:  73%|#######3  | 36531/50000 [23:53:15<6:56:32,  1.86s/it]
Outer training loop:  73%|#######3  | 36532/50000 [23:53:17<7:10:21,  1.92s/it]
Outer training loop:  73%|#######3  | 36533/50000 [23:53:19<7:06:49,  1.90s/it]
Outer training loop:  73%|#######3  | 36534/50000 [23:53:21<6:57:45,  1.86s/it]
Outer training loop:  73%|#######3  | 36535/50000 [23:53:23<7:05:57,  1.90s/it]
Outer training loop:  73%|#######3  | 36536/50000 [23:53:25<7:22:22,  1.97s/it]
Outer training loop:  73%|#######3  | 36537/50000 [23:53:27<7:07:10,  1.90s/it]
Outer training loop:  73%|#######3  | 36538/50000 [23:53:29<6:58:27,  1.87s/it]
Outer training loop:  73%|#######3  | 36539/50000 [23:53:31<7:07:49,  1.91s/it]
Outer training loop:  73%|#######3  | 36540/50000 [23:53:32<7:00:58,  1.88s/it]
Outer training loop:  73%|#######3  | 36541/50000 [23:53:34<7:00:19,  1.87s/it]
Outer training loop:  73%|#######3  | 36542/50000 [23:53:36<7:04:10,  1.89s/it]
Outer training loop:  73%|#######3  | 36543/50000 [23:53:38<6:59:44,  1.87s/it]
Outer training loop:  73%|#######3  | 36544/50000 [23:53:40<6:56:19,  1.86s/it]
Outer training loop:  73%|#######3  | 36545/50000 [23:53:42<6:52:43,  1.84s/it]
Outer training loop:  73%|#######3  | 36546/50000 [23:53:44<6:59:24,  1.87s/it]
Outer training loop:  73%|#######3  | 36547/50000 [23:53:45<7:02:24,  1.88s/it]
Outer training loop:  73%|#######3  | 36548/50000 [23:53:47<6:46:28,  1.81s/it]
Outer training loop:  73%|#######3  | 36549/50000 [23:53:49<7:03:20,  1.89s/it]
Outer training loop:  73%|#######3  | 36550/50000 [23:53:51<6:58:53,  1.87s/it]
Outer training loop:  73%|#######3  | 36551/50000 [23:53:53<6:56:40,  1.86s/it]
Outer training loop:  73%|#######3  | 36552/50000 [23:53:55<6:52:39,  1.84s/it]
Outer training loop:  73%|#######3  | 36553/50000 [23:53:57<7:00:00,  1.87s/it]
Outer training loop:  73%|#######3  | 36554/50000 [23:53:58<6:51:40,  1.84s/it]
Outer training loop:  73%|#######3  | 36555/50000 [23:54:00<6:50:34,  1.83s/it]
Outer training loop:  73%|#######3  | 36556/50000 [23:54:02<7:06:04,  1.90s/it]
Outer training loop:  73%|#######3  | 36557/50000 [23:54:04<7:05:00,  1.90s/it]
Outer training loop:  73%|#######3  | 36558/50000 [23:54:06<7:05:11,  1.90s/it]
Outer training loop:  73%|#######3  | 36559/50000 [23:54:08<6:53:37,  1.85s/it]
Outer training loop:  73%|#######3  | 36560/50000 [23:54:10<7:06:15,  1.90s/it]
Outer training loop:  73%|#######3  | 36561/50000 [23:54:12<7:04:55,  1.90s/it]
Outer training loop:  73%|#######3  | 36562/50000 [23:54:14<7:09:58,  1.92s/it]
Outer training loop:  73%|#######3  | 36563/50000 [23:54:16<7:15:01,  1.94s/it]
Outer training loop:  73%|#######3  | 36564/50000 [23:54:17<7:01:59,  1.88s/it]
Outer training loop:  73%|#######3  | 36565/50000 [23:54:19<6:59:30,  1.87s/it]
Outer training loop:  73%|#######3  | 36566/50000 [23:54:21<6:58:47,  1.87s/it]
Outer training loop:  73%|#######3  | 36567/50000 [23:54:23<7:05:36,  1.90s/it]
Outer training loop:  73%|#######3  | 36568/50000 [23:54:25<7:02:57,  1.89s/it]
Outer training loop:  73%|#######3  | 36569/50000 [23:54:27<7:08:24,  1.91s/it]
Outer training loop:  73%|#######3  | 36570/50000 [23:54:29<7:15:21,  1.94s/it]
Outer training loop:  73%|#######3  | 36571/50000 [23:54:31<7:06:53,  1.91s/it]
Outer training loop:  73%|#######3  | 36572/50000 [23:54:32<6:57:07,  1.86s/it]
Outer training loop:  73%|#######3  | 36573/50000 [23:54:35<7:08:43,  1.92s/it]
Outer training loop:  73%|#######3  | 36574/50000 [23:54:36<7:05:08,  1.90s/it]
Outer training loop:  73%|#######3  | 36575/50000 [23:54:38<6:56:18,  1.86s/it]
Outer training loop:  73%|#######3  | 36576/50000 [23:54:40<7:05:40,  1.90s/it]
Outer training loop:  73%|#######3  | 36577/50000 [23:54:42<6:58:42,  1.87s/it]
Outer training loop:  73%|#######3  | 36578/50000 [23:54:44<6:49:06,  1.83s/it]
Outer training loop:  73%|#######3  | 36579/50000 [23:54:45<6:39:36,  1.79s/it]
Outer training loop:  73%|#######3  | 36580/50000 [23:54:47<6:51:39,  1.84s/it]
Outer training loop:  73%|#######3  | 36581/50000 [23:54:49<6:49:39,  1.83s/it]
Outer training loop:  73%|#######3  | 36582/50000 [23:54:51<6:43:40,  1.81s/it]
Outer training loop:  73%|#######3  | 36583/50000 [23:54:53<6:59:22,  1.88s/it]
Outer training loop:  73%|#######3  | 36584/50000 [23:54:55<6:52:14,  1.84s/it]
Outer training loop:  73%|#######3  | 36585/50000 [23:54:57<7:09:28,  1.92s/it]
Outer training loop:  73%|#######3  | 36586/50000 [23:54:59<7:17:22,  1.96s/it]
Outer training loop:  73%|#######3  | 36587/50000 [23:55:01<6:59:43,  1.88s/it]
Outer training loop:  73%|#######3  | 36588/50000 [23:55:02<6:49:55,  1.83s/it]
Outer training loop:  73%|#######3  | 36589/50000 [23:55:04<7:12:05,  1.93s/it]
Outer training loop:  73%|#######3  | 36590/50000 [23:55:06<7:13:52,  1.94s/it]
Outer training loop:  73%|#######3  | 36591/50000 [23:55:08<7:16:08,  1.95s/it]
Outer training loop:  73%|#######3  | 36592/50000 [23:55:10<7:04:01,  1.90s/it]
Outer training loop:  73%|#######3  | 36593/50000 [23:55:12<7:13:06,  1.94s/it]
Outer training loop:  73%|#######3  | 36594/50000 [23:55:14<7:01:52,  1.89s/it]
Outer training loop:  73%|#######3  | 36595/50000 [23:55:16<6:57:48,  1.87s/it]
Outer training loop:  73%|#######3  | 36596/50000 [23:55:18<7:07:48,  1.92s/it]
Outer training loop:  73%|#######3  | 36597/50000 [23:55:20<7:07:55,  1.92s/it]
Outer training loop:  73%|#######3  | 36598/50000 [23:55:22<7:09:29,  1.92s/it]
Outer training loop:  73%|#######3  | 36599/50000 [23:55:24<7:12:50,  1.94s/it]
Outer training loop:  73%|#######3  | 36600/50000 [23:55:25<7:02:27,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  73%|#######3  | 36601/50000 [23:55:33<13:10:11,  3.54s/it]
Outer training loop:  73%|#######3  | 36602/50000 [23:55:35<11:14:40,  3.02s/it]
Outer training loop:  73%|#######3  | 36603/50000 [23:55:37<10:19:57,  2.78s/it]
Outer training loop:  73%|#######3  | 36604/50000 [23:55:39<9:18:30,  2.50s/it] 
Outer training loop:  73%|#######3  | 36605/50000 [23:55:41<8:35:46,  2.31s/it]
Outer training loop:  73%|#######3  | 36606/50000 [23:55:42<8:09:06,  2.19s/it]
Outer training loop:  73%|#######3  | 36607/50000 [23:55:45<8:03:42,  2.17s/it]
Outer training loop:  73%|#######3  | 36608/50000 [23:55:46<7:39:50,  2.06s/it]
Outer training loop:  73%|#######3  | 36609/50000 [23:55:48<7:24:52,  1.99s/it]
Outer training loop:  73%|#######3  | 36610/50000 [23:55:50<7:31:47,  2.02s/it]
Outer training loop:  73%|#######3  | 36611/50000 [23:55:52<7:19:15,  1.97s/it]
Outer training loop:  73%|#######3  | 36612/50000 [23:55:54<7:03:13,  1.90s/it]
Outer training loop:  73%|#######3  | 36613/50000 [23:55:56<7:13:50,  1.94s/it]
Outer training loop:  73%|#######3  | 36614/50000 [23:55:58<7:11:20,  1.93s/it]
Outer training loop:  73%|#######3  | 36615/50000 [23:56:00<7:05:27,  1.91s/it]
Outer training loop:  73%|#######3  | 36616/50000 [23:56:02<7:11:50,  1.94s/it]
Outer training loop:  73%|#######3  | 36617/50000 [23:56:03<6:57:30,  1.87s/it]
Outer training loop:  73%|#######3  | 36618/50000 [23:56:05<7:04:17,  1.90s/it]
Outer training loop:  73%|#######3  | 36619/50000 [23:56:07<6:56:59,  1.87s/it]
Outer training loop:  73%|#######3  | 36620/50000 [23:56:09<7:14:32,  1.95s/it]
Outer training loop:  73%|#######3  | 36621/50000 [23:56:11<7:07:11,  1.92s/it]
Outer training loop:  73%|#######3  | 36622/50000 [23:56:13<7:02:07,  1.89s/it]
Outer training loop:  73%|#######3  | 36623/50000 [23:56:15<7:14:28,  1.95s/it]
Outer training loop:  73%|#######3  | 36624/50000 [23:56:17<6:56:42,  1.87s/it]
Outer training loop:  73%|#######3  | 36625/50000 [23:56:19<6:50:52,  1.84s/it]
Outer training loop:  73%|#######3  | 36626/50000 [23:56:20<6:47:54,  1.83s/it]
Outer training loop:  73%|#######3  | 36627/50000 [23:56:22<6:56:42,  1.87s/it]
Outer training loop:  73%|#######3  | 36628/50000 [23:56:24<6:49:32,  1.84s/it]
Outer training loop:  73%|#######3  | 36629/50000 [23:56:26<6:41:45,  1.80s/it]
Outer training loop:  73%|#######3  | 36630/50000 [23:56:28<6:59:57,  1.88s/it]
Outer training loop:  73%|#######3  | 36631/50000 [23:56:30<7:13:02,  1.94s/it]
Outer training loop:  73%|#######3  | 36632/50000 [23:56:32<7:20:29,  1.98s/it]
Outer training loop:  73%|#######3  | 36633/50000 [23:56:34<7:06:52,  1.92s/it]
Outer training loop:  73%|#######3  | 36634/50000 [23:56:36<7:11:45,  1.94s/it]
Outer training loop:  73%|#######3  | 36635/50000 [23:56:38<6:58:37,  1.88s/it]
Outer training loop:  73%|#######3  | 36636/50000 [23:56:39<6:57:20,  1.87s/it]
Outer training loop:  73%|#######3  | 36637/50000 [23:56:41<7:03:33,  1.90s/it]
Outer training loop:  73%|#######3  | 36638/50000 [23:56:43<7:05:18,  1.91s/it]
Outer training loop:  73%|#######3  | 36639/50000 [23:56:45<6:51:25,  1.85s/it]
Outer training loop:  73%|#######3  | 36640/50000 [23:56:47<6:39:48,  1.80s/it]
Outer training loop:  73%|#######3  | 36641/50000 [23:56:49<6:54:23,  1.86s/it]
Outer training loop:  73%|#######3  | 36642/50000 [23:56:51<7:02:34,  1.90s/it]
Outer training loop:  73%|#######3  | 36643/50000 [23:56:52<6:58:05,  1.88s/it]
Outer training loop:  73%|#######3  | 36644/50000 [23:56:55<7:12:02,  1.94s/it]
Outer training loop:  73%|#######3  | 36645/50000 [23:56:56<7:11:35,  1.94s/it]
Outer training loop:  73%|#######3  | 36646/50000 [23:56:58<7:11:24,  1.94s/it]
Outer training loop:  73%|#######3  | 36647/50000 [23:57:01<7:25:14,  2.00s/it]
Outer training loop:  73%|#######3  | 36648/50000 [23:57:03<7:22:19,  1.99s/it]
Outer training loop:  73%|#######3  | 36649/50000 [23:57:04<7:14:13,  1.95s/it]
Outer training loop:  73%|#######3  | 36650/50000 [23:57:06<7:22:09,  1.99s/it]
Outer training loop:  73%|#######3  | 36651/50000 [23:57:08<7:17:08,  1.96s/it]
Outer training loop:  73%|#######3  | 36652/50000 [23:57:10<7:19:06,  1.97s/it]
Outer training loop:  73%|#######3  | 36653/50000 [23:57:12<7:13:16,  1.95s/it]
Outer training loop:  73%|#######3  | 36654/50000 [23:57:14<7:28:13,  2.02s/it]
Outer training loop:  73%|#######3  | 36655/50000 [23:57:16<7:27:45,  2.01s/it]
Outer training loop:  73%|#######3  | 36656/50000 [23:57:18<7:14:32,  1.95s/it]
Outer training loop:  73%|#######3  | 36657/50000 [23:57:20<7:21:24,  1.98s/it]
Outer training loop:  73%|#######3  | 36658/50000 [23:57:22<7:11:09,  1.94s/it]
Outer training loop:  73%|#######3  | 36659/50000 [23:57:24<7:03:35,  1.91s/it]
Outer training loop:  73%|#######3  | 36660/50000 [23:57:26<7:16:29,  1.96s/it]
Outer training loop:  73%|#######3  | 36661/50000 [23:57:28<7:07:53,  1.92s/it]
Outer training loop:  73%|#######3  | 36662/50000 [23:57:30<7:06:04,  1.92s/it]
Outer training loop:  73%|#######3  | 36663/50000 [23:57:32<7:19:34,  1.98s/it]
Outer training loop:  73%|#######3  | 36664/50000 [23:57:34<7:11:55,  1.94s/it]
Outer training loop:  73%|#######3  | 36665/50000 [23:57:36<7:02:18,  1.90s/it]
Outer training loop:  73%|#######3  | 36666/50000 [23:57:37<6:57:17,  1.88s/it]
Outer training loop:  73%|#######3  | 36667/50000 [23:57:39<7:05:21,  1.91s/it]
Outer training loop:  73%|#######3  | 36668/50000 [23:57:41<7:01:00,  1.89s/it]
Outer training loop:  73%|#######3  | 36669/50000 [23:57:43<6:52:52,  1.86s/it]
Outer training loop:  73%|#######3  | 36670/50000 [23:57:45<6:53:36,  1.86s/it]
Outer training loop:  73%|#######3  | 36671/50000 [23:57:47<6:47:15,  1.83s/it]
Outer training loop:  73%|#######3  | 36672/50000 [23:57:49<6:50:27,  1.85s/it]
Outer training loop:  73%|#######3  | 36673/50000 [23:57:50<6:55:22,  1.87s/it]
Outer training loop:  73%|#######3  | 36674/50000 [23:57:52<6:54:33,  1.87s/it]
Outer training loop:  73%|#######3  | 36675/50000 [23:57:54<6:46:04,  1.83s/it]
Outer training loop:  73%|#######3  | 36676/50000 [23:57:56<6:43:45,  1.82s/it]
Outer training loop:  73%|#######3  | 36677/50000 [23:57:58<6:41:31,  1.81s/it]
Outer training loop:  73%|#######3  | 36678/50000 [23:58:00<6:56:37,  1.88s/it]
Outer training loop:  73%|#######3  | 36679/50000 [23:58:01<6:44:40,  1.82s/it]
Outer training loop:  73%|#######3  | 36680/50000 [23:58:03<6:43:08,  1.82s/it]
Outer training loop:  73%|#######3  | 36681/50000 [23:58:05<7:05:25,  1.92s/it]
Outer training loop:  73%|#######3  | 36682/50000 [23:58:07<7:07:02,  1.92s/it]
Outer training loop:  73%|#######3  | 36683/50000 [23:58:09<6:57:25,  1.88s/it]
Outer training loop:  73%|#######3  | 36684/50000 [23:58:11<7:03:01,  1.91s/it]
Outer training loop:  73%|#######3  | 36685/50000 [23:58:13<7:04:10,  1.91s/it]
Outer training loop:  73%|#######3  | 36686/50000 [23:58:15<6:55:56,  1.87s/it]
Outer training loop:  73%|#######3  | 36687/50000 [23:58:17<7:09:48,  1.94s/it]
Outer training loop:  73%|#######3  | 36688/50000 [23:58:19<7:11:37,  1.95s/it]
Outer training loop:  73%|#######3  | 36689/50000 [23:58:21<7:08:47,  1.93s/it]
Outer training loop:  73%|#######3  | 36690/50000 [23:58:22<7:00:14,  1.89s/it]
Outer training loop:  73%|#######3  | 36691/50000 [23:58:25<7:11:08,  1.94s/it]
Outer training loop:  73%|#######3  | 36692/50000 [23:58:26<7:01:09,  1.90s/it]
Outer training loop:  73%|#######3  | 36693/50000 [23:58:28<6:53:11,  1.86s/it]
Outer training loop:  73%|#######3  | 36694/50000 [23:58:30<6:47:14,  1.84s/it]
Outer training loop:  73%|#######3  | 36695/50000 [23:58:32<6:55:17,  1.87s/it]
Outer training loop:  73%|#######3  | 36696/50000 [23:58:34<6:53:21,  1.86s/it]
Outer training loop:  73%|#######3  | 36697/50000 [23:58:36<7:01:23,  1.90s/it]
Outer training loop:  73%|#######3  | 36698/50000 [23:58:38<7:08:09,  1.93s/it]
Outer training loop:  73%|#######3  | 36699/50000 [23:58:39<6:57:13,  1.88s/it]
Outer training loop:  73%|#######3  | 36700/50000 [23:58:41<6:45:43,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  73%|#######3  | 36701/50000 [23:58:49<12:57:38,  3.51s/it]
Outer training loop:  73%|#######3  | 36702/50000 [23:58:51<11:23:21,  3.08s/it]
Outer training loop:  73%|#######3  | 36703/50000 [23:58:52<9:53:36,  2.68s/it] 
Outer training loop:  73%|#######3  | 36704/50000 [23:58:54<8:59:58,  2.44s/it]
Outer training loop:  73%|#######3  | 36705/50000 [23:58:56<8:35:48,  2.33s/it]
Outer training loop:  73%|#######3  | 36706/50000 [23:58:58<8:04:30,  2.19s/it]
Outer training loop:  73%|#######3  | 36707/50000 [23:59:00<7:47:52,  2.11s/it]
Outer training loop:  73%|#######3  | 36708/50000 [23:59:02<7:27:06,  2.02s/it]
Outer training loop:  73%|#######3  | 36709/50000 [23:59:04<7:18:05,  1.98s/it]
Outer training loop:  73%|#######3  | 36710/50000 [23:59:06<7:04:21,  1.92s/it]
Outer training loop:  73%|#######3  | 36711/50000 [23:59:07<7:00:08,  1.90s/it]
Outer training loop:  73%|#######3  | 36712/50000 [23:59:09<7:00:52,  1.90s/it]
Outer training loop:  73%|#######3  | 36713/50000 [23:59:11<6:53:56,  1.87s/it]
Outer training loop:  73%|#######3  | 36714/50000 [23:59:13<6:54:32,  1.87s/it]
Outer training loop:  73%|#######3  | 36715/50000 [23:59:15<6:55:07,  1.87s/it]
Outer training loop:  73%|#######3  | 36716/50000 [23:59:17<6:43:24,  1.82s/it]
Outer training loop:  73%|#######3  | 36717/50000 [23:59:18<6:46:14,  1.84s/it]
Outer training loop:  73%|#######3  | 36718/50000 [23:59:20<6:48:23,  1.84s/it]
Outer training loop:  73%|#######3  | 36719/50000 [23:59:22<7:07:16,  1.93s/it]
Outer training loop:  73%|#######3  | 36720/50000 [23:59:24<7:06:46,  1.93s/it]
Outer training loop:  73%|#######3  | 36721/50000 [23:59:26<7:03:55,  1.92s/it]
Outer training loop:  73%|#######3  | 36722/50000 [23:59:28<7:12:32,  1.95s/it]
Outer training loop:  73%|#######3  | 36723/50000 [23:59:30<7:03:08,  1.91s/it]
Outer training loop:  73%|#######3  | 36724/50000 [23:59:32<6:50:27,  1.86s/it]
Outer training loop:  73%|#######3  | 36725/50000 [23:59:34<6:42:37,  1.82s/it]
Outer training loop:  73%|#######3  | 36726/50000 [23:59:36<7:03:13,  1.91s/it]
Outer training loop:  73%|#######3  | 36727/50000 [23:59:38<6:56:50,  1.88s/it]
Outer training loop:  73%|#######3  | 36728/50000 [23:59:39<6:51:45,  1.86s/it]
Outer training loop:  73%|#######3  | 36729/50000 [23:59:42<7:10:33,  1.95s/it]
Outer training loop:  73%|#######3  | 36730/50000 [23:59:43<7:02:33,  1.91s/it]
Outer training loop:  73%|#######3  | 36731/50000 [23:59:45<6:55:28,  1.88s/it]
Outer training loop:  73%|#######3  | 36732/50000 [23:59:47<6:51:04,  1.86s/it]
Outer training loop:  73%|#######3  | 36733/50000 [23:59:49<6:59:32,  1.90s/it]
Outer training loop:  73%|#######3  | 36734/50000 [23:59:51<6:53:38,  1.87s/it]
Outer training loop:  73%|#######3  | 36735/50000 [23:59:53<6:57:22,  1.89s/it]
Outer training loop:  73%|#######3  | 36736/50000 [23:59:54<6:49:43,  1.85s/it]
Outer training loop:  73%|#######3  | 36737/50000 [23:59:57<7:08:46,  1.94s/it]
Outer training loop:  73%|#######3  | 36738/50000 [23:59:58<7:00:57,  1.90s/it]
Outer training loop:  73%|#######3  | 36739/50000 [24:00:00<6:56:47,  1.89s/it]
Outer training loop:  73%|#######3  | 36740/50000 [24:00:02<7:07:51,  1.94s/it]
Outer training loop:  73%|#######3  | 36741/50000 [24:00:04<7:00:31,  1.90s/it]
Outer training loop:  73%|#######3  | 36742/50000 [24:00:06<7:03:31,  1.92s/it]
Outer training loop:  73%|#######3  | 36743/50000 [24:00:08<7:07:23,  1.93s/it]
Outer training loop:  73%|#######3  | 36744/50000 [24:00:10<7:02:34,  1.91s/it]
Outer training loop:  73%|#######3  | 36745/50000 [24:00:12<6:59:27,  1.90s/it]
Outer training loop:  73%|#######3  | 36746/50000 [24:00:14<7:06:56,  1.93s/it]
Outer training loop:  73%|#######3  | 36747/50000 [24:00:16<7:01:57,  1.91s/it]
Outer training loop:  73%|#######3  | 36748/50000 [24:00:17<6:53:28,  1.87s/it]
Outer training loop:  73%|#######3  | 36749/50000 [24:00:19<6:51:33,  1.86s/it]
Outer training loop:  74%|#######3  | 36750/50000 [24:00:21<7:01:55,  1.91s/it]
Outer training loop:  74%|#######3  | 36751/50000 [24:00:23<6:49:43,  1.86s/it]
Outer training loop:  74%|#######3  | 36752/50000 [24:00:25<6:53:23,  1.87s/it]
Outer training loop:  74%|#######3  | 36753/50000 [24:00:27<6:58:22,  1.89s/it]
Outer training loop:  74%|#######3  | 36754/50000 [24:00:29<6:53:17,  1.87s/it]
Outer training loop:  74%|#######3  | 36755/50000 [24:00:31<6:48:13,  1.85s/it]
Outer training loop:  74%|#######3  | 36756/50000 [24:00:33<7:00:17,  1.90s/it]
Outer training loop:  74%|#######3  | 36757/50000 [24:00:35<7:08:04,  1.94s/it]
Outer training loop:  74%|#######3  | 36758/50000 [24:00:36<7:03:41,  1.92s/it]
Outer training loop:  74%|#######3  | 36759/50000 [24:00:38<7:08:57,  1.94s/it]
Outer training loop:  74%|#######3  | 36760/50000 [24:00:40<6:54:58,  1.88s/it]
Outer training loop:  74%|#######3  | 36761/50000 [24:00:42<6:44:47,  1.83s/it]
Outer training loop:  74%|#######3  | 36762/50000 [24:00:44<6:51:08,  1.86s/it]
Outer training loop:  74%|#######3  | 36763/50000 [24:00:46<6:55:05,  1.88s/it]
Outer training loop:  74%|#######3  | 36764/50000 [24:00:48<6:49:34,  1.86s/it]
Outer training loop:  74%|#######3  | 36765/50000 [24:00:49<6:48:09,  1.85s/it]
Outer training loop:  74%|#######3  | 36766/50000 [24:00:51<7:03:33,  1.92s/it]
Outer training loop:  74%|#######3  | 36767/50000 [24:00:53<6:50:56,  1.86s/it]
Outer training loop:  74%|#######3  | 36768/50000 [24:00:55<6:49:23,  1.86s/it]
Outer training loop:  74%|#######3  | 36769/50000 [24:00:57<6:45:26,  1.84s/it]
Outer training loop:  74%|#######3  | 36770/50000 [24:00:59<6:54:06,  1.88s/it]
Outer training loop:  74%|#######3  | 36771/50000 [24:01:01<6:48:33,  1.85s/it]
Outer training loop:  74%|#######3  | 36772/50000 [24:01:02<6:45:10,  1.84s/it]
Outer training loop:  74%|#######3  | 36773/50000 [24:01:04<6:48:31,  1.85s/it]
Outer training loop:  74%|#######3  | 36774/50000 [24:01:06<6:53:27,  1.88s/it]
Outer training loop:  74%|#######3  | 36775/50000 [24:01:08<6:52:30,  1.87s/it]
Outer training loop:  74%|#######3  | 36776/50000 [24:01:10<6:51:07,  1.87s/it]
Outer training loop:  74%|#######3  | 36777/50000 [24:01:12<7:02:14,  1.92s/it]
Outer training loop:  74%|#######3  | 36778/50000 [24:01:14<6:51:59,  1.87s/it]
Outer training loop:  74%|#######3  | 36779/50000 [24:01:16<6:59:22,  1.90s/it]
Outer training loop:  74%|#######3  | 36780/50000 [24:01:18<7:04:58,  1.93s/it]
Outer training loop:  74%|#######3  | 36781/50000 [24:01:20<7:04:20,  1.93s/it]
Outer training loop:  74%|#######3  | 36782/50000 [24:01:21<6:56:49,  1.89s/it]
Outer training loop:  74%|#######3  | 36783/50000 [24:01:23<7:04:44,  1.93s/it]
Outer training loop:  74%|#######3  | 36784/50000 [24:01:25<7:08:35,  1.95s/it]
Outer training loop:  74%|#######3  | 36785/50000 [24:01:27<7:02:42,  1.92s/it]
Outer training loop:  74%|#######3  | 36786/50000 [24:01:29<7:18:54,  1.99s/it]
Outer training loop:  74%|#######3  | 36787/50000 [24:01:31<7:18:06,  1.99s/it]
Outer training loop:  74%|#######3  | 36788/50000 [24:01:33<7:06:29,  1.94s/it]
Outer training loop:  74%|#######3  | 36789/50000 [24:01:35<7:00:34,  1.91s/it]
Outer training loop:  74%|#######3  | 36790/50000 [24:01:37<7:08:36,  1.95s/it]
Outer training loop:  74%|#######3  | 36791/50000 [24:01:39<7:06:12,  1.94s/it]
Outer training loop:  74%|#######3  | 36792/50000 [24:01:41<7:04:55,  1.93s/it]
Outer training loop:  74%|#######3  | 36793/50000 [24:01:43<7:17:16,  1.99s/it]
Outer training loop:  74%|#######3  | 36794/50000 [24:01:45<7:02:02,  1.92s/it]
Outer training loop:  74%|#######3  | 36795/50000 [24:01:47<6:51:18,  1.87s/it]
Outer training loop:  74%|#######3  | 36796/50000 [24:01:48<6:51:44,  1.87s/it]
Outer training loop:  74%|#######3  | 36797/50000 [24:01:50<6:54:14,  1.88s/it]
Outer training loop:  74%|#######3  | 36798/50000 [24:01:52<6:48:56,  1.86s/it]
Outer training loop:  74%|#######3  | 36799/50000 [24:01:54<6:40:29,  1.82s/it]
Outer training loop:  74%|#######3  | 36800/50000 [24:01:56<6:53:31,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  74%|#######3  | 36801/50000 [24:02:03<12:59:09,  3.54s/it]
Outer training loop:  74%|#######3  | 36802/50000 [24:02:05<11:08:35,  3.04s/it]
Outer training loop:  74%|#######3  | 36803/50000 [24:02:07<9:44:01,  2.66s/it] 
Outer training loop:  74%|#######3  | 36804/50000 [24:02:09<9:05:38,  2.48s/it]
Outer training loop:  74%|#######3  | 36805/50000 [24:02:11<8:23:58,  2.29s/it]
Outer training loop:  74%|#######3  | 36806/50000 [24:02:13<7:53:39,  2.15s/it]
Outer training loop:  74%|#######3  | 36807/50000 [24:02:15<7:40:27,  2.09s/it]
Outer training loop:  74%|#######3  | 36808/50000 [24:02:16<7:19:39,  2.00s/it]
Outer training loop:  74%|#######3  | 36809/50000 [24:02:18<7:11:25,  1.96s/it]
Outer training loop:  74%|#######3  | 36810/50000 [24:02:20<7:23:30,  2.02s/it]
Outer training loop:  74%|#######3  | 36811/50000 [24:02:22<7:09:14,  1.95s/it]
Outer training loop:  74%|#######3  | 36812/50000 [24:02:24<6:59:54,  1.91s/it]
Outer training loop:  74%|#######3  | 36813/50000 [24:02:26<6:54:06,  1.88s/it]
Outer training loop:  74%|#######3  | 36814/50000 [24:02:28<7:00:29,  1.91s/it]
Outer training loop:  74%|#######3  | 36815/50000 [24:02:30<6:54:50,  1.89s/it]
Outer training loop:  74%|#######3  | 36816/50000 [24:02:32<6:48:47,  1.86s/it]
Outer training loop:  74%|#######3  | 36817/50000 [24:02:34<6:57:06,  1.90s/it]
Outer training loop:  74%|#######3  | 36818/50000 [24:02:35<6:55:32,  1.89s/it]
Outer training loop:  74%|#######3  | 36819/50000 [24:02:37<6:51:17,  1.87s/it]
Outer training loop:  74%|#######3  | 36820/50000 [24:02:39<6:48:50,  1.86s/it]
Outer training loop:  74%|#######3  | 36821/50000 [24:02:41<6:58:19,  1.90s/it]
Outer training loop:  74%|#######3  | 36822/50000 [24:02:43<6:50:29,  1.87s/it]
Outer training loop:  74%|#######3  | 36823/50000 [24:02:45<6:41:16,  1.83s/it]
Outer training loop:  74%|#######3  | 36824/50000 [24:02:47<6:51:32,  1.87s/it]
Outer training loop:  74%|#######3  | 36825/50000 [24:02:48<6:47:18,  1.85s/it]
Outer training loop:  74%|#######3  | 36826/50000 [24:02:50<6:51:55,  1.88s/it]
Outer training loop:  74%|#######3  | 36827/50000 [24:02:52<6:42:26,  1.83s/it]
Outer training loop:  74%|#######3  | 36828/50000 [24:02:54<6:45:03,  1.85s/it]
Outer training loop:  74%|#######3  | 36829/50000 [24:02:56<6:45:12,  1.85s/it]
Outer training loop:  74%|#######3  | 36830/50000 [24:02:58<6:43:17,  1.84s/it]
Outer training loop:  74%|#######3  | 36831/50000 [24:02:59<6:46:11,  1.85s/it]
Outer training loop:  74%|#######3  | 36832/50000 [24:03:01<6:48:49,  1.86s/it]
Outer training loop:  74%|#######3  | 36833/50000 [24:03:03<6:48:01,  1.86s/it]
Outer training loop:  74%|#######3  | 36834/50000 [24:03:05<6:40:37,  1.83s/it]
Outer training loop:  74%|#######3  | 36835/50000 [24:03:07<6:57:16,  1.90s/it]
Outer training loop:  74%|#######3  | 36836/50000 [24:03:09<6:52:56,  1.88s/it]
Outer training loop:  74%|#######3  | 36837/50000 [24:03:11<6:48:31,  1.86s/it]
Outer training loop:  74%|#######3  | 36838/50000 [24:03:13<7:05:28,  1.94s/it]
Outer training loop:  74%|#######3  | 36839/50000 [24:03:15<6:55:21,  1.89s/it]
Outer training loop:  74%|#######3  | 36840/50000 [24:03:17<6:57:59,  1.91s/it]
Outer training loop:  74%|#######3  | 36841/50000 [24:03:18<6:46:18,  1.85s/it]
Outer training loop:  74%|#######3  | 36842/50000 [24:03:20<6:58:35,  1.91s/it]
Outer training loop:  74%|#######3  | 36843/50000 [24:03:22<6:54:20,  1.89s/it]
Outer training loop:  74%|#######3  | 36844/50000 [24:03:24<6:55:52,  1.90s/it]
Outer training loop:  74%|#######3  | 36845/50000 [24:03:26<7:06:51,  1.95s/it]
Outer training loop:  74%|#######3  | 36846/50000 [24:03:28<6:52:29,  1.88s/it]
Outer training loop:  74%|#######3  | 36847/50000 [24:03:30<6:46:04,  1.85s/it]
Outer training loop:  74%|#######3  | 36848/50000 [24:03:31<6:45:44,  1.85s/it]
Outer training loop:  74%|#######3  | 36849/50000 [24:03:33<6:57:00,  1.90s/it]
Outer training loop:  74%|#######3  | 36850/50000 [24:03:35<6:59:37,  1.91s/it]
Outer training loop:  74%|#######3  | 36851/50000 [24:03:37<6:54:08,  1.89s/it]
Outer training loop:  74%|#######3  | 36852/50000 [24:03:39<7:00:23,  1.92s/it]
Outer training loop:  74%|#######3  | 36853/50000 [24:03:41<6:56:28,  1.90s/it]
Outer training loop:  74%|#######3  | 36854/50000 [24:03:43<6:52:06,  1.88s/it]
Outer training loop:  74%|#######3  | 36855/50000 [24:03:45<6:42:39,  1.84s/it]
Outer training loop:  74%|#######3  | 36856/50000 [24:03:47<6:50:18,  1.87s/it]
Outer training loop:  74%|#######3  | 36857/50000 [24:03:48<6:46:05,  1.85s/it]
Outer training loop:  74%|#######3  | 36858/50000 [24:03:50<6:42:44,  1.84s/it]
Outer training loop:  74%|#######3  | 36859/50000 [24:03:52<6:46:36,  1.86s/it]
Outer training loop:  74%|#######3  | 36860/50000 [24:03:54<6:37:13,  1.81s/it]
Outer training loop:  74%|#######3  | 36861/50000 [24:03:56<6:38:14,  1.82s/it]
Outer training loop:  74%|#######3  | 36862/50000 [24:03:57<6:32:43,  1.79s/it]
Outer training loop:  74%|#######3  | 36863/50000 [24:03:59<6:45:20,  1.85s/it]
Outer training loop:  74%|#######3  | 36864/50000 [24:04:01<6:42:09,  1.84s/it]
Outer training loop:  74%|#######3  | 36865/50000 [24:04:03<6:40:52,  1.83s/it]
Outer training loop:  74%|#######3  | 36866/50000 [24:04:05<6:53:26,  1.89s/it]
Outer training loop:  74%|#######3  | 36867/50000 [24:04:07<6:40:06,  1.83s/it]
Outer training loop:  74%|#######3  | 36868/50000 [24:04:08<6:31:38,  1.79s/it]
Outer training loop:  74%|#######3  | 36869/50000 [24:04:10<6:47:31,  1.86s/it]
Outer training loop:  74%|#######3  | 36870/50000 [24:04:12<6:41:26,  1.83s/it]
Outer training loop:  74%|#######3  | 36871/50000 [24:04:14<6:35:02,  1.81s/it]
Outer training loop:  74%|#######3  | 36872/50000 [24:04:16<6:45:08,  1.85s/it]
Outer training loop:  74%|#######3  | 36873/50000 [24:04:18<6:35:03,  1.81s/it]
Outer training loop:  74%|#######3  | 36874/50000 [24:04:19<6:30:45,  1.79s/it]
Outer training loop:  74%|#######3  | 36875/50000 [24:04:21<6:49:42,  1.87s/it]
Outer training loop:  74%|#######3  | 36876/50000 [24:04:23<6:55:37,  1.90s/it]
Outer training loop:  74%|#######3  | 36877/50000 [24:04:25<6:54:09,  1.89s/it]
Outer training loop:  74%|#######3  | 36878/50000 [24:04:27<7:00:32,  1.92s/it]
Outer training loop:  74%|#######3  | 36879/50000 [24:04:29<6:48:57,  1.87s/it]
Outer training loop:  74%|#######3  | 36880/50000 [24:04:31<6:54:14,  1.89s/it]
Outer training loop:  74%|#######3  | 36881/50000 [24:04:33<6:53:48,  1.89s/it]
Outer training loop:  74%|#######3  | 36882/50000 [24:04:35<7:18:54,  2.01s/it]
Outer training loop:  74%|#######3  | 36883/50000 [24:04:37<7:05:02,  1.94s/it]
Outer training loop:  74%|#######3  | 36884/50000 [24:04:39<7:01:40,  1.93s/it]
Outer training loop:  74%|#######3  | 36885/50000 [24:04:41<7:19:20,  2.01s/it]
Outer training loop:  74%|#######3  | 36886/50000 [24:04:43<7:12:50,  1.98s/it]
Outer training loop:  74%|#######3  | 36887/50000 [24:04:45<7:06:51,  1.95s/it]
Outer training loop:  74%|#######3  | 36888/50000 [24:04:47<7:02:20,  1.93s/it]
Outer training loop:  74%|#######3  | 36889/50000 [24:04:49<7:08:49,  1.96s/it]
Outer training loop:  74%|#######3  | 36890/50000 [24:04:50<6:54:28,  1.90s/it]
Outer training loop:  74%|#######3  | 36891/50000 [24:04:52<6:43:36,  1.85s/it]
Outer training loop:  74%|#######3  | 36892/50000 [24:04:54<6:58:10,  1.91s/it]
Outer training loop:  74%|#######3  | 36893/50000 [24:04:56<6:49:22,  1.87s/it]
Outer training loop:  74%|#######3  | 36894/50000 [24:04:58<6:53:55,  1.89s/it]
Outer training loop:  74%|#######3  | 36895/50000 [24:05:00<6:46:25,  1.86s/it]
Outer training loop:  74%|#######3  | 36896/50000 [24:05:02<7:02:35,  1.93s/it]
Outer training loop:  74%|#######3  | 36897/50000 [24:05:04<6:55:49,  1.90s/it]
Outer training loop:  74%|#######3  | 36898/50000 [24:05:06<6:57:52,  1.91s/it]
Outer training loop:  74%|#######3  | 36899/50000 [24:05:08<7:13:40,  1.99s/it]
Outer training loop:  74%|#######3  | 36900/50000 [24:05:10<7:02:55,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  74%|#######3  | 36901/50000 [24:05:17<12:53:40,  3.54s/it]
Outer training loop:  74%|#######3  | 36902/50000 [24:05:19<11:04:22,  3.04s/it]
Outer training loop:  74%|#######3  | 36903/50000 [24:05:21<9:42:48,  2.67s/it] 
Outer training loop:  74%|#######3  | 36904/50000 [24:05:23<8:58:49,  2.47s/it]
Outer training loop:  74%|#######3  | 36905/50000 [24:05:25<8:21:19,  2.30s/it]
Outer training loop:  74%|#######3  | 36906/50000 [24:05:26<7:49:58,  2.15s/it]
Outer training loop:  74%|#######3  | 36907/50000 [24:05:28<7:28:13,  2.05s/it]
Outer training loop:  74%|#######3  | 36908/50000 [24:05:30<7:28:44,  2.06s/it]
Outer training loop:  74%|#######3  | 36909/50000 [24:05:32<7:15:25,  2.00s/it]
Outer training loop:  74%|#######3  | 36910/50000 [24:05:34<7:03:00,  1.94s/it]
Outer training loop:  74%|#######3  | 36911/50000 [24:05:36<7:08:50,  1.97s/it]
Outer training loop:  74%|#######3  | 36912/50000 [24:05:38<7:01:28,  1.93s/it]
Outer training loop:  74%|#######3  | 36913/50000 [24:05:40<6:49:24,  1.88s/it]
Outer training loop:  74%|#######3  | 36914/50000 [24:05:41<6:39:06,  1.83s/it]
Outer training loop:  74%|#######3  | 36915/50000 [24:05:43<6:52:37,  1.89s/it]
Outer training loop:  74%|#######3  | 36916/50000 [24:05:45<6:48:34,  1.87s/it]
Outer training loop:  74%|#######3  | 36917/50000 [24:05:47<6:42:42,  1.85s/it]
Outer training loop:  74%|#######3  | 36918/50000 [24:05:49<6:59:31,  1.92s/it]
Outer training loop:  74%|#######3  | 36919/50000 [24:05:51<6:53:55,  1.90s/it]
Outer training loop:  74%|#######3  | 36920/50000 [24:05:53<6:44:49,  1.86s/it]
Outer training loop:  74%|#######3  | 36921/50000 [24:05:55<6:53:31,  1.90s/it]
Outer training loop:  74%|#######3  | 36922/50000 [24:05:56<6:47:33,  1.87s/it]
Outer training loop:  74%|#######3  | 36923/50000 [24:05:58<7:03:42,  1.94s/it]
Outer training loop:  74%|#######3  | 36924/50000 [24:06:01<7:09:49,  1.97s/it]
Outer training loop:  74%|#######3  | 36925/50000 [24:06:02<6:57:32,  1.92s/it]
Outer training loop:  74%|#######3  | 36926/50000 [24:06:04<6:53:55,  1.90s/it]
Outer training loop:  74%|#######3  | 36927/50000 [24:06:06<7:13:43,  1.99s/it]
Outer training loop:  74%|#######3  | 36928/50000 [24:06:08<7:03:03,  1.94s/it]
Outer training loop:  74%|#######3  | 36929/50000 [24:06:10<6:57:58,  1.92s/it]
Outer training loop:  74%|#######3  | 36930/50000 [24:06:12<7:03:33,  1.94s/it]
Outer training loop:  74%|#######3  | 36931/50000 [24:06:14<6:52:17,  1.89s/it]
Outer training loop:  74%|#######3  | 36932/50000 [24:06:16<6:49:19,  1.88s/it]
Outer training loop:  74%|#######3  | 36933/50000 [24:06:17<6:42:11,  1.85s/it]
Outer training loop:  74%|#######3  | 36934/50000 [24:06:19<6:49:32,  1.88s/it]
Outer training loop:  74%|#######3  | 36935/50000 [24:06:21<6:48:30,  1.88s/it]
Outer training loop:  74%|#######3  | 36936/50000 [24:06:23<6:58:10,  1.92s/it]
Outer training loop:  74%|#######3  | 36937/50000 [24:06:25<7:03:13,  1.94s/it]
Outer training loop:  74%|#######3  | 36938/50000 [24:06:27<7:03:48,  1.95s/it]
Outer training loop:  74%|#######3  | 36939/50000 [24:06:29<6:56:52,  1.92s/it]
Outer training loop:  74%|#######3  | 36940/50000 [24:06:31<6:59:56,  1.93s/it]
Outer training loop:  74%|#######3  | 36941/50000 [24:06:33<6:48:15,  1.88s/it]
Outer training loop:  74%|#######3  | 36942/50000 [24:06:35<6:49:34,  1.88s/it]
Outer training loop:  74%|#######3  | 36943/50000 [24:06:36<6:42:31,  1.85s/it]
Outer training loop:  74%|#######3  | 36944/50000 [24:06:39<6:54:08,  1.90s/it]
Outer training loop:  74%|#######3  | 36945/50000 [24:06:40<6:47:54,  1.87s/it]
Outer training loop:  74%|#######3  | 36946/50000 [24:06:42<6:53:55,  1.90s/it]
Outer training loop:  74%|#######3  | 36947/50000 [24:06:44<6:57:15,  1.92s/it]
Outer training loop:  74%|#######3  | 36948/50000 [24:06:46<6:44:58,  1.86s/it]
Outer training loop:  74%|#######3  | 36949/50000 [24:06:48<6:37:57,  1.83s/it]
Outer training loop:  74%|#######3  | 36950/50000 [24:06:50<6:52:24,  1.90s/it]
Outer training loop:  74%|#######3  | 36951/50000 [24:06:52<6:47:14,  1.87s/it]
Outer training loop:  74%|#######3  | 36952/50000 [24:06:53<6:45:30,  1.86s/it]
Outer training loop:  74%|#######3  | 36953/50000 [24:06:55<6:34:15,  1.81s/it]
Outer training loop:  74%|#######3  | 36954/50000 [24:06:57<6:44:02,  1.86s/it]
Outer training loop:  74%|#######3  | 36955/50000 [24:06:59<6:40:52,  1.84s/it]
Outer training loop:  74%|#######3  | 36956/50000 [24:07:01<6:39:00,  1.84s/it]
Outer training loop:  74%|#######3  | 36957/50000 [24:07:03<6:44:51,  1.86s/it]
Outer training loop:  74%|#######3  | 36958/50000 [24:07:05<6:49:12,  1.88s/it]
Outer training loop:  74%|#######3  | 36959/50000 [24:07:06<6:39:04,  1.84s/it]
Outer training loop:  74%|#######3  | 36960/50000 [24:07:08<6:57:16,  1.92s/it]
Outer training loop:  74%|#######3  | 36961/50000 [24:07:10<6:51:53,  1.90s/it]
Outer training loop:  74%|#######3  | 36962/50000 [24:07:12<6:48:45,  1.88s/it]
Outer training loop:  74%|#######3  | 36963/50000 [24:07:14<6:45:20,  1.87s/it]
Outer training loop:  74%|#######3  | 36964/50000 [24:07:16<6:57:33,  1.92s/it]
Outer training loop:  74%|#######3  | 36965/50000 [24:07:18<6:48:57,  1.88s/it]
Outer training loop:  74%|#######3  | 36966/50000 [24:07:20<6:40:53,  1.85s/it]
Outer training loop:  74%|#######3  | 36967/50000 [24:07:22<6:50:26,  1.89s/it]
Outer training loop:  74%|#######3  | 36968/50000 [24:07:23<6:42:56,  1.86s/it]
Outer training loop:  74%|#######3  | 36969/50000 [24:07:25<6:47:13,  1.88s/it]
Outer training loop:  74%|#######3  | 36970/50000 [24:07:27<6:59:21,  1.93s/it]
Outer training loop:  74%|#######3  | 36971/50000 [24:07:29<6:54:11,  1.91s/it]
Outer training loop:  74%|#######3  | 36972/50000 [24:07:31<6:49:26,  1.89s/it]
Outer training loop:  74%|#######3  | 36973/50000 [24:07:33<6:50:17,  1.89s/it]
Outer training loop:  74%|#######3  | 36974/50000 [24:07:35<7:03:15,  1.95s/it]
Outer training loop:  74%|#######3  | 36975/50000 [24:07:37<6:53:21,  1.90s/it]
Outer training loop:  74%|#######3  | 36976/50000 [24:07:39<6:49:20,  1.89s/it]
Outer training loop:  74%|#######3  | 36977/50000 [24:07:41<7:01:55,  1.94s/it]
Outer training loop:  74%|#######3  | 36978/50000 [24:07:43<6:54:43,  1.91s/it]
Outer training loop:  74%|#######3  | 36979/50000 [24:07:44<6:50:06,  1.89s/it]
Outer training loop:  74%|#######3  | 36980/50000 [24:07:46<6:49:41,  1.89s/it]
Outer training loop:  74%|#######3  | 36981/50000 [24:07:48<6:55:33,  1.92s/it]
Outer training loop:  74%|#######3  | 36982/50000 [24:07:50<6:49:23,  1.89s/it]
Outer training loop:  74%|#######3  | 36983/50000 [24:07:52<6:42:13,  1.85s/it]
Outer training loop:  74%|#######3  | 36984/50000 [24:07:54<7:04:04,  1.95s/it]
Outer training loop:  74%|#######3  | 36985/50000 [24:07:56<6:53:09,  1.90s/it]
Outer training loop:  74%|#######3  | 36986/50000 [24:07:58<6:50:46,  1.89s/it]
Outer training loop:  74%|#######3  | 36987/50000 [24:08:00<6:47:00,  1.88s/it]
Outer training loop:  74%|#######3  | 36988/50000 [24:08:01<6:39:57,  1.84s/it]
Outer training loop:  74%|#######3  | 36989/50000 [24:08:03<6:34:42,  1.82s/it]
Outer training loop:  74%|#######3  | 36990/50000 [24:08:05<6:34:10,  1.82s/it]
Outer training loop:  74%|#######3  | 36991/50000 [24:08:07<6:43:48,  1.86s/it]
Outer training loop:  74%|#######3  | 36992/50000 [24:08:09<6:36:11,  1.83s/it]
Outer training loop:  74%|#######3  | 36993/50000 [24:08:11<6:43:34,  1.86s/it]
Outer training loop:  74%|#######3  | 36994/50000 [24:08:13<6:55:11,  1.92s/it]
Outer training loop:  74%|#######3  | 36995/50000 [24:08:14<6:46:14,  1.87s/it]
Outer training loop:  74%|#######3  | 36996/50000 [24:08:16<6:42:08,  1.86s/it]
Outer training loop:  74%|#######3  | 36997/50000 [24:08:18<6:45:57,  1.87s/it]
Outer training loop:  74%|#######3  | 36998/50000 [24:08:20<6:47:38,  1.88s/it]
Outer training loop:  74%|#######3  | 36999/50000 [24:08:22<6:43:38,  1.86s/it]
Outer training loop:  74%|#######4  | 37000/50000 [24:08:24<6:38:56,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:28,  1.82s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.83s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:26,  1.83s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.89s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.88s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.86s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:25,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:20,  1.89s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:17,  1.88s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:10,  1.86s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:07,  1.85s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:04,  1.86s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:01,  1.84s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:59,  1.84s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:01,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:04,  1.98s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:59,  1.93s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.90s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.90s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:53,  1.95s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:50,  1.93s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.92s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:42,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:43,  1.95s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:39,  1.92s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:27,  1.83s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:31,  1.96s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.86s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:17,  1.81s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.81s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:17,  1.93s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:13,  1.89s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:09,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:07,  1.82s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:05,  1.81s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.85s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:05,  1.98s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.92s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.88s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:54,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.85s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.88s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:50,  2.02s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:46,  1.96s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:44,  1.92s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:42,  1.92s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:33,  1.96s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.90s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.90s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.92s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.93s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.87s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:16,  2.04s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.99s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.95s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.94s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.94s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.91s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<03:07,  2.29s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:51,  2.12s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:43,  2.04s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:37,  2.00s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:35,  2.00s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:30,  1.96s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:20,  1.90s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:15,  1.85s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:14,  1.87s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:11,  1.85s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.87s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:08,  1.94s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.91s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:59,  1.95s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.92s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:46,  1.87s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.94s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.91s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:34,  1.82s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.89s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.93s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:26,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:28,  1.97s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.91s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:21,  1.90s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.87s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.90s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:15,  1.98s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:11,  1.92s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:08,  1.91s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:03,  1.99s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.93s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.89s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:47,  1.99s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:45,  1.96s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.91s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.80s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.82s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:31,  1.96s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.90s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.78s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:17,  1.78s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:15,  1.76s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.79s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.82s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.82s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.81s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.78s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.83s/it]

Outer training loop:  74%|#######4  | 37001/50000 [24:13:44<351:39:27, 97.39s/it]
Outer training loop:  74%|#######4  | 37002/50000 [24:13:46<247:55:40, 68.67s/it]
Outer training loop:  74%|#######4  | 37003/50000 [24:13:48<176:05:31, 48.78s/it]
Outer training loop:  74%|#######4  | 37004/50000 [24:13:50<125:37:48, 34.80s/it]
Outer training loop:  74%|#######4  | 37005/50000 [24:13:52<89:54:33, 24.91s/it] 
Outer training loop:  74%|#######4  | 37006/50000 [24:13:54<64:42:30, 17.93s/it]
Outer training loop:  74%|#######4  | 37007/50000 [24:13:56<47:27:28, 13.15s/it]
Outer training loop:  74%|#######4  | 37008/50000 [24:13:58<35:21:58,  9.80s/it]
Outer training loop:  74%|#######4  | 37009/50000 [24:13:59<26:40:31,  7.39s/it]
Outer training loop:  74%|#######4  | 37010/50000 [24:14:01<20:48:34,  5.77s/it]
Outer training loop:  74%|#######4  | 37011/50000 [24:14:03<16:32:42,  4.59s/it]
Outer training loop:  74%|#######4  | 37012/50000 [24:14:05<13:35:00,  3.77s/it]
Outer training loop:  74%|#######4  | 37013/50000 [24:14:07<11:25:58,  3.17s/it]
Outer training loop:  74%|#######4  | 37014/50000 [24:14:09<10:07:54,  2.81s/it]
Outer training loop:  74%|#######4  | 37015/50000 [24:14:11<9:07:00,  2.53s/it] 
Outer training loop:  74%|#######4  | 37016/50000 [24:14:12<8:18:00,  2.30s/it]
Outer training loop:  74%|#######4  | 37017/50000 [24:14:14<7:55:06,  2.20s/it]
Outer training loop:  74%|#######4  | 37018/50000 [24:14:16<7:26:18,  2.06s/it]
Outer training loop:  74%|#######4  | 37019/50000 [24:14:18<7:07:17,  1.97s/it]
Outer training loop:  74%|#######4  | 37020/50000 [24:14:20<7:13:24,  2.00s/it]
Outer training loop:  74%|#######4  | 37021/50000 [24:14:22<7:04:56,  1.96s/it]
Outer training loop:  74%|#######4  | 37022/50000 [24:14:24<6:51:35,  1.90s/it]
Outer training loop:  74%|#######4  | 37023/50000 [24:14:26<6:55:01,  1.92s/it]
Outer training loop:  74%|#######4  | 37024/50000 [24:14:27<6:51:09,  1.90s/it]
Outer training loop:  74%|#######4  | 37025/50000 [24:14:29<6:38:08,  1.84s/it]
Outer training loop:  74%|#######4  | 37026/50000 [24:14:31<6:26:08,  1.79s/it]
Outer training loop:  74%|#######4  | 37027/50000 [24:14:33<6:41:54,  1.86s/it]
Outer training loop:  74%|#######4  | 37028/50000 [24:14:35<6:41:34,  1.86s/it]
Outer training loop:  74%|#######4  | 37029/50000 [24:14:36<6:39:47,  1.85s/it]
Outer training loop:  74%|#######4  | 37030/50000 [24:14:38<6:42:17,  1.86s/it]
Outer training loop:  74%|#######4  | 37031/50000 [24:14:40<6:33:41,  1.82s/it]
Outer training loop:  74%|#######4  | 37032/50000 [24:14:42<6:25:23,  1.78s/it]
Outer training loop:  74%|#######4  | 37033/50000 [24:14:44<6:40:24,  1.85s/it]
Outer training loop:  74%|#######4  | 37034/50000 [24:14:46<6:37:45,  1.84s/it]
Outer training loop:  74%|#######4  | 37035/50000 [24:14:47<6:35:38,  1.83s/it]
Outer training loop:  74%|#######4  | 37036/50000 [24:14:49<6:42:14,  1.86s/it]
Outer training loop:  74%|#######4  | 37037/50000 [24:14:51<6:51:36,  1.91s/it]
Outer training loop:  74%|#######4  | 37038/50000 [24:14:53<6:44:01,  1.87s/it]
Outer training loop:  74%|#######4  | 37039/50000 [24:14:55<6:44:29,  1.87s/it]
Outer training loop:  74%|#######4  | 37040/50000 [24:14:57<6:50:19,  1.90s/it]
Outer training loop:  74%|#######4  | 37041/50000 [24:14:59<6:44:09,  1.87s/it]
Outer training loop:  74%|#######4  | 37042/50000 [24:15:01<6:43:50,  1.87s/it]
Outer training loop:  74%|#######4  | 37043/50000 [24:15:02<6:36:46,  1.84s/it]
Outer training loop:  74%|#######4  | 37044/50000 [24:15:04<6:51:24,  1.91s/it]
Outer training loop:  74%|#######4  | 37045/50000 [24:15:06<6:51:38,  1.91s/it]
Outer training loop:  74%|#######4  | 37046/50000 [24:15:08<6:42:13,  1.86s/it]
Outer training loop:  74%|#######4  | 37047/50000 [24:15:10<6:55:29,  1.92s/it]
Outer training loop:  74%|#######4  | 37048/50000 [24:15:12<6:47:55,  1.89s/it]
Outer training loop:  74%|#######4  | 37049/50000 [24:15:14<6:44:41,  1.87s/it]
Outer training loop:  74%|#######4  | 37050/50000 [24:15:16<6:36:16,  1.84s/it]
Outer training loop:  74%|#######4  | 37051/50000 [24:15:18<6:48:45,  1.89s/it]
Outer training loop:  74%|#######4  | 37052/50000 [24:15:19<6:44:03,  1.87s/it]
Outer training loop:  74%|#######4  | 37053/50000 [24:15:21<6:39:51,  1.85s/it]
Outer training loop:  74%|#######4  | 37054/50000 [24:15:23<7:05:13,  1.97s/it]
Outer training loop:  74%|#######4  | 37055/50000 [24:15:25<7:07:06,  1.98s/it]
Outer training loop:  74%|#######4  | 37056/50000 [24:15:27<6:53:19,  1.92s/it]
Outer training loop:  74%|#######4  | 37057/50000 [24:15:29<6:43:32,  1.87s/it]
Outer training loop:  74%|#######4  | 37058/50000 [24:15:31<6:46:38,  1.89s/it]
Outer training loop:  74%|#######4  | 37059/50000 [24:15:33<6:39:11,  1.85s/it]
Outer training loop:  74%|#######4  | 37060/50000 [24:15:35<6:43:50,  1.87s/it]
Outer training loop:  74%|#######4  | 37061/50000 [24:15:37<6:44:30,  1.88s/it]
Outer training loop:  74%|#######4  | 37062/50000 [24:15:38<6:46:10,  1.88s/it]
Outer training loop:  74%|#######4  | 37063/50000 [24:15:40<6:42:10,  1.87s/it]
Outer training loop:  74%|#######4  | 37064/50000 [24:15:42<6:30:18,  1.81s/it]
Outer training loop:  74%|#######4  | 37065/50000 [24:15:44<6:42:41,  1.87s/it]
Outer training loop:  74%|#######4  | 37066/50000 [24:15:46<6:40:22,  1.86s/it]
Outer training loop:  74%|#######4  | 37067/50000 [24:15:48<6:35:49,  1.84s/it]
Outer training loop:  74%|#######4  | 37068/50000 [24:15:50<6:47:52,  1.89s/it]
Outer training loop:  74%|#######4  | 37069/50000 [24:15:51<6:41:32,  1.86s/it]
Outer training loop:  74%|#######4  | 37070/50000 [24:15:53<6:40:57,  1.86s/it]
Outer training loop:  74%|#######4  | 37071/50000 [24:15:55<6:40:02,  1.86s/it]
Outer training loop:  74%|#######4  | 37072/50000 [24:15:57<6:47:34,  1.89s/it]
Outer training loop:  74%|#######4  | 37073/50000 [24:15:59<6:49:34,  1.90s/it]
Outer training loop:  74%|#######4  | 37074/50000 [24:16:01<6:47:22,  1.89s/it]
Outer training loop:  74%|#######4  | 37075/50000 [24:16:03<7:00:12,  1.95s/it]
Outer training loop:  74%|#######4  | 37076/50000 [24:16:05<6:55:25,  1.93s/it]
Outer training loop:  74%|#######4  | 37077/50000 [24:16:07<6:49:10,  1.90s/it]
Outer training loop:  74%|#######4  | 37078/50000 [24:16:08<6:42:22,  1.87s/it]
Outer training loop:  74%|#######4  | 37079/50000 [24:16:11<6:58:12,  1.94s/it]
Outer training loop:  74%|#######4  | 37080/50000 [24:16:12<6:43:18,  1.87s/it]
Outer training loop:  74%|#######4  | 37081/50000 [24:16:14<6:39:05,  1.85s/it]
Outer training loop:  74%|#######4  | 37082/50000 [24:16:16<6:46:32,  1.89s/it]
Outer training loop:  74%|#######4  | 37083/50000 [24:16:18<6:40:16,  1.86s/it]
Outer training loop:  74%|#######4  | 37084/50000 [24:16:20<6:40:25,  1.86s/it]
Outer training loop:  74%|#######4  | 37085/50000 [24:16:22<6:51:17,  1.91s/it]
Outer training loop:  74%|#######4  | 37086/50000 [24:16:23<6:42:00,  1.87s/it]
Outer training loop:  74%|#######4  | 37087/50000 [24:16:25<6:42:45,  1.87s/it]
Outer training loop:  74%|#######4  | 37088/50000 [24:16:27<6:45:55,  1.89s/it]
Outer training loop:  74%|#######4  | 37089/50000 [24:16:29<6:48:08,  1.90s/it]
Outer training loop:  74%|#######4  | 37090/50000 [24:16:31<6:41:31,  1.87s/it]
Outer training loop:  74%|#######4  | 37091/50000 [24:16:33<6:33:49,  1.83s/it]
Outer training loop:  74%|#######4  | 37092/50000 [24:16:35<6:57:47,  1.94s/it]
Outer training loop:  74%|#######4  | 37093/50000 [24:16:37<6:54:32,  1.93s/it]
Outer training loop:  74%|#######4  | 37094/50000 [24:16:39<6:44:25,  1.88s/it]
Outer training loop:  74%|#######4  | 37095/50000 [24:16:41<6:55:14,  1.93s/it]
Outer training loop:  74%|#######4  | 37096/50000 [24:16:43<6:52:37,  1.92s/it]
Outer training loop:  74%|#######4  | 37097/50000 [24:16:44<6:48:17,  1.90s/it]
Outer training loop:  74%|#######4  | 37098/50000 [24:16:46<6:59:20,  1.95s/it]
Outer training loop:  74%|#######4  | 37099/50000 [24:16:48<6:45:16,  1.88s/it]
Outer training loop:  74%|#######4  | 37100/50000 [24:16:50<6:47:21,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.02s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.01s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.94s/it]

Outer training loop:  74%|#######4  | 37101/50000 [24:16:58<12:57:53,  3.62s/it]
Outer training loop:  74%|#######4  | 37102/50000 [24:17:00<11:11:02,  3.12s/it]
Outer training loop:  74%|#######4  | 37103/50000 [24:17:01<9:37:02,  2.68s/it] 
Outer training loop:  74%|#######4  | 37104/50000 [24:17:03<8:49:23,  2.46s/it]
Outer training loop:  74%|#######4  | 37105/50000 [24:17:05<8:26:26,  2.36s/it]
Outer training loop:  74%|#######4  | 37106/50000 [24:17:07<7:54:02,  2.21s/it]
Outer training loop:  74%|#######4  | 37107/50000 [24:17:09<7:30:37,  2.10s/it]
Outer training loop:  74%|#######4  | 37108/50000 [24:17:11<7:10:18,  2.00s/it]
Outer training loop:  74%|#######4  | 37109/50000 [24:17:13<7:20:25,  2.05s/it]
Outer training loop:  74%|#######4  | 37110/50000 [24:17:15<7:08:49,  2.00s/it]
Outer training loop:  74%|#######4  | 37111/50000 [24:17:17<7:11:07,  2.01s/it]
Outer training loop:  74%|#######4  | 37112/50000 [24:17:19<7:01:31,  1.96s/it]
Outer training loop:  74%|#######4  | 37113/50000 [24:17:21<6:53:41,  1.93s/it]
Outer training loop:  74%|#######4  | 37114/50000 [24:17:23<7:00:35,  1.96s/it]
Outer training loop:  74%|#######4  | 37115/50000 [24:17:25<7:08:42,  2.00s/it]
Outer training loop:  74%|#######4  | 37116/50000 [24:17:27<6:57:53,  1.95s/it]
Outer training loop:  74%|#######4  | 37117/50000 [24:17:29<6:52:38,  1.92s/it]
Outer training loop:  74%|#######4  | 37118/50000 [24:17:31<6:59:44,  1.96s/it]
Outer training loop:  74%|#######4  | 37119/50000 [24:17:32<6:45:53,  1.89s/it]
Outer training loop:  74%|#######4  | 37120/50000 [24:17:34<6:38:25,  1.86s/it]
Outer training loop:  74%|#######4  | 37121/50000 [24:17:36<6:42:14,  1.87s/it]
Outer training loop:  74%|#######4  | 37122/50000 [24:17:38<6:36:30,  1.85s/it]
Outer training loop:  74%|#######4  | 37123/50000 [24:17:40<6:38:36,  1.86s/it]
Outer training loop:  74%|#######4  | 37124/50000 [24:17:42<6:40:33,  1.87s/it]
Outer training loop:  74%|#######4  | 37125/50000 [24:17:44<6:54:54,  1.93s/it]
Outer training loop:  74%|#######4  | 37126/50000 [24:17:45<6:46:18,  1.89s/it]
Outer training loop:  74%|#######4  | 37127/50000 [24:17:47<6:48:25,  1.90s/it]
Outer training loop:  74%|#######4  | 37128/50000 [24:17:49<6:56:27,  1.94s/it]
Outer training loop:  74%|#######4  | 37129/50000 [24:17:51<6:52:45,  1.92s/it]
Outer training loop:  74%|#######4  | 37130/50000 [24:17:53<6:48:55,  1.91s/it]
Outer training loop:  74%|#######4  | 37131/50000 [24:17:55<7:06:29,  1.99s/it]
Outer training loop:  74%|#######4  | 37132/50000 [24:17:57<6:52:33,  1.92s/it]
Outer training loop:  74%|#######4  | 37133/50000 [24:17:59<6:50:59,  1.92s/it]
Outer training loop:  74%|#######4  | 37134/50000 [24:18:01<6:54:11,  1.93s/it]
Outer training loop:  74%|#######4  | 37135/50000 [24:18:03<6:50:04,  1.91s/it]
Outer training loop:  74%|#######4  | 37136/50000 [24:18:05<6:53:55,  1.93s/it]
Outer training loop:  74%|#######4  | 37137/50000 [24:18:07<6:46:32,  1.90s/it]
Outer training loop:  74%|#######4  | 37138/50000 [24:18:09<6:57:48,  1.95s/it]
Outer training loop:  74%|#######4  | 37139/50000 [24:18:10<6:49:00,  1.91s/it]
Outer training loop:  74%|#######4  | 37140/50000 [24:18:12<6:45:45,  1.89s/it]
Outer training loop:  74%|#######4  | 37141/50000 [24:18:14<6:54:09,  1.93s/it]
Outer training loop:  74%|#######4  | 37142/50000 [24:18:16<6:51:55,  1.92s/it]
Outer training loop:  74%|#######4  | 37143/50000 [24:18:18<6:40:00,  1.87s/it]
Outer training loop:  74%|#######4  | 37144/50000 [24:18:20<6:31:52,  1.83s/it]
Outer training loop:  74%|#######4  | 37145/50000 [24:18:22<6:45:12,  1.89s/it]
Outer training loop:  74%|#######4  | 37146/50000 [24:18:24<6:42:46,  1.88s/it]
Outer training loop:  74%|#######4  | 37147/50000 [24:18:25<6:41:01,  1.87s/it]
Outer training loop:  74%|#######4  | 37148/50000 [24:18:28<6:57:59,  1.95s/it]
Outer training loop:  74%|#######4  | 37149/50000 [24:18:29<6:46:27,  1.90s/it]
Outer training loop:  74%|#######4  | 37150/50000 [24:18:31<6:39:41,  1.87s/it]
Outer training loop:  74%|#######4  | 37151/50000 [24:18:33<7:02:43,  1.97s/it]
Outer training loop:  74%|#######4  | 37152/50000 [24:18:35<6:58:39,  1.96s/it]
Outer training loop:  74%|#######4  | 37153/50000 [24:18:37<7:00:40,  1.96s/it]
Outer training loop:  74%|#######4  | 37154/50000 [24:18:39<6:52:41,  1.93s/it]
Outer training loop:  74%|#######4  | 37155/50000 [24:18:41<7:06:01,  1.99s/it]
Outer training loop:  74%|#######4  | 37156/50000 [24:18:43<6:46:18,  1.90s/it]
Outer training loop:  74%|#######4  | 37157/50000 [24:18:45<6:34:25,  1.84s/it]
Outer training loop:  74%|#######4  | 37158/50000 [24:18:47<6:44:14,  1.89s/it]
Outer training loop:  74%|#######4  | 37159/50000 [24:18:48<6:32:50,  1.84s/it]
Outer training loop:  74%|#######4  | 37160/50000 [24:18:50<6:34:53,  1.85s/it]
Outer training loop:  74%|#######4  | 37161/50000 [24:18:52<6:45:38,  1.90s/it]
Outer training loop:  74%|#######4  | 37162/50000 [24:18:54<6:45:03,  1.89s/it]
Outer training loop:  74%|#######4  | 37163/50000 [24:18:56<6:37:08,  1.86s/it]
Outer training loop:  74%|#######4  | 37164/50000 [24:18:58<6:48:10,  1.91s/it]
Outer training loop:  74%|#######4  | 37165/50000 [24:19:00<6:51:02,  1.92s/it]
Outer training loop:  74%|#######4  | 37166/50000 [24:19:02<6:47:06,  1.90s/it]
Outer training loop:  74%|#######4  | 37167/50000 [24:19:04<6:39:56,  1.87s/it]
Outer training loop:  74%|#######4  | 37168/50000 [24:19:05<6:40:18,  1.87s/it]
Outer training loop:  74%|#######4  | 37169/50000 [24:19:07<6:48:46,  1.91s/it]
Outer training loop:  74%|#######4  | 37170/50000 [24:19:09<6:48:23,  1.91s/it]
Outer training loop:  74%|#######4  | 37171/50000 [24:19:11<6:54:12,  1.94s/it]
Outer training loop:  74%|#######4  | 37172/50000 [24:19:13<7:06:42,  2.00s/it]
Outer training loop:  74%|#######4  | 37173/50000 [24:19:15<6:56:18,  1.95s/it]
Outer training loop:  74%|#######4  | 37174/50000 [24:19:17<6:49:40,  1.92s/it]
Outer training loop:  74%|#######4  | 37175/50000 [24:19:19<7:02:30,  1.98s/it]
Outer training loop:  74%|#######4  | 37176/50000 [24:19:21<6:52:39,  1.93s/it]
Outer training loop:  74%|#######4  | 37177/50000 [24:19:23<6:43:19,  1.89s/it]
Outer training loop:  74%|#######4  | 37178/50000 [24:19:25<6:41:15,  1.88s/it]
Outer training loop:  74%|#######4  | 37179/50000 [24:19:27<6:57:56,  1.96s/it]
Outer training loop:  74%|#######4  | 37180/50000 [24:19:29<6:43:38,  1.89s/it]
Outer training loop:  74%|#######4  | 37181/50000 [24:19:30<6:34:25,  1.85s/it]
Outer training loop:  74%|#######4  | 37182/50000 [24:19:32<6:40:22,  1.87s/it]
Outer training loop:  74%|#######4  | 37183/50000 [24:19:34<6:40:26,  1.87s/it]
Outer training loop:  74%|#######4  | 37184/50000 [24:19:36<6:31:00,  1.83s/it]
Outer training loop:  74%|#######4  | 37185/50000 [24:19:38<6:33:30,  1.84s/it]
Outer training loop:  74%|#######4  | 37186/50000 [24:19:40<6:42:23,  1.88s/it]
Outer training loop:  74%|#######4  | 37187/50000 [24:19:41<6:30:27,  1.83s/it]
Outer training loop:  74%|#######4  | 37188/50000 [24:19:43<6:32:13,  1.84s/it]
Outer training loop:  74%|#######4  | 37189/50000 [24:19:45<6:36:31,  1.86s/it]
Outer training loop:  74%|#######4  | 37190/50000 [24:19:47<6:31:08,  1.83s/it]
Outer training loop:  74%|#######4  | 37191/50000 [24:19:49<6:26:28,  1.81s/it]
Outer training loop:  74%|#######4  | 37192/50000 [24:19:51<6:32:31,  1.84s/it]
Outer training loop:  74%|#######4  | 37193/50000 [24:19:53<6:39:58,  1.87s/it]
Outer training loop:  74%|#######4  | 37194/50000 [24:19:54<6:28:43,  1.82s/it]
Outer training loop:  74%|#######4  | 37195/50000 [24:19:56<6:24:00,  1.80s/it]
Outer training loop:  74%|#######4  | 37196/50000 [24:19:58<6:51:11,  1.93s/it]
Outer training loop:  74%|#######4  | 37197/50000 [24:20:00<6:54:28,  1.94s/it]
Outer training loop:  74%|#######4  | 37198/50000 [24:20:02<6:50:24,  1.92s/it]
Outer training loop:  74%|#######4  | 37199/50000 [24:20:04<6:48:00,  1.91s/it]
Outer training loop:  74%|#######4  | 37200/50000 [24:20:06<6:59:45,  1.97s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  74%|#######4  | 37201/50000 [24:20:14<13:02:18,  3.67s/it]
Outer training loop:  74%|#######4  | 37202/50000 [24:20:16<11:06:35,  3.13s/it]
Outer training loop:  74%|#######4  | 37203/50000 [24:20:18<9:57:20,  2.80s/it] 
Outer training loop:  74%|#######4  | 37204/50000 [24:20:19<8:50:00,  2.49s/it]
Outer training loop:  74%|#######4  | 37205/50000 [24:20:21<8:07:38,  2.29s/it]
Outer training loop:  74%|#######4  | 37206/50000 [24:20:23<7:44:29,  2.18s/it]
Outer training loop:  74%|#######4  | 37207/50000 [24:20:25<7:43:25,  2.17s/it]
Outer training loop:  74%|#######4  | 37208/50000 [24:20:27<7:15:38,  2.04s/it]
Outer training loop:  74%|#######4  | 37209/50000 [24:20:29<6:59:17,  1.97s/it]
Outer training loop:  74%|#######4  | 37210/50000 [24:20:31<7:04:27,  1.99s/it]
Outer training loop:  74%|#######4  | 37211/50000 [24:20:33<6:56:01,  1.95s/it]
Outer training loop:  74%|#######4  | 37212/50000 [24:20:35<6:47:40,  1.91s/it]
Outer training loop:  74%|#######4  | 37213/50000 [24:20:37<6:54:45,  1.95s/it]
Outer training loop:  74%|#######4  | 37214/50000 [24:20:38<6:43:29,  1.89s/it]
Outer training loop:  74%|#######4  | 37215/50000 [24:20:40<6:44:03,  1.90s/it]
Outer training loop:  74%|#######4  | 37216/50000 [24:20:42<6:36:39,  1.86s/it]
Outer training loop:  74%|#######4  | 37217/50000 [24:20:44<6:42:56,  1.89s/it]
Outer training loop:  74%|#######4  | 37218/50000 [24:20:46<6:44:58,  1.90s/it]
Outer training loop:  74%|#######4  | 37219/50000 [24:20:48<6:36:59,  1.86s/it]
Outer training loop:  74%|#######4  | 37220/50000 [24:20:50<6:49:22,  1.92s/it]
Outer training loop:  74%|#######4  | 37221/50000 [24:20:52<6:38:30,  1.87s/it]
Outer training loop:  74%|#######4  | 37222/50000 [24:20:53<6:31:57,  1.84s/it]
Outer training loop:  74%|#######4  | 37223/50000 [24:20:55<6:36:44,  1.86s/it]
Outer training loop:  74%|#######4  | 37224/50000 [24:20:57<6:40:34,  1.88s/it]
Outer training loop:  74%|#######4  | 37225/50000 [24:20:59<6:38:03,  1.87s/it]
Outer training loop:  74%|#######4  | 37226/50000 [24:21:01<6:30:36,  1.83s/it]
Outer training loop:  74%|#######4  | 37227/50000 [24:21:03<6:38:12,  1.87s/it]
Outer training loop:  74%|#######4  | 37228/50000 [24:21:05<6:41:13,  1.88s/it]
Outer training loop:  74%|#######4  | 37229/50000 [24:21:06<6:32:01,  1.84s/it]
Outer training loop:  74%|#######4  | 37230/50000 [24:21:08<6:42:43,  1.89s/it]
Outer training loop:  74%|#######4  | 37231/50000 [24:21:10<6:38:17,  1.87s/it]
Outer training loop:  74%|#######4  | 37232/50000 [24:21:12<6:36:07,  1.86s/it]
Outer training loop:  74%|#######4  | 37233/50000 [24:21:14<6:49:07,  1.92s/it]
Outer training loop:  74%|#######4  | 37234/50000 [24:21:16<6:45:07,  1.90s/it]
Outer training loop:  74%|#######4  | 37235/50000 [24:21:18<6:39:01,  1.88s/it]
Outer training loop:  74%|#######4  | 37236/50000 [24:21:19<6:27:26,  1.82s/it]
Outer training loop:  74%|#######4  | 37237/50000 [24:21:21<6:38:16,  1.87s/it]
Outer training loop:  74%|#######4  | 37238/50000 [24:21:23<6:39:59,  1.88s/it]
Outer training loop:  74%|#######4  | 37239/50000 [24:21:25<6:36:40,  1.87s/it]
Outer training loop:  74%|#######4  | 37240/50000 [24:21:27<6:39:27,  1.88s/it]
Outer training loop:  74%|#######4  | 37241/50000 [24:21:29<6:30:13,  1.84s/it]
Outer training loop:  74%|#######4  | 37242/50000 [24:21:31<6:35:15,  1.86s/it]
Outer training loop:  74%|#######4  | 37243/50000 [24:21:33<6:43:23,  1.90s/it]
Outer training loop:  74%|#######4  | 37244/50000 [24:21:35<6:43:10,  1.90s/it]
Outer training loop:  74%|#######4  | 37245/50000 [24:21:36<6:31:15,  1.84s/it]
Outer training loop:  74%|#######4  | 37246/50000 [24:21:38<6:22:56,  1.80s/it]
Outer training loop:  74%|#######4  | 37247/50000 [24:21:40<6:49:57,  1.93s/it]
Outer training loop:  74%|#######4  | 37248/50000 [24:21:42<6:55:13,  1.95s/it]
Outer training loop:  74%|#######4  | 37249/50000 [24:21:44<6:49:43,  1.93s/it]
Outer training loop:  74%|#######4  | 37250/50000 [24:21:46<6:51:18,  1.94s/it]
Outer training loop:  75%|#######4  | 37251/50000 [24:21:48<6:38:22,  1.87s/it]
Outer training loop:  75%|#######4  | 37252/50000 [24:21:50<6:41:12,  1.89s/it]
Outer training loop:  75%|#######4  | 37253/50000 [24:21:52<6:52:44,  1.94s/it]
Outer training loop:  75%|#######4  | 37254/50000 [24:21:54<6:49:32,  1.93s/it]
Outer training loop:  75%|#######4  | 37255/50000 [24:21:56<6:46:31,  1.91s/it]
Outer training loop:  75%|#######4  | 37256/50000 [24:21:58<6:56:09,  1.96s/it]
Outer training loop:  75%|#######4  | 37257/50000 [24:21:59<6:47:48,  1.92s/it]
Outer training loop:  75%|#######4  | 37258/50000 [24:22:01<6:35:28,  1.86s/it]
Outer training loop:  75%|#######4  | 37259/50000 [24:22:03<6:43:54,  1.90s/it]
Outer training loop:  75%|#######4  | 37260/50000 [24:22:05<6:39:22,  1.88s/it]
Outer training loop:  75%|#######4  | 37261/50000 [24:22:07<6:42:42,  1.90s/it]
Outer training loop:  75%|#######4  | 37262/50000 [24:22:09<6:37:21,  1.87s/it]
Outer training loop:  75%|#######4  | 37263/50000 [24:22:11<6:45:34,  1.91s/it]
Outer training loop:  75%|#######4  | 37264/50000 [24:22:13<6:42:39,  1.90s/it]
Outer training loop:  75%|#######4  | 37265/50000 [24:22:14<6:35:06,  1.86s/it]
Outer training loop:  75%|#######4  | 37266/50000 [24:22:16<6:42:04,  1.89s/it]
Outer training loop:  75%|#######4  | 37267/50000 [24:22:18<6:30:00,  1.84s/it]
Outer training loop:  75%|#######4  | 37268/50000 [24:22:20<6:24:23,  1.81s/it]
Outer training loop:  75%|#######4  | 37269/50000 [24:22:22<6:23:19,  1.81s/it]
Outer training loop:  75%|#######4  | 37270/50000 [24:22:24<6:30:36,  1.84s/it]
Outer training loop:  75%|#######4  | 37271/50000 [24:22:25<6:34:31,  1.86s/it]
Outer training loop:  75%|#######4  | 37272/50000 [24:22:27<6:30:00,  1.84s/it]
Outer training loop:  75%|#######4  | 37273/50000 [24:22:29<6:27:48,  1.83s/it]
Outer training loop:  75%|#######4  | 37274/50000 [24:22:31<6:49:58,  1.93s/it]
Outer training loop:  75%|#######4  | 37275/50000 [24:22:33<6:47:22,  1.92s/it]
Outer training loop:  75%|#######4  | 37276/50000 [24:22:35<6:48:51,  1.93s/it]
Outer training loop:  75%|#######4  | 37277/50000 [24:22:37<6:59:36,  1.98s/it]
Outer training loop:  75%|#######4  | 37278/50000 [24:22:39<6:47:41,  1.92s/it]
Outer training loop:  75%|#######4  | 37279/50000 [24:22:41<6:37:28,  1.87s/it]
Outer training loop:  75%|#######4  | 37280/50000 [24:22:42<6:28:12,  1.83s/it]
Outer training loop:  75%|#######4  | 37281/50000 [24:22:44<6:36:56,  1.87s/it]
Outer training loop:  75%|#######4  | 37282/50000 [24:22:46<6:32:14,  1.85s/it]
Outer training loop:  75%|#######4  | 37283/50000 [24:22:48<6:28:30,  1.83s/it]
Outer training loop:  75%|#######4  | 37284/50000 [24:22:50<6:45:15,  1.91s/it]
Outer training loop:  75%|#######4  | 37285/50000 [24:22:52<6:34:52,  1.86s/it]
Outer training loop:  75%|#######4  | 37286/50000 [24:22:54<6:30:25,  1.84s/it]
Outer training loop:  75%|#######4  | 37287/50000 [24:22:55<6:28:52,  1.84s/it]
Outer training loop:  75%|#######4  | 37288/50000 [24:22:58<6:43:16,  1.90s/it]
Outer training loop:  75%|#######4  | 37289/50000 [24:22:59<6:36:30,  1.87s/it]
Outer training loop:  75%|#######4  | 37290/50000 [24:23:01<6:22:51,  1.81s/it]
Outer training loop:  75%|#######4  | 37291/50000 [24:23:03<6:38:38,  1.88s/it]
Outer training loop:  75%|#######4  | 37292/50000 [24:23:05<6:30:21,  1.84s/it]
Outer training loop:  75%|#######4  | 37293/50000 [24:23:06<6:21:26,  1.80s/it]
Outer training loop:  75%|#######4  | 37294/50000 [24:23:08<6:21:15,  1.80s/it]
Outer training loop:  75%|#######4  | 37295/50000 [24:23:10<6:33:03,  1.86s/it]
Outer training loop:  75%|#######4  | 37296/50000 [24:23:12<6:26:53,  1.83s/it]
Outer training loop:  75%|#######4  | 37297/50000 [24:23:14<6:24:44,  1.82s/it]
Outer training loop:  75%|#######4  | 37298/50000 [24:23:16<6:54:38,  1.96s/it]
Outer training loop:  75%|#######4  | 37299/50000 [24:23:18<6:42:21,  1.90s/it]
Outer training loop:  75%|#######4  | 37300/50000 [24:23:20<6:38:46,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  75%|#######4  | 37301/50000 [24:23:27<12:40:42,  3.59s/it]
Outer training loop:  75%|#######4  | 37302/50000 [24:23:29<11:02:48,  3.13s/it]
Outer training loop:  75%|#######4  | 37303/50000 [24:23:31<9:42:12,  2.75s/it] 
Outer training loop:  75%|#######4  | 37304/50000 [24:23:33<8:42:57,  2.47s/it]
Outer training loop:  75%|#######4  | 37305/50000 [24:23:35<8:22:05,  2.37s/it]
Outer training loop:  75%|#######4  | 37306/50000 [24:23:37<7:41:11,  2.18s/it]
Outer training loop:  75%|#######4  | 37307/50000 [24:23:39<7:19:35,  2.08s/it]
Outer training loop:  75%|#######4  | 37308/50000 [24:23:41<7:13:05,  2.05s/it]
Outer training loop:  75%|#######4  | 37309/50000 [24:23:43<6:57:48,  1.98s/it]
Outer training loop:  75%|#######4  | 37310/50000 [24:23:44<6:46:05,  1.92s/it]
Outer training loop:  75%|#######4  | 37311/50000 [24:23:46<6:34:37,  1.87s/it]
Outer training loop:  75%|#######4  | 37312/50000 [24:23:48<6:39:19,  1.89s/it]
Outer training loop:  75%|#######4  | 37313/50000 [24:23:50<6:45:40,  1.92s/it]
Outer training loop:  75%|#######4  | 37314/50000 [24:23:52<6:33:14,  1.86s/it]
Outer training loop:  75%|#######4  | 37315/50000 [24:23:54<6:45:20,  1.92s/it]
Outer training loop:  75%|#######4  | 37316/50000 [24:23:56<6:42:45,  1.91s/it]
Outer training loop:  75%|#######4  | 37317/50000 [24:23:58<6:39:32,  1.89s/it]
Outer training loop:  75%|#######4  | 37318/50000 [24:24:00<6:48:56,  1.93s/it]
Outer training loop:  75%|#######4  | 37319/50000 [24:24:01<6:38:34,  1.89s/it]
Outer training loop:  75%|#######4  | 37320/50000 [24:24:03<6:27:42,  1.83s/it]
Outer training loop:  75%|#######4  | 37321/50000 [24:24:05<6:28:43,  1.84s/it]
Outer training loop:  75%|#######4  | 37322/50000 [24:24:07<6:40:11,  1.89s/it]
Outer training loop:  75%|#######4  | 37323/50000 [24:24:09<6:33:15,  1.86s/it]
Outer training loop:  75%|#######4  | 37324/50000 [24:24:11<6:31:07,  1.85s/it]
Outer training loop:  75%|#######4  | 37325/50000 [24:24:12<6:37:10,  1.88s/it]
Outer training loop:  75%|#######4  | 37326/50000 [24:24:14<6:32:06,  1.86s/it]
Outer training loop:  75%|#######4  | 37327/50000 [24:24:16<6:24:46,  1.82s/it]
Outer training loop:  75%|#######4  | 37328/50000 [24:24:18<6:23:42,  1.82s/it]
Outer training loop:  75%|#######4  | 37329/50000 [24:24:20<6:41:10,  1.90s/it]
Outer training loop:  75%|#######4  | 37330/50000 [24:24:22<6:35:10,  1.87s/it]
Outer training loop:  75%|#######4  | 37331/50000 [24:24:24<6:33:00,  1.86s/it]
Outer training loop:  75%|#######4  | 37332/50000 [24:24:26<6:51:18,  1.95s/it]
Outer training loop:  75%|#######4  | 37333/50000 [24:24:28<6:49:09,  1.94s/it]
Outer training loop:  75%|#######4  | 37334/50000 [24:24:29<6:41:13,  1.90s/it]
Outer training loop:  75%|#######4  | 37335/50000 [24:24:31<6:35:52,  1.88s/it]
Outer training loop:  75%|#######4  | 37336/50000 [24:24:33<6:46:05,  1.92s/it]
Outer training loop:  75%|#######4  | 37337/50000 [24:24:35<6:49:31,  1.94s/it]
Outer training loop:  75%|#######4  | 37338/50000 [24:24:37<6:37:53,  1.89s/it]
Outer training loop:  75%|#######4  | 37339/50000 [24:24:39<6:46:46,  1.93s/it]
Outer training loop:  75%|#######4  | 37340/50000 [24:24:41<6:41:53,  1.90s/it]
Outer training loop:  75%|#######4  | 37341/50000 [24:24:43<6:41:08,  1.90s/it]
Outer training loop:  75%|#######4  | 37342/50000 [24:24:45<6:51:00,  1.95s/it]
Outer training loop:  75%|#######4  | 37343/50000 [24:24:47<6:41:02,  1.90s/it]
Outer training loop:  75%|#######4  | 37344/50000 [24:24:48<6:31:11,  1.85s/it]
Outer training loop:  75%|#######4  | 37345/50000 [24:24:51<6:52:24,  1.96s/it]
Outer training loop:  75%|#######4  | 37346/50000 [24:24:53<6:50:29,  1.95s/it]
Outer training loop:  75%|#######4  | 37347/50000 [24:24:54<6:39:56,  1.90s/it]
Outer training loop:  75%|#######4  | 37348/50000 [24:24:56<6:31:51,  1.86s/it]
Outer training loop:  75%|#######4  | 37349/50000 [24:24:58<6:46:58,  1.93s/it]
Outer training loop:  75%|#######4  | 37350/50000 [24:25:00<6:42:40,  1.91s/it]
Outer training loop:  75%|#######4  | 37351/50000 [24:25:02<6:40:28,  1.90s/it]
Outer training loop:  75%|#######4  | 37352/50000 [24:25:04<6:51:16,  1.95s/it]
Outer training loop:  75%|#######4  | 37353/50000 [24:25:06<6:41:45,  1.91s/it]
Outer training loop:  75%|#######4  | 37354/50000 [24:25:08<6:37:02,  1.88s/it]
Outer training loop:  75%|#######4  | 37355/50000 [24:25:09<6:37:03,  1.88s/it]
Outer training loop:  75%|#######4  | 37356/50000 [24:25:12<6:47:22,  1.93s/it]
Outer training loop:  75%|#######4  | 37357/50000 [24:25:13<6:38:01,  1.89s/it]
Outer training loop:  75%|#######4  | 37358/50000 [24:25:15<6:35:45,  1.88s/it]
Outer training loop:  75%|#######4  | 37359/50000 [24:25:17<6:42:28,  1.91s/it]
Outer training loop:  75%|#######4  | 37360/50000 [24:25:19<6:48:05,  1.94s/it]
Outer training loop:  75%|#######4  | 37361/50000 [24:25:21<6:39:28,  1.90s/it]
Outer training loop:  75%|#######4  | 37362/50000 [24:25:23<6:35:59,  1.88s/it]
Outer training loop:  75%|#######4  | 37363/50000 [24:25:25<6:46:03,  1.93s/it]
Outer training loop:  75%|#######4  | 37364/50000 [24:25:27<6:42:12,  1.91s/it]
Outer training loop:  75%|#######4  | 37365/50000 [24:25:28<6:34:33,  1.87s/it]
Outer training loop:  75%|#######4  | 37366/50000 [24:25:31<6:46:57,  1.93s/it]
Outer training loop:  75%|#######4  | 37367/50000 [24:25:32<6:37:32,  1.89s/it]
Outer training loop:  75%|#######4  | 37368/50000 [24:25:34<6:33:02,  1.87s/it]
Outer training loop:  75%|#######4  | 37369/50000 [24:25:36<6:40:24,  1.90s/it]
Outer training loop:  75%|#######4  | 37370/50000 [24:25:38<6:34:27,  1.87s/it]
Outer training loop:  75%|#######4  | 37371/50000 [24:25:40<6:35:24,  1.88s/it]
Outer training loop:  75%|#######4  | 37372/50000 [24:25:42<6:37:25,  1.89s/it]
Outer training loop:  75%|#######4  | 37373/50000 [24:25:44<6:57:15,  1.98s/it]
Outer training loop:  75%|#######4  | 37374/50000 [24:25:46<6:41:49,  1.91s/it]
Outer training loop:  75%|#######4  | 37375/50000 [24:25:47<6:34:38,  1.88s/it]
Outer training loop:  75%|#######4  | 37376/50000 [24:25:50<6:44:17,  1.92s/it]
Outer training loop:  75%|#######4  | 37377/50000 [24:25:51<6:37:16,  1.89s/it]
Outer training loop:  75%|#######4  | 37378/50000 [24:25:53<6:31:47,  1.86s/it]
Outer training loop:  75%|#######4  | 37379/50000 [24:25:55<6:40:36,  1.90s/it]
Outer training loop:  75%|#######4  | 37380/50000 [24:25:57<6:27:02,  1.84s/it]
Outer training loop:  75%|#######4  | 37381/50000 [24:25:59<6:25:10,  1.83s/it]
Outer training loop:  75%|#######4  | 37382/50000 [24:26:00<6:20:42,  1.81s/it]
Outer training loop:  75%|#######4  | 37383/50000 [24:26:02<6:33:25,  1.87s/it]
Outer training loop:  75%|#######4  | 37384/50000 [24:26:04<6:32:14,  1.87s/it]
Outer training loop:  75%|#######4  | 37385/50000 [24:26:06<6:21:23,  1.81s/it]
Outer training loop:  75%|#######4  | 37386/50000 [24:26:08<6:19:12,  1.80s/it]
Outer training loop:  75%|#######4  | 37387/50000 [24:26:10<6:25:34,  1.83s/it]
Outer training loop:  75%|#######4  | 37388/50000 [24:26:11<6:23:32,  1.82s/it]
Outer training loop:  75%|#######4  | 37389/50000 [24:26:13<6:23:01,  1.82s/it]
Outer training loop:  75%|#######4  | 37390/50000 [24:26:15<6:37:00,  1.89s/it]
Outer training loop:  75%|#######4  | 37391/50000 [24:26:17<6:34:41,  1.88s/it]
Outer training loop:  75%|#######4  | 37392/50000 [24:26:19<6:26:04,  1.84s/it]
Outer training loop:  75%|#######4  | 37393/50000 [24:26:21<6:27:06,  1.84s/it]
Outer training loop:  75%|#######4  | 37394/50000 [24:26:23<6:36:08,  1.89s/it]
Outer training loop:  75%|#######4  | 37395/50000 [24:26:25<6:47:48,  1.94s/it]
Outer training loop:  75%|#######4  | 37396/50000 [24:26:27<6:34:31,  1.88s/it]
Outer training loop:  75%|#######4  | 37397/50000 [24:26:29<6:53:35,  1.97s/it]
Outer training loop:  75%|#######4  | 37398/50000 [24:26:30<6:38:20,  1.90s/it]
Outer training loop:  75%|#######4  | 37399/50000 [24:26:32<6:35:43,  1.88s/it]
Outer training loop:  75%|#######4  | 37400/50000 [24:26:34<6:49:23,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  75%|#######4  | 37401/50000 [24:26:42<12:32:16,  3.58s/it]
Outer training loop:  75%|#######4  | 37402/50000 [24:26:44<10:49:26,  3.09s/it]
Outer training loop:  75%|#######4  | 37403/50000 [24:26:46<9:32:23,  2.73s/it] 
Outer training loop:  75%|#######4  | 37404/50000 [24:26:48<8:47:40,  2.51s/it]
Outer training loop:  75%|#######4  | 37405/50000 [24:26:49<8:04:44,  2.31s/it]
Outer training loop:  75%|#######4  | 37406/50000 [24:26:51<7:40:46,  2.20s/it]
Outer training loop:  75%|#######4  | 37407/50000 [24:26:53<7:14:59,  2.07s/it]
Outer training loop:  75%|#######4  | 37408/50000 [24:26:55<7:11:07,  2.05s/it]
Outer training loop:  75%|#######4  | 37409/50000 [24:26:57<6:53:34,  1.97s/it]
Outer training loop:  75%|#######4  | 37410/50000 [24:26:59<6:48:34,  1.95s/it]
Outer training loop:  75%|#######4  | 37411/50000 [24:27:01<6:41:35,  1.91s/it]
Outer training loop:  75%|#######4  | 37412/50000 [24:27:03<6:41:20,  1.91s/it]
Outer training loop:  75%|#######4  | 37413/50000 [24:27:04<6:34:22,  1.88s/it]
Outer training loop:  75%|#######4  | 37414/50000 [24:27:06<6:32:14,  1.87s/it]
Outer training loop:  75%|#######4  | 37415/50000 [24:27:08<6:40:14,  1.91s/it]
Outer training loop:  75%|#######4  | 37416/50000 [24:27:10<6:30:15,  1.86s/it]
Outer training loop:  75%|#######4  | 37417/50000 [24:27:12<6:30:37,  1.86s/it]
Outer training loop:  75%|#######4  | 37418/50000 [24:27:14<6:29:41,  1.86s/it]
Outer training loop:  75%|#######4  | 37419/50000 [24:27:16<6:40:22,  1.91s/it]
Outer training loop:  75%|#######4  | 37420/50000 [24:27:18<6:33:10,  1.88s/it]
Outer training loop:  75%|#######4  | 37421/50000 [24:27:19<6:34:27,  1.88s/it]
Outer training loop:  75%|#######4  | 37422/50000 [24:27:21<6:38:33,  1.90s/it]
Outer training loop:  75%|#######4  | 37423/50000 [24:27:23<6:36:18,  1.89s/it]
Outer training loop:  75%|#######4  | 37424/50000 [24:27:25<6:29:23,  1.86s/it]
Outer training loop:  75%|#######4  | 37425/50000 [24:27:27<6:22:50,  1.83s/it]
Outer training loop:  75%|#######4  | 37426/50000 [24:27:29<6:41:08,  1.91s/it]
Outer training loop:  75%|#######4  | 37427/50000 [24:27:31<6:44:24,  1.93s/it]
Outer training loop:  75%|#######4  | 37428/50000 [24:27:33<6:33:16,  1.88s/it]
Outer training loop:  75%|#######4  | 37429/50000 [24:27:35<6:38:00,  1.90s/it]
Outer training loop:  75%|#######4  | 37430/50000 [24:27:36<6:31:23,  1.87s/it]
Outer training loop:  75%|#######4  | 37431/50000 [24:27:38<6:21:43,  1.82s/it]
Outer training loop:  75%|#######4  | 37432/50000 [24:27:40<6:35:30,  1.89s/it]
Outer training loop:  75%|#######4  | 37433/50000 [24:27:42<6:22:17,  1.83s/it]
Outer training loop:  75%|#######4  | 37434/50000 [24:27:44<6:22:19,  1.83s/it]
Outer training loop:  75%|#######4  | 37435/50000 [24:27:45<6:21:16,  1.82s/it]
Outer training loop:  75%|#######4  | 37436/50000 [24:27:47<6:28:04,  1.85s/it]
Outer training loop:  75%|#######4  | 37437/50000 [24:27:49<6:28:48,  1.86s/it]
Outer training loop:  75%|#######4  | 37438/50000 [24:27:51<6:21:39,  1.82s/it]
Outer training loop:  75%|#######4  | 37439/50000 [24:27:53<6:36:53,  1.90s/it]
Outer training loop:  75%|#######4  | 37440/50000 [24:27:55<6:25:20,  1.84s/it]
Outer training loop:  75%|#######4  | 37441/50000 [24:27:57<6:24:20,  1.84s/it]
Outer training loop:  75%|#######4  | 37442/50000 [24:27:59<6:53:25,  1.98s/it]
Outer training loop:  75%|#######4  | 37443/50000 [24:28:01<6:46:54,  1.94s/it]
Outer training loop:  75%|#######4  | 37444/50000 [24:28:03<6:44:01,  1.93s/it]
Outer training loop:  75%|#######4  | 37445/50000 [24:28:05<6:38:30,  1.90s/it]
Outer training loop:  75%|#######4  | 37446/50000 [24:28:07<6:49:49,  1.96s/it]
Outer training loop:  75%|#######4  | 37447/50000 [24:28:09<6:51:34,  1.97s/it]
Outer training loop:  75%|#######4  | 37448/50000 [24:28:11<6:51:02,  1.96s/it]
Outer training loop:  75%|#######4  | 37449/50000 [24:28:13<6:58:39,  2.00s/it]
Outer training loop:  75%|#######4  | 37450/50000 [24:28:14<6:49:58,  1.96s/it]
Outer training loop:  75%|#######4  | 37451/50000 [24:28:16<6:46:50,  1.95s/it]
Outer training loop:  75%|#######4  | 37452/50000 [24:28:18<6:41:43,  1.92s/it]
Outer training loop:  75%|#######4  | 37453/50000 [24:28:20<6:53:31,  1.98s/it]
Outer training loop:  75%|#######4  | 37454/50000 [24:28:22<6:41:21,  1.92s/it]
Outer training loop:  75%|#######4  | 37455/50000 [24:28:24<6:33:41,  1.88s/it]
Outer training loop:  75%|#######4  | 37456/50000 [24:28:26<6:40:41,  1.92s/it]
Outer training loop:  75%|#######4  | 37457/50000 [24:28:28<6:35:37,  1.89s/it]
Outer training loop:  75%|#######4  | 37458/50000 [24:28:30<6:29:10,  1.86s/it]
Outer training loop:  75%|#######4  | 37459/50000 [24:28:31<6:20:09,  1.82s/it]
Outer training loop:  75%|#######4  | 37460/50000 [24:28:33<6:25:41,  1.85s/it]
Outer training loop:  75%|#######4  | 37461/50000 [24:28:35<6:33:27,  1.88s/it]
Outer training loop:  75%|#######4  | 37462/50000 [24:28:37<6:30:08,  1.87s/it]
Outer training loop:  75%|#######4  | 37463/50000 [24:28:39<6:26:14,  1.85s/it]
Outer training loop:  75%|#######4  | 37464/50000 [24:28:41<6:44:35,  1.94s/it]
Outer training loop:  75%|#######4  | 37465/50000 [24:28:43<6:48:38,  1.96s/it]
Outer training loop:  75%|#######4  | 37466/50000 [24:28:45<6:34:40,  1.89s/it]
Outer training loop:  75%|#######4  | 37467/50000 [24:28:47<6:39:37,  1.91s/it]
Outer training loop:  75%|#######4  | 37468/50000 [24:28:49<6:47:13,  1.95s/it]
Outer training loop:  75%|#######4  | 37469/50000 [24:28:51<6:41:43,  1.92s/it]
Outer training loop:  75%|#######4  | 37470/50000 [24:28:53<6:43:36,  1.93s/it]
Outer training loop:  75%|#######4  | 37471/50000 [24:28:54<6:43:53,  1.93s/it]
Outer training loop:  75%|#######4  | 37472/50000 [24:28:56<6:35:26,  1.89s/it]
Outer training loop:  75%|#######4  | 37473/50000 [24:28:58<6:55:52,  1.99s/it]
Outer training loop:  75%|#######4  | 37474/50000 [24:29:00<6:43:33,  1.93s/it]
Outer training loop:  75%|#######4  | 37475/50000 [24:29:02<6:44:53,  1.94s/it]
Outer training loop:  75%|#######4  | 37476/50000 [24:29:04<6:54:19,  1.98s/it]
Outer training loop:  75%|#######4  | 37477/50000 [24:29:06<6:46:29,  1.95s/it]
Outer training loop:  75%|#######4  | 37478/50000 [24:29:08<6:39:47,  1.92s/it]
Outer training loop:  75%|#######4  | 37479/50000 [24:29:10<6:43:30,  1.93s/it]
Outer training loop:  75%|#######4  | 37480/50000 [24:29:12<6:53:26,  1.98s/it]
Outer training loop:  75%|#######4  | 37481/50000 [24:29:14<6:45:43,  1.94s/it]
Outer training loop:  75%|#######4  | 37482/50000 [24:29:16<6:41:47,  1.93s/it]
Outer training loop:  75%|#######4  | 37483/50000 [24:29:18<6:56:05,  1.99s/it]
Outer training loop:  75%|#######4  | 37484/50000 [24:29:20<6:48:52,  1.96s/it]
Outer training loop:  75%|#######4  | 37485/50000 [24:29:22<6:42:45,  1.93s/it]
Outer training loop:  75%|#######4  | 37486/50000 [24:29:24<6:45:33,  1.94s/it]
Outer training loop:  75%|#######4  | 37487/50000 [24:29:26<6:41:33,  1.93s/it]
Outer training loop:  75%|#######4  | 37488/50000 [24:29:27<6:30:20,  1.87s/it]
Outer training loop:  75%|#######4  | 37489/50000 [24:29:29<6:29:22,  1.87s/it]
Outer training loop:  75%|#######4  | 37490/50000 [24:29:31<6:35:12,  1.90s/it]
Outer training loop:  75%|#######4  | 37491/50000 [24:29:33<6:25:04,  1.85s/it]
Outer training loop:  75%|#######4  | 37492/50000 [24:29:35<6:38:45,  1.91s/it]
Outer training loop:  75%|#######4  | 37493/50000 [24:29:37<6:53:35,  1.98s/it]
Outer training loop:  75%|#######4  | 37494/50000 [24:29:39<6:40:01,  1.92s/it]
Outer training loop:  75%|#######4  | 37495/50000 [24:29:41<6:45:29,  1.95s/it]
Outer training loop:  75%|#######4  | 37496/50000 [24:29:43<6:58:24,  2.01s/it]
Outer training loop:  75%|#######4  | 37497/50000 [24:29:45<6:44:22,  1.94s/it]
Outer training loop:  75%|#######4  | 37498/50000 [24:29:47<6:43:31,  1.94s/it]
Outer training loop:  75%|#######4  | 37499/50000 [24:29:49<6:53:02,  1.98s/it]
Outer training loop:  75%|#######5  | 37500/50000 [24:29:51<6:52:07,  1.98s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:31,  1.85s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:33,  1.89s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:32,  1.91s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.89s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:31,  1.96s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:25,  1.94s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:28,  2.00s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:21,  1.95s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:15,  1.91s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:16,  1.95s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:14,  1.95s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.85s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.93s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.93s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:57,  1.92s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<02:00,  2.01s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:54,  1.94s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:51,  1.92s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:43,  1.89s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.92s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:41,  1.92s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:41,  1.94s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:36,  1.96s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:33,  1.95s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:32,  1.96s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:28,  1.93s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:24,  1.92s/it]

Evaluating policy:  48%|####8     | 40/83 [01:17<01:23,  1.94s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:16,  1.92s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:11,  1.88s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:08,  1.85s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:10,  1.95s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:06,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:02,  1.85s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:57,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:54,  1.95s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:52,  1.94s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:49,  1.92s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:48,  1.92s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:44,  1.92s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:43,  2.06s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:01<00:39,  2.00s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:37,  1.96s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:34,  1.90s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:06<00:32,  1.91s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:30,  1.88s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:28,  1.87s/it]

Evaluating policy:  83%|########3 | 69/83 [02:12<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:26,  2.02s/it]

Evaluating policy:  86%|########5 | 71/83 [02:16<00:23,  1.94s/it]

Evaluating policy:  87%|########6 | 72/83 [02:18<00:20,  1.90s/it]

Evaluating policy:  88%|########7 | 73/83 [02:19<00:18,  1.87s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:17,  1.91s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:15,  1.92s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:11,  1.92s/it]

Evaluating policy:  94%|#########3| 78/83 [02:29<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:32<00:08,  2.10s/it]

Evaluating policy:  96%|#########6| 80/83 [02:33<00:06,  2.03s/it]

Evaluating policy:  98%|#########7| 81/83 [02:35<00:03,  1.96s/it]

Evaluating policy:  99%|#########8| 82/83 [02:37<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 83/83 [02:39<00:00,  1.89s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:29,  1.82s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:28,  1.83s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:27,  1.85s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:10<02:40,  2.06s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:35,  2.02s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:29,  1.97s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:26,  1.98s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:23,  1.96s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:16,  1.93s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:17,  1.96s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:13,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:09,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:31<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:33<02:05,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:35<02:08,  1.98s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:02,  1.92s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:58,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:58,  1.94s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:46,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:48,  1.93s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.91s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:38,  1.87s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:35,  1.84s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:36,  1.93s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:32,  1.89s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:27,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:25,  1.94s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:24,  1.95s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:19,  1.90s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:18,  1.91s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:11,  1.84s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:15,  1.98s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:11,  1.92s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:08,  1.89s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.86s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:03,  1.86s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<00:58,  1.84s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:56,  1.83s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:58,  1.94s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.88s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:53,  1.89s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.84s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:43,  1.97s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:40,  1.93s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.89s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.89s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:27,  1.98s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:25,  1.94s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.89s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.84s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.82s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:10,  1.81s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.83s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:08,  2.01s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.88s/it]

Outer training loop:  75%|#######5  | 37501/50000 [24:35:15<342:48:25, 98.74s/it]
Outer training loop:  75%|#######5  | 37502/50000 [24:35:17<241:55:44, 69.69s/it]
Outer training loop:  75%|#######5  | 37503/50000 [24:35:19<171:10:15, 49.31s/it]
Outer training loop:  75%|#######5  | 37504/50000 [24:35:21<121:43:29, 35.07s/it]
Outer training loop:  75%|#######5  | 37505/50000 [24:35:23<87:02:45, 25.08s/it] 
Outer training loop:  75%|#######5  | 37506/50000 [24:35:25<63:23:07, 18.26s/it]
Outer training loop:  75%|#######5  | 37507/50000 [24:35:27<46:17:20, 13.34s/it]
Outer training loop:  75%|#######5  | 37508/50000 [24:35:29<34:20:59,  9.90s/it]
Outer training loop:  75%|#######5  | 37509/50000 [24:35:30<25:53:10,  7.46s/it]
Outer training loop:  75%|#######5  | 37510/50000 [24:35:32<20:09:09,  5.81s/it]
Outer training loop:  75%|#######5  | 37511/50000 [24:35:34<16:00:34,  4.61s/it]
Outer training loop:  75%|#######5  | 37512/50000 [24:35:36<13:09:19,  3.79s/it]
Outer training loop:  75%|#######5  | 37513/50000 [24:35:38<11:05:49,  3.20s/it]
Outer training loop:  75%|#######5  | 37514/50000 [24:35:40<9:54:51,  2.86s/it] 
Outer training loop:  75%|#######5  | 37515/50000 [24:35:42<8:44:48,  2.52s/it]
Outer training loop:  75%|#######5  | 37516/50000 [24:35:43<7:57:58,  2.30s/it]
Outer training loop:  75%|#######5  | 37517/50000 [24:35:45<7:34:05,  2.18s/it]
Outer training loop:  75%|#######5  | 37518/50000 [24:35:47<7:07:15,  2.05s/it]
Outer training loop:  75%|#######5  | 37519/50000 [24:35:49<7:00:06,  2.02s/it]
Outer training loop:  75%|#######5  | 37520/50000 [24:35:51<7:05:30,  2.05s/it]
Outer training loop:  75%|#######5  | 37521/50000 [24:35:53<6:52:35,  1.98s/it]
Outer training loop:  75%|#######5  | 37522/50000 [24:35:55<6:46:13,  1.95s/it]
Outer training loop:  75%|#######5  | 37523/50000 [24:35:57<6:45:06,  1.95s/it]
Outer training loop:  75%|#######5  | 37524/50000 [24:35:59<6:56:08,  2.00s/it]
Outer training loop:  75%|#######5  | 37525/50000 [24:36:01<6:40:24,  1.93s/it]
Outer training loop:  75%|#######5  | 37526/50000 [24:36:03<6:32:52,  1.89s/it]
Outer training loop:  75%|#######5  | 37527/50000 [24:36:04<6:37:41,  1.91s/it]
Outer training loop:  75%|#######5  | 37528/50000 [24:36:06<6:37:18,  1.91s/it]
Outer training loop:  75%|#######5  | 37529/50000 [24:36:08<6:34:04,  1.90s/it]
Outer training loop:  75%|#######5  | 37530/50000 [24:36:10<6:50:15,  1.97s/it]
Outer training loop:  75%|#######5  | 37531/50000 [24:36:12<6:42:52,  1.94s/it]
Outer training loop:  75%|#######5  | 37532/50000 [24:36:14<6:33:13,  1.89s/it]
Outer training loop:  75%|#######5  | 37533/50000 [24:36:16<6:28:04,  1.87s/it]
Outer training loop:  75%|#######5  | 37534/50000 [24:36:18<6:42:27,  1.94s/it]
Outer training loop:  75%|#######5  | 37535/50000 [24:36:20<6:37:34,  1.91s/it]
Outer training loop:  75%|#######5  | 37536/50000 [24:36:22<6:32:31,  1.89s/it]
Outer training loop:  75%|#######5  | 37537/50000 [24:36:24<6:43:54,  1.94s/it]
Outer training loop:  75%|#######5  | 37538/50000 [24:36:26<6:44:33,  1.95s/it]
Outer training loop:  75%|#######5  | 37539/50000 [24:36:28<6:46:38,  1.96s/it]
Outer training loop:  75%|#######5  | 37540/50000 [24:36:29<6:34:54,  1.90s/it]
Outer training loop:  75%|#######5  | 37541/50000 [24:36:31<6:42:57,  1.94s/it]
Outer training loop:  75%|#######5  | 37542/50000 [24:36:33<6:41:04,  1.93s/it]
Outer training loop:  75%|#######5  | 37543/50000 [24:36:35<6:41:06,  1.93s/it]
Outer training loop:  75%|#######5  | 37544/50000 [24:36:37<6:40:40,  1.93s/it]
Outer training loop:  75%|#######5  | 37545/50000 [24:36:39<6:34:07,  1.90s/it]
Outer training loop:  75%|#######5  | 37546/50000 [24:36:41<6:25:26,  1.86s/it]
Outer training loop:  75%|#######5  | 37547/50000 [24:36:43<6:22:17,  1.84s/it]
Outer training loop:  75%|#######5  | 37548/50000 [24:36:45<6:31:05,  1.88s/it]
Outer training loop:  75%|#######5  | 37549/50000 [24:36:47<6:32:48,  1.89s/it]
Outer training loop:  75%|#######5  | 37550/50000 [24:36:48<6:33:44,  1.90s/it]
Outer training loop:  75%|#######5  | 37551/50000 [24:36:50<6:39:15,  1.92s/it]
Outer training loop:  75%|#######5  | 37552/50000 [24:36:52<6:40:27,  1.93s/it]
Outer training loop:  75%|#######5  | 37553/50000 [24:36:54<6:35:40,  1.91s/it]
Outer training loop:  75%|#######5  | 37554/50000 [24:36:56<6:44:33,  1.95s/it]
Outer training loop:  75%|#######5  | 37555/50000 [24:36:58<6:35:40,  1.91s/it]
Outer training loop:  75%|#######5  | 37556/50000 [24:37:00<6:33:22,  1.90s/it]
Outer training loop:  75%|#######5  | 37557/50000 [24:37:02<6:32:52,  1.89s/it]
Outer training loop:  75%|#######5  | 37558/50000 [24:37:04<6:36:02,  1.91s/it]
Outer training loop:  75%|#######5  | 37559/50000 [24:37:06<6:29:28,  1.88s/it]
Outer training loop:  75%|#######5  | 37560/50000 [24:37:07<6:23:06,  1.85s/it]
Outer training loop:  75%|#######5  | 37561/50000 [24:37:09<6:31:26,  1.89s/it]
Outer training loop:  75%|#######5  | 37562/50000 [24:37:11<6:24:29,  1.85s/it]
Outer training loop:  75%|#######5  | 37563/50000 [24:37:13<6:21:14,  1.84s/it]
Outer training loop:  75%|#######5  | 37564/50000 [24:37:15<6:29:08,  1.88s/it]
Outer training loop:  75%|#######5  | 37565/50000 [24:37:17<6:25:42,  1.86s/it]
Outer training loop:  75%|#######5  | 37566/50000 [24:37:19<6:23:34,  1.85s/it]
Outer training loop:  75%|#######5  | 37567/50000 [24:37:20<6:22:40,  1.85s/it]
Outer training loop:  75%|#######5  | 37568/50000 [24:37:22<6:33:36,  1.90s/it]
Outer training loop:  75%|#######5  | 37569/50000 [24:37:24<6:36:44,  1.91s/it]
Outer training loop:  75%|#######5  | 37570/50000 [24:37:26<6:36:38,  1.91s/it]
Outer training loop:  75%|#######5  | 37571/50000 [24:37:28<6:43:34,  1.95s/it]
Outer training loop:  75%|#######5  | 37572/50000 [24:37:30<6:23:44,  1.85s/it]
Outer training loop:  75%|#######5  | 37573/50000 [24:37:32<6:34:34,  1.91s/it]
Outer training loop:  75%|#######5  | 37574/50000 [24:37:34<6:44:52,  1.95s/it]
Outer training loop:  75%|#######5  | 37575/50000 [24:37:36<6:28:17,  1.88s/it]
Outer training loop:  75%|#######5  | 37576/50000 [24:37:37<6:21:06,  1.84s/it]
Outer training loop:  75%|#######5  | 37577/50000 [24:37:39<6:13:25,  1.80s/it]
Outer training loop:  75%|#######5  | 37578/50000 [24:37:41<6:26:14,  1.87s/it]
Outer training loop:  75%|#######5  | 37579/50000 [24:37:43<6:24:37,  1.86s/it]
Outer training loop:  75%|#######5  | 37580/50000 [24:37:45<6:20:35,  1.84s/it]
Outer training loop:  75%|#######5  | 37581/50000 [24:37:47<6:40:41,  1.94s/it]
Outer training loop:  75%|#######5  | 37582/50000 [24:37:49<6:37:47,  1.92s/it]
Outer training loop:  75%|#######5  | 37583/50000 [24:37:51<6:23:36,  1.85s/it]
Outer training loop:  75%|#######5  | 37584/50000 [24:37:53<6:38:47,  1.93s/it]
Outer training loop:  75%|#######5  | 37585/50000 [24:37:54<6:31:52,  1.89s/it]
Outer training loop:  75%|#######5  | 37586/50000 [24:37:56<6:24:02,  1.86s/it]
Outer training loop:  75%|#######5  | 37587/50000 [24:37:58<6:21:09,  1.84s/it]
Outer training loop:  75%|#######5  | 37588/50000 [24:38:00<6:36:14,  1.92s/it]
Outer training loop:  75%|#######5  | 37589/50000 [24:38:02<6:24:24,  1.86s/it]
Outer training loop:  75%|#######5  | 37590/50000 [24:38:04<6:24:17,  1.86s/it]
Outer training loop:  75%|#######5  | 37591/50000 [24:38:06<6:39:13,  1.93s/it]
Outer training loop:  75%|#######5  | 37592/50000 [24:38:08<6:41:39,  1.94s/it]
Outer training loop:  75%|#######5  | 37593/50000 [24:38:10<6:41:47,  1.94s/it]
Outer training loop:  75%|#######5  | 37594/50000 [24:38:12<6:34:08,  1.91s/it]
Outer training loop:  75%|#######5  | 37595/50000 [24:38:14<6:49:07,  1.98s/it]
Outer training loop:  75%|#######5  | 37596/50000 [24:38:16<6:36:50,  1.92s/it]
Outer training loop:  75%|#######5  | 37597/50000 [24:38:17<6:32:15,  1.90s/it]
Outer training loop:  75%|#######5  | 37598/50000 [24:38:19<6:41:52,  1.94s/it]
Outer training loop:  75%|#######5  | 37599/50000 [24:38:21<6:36:50,  1.92s/it]
Outer training loop:  75%|#######5  | 37600/50000 [24:38:23<6:27:13,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  75%|#######5  | 37601/50000 [24:38:30<12:12:53,  3.55s/it]
Outer training loop:  75%|#######5  | 37602/50000 [24:38:33<10:46:45,  3.13s/it]
Outer training loop:  75%|#######5  | 37603/50000 [24:38:34<9:20:32,  2.71s/it] 
Outer training loop:  75%|#######5  | 37604/50000 [24:38:36<8:31:14,  2.47s/it]
Outer training loop:  75%|#######5  | 37605/50000 [24:38:38<8:04:27,  2.35s/it]
Outer training loop:  75%|#######5  | 37606/50000 [24:38:40<7:35:15,  2.20s/it]
Outer training loop:  75%|#######5  | 37607/50000 [24:38:42<7:08:46,  2.08s/it]
Outer training loop:  75%|#######5  | 37608/50000 [24:38:44<7:02:41,  2.05s/it]
Outer training loop:  75%|#######5  | 37609/50000 [24:38:46<6:54:07,  2.01s/it]
Outer training loop:  75%|#######5  | 37610/50000 [24:38:48<6:52:33,  2.00s/it]
Outer training loop:  75%|#######5  | 37611/50000 [24:38:50<6:54:14,  2.01s/it]
Outer training loop:  75%|#######5  | 37612/50000 [24:38:52<6:55:15,  2.01s/it]
Outer training loop:  75%|#######5  | 37613/50000 [24:38:54<6:47:23,  1.97s/it]
Outer training loop:  75%|#######5  | 37614/50000 [24:38:56<6:40:12,  1.94s/it]
Outer training loop:  75%|#######5  | 37615/50000 [24:38:57<6:28:20,  1.88s/it]
Outer training loop:  75%|#######5  | 37616/50000 [24:38:59<6:40:16,  1.94s/it]
Outer training loop:  75%|#######5  | 37617/50000 [24:39:01<6:34:26,  1.91s/it]
Outer training loop:  75%|#######5  | 37618/50000 [24:39:03<6:30:52,  1.89s/it]
Outer training loop:  75%|#######5  | 37619/50000 [24:39:05<6:47:08,  1.97s/it]
Outer training loop:  75%|#######5  | 37620/50000 [24:39:07<6:54:48,  2.01s/it]
Outer training loop:  75%|#######5  | 37621/50000 [24:39:09<6:42:44,  1.95s/it]
Outer training loop:  75%|#######5  | 37622/50000 [24:39:11<6:49:49,  1.99s/it]
Outer training loop:  75%|#######5  | 37623/50000 [24:39:13<6:43:27,  1.96s/it]
Outer training loop:  75%|#######5  | 37624/50000 [24:39:15<6:35:36,  1.92s/it]
Outer training loop:  75%|#######5  | 37625/50000 [24:39:17<6:28:06,  1.88s/it]
Outer training loop:  75%|#######5  | 37626/50000 [24:39:19<6:39:54,  1.94s/it]
Outer training loop:  75%|#######5  | 37627/50000 [24:39:21<6:31:13,  1.90s/it]
Outer training loop:  75%|#######5  | 37628/50000 [24:39:22<6:19:27,  1.84s/it]
Outer training loop:  75%|#######5  | 37629/50000 [24:39:24<6:33:01,  1.91s/it]
Outer training loop:  75%|#######5  | 37630/50000 [24:39:26<6:34:59,  1.92s/it]
Outer training loop:  75%|#######5  | 37631/50000 [24:39:28<6:33:14,  1.91s/it]
Outer training loop:  75%|#######5  | 37632/50000 [24:39:30<6:51:12,  1.99s/it]
Outer training loop:  75%|#######5  | 37633/50000 [24:39:32<6:48:06,  1.98s/it]
Outer training loop:  75%|#######5  | 37634/50000 [24:39:34<6:42:38,  1.95s/it]
Outer training loop:  75%|#######5  | 37635/50000 [24:39:36<6:36:01,  1.92s/it]
Outer training loop:  75%|#######5  | 37636/50000 [24:39:38<6:41:29,  1.95s/it]
Outer training loop:  75%|#######5  | 37637/50000 [24:39:40<6:31:23,  1.90s/it]
Outer training loop:  75%|#######5  | 37638/50000 [24:39:42<6:23:58,  1.86s/it]
Outer training loop:  75%|#######5  | 37639/50000 [24:39:44<6:39:59,  1.94s/it]
Outer training loop:  75%|#######5  | 37640/50000 [24:39:46<6:31:21,  1.90s/it]
Outer training loop:  75%|#######5  | 37641/50000 [24:39:47<6:21:05,  1.85s/it]
Outer training loop:  75%|#######5  | 37642/50000 [24:39:49<6:27:35,  1.88s/it]
Outer training loop:  75%|#######5  | 37643/50000 [24:39:51<6:28:10,  1.88s/it]
Outer training loop:  75%|#######5  | 37644/50000 [24:39:53<6:19:07,  1.84s/it]
Outer training loop:  75%|#######5  | 37645/50000 [24:39:55<6:24:53,  1.87s/it]
Outer training loop:  75%|#######5  | 37646/50000 [24:39:57<6:28:03,  1.88s/it]
Outer training loop:  75%|#######5  | 37647/50000 [24:39:59<6:24:38,  1.87s/it]
Outer training loop:  75%|#######5  | 37648/50000 [24:40:00<6:19:56,  1.85s/it]
Outer training loop:  75%|#######5  | 37649/50000 [24:40:02<6:27:23,  1.88s/it]
Outer training loop:  75%|#######5  | 37650/50000 [24:40:04<6:25:19,  1.87s/it]
Outer training loop:  75%|#######5  | 37651/50000 [24:40:06<6:21:44,  1.85s/it]
Outer training loop:  75%|#######5  | 37652/50000 [24:40:08<6:29:27,  1.89s/it]
Outer training loop:  75%|#######5  | 37653/50000 [24:40:10<6:19:58,  1.85s/it]
Outer training loop:  75%|#######5  | 37654/50000 [24:40:12<6:17:23,  1.83s/it]
Outer training loop:  75%|#######5  | 37655/50000 [24:40:13<6:18:16,  1.84s/it]
Outer training loop:  75%|#######5  | 37656/50000 [24:40:15<6:20:04,  1.85s/it]
Outer training loop:  75%|#######5  | 37657/50000 [24:40:17<6:18:49,  1.84s/it]
Outer training loop:  75%|#######5  | 37658/50000 [24:40:19<6:12:16,  1.81s/it]
Outer training loop:  75%|#######5  | 37659/50000 [24:40:21<6:22:19,  1.86s/it]
Outer training loop:  75%|#######5  | 37660/50000 [24:40:23<6:14:30,  1.82s/it]
Outer training loop:  75%|#######5  | 37661/50000 [24:40:24<6:14:38,  1.82s/it]
Outer training loop:  75%|#######5  | 37662/50000 [24:40:26<6:15:22,  1.83s/it]
Outer training loop:  75%|#######5  | 37663/50000 [24:40:28<6:26:47,  1.88s/it]
Outer training loop:  75%|#######5  | 37664/50000 [24:40:30<6:25:00,  1.87s/it]
Outer training loop:  75%|#######5  | 37665/50000 [24:40:32<6:11:18,  1.81s/it]
Outer training loop:  75%|#######5  | 37666/50000 [24:40:34<6:28:46,  1.89s/it]
Outer training loop:  75%|#######5  | 37667/50000 [24:40:36<6:25:23,  1.87s/it]
Outer training loop:  75%|#######5  | 37668/50000 [24:40:38<6:23:41,  1.87s/it]
Outer training loop:  75%|#######5  | 37669/50000 [24:40:40<6:40:59,  1.95s/it]
Outer training loop:  75%|#######5  | 37670/50000 [24:40:42<6:40:02,  1.95s/it]
Outer training loop:  75%|#######5  | 37671/50000 [24:40:43<6:33:07,  1.91s/it]
Outer training loop:  75%|#######5  | 37672/50000 [24:40:45<6:24:00,  1.87s/it]
Outer training loop:  75%|#######5  | 37673/50000 [24:40:47<6:32:59,  1.91s/it]
Outer training loop:  75%|#######5  | 37674/50000 [24:40:49<6:27:06,  1.88s/it]
Outer training loop:  75%|#######5  | 37675/50000 [24:40:51<6:22:29,  1.86s/it]
Outer training loop:  75%|#######5  | 37676/50000 [24:40:53<6:36:26,  1.93s/it]
Outer training loop:  75%|#######5  | 37677/50000 [24:40:55<6:31:29,  1.91s/it]
Outer training loop:  75%|#######5  | 37678/50000 [24:40:57<6:25:22,  1.88s/it]
Outer training loop:  75%|#######5  | 37679/50000 [24:40:59<6:30:44,  1.90s/it]
Outer training loop:  75%|#######5  | 37680/50000 [24:41:01<6:33:43,  1.92s/it]
Outer training loop:  75%|#######5  | 37681/50000 [24:41:02<6:25:43,  1.88s/it]
Outer training loop:  75%|#######5  | 37682/50000 [24:41:04<6:27:26,  1.89s/it]
Outer training loop:  75%|#######5  | 37683/50000 [24:41:06<6:43:10,  1.96s/it]
Outer training loop:  75%|#######5  | 37684/50000 [24:41:08<6:28:46,  1.89s/it]
Outer training loop:  75%|#######5  | 37685/50000 [24:41:10<6:28:38,  1.89s/it]
Outer training loop:  75%|#######5  | 37686/50000 [24:41:12<6:25:13,  1.88s/it]
Outer training loop:  75%|#######5  | 37687/50000 [24:41:14<6:38:13,  1.94s/it]
Outer training loop:  75%|#######5  | 37688/50000 [24:41:16<6:30:00,  1.90s/it]
Outer training loop:  75%|#######5  | 37689/50000 [24:41:18<6:24:50,  1.88s/it]
Outer training loop:  75%|#######5  | 37690/50000 [24:41:20<6:38:46,  1.94s/it]
Outer training loop:  75%|#######5  | 37691/50000 [24:41:21<6:24:18,  1.87s/it]
Outer training loop:  75%|#######5  | 37692/50000 [24:41:23<6:20:13,  1.85s/it]
Outer training loop:  75%|#######5  | 37693/50000 [24:41:25<6:33:49,  1.92s/it]
Outer training loop:  75%|#######5  | 37694/50000 [24:41:27<6:32:16,  1.91s/it]
Outer training loop:  75%|#######5  | 37695/50000 [24:41:29<6:30:50,  1.91s/it]
Outer training loop:  75%|#######5  | 37696/50000 [24:41:31<6:27:36,  1.89s/it]
Outer training loop:  75%|#######5  | 37697/50000 [24:41:33<6:35:57,  1.93s/it]
Outer training loop:  75%|#######5  | 37698/50000 [24:41:35<6:32:19,  1.91s/it]
Outer training loop:  75%|#######5  | 37699/50000 [24:41:37<6:26:01,  1.88s/it]
Outer training loop:  75%|#######5  | 37700/50000 [24:41:39<6:30:50,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  75%|#######5  | 37701/50000 [24:41:46<12:04:18,  3.53s/it]
Outer training loop:  75%|#######5  | 37702/50000 [24:41:48<10:20:12,  3.03s/it]
Outer training loop:  75%|#######5  | 37703/50000 [24:41:50<9:06:21,  2.67s/it] 
Outer training loop:  75%|#######5  | 37704/50000 [24:41:51<8:11:04,  2.40s/it]
Outer training loop:  75%|#######5  | 37705/50000 [24:41:53<7:52:50,  2.31s/it]
Outer training loop:  75%|#######5  | 37706/50000 [24:41:55<7:29:29,  2.19s/it]
Outer training loop:  75%|#######5  | 37707/50000 [24:41:57<7:04:29,  2.07s/it]
Outer training loop:  75%|#######5  | 37708/50000 [24:41:59<7:02:33,  2.06s/it]
Outer training loop:  75%|#######5  | 37709/50000 [24:42:01<6:47:09,  1.99s/it]
Outer training loop:  75%|#######5  | 37710/50000 [24:42:03<6:35:16,  1.93s/it]
Outer training loop:  75%|#######5  | 37711/50000 [24:42:05<6:39:40,  1.95s/it]
Outer training loop:  75%|#######5  | 37712/50000 [24:42:07<6:38:21,  1.95s/it]
Outer training loop:  75%|#######5  | 37713/50000 [24:42:09<6:28:22,  1.90s/it]
Outer training loop:  75%|#######5  | 37714/50000 [24:42:11<6:34:36,  1.93s/it]
Outer training loop:  75%|#######5  | 37715/50000 [24:42:12<6:29:20,  1.90s/it]
Outer training loop:  75%|#######5  | 37716/50000 [24:42:14<6:17:32,  1.84s/it]
Outer training loop:  75%|#######5  | 37717/50000 [24:42:16<6:28:19,  1.90s/it]
Outer training loop:  75%|#######5  | 37718/50000 [24:42:18<6:36:48,  1.94s/it]
Outer training loop:  75%|#######5  | 37719/50000 [24:42:20<6:37:16,  1.94s/it]
Outer training loop:  75%|#######5  | 37720/50000 [24:42:22<6:25:33,  1.88s/it]
Outer training loop:  75%|#######5  | 37721/50000 [24:42:24<6:35:41,  1.93s/it]
Outer training loop:  75%|#######5  | 37722/50000 [24:42:26<6:30:17,  1.91s/it]
Outer training loop:  75%|#######5  | 37723/50000 [24:42:28<6:27:13,  1.89s/it]
Outer training loop:  75%|#######5  | 37724/50000 [24:42:30<6:40:13,  1.96s/it]
Outer training loop:  75%|#######5  | 37725/50000 [24:42:32<6:39:06,  1.95s/it]
Outer training loop:  75%|#######5  | 37726/50000 [24:42:33<6:33:34,  1.92s/it]
Outer training loop:  75%|#######5  | 37727/50000 [24:42:35<6:29:14,  1.90s/it]
Outer training loop:  75%|#######5  | 37728/50000 [24:42:37<6:40:04,  1.96s/it]
Outer training loop:  75%|#######5  | 37729/50000 [24:42:39<6:27:05,  1.89s/it]
Outer training loop:  75%|#######5  | 37730/50000 [24:42:41<6:21:39,  1.87s/it]
Outer training loop:  75%|#######5  | 37731/50000 [24:42:43<6:29:08,  1.90s/it]
Outer training loop:  75%|#######5  | 37732/50000 [24:42:45<6:23:34,  1.88s/it]
Outer training loop:  75%|#######5  | 37733/50000 [24:42:47<6:20:43,  1.86s/it]
Outer training loop:  75%|#######5  | 37734/50000 [24:42:49<6:38:35,  1.95s/it]
Outer training loop:  75%|#######5  | 37735/50000 [24:42:51<6:30:39,  1.91s/it]
Outer training loop:  75%|#######5  | 37736/50000 [24:42:52<6:24:51,  1.88s/it]
Outer training loop:  75%|#######5  | 37737/50000 [24:42:55<6:44:46,  1.98s/it]
Outer training loop:  75%|#######5  | 37738/50000 [24:42:56<6:37:42,  1.95s/it]
Outer training loop:  75%|#######5  | 37739/50000 [24:42:58<6:33:41,  1.93s/it]
Outer training loop:  75%|#######5  | 37740/50000 [24:43:00<6:21:05,  1.87s/it]
Outer training loop:  75%|#######5  | 37741/50000 [24:43:02<6:34:30,  1.93s/it]
Outer training loop:  75%|#######5  | 37742/50000 [24:43:04<6:26:47,  1.89s/it]
Outer training loop:  75%|#######5  | 37743/50000 [24:43:06<6:21:57,  1.87s/it]
Outer training loop:  75%|#######5  | 37744/50000 [24:43:08<6:34:53,  1.93s/it]
Outer training loop:  75%|#######5  | 37745/50000 [24:43:10<6:28:02,  1.90s/it]
Outer training loop:  75%|#######5  | 37746/50000 [24:43:11<6:18:25,  1.85s/it]
Outer training loop:  75%|#######5  | 37747/50000 [24:43:13<6:13:43,  1.83s/it]
Outer training loop:  75%|#######5  | 37748/50000 [24:43:15<6:33:25,  1.93s/it]
Outer training loop:  75%|#######5  | 37749/50000 [24:43:17<6:31:03,  1.92s/it]
Outer training loop:  76%|#######5  | 37750/50000 [24:43:19<6:29:39,  1.91s/it]
Outer training loop:  76%|#######5  | 37751/50000 [24:43:21<6:36:50,  1.94s/it]
Outer training loop:  76%|#######5  | 37752/50000 [24:43:23<6:26:33,  1.89s/it]
Outer training loop:  76%|#######5  | 37753/50000 [24:43:25<6:15:15,  1.84s/it]
Outer training loop:  76%|#######5  | 37754/50000 [24:43:27<6:20:17,  1.86s/it]
Outer training loop:  76%|#######5  | 37755/50000 [24:43:28<6:17:00,  1.85s/it]
Outer training loop:  76%|#######5  | 37756/50000 [24:43:30<6:09:16,  1.81s/it]
Outer training loop:  76%|#######5  | 37757/50000 [24:43:32<6:07:45,  1.80s/it]
Outer training loop:  76%|#######5  | 37758/50000 [24:43:34<6:20:32,  1.87s/it]
Outer training loop:  76%|#######5  | 37759/50000 [24:43:36<6:18:30,  1.86s/it]
Outer training loop:  76%|#######5  | 37760/50000 [24:43:37<6:13:49,  1.83s/it]
Outer training loop:  76%|#######5  | 37761/50000 [24:43:40<6:29:08,  1.91s/it]
Outer training loop:  76%|#######5  | 37762/50000 [24:43:41<6:18:55,  1.86s/it]
Outer training loop:  76%|#######5  | 37763/50000 [24:43:43<6:15:32,  1.84s/it]
Outer training loop:  76%|#######5  | 37764/50000 [24:43:45<6:27:54,  1.90s/it]
Outer training loop:  76%|#######5  | 37765/50000 [24:43:47<6:20:40,  1.87s/it]
Outer training loop:  76%|#######5  | 37766/50000 [24:43:49<6:19:36,  1.86s/it]
Outer training loop:  76%|#######5  | 37767/50000 [24:43:51<6:30:53,  1.92s/it]
Outer training loop:  76%|#######5  | 37768/50000 [24:43:53<6:35:19,  1.94s/it]
Outer training loop:  76%|#######5  | 37769/50000 [24:43:55<6:31:19,  1.92s/it]
Outer training loop:  76%|#######5  | 37770/50000 [24:43:56<6:20:50,  1.87s/it]
Outer training loop:  76%|#######5  | 37771/50000 [24:43:58<6:31:18,  1.92s/it]
Outer training loop:  76%|#######5  | 37772/50000 [24:44:00<6:26:08,  1.89s/it]
Outer training loop:  76%|#######5  | 37773/50000 [24:44:02<6:22:02,  1.87s/it]
Outer training loop:  76%|#######5  | 37774/50000 [24:44:04<6:34:07,  1.93s/it]
Outer training loop:  76%|#######5  | 37775/50000 [24:44:06<6:28:49,  1.91s/it]
Outer training loop:  76%|#######5  | 37776/50000 [24:44:08<6:25:36,  1.89s/it]
Outer training loop:  76%|#######5  | 37777/50000 [24:44:10<6:19:33,  1.86s/it]
Outer training loop:  76%|#######5  | 37778/50000 [24:44:12<6:26:18,  1.90s/it]
Outer training loop:  76%|#######5  | 37779/50000 [24:44:13<6:19:47,  1.86s/it]
Outer training loop:  76%|#######5  | 37780/50000 [24:44:15<6:22:32,  1.88s/it]
Outer training loop:  76%|#######5  | 37781/50000 [24:44:17<6:33:26,  1.93s/it]
Outer training loop:  76%|#######5  | 37782/50000 [24:44:19<6:25:40,  1.89s/it]
Outer training loop:  76%|#######5  | 37783/50000 [24:44:21<6:23:30,  1.88s/it]
Outer training loop:  76%|#######5  | 37784/50000 [24:44:23<6:17:00,  1.85s/it]
Outer training loop:  76%|#######5  | 37785/50000 [24:44:25<6:34:26,  1.94s/it]
Outer training loop:  76%|#######5  | 37786/50000 [24:44:27<6:34:47,  1.94s/it]
Outer training loop:  76%|#######5  | 37787/50000 [24:44:29<6:29:41,  1.91s/it]
Outer training loop:  76%|#######5  | 37788/50000 [24:44:31<6:32:54,  1.93s/it]
Outer training loop:  76%|#######5  | 37789/50000 [24:44:33<6:29:37,  1.91s/it]
Outer training loop:  76%|#######5  | 37790/50000 [24:44:35<6:33:33,  1.93s/it]
Outer training loop:  76%|#######5  | 37791/50000 [24:44:37<6:47:10,  2.00s/it]
Outer training loop:  76%|#######5  | 37792/50000 [24:44:39<6:35:45,  1.95s/it]
Outer training loop:  76%|#######5  | 37793/50000 [24:44:41<6:30:59,  1.92s/it]
Outer training loop:  76%|#######5  | 37794/50000 [24:44:43<6:41:57,  1.98s/it]
Outer training loop:  76%|#######5  | 37795/50000 [24:44:44<6:34:56,  1.94s/it]
Outer training loop:  76%|#######5  | 37796/50000 [24:44:46<6:25:33,  1.90s/it]
Outer training loop:  76%|#######5  | 37797/50000 [24:44:48<6:38:43,  1.96s/it]
Outer training loop:  76%|#######5  | 37798/50000 [24:44:50<6:37:45,  1.96s/it]
Outer training loop:  76%|#######5  | 37799/50000 [24:44:52<6:23:38,  1.89s/it]
Outer training loop:  76%|#######5  | 37800/50000 [24:44:54<6:22:40,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.82s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  76%|#######5  | 37801/50000 [24:45:02<12:13:05,  3.61s/it]
Outer training loop:  76%|#######5  | 37802/50000 [24:45:04<10:32:56,  3.11s/it]
Outer training loop:  76%|#######5  | 37803/50000 [24:45:05<9:19:05,  2.75s/it] 
Outer training loop:  76%|#######5  | 37804/50000 [24:45:07<8:18:02,  2.45s/it]
Outer training loop:  76%|#######5  | 37805/50000 [24:45:09<7:54:43,  2.34s/it]
Outer training loop:  76%|#######5  | 37806/50000 [24:45:11<7:21:17,  2.17s/it]
Outer training loop:  76%|#######5  | 37807/50000 [24:45:13<6:59:55,  2.07s/it]
Outer training loop:  76%|#######5  | 37808/50000 [24:45:15<6:55:05,  2.04s/it]
Outer training loop:  76%|#######5  | 37809/50000 [24:45:17<6:38:37,  1.96s/it]
Outer training loop:  76%|#######5  | 37810/50000 [24:45:18<6:33:53,  1.94s/it]
Outer training loop:  76%|#######5  | 37811/50000 [24:45:21<6:45:13,  1.99s/it]
Outer training loop:  76%|#######5  | 37812/50000 [24:45:22<6:35:51,  1.95s/it]
Outer training loop:  76%|#######5  | 37813/50000 [24:45:24<6:32:55,  1.93s/it]
Outer training loop:  76%|#######5  | 37814/50000 [24:45:26<6:38:36,  1.96s/it]
Outer training loop:  76%|#######5  | 37815/50000 [24:45:28<6:44:03,  1.99s/it]
Outer training loop:  76%|#######5  | 37816/50000 [24:45:30<6:29:10,  1.92s/it]
Outer training loop:  76%|#######5  | 37817/50000 [24:45:32<6:24:09,  1.89s/it]
Outer training loop:  76%|#######5  | 37818/50000 [24:45:34<6:36:18,  1.95s/it]
Outer training loop:  76%|#######5  | 37819/50000 [24:45:36<6:26:21,  1.90s/it]
Outer training loop:  76%|#######5  | 37820/50000 [24:45:38<6:24:40,  1.89s/it]
Outer training loop:  76%|#######5  | 37821/50000 [24:45:40<6:34:05,  1.94s/it]
Outer training loop:  76%|#######5  | 37822/50000 [24:45:42<6:43:15,  1.99s/it]
Outer training loop:  76%|#######5  | 37823/50000 [24:45:44<6:33:27,  1.94s/it]
Outer training loop:  76%|#######5  | 37824/50000 [24:45:45<6:21:19,  1.88s/it]
Outer training loop:  76%|#######5  | 37825/50000 [24:45:47<6:29:33,  1.92s/it]
Outer training loop:  76%|#######5  | 37826/50000 [24:45:49<6:21:50,  1.88s/it]
Outer training loop:  76%|#######5  | 37827/50000 [24:45:51<6:21:41,  1.88s/it]
Outer training loop:  76%|#######5  | 37828/50000 [24:45:53<6:20:04,  1.87s/it]
Outer training loop:  76%|#######5  | 37829/50000 [24:45:55<6:34:08,  1.94s/it]
Outer training loop:  76%|#######5  | 37830/50000 [24:45:57<6:29:06,  1.92s/it]
Outer training loop:  76%|#######5  | 37831/50000 [24:45:59<6:28:12,  1.91s/it]
Outer training loop:  76%|#######5  | 37832/50000 [24:46:01<6:41:05,  1.98s/it]
Outer training loop:  76%|#######5  | 37833/50000 [24:46:03<6:32:30,  1.94s/it]
Outer training loop:  76%|#######5  | 37834/50000 [24:46:05<6:27:34,  1.91s/it]
Outer training loop:  76%|#######5  | 37835/50000 [24:46:07<6:34:48,  1.95s/it]
Outer training loop:  76%|#######5  | 37836/50000 [24:46:09<6:34:17,  1.94s/it]
Outer training loop:  76%|#######5  | 37837/50000 [24:46:10<6:22:36,  1.89s/it]
Outer training loop:  76%|#######5  | 37838/50000 [24:46:12<6:23:32,  1.89s/it]
Outer training loop:  76%|#######5  | 37839/50000 [24:46:14<6:29:56,  1.92s/it]
Outer training loop:  76%|#######5  | 37840/50000 [24:46:16<6:19:11,  1.87s/it]
Outer training loop:  76%|#######5  | 37841/50000 [24:46:18<6:18:53,  1.87s/it]
Outer training loop:  76%|#######5  | 37842/50000 [24:46:20<6:30:50,  1.93s/it]
Outer training loop:  76%|#######5  | 37843/50000 [24:46:22<6:23:33,  1.89s/it]
Outer training loop:  76%|#######5  | 37844/50000 [24:46:24<6:20:28,  1.88s/it]
Outer training loop:  76%|#######5  | 37845/50000 [24:46:26<6:23:24,  1.89s/it]
Outer training loop:  76%|#######5  | 37846/50000 [24:46:28<6:30:47,  1.93s/it]
Outer training loop:  76%|#######5  | 37847/50000 [24:46:29<6:24:44,  1.90s/it]
Outer training loop:  76%|#######5  | 37848/50000 [24:46:31<6:14:45,  1.85s/it]
Outer training loop:  76%|#######5  | 37849/50000 [24:46:33<6:24:30,  1.90s/it]
Outer training loop:  76%|#######5  | 37850/50000 [24:46:35<6:28:30,  1.92s/it]
Outer training loop:  76%|#######5  | 37851/50000 [24:46:37<6:23:11,  1.89s/it]
Outer training loop:  76%|#######5  | 37852/50000 [24:46:39<6:27:41,  1.91s/it]
Outer training loop:  76%|#######5  | 37853/50000 [24:46:41<6:27:05,  1.91s/it]
Outer training loop:  76%|#######5  | 37854/50000 [24:46:43<6:24:45,  1.90s/it]
Outer training loop:  76%|#######5  | 37855/50000 [24:46:45<6:27:33,  1.91s/it]
Outer training loop:  76%|#######5  | 37856/50000 [24:46:47<6:23:45,  1.90s/it]
Outer training loop:  76%|#######5  | 37857/50000 [24:46:48<6:16:44,  1.86s/it]
Outer training loop:  76%|#######5  | 37858/50000 [24:46:50<6:17:09,  1.86s/it]
Outer training loop:  76%|#######5  | 37859/50000 [24:46:52<6:23:34,  1.90s/it]
Outer training loop:  76%|#######5  | 37860/50000 [24:46:54<6:15:26,  1.86s/it]
Outer training loop:  76%|#######5  | 37861/50000 [24:46:56<6:14:08,  1.85s/it]
Outer training loop:  76%|#######5  | 37862/50000 [24:46:58<6:31:38,  1.94s/it]
Outer training loop:  76%|#######5  | 37863/50000 [24:47:00<6:20:27,  1.88s/it]
Outer training loop:  76%|#######5  | 37864/50000 [24:47:02<6:26:01,  1.91s/it]
Outer training loop:  76%|#######5  | 37865/50000 [24:47:04<6:31:21,  1.94s/it]
Outer training loop:  76%|#######5  | 37866/50000 [24:47:06<6:30:45,  1.93s/it]
Outer training loop:  76%|#######5  | 37867/50000 [24:47:07<6:19:39,  1.88s/it]
Outer training loop:  76%|#######5  | 37868/50000 [24:47:09<6:29:13,  1.92s/it]
Outer training loop:  76%|#######5  | 37869/50000 [24:47:11<6:21:01,  1.88s/it]
Outer training loop:  76%|#######5  | 37870/50000 [24:47:13<6:29:38,  1.93s/it]
Outer training loop:  76%|#######5  | 37871/50000 [24:47:15<6:32:04,  1.94s/it]
Outer training loop:  76%|#######5  | 37872/50000 [24:47:17<6:17:55,  1.87s/it]
Outer training loop:  76%|#######5  | 37873/50000 [24:47:19<6:21:27,  1.89s/it]
Outer training loop:  76%|#######5  | 37874/50000 [24:47:21<6:24:34,  1.90s/it]
Outer training loop:  76%|#######5  | 37875/50000 [24:47:23<6:32:30,  1.94s/it]
Outer training loop:  76%|#######5  | 37876/50000 [24:47:24<6:22:14,  1.89s/it]
Outer training loop:  76%|#######5  | 37877/50000 [24:47:26<6:29:11,  1.93s/it]
Outer training loop:  76%|#######5  | 37878/50000 [24:47:29<6:37:32,  1.97s/it]
Outer training loop:  76%|#######5  | 37879/50000 [24:47:30<6:32:51,  1.94s/it]
Outer training loop:  76%|#######5  | 37880/50000 [24:47:32<6:18:16,  1.87s/it]
Outer training loop:  76%|#######5  | 37881/50000 [24:47:34<6:26:41,  1.91s/it]
Outer training loop:  76%|#######5  | 37882/50000 [24:47:36<6:28:18,  1.92s/it]
Outer training loop:  76%|#######5  | 37883/50000 [24:47:38<6:21:04,  1.89s/it]
Outer training loop:  76%|#######5  | 37884/50000 [24:47:40<6:15:11,  1.86s/it]
Outer training loop:  76%|#######5  | 37885/50000 [24:47:42<6:32:57,  1.95s/it]
Outer training loop:  76%|#######5  | 37886/50000 [24:47:44<6:22:09,  1.89s/it]
Outer training loop:  76%|#######5  | 37887/50000 [24:47:45<6:12:59,  1.85s/it]
Outer training loop:  76%|#######5  | 37888/50000 [24:47:47<6:02:01,  1.79s/it]
Outer training loop:  76%|#######5  | 37889/50000 [24:47:49<6:17:33,  1.87s/it]
Outer training loop:  76%|#######5  | 37890/50000 [24:47:51<6:20:47,  1.89s/it]
Outer training loop:  76%|#######5  | 37891/50000 [24:47:53<6:20:29,  1.89s/it]
Outer training loop:  76%|#######5  | 37892/50000 [24:47:55<6:27:57,  1.92s/it]
Outer training loop:  76%|#######5  | 37893/50000 [24:47:57<6:24:17,  1.90s/it]
Outer training loop:  76%|#######5  | 37894/50000 [24:47:59<6:22:07,  1.89s/it]
Outer training loop:  76%|#######5  | 37895/50000 [24:48:00<6:13:12,  1.85s/it]
Outer training loop:  76%|#######5  | 37896/50000 [24:48:02<6:25:22,  1.91s/it]
Outer training loop:  76%|#######5  | 37897/50000 [24:48:04<6:22:06,  1.89s/it]
Outer training loop:  76%|#######5  | 37898/50000 [24:48:06<6:18:18,  1.88s/it]
Outer training loop:  76%|#######5  | 37899/50000 [24:48:08<6:33:10,  1.95s/it]
Outer training loop:  76%|#######5  | 37900/50000 [24:48:10<6:22:58,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  76%|#######5  | 37901/50000 [24:48:17<11:58:17,  3.56s/it]
Outer training loop:  76%|#######5  | 37902/50000 [24:48:19<10:10:32,  3.03s/it]
Outer training loop:  76%|#######5  | 37903/50000 [24:48:21<9:05:40,  2.71s/it] 
Outer training loop:  76%|#######5  | 37904/50000 [24:48:23<8:14:04,  2.45s/it]
Outer training loop:  76%|#######5  | 37905/50000 [24:48:25<7:38:12,  2.27s/it]
Outer training loop:  76%|#######5  | 37906/50000 [24:48:27<7:23:05,  2.20s/it]
Outer training loop:  76%|#######5  | 37907/50000 [24:48:29<7:00:26,  2.09s/it]
Outer training loop:  76%|#######5  | 37908/50000 [24:48:31<6:44:33,  2.01s/it]
Outer training loop:  76%|#######5  | 37909/50000 [24:48:32<6:29:46,  1.93s/it]
Outer training loop:  76%|#######5  | 37910/50000 [24:48:34<6:42:50,  2.00s/it]
Outer training loop:  76%|#######5  | 37911/50000 [24:48:37<6:49:57,  2.03s/it]
Outer training loop:  76%|#######5  | 37912/50000 [24:48:39<6:42:42,  2.00s/it]
Outer training loop:  76%|#######5  | 37913/50000 [24:48:41<6:41:18,  1.99s/it]
Outer training loop:  76%|#######5  | 37914/50000 [24:48:42<6:32:01,  1.95s/it]
Outer training loop:  76%|#######5  | 37915/50000 [24:48:44<6:20:46,  1.89s/it]
Outer training loop:  76%|#######5  | 37916/50000 [24:48:46<6:19:45,  1.89s/it]
Outer training loop:  76%|#######5  | 37917/50000 [24:48:48<6:31:52,  1.95s/it]
Outer training loop:  76%|#######5  | 37918/50000 [24:48:50<6:22:15,  1.90s/it]
Outer training loop:  76%|#######5  | 37919/50000 [24:48:52<6:19:09,  1.88s/it]
Outer training loop:  76%|#######5  | 37920/50000 [24:48:54<6:25:34,  1.92s/it]
Outer training loop:  76%|#######5  | 37921/50000 [24:48:56<6:24:24,  1.91s/it]
Outer training loop:  76%|#######5  | 37922/50000 [24:48:57<6:09:59,  1.84s/it]
Outer training loop:  76%|#######5  | 37923/50000 [24:48:59<6:09:39,  1.84s/it]
Outer training loop:  76%|#######5  | 37924/50000 [24:49:01<6:20:20,  1.89s/it]
Outer training loop:  76%|#######5  | 37925/50000 [24:49:03<6:17:40,  1.88s/it]
Outer training loop:  76%|#######5  | 37926/50000 [24:49:05<6:19:47,  1.89s/it]
Outer training loop:  76%|#######5  | 37927/50000 [24:49:07<6:25:36,  1.92s/it]
Outer training loop:  76%|#######5  | 37928/50000 [24:49:09<6:19:12,  1.88s/it]
Outer training loop:  76%|#######5  | 37929/50000 [24:49:11<6:21:09,  1.89s/it]
Outer training loop:  76%|#######5  | 37930/50000 [24:49:13<6:30:10,  1.94s/it]
Outer training loop:  76%|#######5  | 37931/50000 [24:49:14<6:22:55,  1.90s/it]
Outer training loop:  76%|#######5  | 37932/50000 [24:49:16<6:18:46,  1.88s/it]
Outer training loop:  76%|#######5  | 37933/50000 [24:49:18<6:17:42,  1.88s/it]
Outer training loop:  76%|#######5  | 37934/50000 [24:49:20<6:30:00,  1.94s/it]
Outer training loop:  76%|#######5  | 37935/50000 [24:49:22<6:22:56,  1.90s/it]
Outer training loop:  76%|#######5  | 37936/50000 [24:49:24<6:18:06,  1.88s/it]
Outer training loop:  76%|#######5  | 37937/50000 [24:49:26<6:17:48,  1.88s/it]
Outer training loop:  76%|#######5  | 37938/50000 [24:49:28<6:28:39,  1.93s/it]
Outer training loop:  76%|#######5  | 37939/50000 [24:49:30<6:27:24,  1.93s/it]
Outer training loop:  76%|#######5  | 37940/50000 [24:49:32<6:19:51,  1.89s/it]
Outer training loop:  76%|#######5  | 37941/50000 [24:49:34<6:35:28,  1.97s/it]
Outer training loop:  76%|#######5  | 37942/50000 [24:49:36<6:27:14,  1.93s/it]
Outer training loop:  76%|#######5  | 37943/50000 [24:49:37<6:24:30,  1.91s/it]
Outer training loop:  76%|#######5  | 37944/50000 [24:49:39<6:19:50,  1.89s/it]
Outer training loop:  76%|#######5  | 37945/50000 [24:49:41<6:29:36,  1.94s/it]
Outer training loop:  76%|#######5  | 37946/50000 [24:49:43<6:24:42,  1.91s/it]
Outer training loop:  76%|#######5  | 37947/50000 [24:49:45<6:11:07,  1.85s/it]
Outer training loop:  76%|#######5  | 37948/50000 [24:49:47<6:20:53,  1.90s/it]
Outer training loop:  76%|#######5  | 37949/50000 [24:49:49<6:10:59,  1.85s/it]
Outer training loop:  76%|#######5  | 37950/50000 [24:49:50<6:14:12,  1.86s/it]
Outer training loop:  76%|#######5  | 37951/50000 [24:49:52<6:19:41,  1.89s/it]
Outer training loop:  76%|#######5  | 37952/50000 [24:49:54<6:11:37,  1.85s/it]
Outer training loop:  76%|#######5  | 37953/50000 [24:49:56<6:13:30,  1.86s/it]
Outer training loop:  76%|#######5  | 37954/50000 [24:49:58<6:25:33,  1.92s/it]
Outer training loop:  76%|#######5  | 37955/50000 [24:50:00<6:15:27,  1.87s/it]
Outer training loop:  76%|#######5  | 37956/50000 [24:50:02<6:03:26,  1.81s/it]
Outer training loop:  76%|#######5  | 37957/50000 [24:50:03<6:02:24,  1.81s/it]
Outer training loop:  76%|#######5  | 37958/50000 [24:50:05<6:16:13,  1.87s/it]
Outer training loop:  76%|#######5  | 37959/50000 [24:50:07<6:07:13,  1.83s/it]
Outer training loop:  76%|#######5  | 37960/50000 [24:50:09<6:07:18,  1.83s/it]
Outer training loop:  76%|#######5  | 37961/50000 [24:50:11<6:38:18,  1.99s/it]
Outer training loop:  76%|#######5  | 37962/50000 [24:50:13<6:33:30,  1.96s/it]
Outer training loop:  76%|#######5  | 37963/50000 [24:50:15<6:19:07,  1.89s/it]
Outer training loop:  76%|#######5  | 37964/50000 [24:50:17<6:25:27,  1.92s/it]
Outer training loop:  76%|#######5  | 37965/50000 [24:50:19<6:26:58,  1.93s/it]
Outer training loop:  76%|#######5  | 37966/50000 [24:50:21<6:27:37,  1.93s/it]
Outer training loop:  76%|#######5  | 37967/50000 [24:50:23<6:22:54,  1.91s/it]
Outer training loop:  76%|#######5  | 37968/50000 [24:50:25<6:31:39,  1.95s/it]
Outer training loop:  76%|#######5  | 37969/50000 [24:50:27<6:24:44,  1.92s/it]
Outer training loop:  76%|#######5  | 37970/50000 [24:50:28<6:19:51,  1.89s/it]
Outer training loop:  76%|#######5  | 37971/50000 [24:50:30<6:22:02,  1.91s/it]
Outer training loop:  76%|#######5  | 37972/50000 [24:50:32<6:19:02,  1.89s/it]
Outer training loop:  76%|#######5  | 37973/50000 [24:50:34<6:11:48,  1.85s/it]
Outer training loop:  76%|#######5  | 37974/50000 [24:50:36<6:25:13,  1.92s/it]
Outer training loop:  76%|#######5  | 37975/50000 [24:50:38<6:18:00,  1.89s/it]
Outer training loop:  76%|#######5  | 37976/50000 [24:50:40<6:23:47,  1.92s/it]
Outer training loop:  76%|#######5  | 37977/50000 [24:50:42<6:21:21,  1.90s/it]
Outer training loop:  76%|#######5  | 37978/50000 [24:50:44<6:33:01,  1.96s/it]
Outer training loop:  76%|#######5  | 37979/50000 [24:50:46<6:18:37,  1.89s/it]
Outer training loop:  76%|#######5  | 37980/50000 [24:50:47<6:19:29,  1.89s/it]
Outer training loop:  76%|#######5  | 37981/50000 [24:50:49<6:31:04,  1.95s/it]
Outer training loop:  76%|#######5  | 37982/50000 [24:50:51<6:26:06,  1.93s/it]
Outer training loop:  76%|#######5  | 37983/50000 [24:50:53<6:18:04,  1.89s/it]
Outer training loop:  76%|#######5  | 37984/50000 [24:50:55<6:15:01,  1.87s/it]
Outer training loop:  76%|#######5  | 37985/50000 [24:50:57<6:21:17,  1.90s/it]
Outer training loop:  76%|#######5  | 37986/50000 [24:50:59<6:21:11,  1.90s/it]
Outer training loop:  76%|#######5  | 37987/50000 [24:51:01<6:15:20,  1.87s/it]
Outer training loop:  76%|#######5  | 37988/50000 [24:51:03<6:20:17,  1.90s/it]
Outer training loop:  76%|#######5  | 37989/50000 [24:51:04<6:10:43,  1.85s/it]
Outer training loop:  76%|#######5  | 37990/50000 [24:51:06<6:11:43,  1.86s/it]
Outer training loop:  76%|#######5  | 37991/50000 [24:51:08<6:14:17,  1.87s/it]
Outer training loop:  76%|#######5  | 37992/50000 [24:51:10<6:17:10,  1.88s/it]
Outer training loop:  76%|#######5  | 37993/50000 [24:51:12<6:17:10,  1.88s/it]
Outer training loop:  76%|#######5  | 37994/50000 [24:51:14<6:20:18,  1.90s/it]
Outer training loop:  76%|#######5  | 37995/50000 [24:51:16<6:08:26,  1.84s/it]
Outer training loop:  76%|#######5  | 37996/50000 [24:51:17<6:08:47,  1.84s/it]
Outer training loop:  76%|#######5  | 37997/50000 [24:51:19<6:09:00,  1.84s/it]
Outer training loop:  76%|#######5  | 37998/50000 [24:51:21<6:21:33,  1.91s/it]
Outer training loop:  76%|#######5  | 37999/50000 [24:51:23<6:14:46,  1.87s/it]
Outer training loop:  76%|#######6  | 38000/50000 [24:51:25<6:17:32,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:46,  2.03s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:40,  1.98s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:35,  1.95s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:34,  1.96s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:29,  1.92s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.90s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.94s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:26,  1.95s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.96s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.97s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:15,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.93s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:58,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:03,  1.97s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.92s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.86s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.88s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.85s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:53,  1.96s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:44,  1.87s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.89s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:39,  1.92s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:35,  1.87s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:29,  1.82s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:26,  1.80s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.92s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:21,  1.82s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:17,  1.80s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:15,  1.80s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:14,  1.81s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:16,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:12,  1.87s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:09,  1.82s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:06,  1.79s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:03,  1.77s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:29<01:01,  1.76s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:00,  1.78s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<01:03,  1.91s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:57,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:54,  1.83s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.92s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:50,  2.02s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:47,  1.97s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:44,  1.94s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:42,  1.91s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.90s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:33,  1.97s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.92s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.91s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.86s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.83s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:16,  2.02s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.97s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.94s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.87s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.89s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.84s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.79s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.77s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:36,  1.93s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.90s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:22,  1.83s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:17,  1.84s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:14,  1.86s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:11,  1.85s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:10,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.91s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:08,  1.97s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.92s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:54,  1.85s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:46,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.94s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:42,  1.90s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:36,  1.96s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.86s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:20,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:24,  1.96s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.91s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.88s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:13,  1.98s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:09,  1.94s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:04,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.87s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.85s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:55,  1.91s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.83s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.83s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:41,  1.96s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:38,  1.91s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.88s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.85s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.82s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:28,  1.81s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:26,  1.80s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.81s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.98s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.92s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.86s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:17,  1.92s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.91s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.86s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.84s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.82s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:08,  2.02s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.93s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.95s/it]

Outer training loop:  76%|#######6  | 38001/50000 [24:56:45<324:45:36, 97.44s/it]
Outer training loop:  76%|#######6  | 38002/50000 [24:56:47<229:09:07, 68.76s/it]
Outer training loop:  76%|#######6  | 38003/50000 [24:56:49<162:21:52, 48.72s/it]
Outer training loop:  76%|#######6  | 38004/50000 [24:56:51<115:25:48, 34.64s/it]
Outer training loop:  76%|#######6  | 38005/50000 [24:56:53<82:38:24, 24.80s/it] 
Outer training loop:  76%|#######6  | 38006/50000 [24:56:55<59:43:54, 17.93s/it]
Outer training loop:  76%|#######6  | 38007/50000 [24:56:57<44:11:20, 13.26s/it]
Outer training loop:  76%|#######6  | 38008/50000 [24:56:59<32:42:37,  9.82s/it]
Outer training loop:  76%|#######6  | 38009/50000 [24:57:01<24:41:14,  7.41s/it]
Outer training loop:  76%|#######6  | 38010/50000 [24:57:03<19:04:10,  5.73s/it]
Outer training loop:  76%|#######6  | 38011/50000 [24:57:05<15:19:51,  4.60s/it]
Outer training loop:  76%|#######6  | 38012/50000 [24:57:06<12:31:58,  3.76s/it]
Outer training loop:  76%|#######6  | 38013/50000 [24:57:08<10:40:02,  3.20s/it]
Outer training loop:  76%|#######6  | 38014/50000 [24:57:10<9:32:12,  2.86s/it] 
Outer training loop:  76%|#######6  | 38015/50000 [24:57:12<8:27:22,  2.54s/it]
Outer training loop:  76%|#######6  | 38016/50000 [24:57:14<7:45:46,  2.33s/it]
Outer training loop:  76%|#######6  | 38017/50000 [24:57:16<7:14:27,  2.18s/it]
Outer training loop:  76%|#######6  | 38018/50000 [24:57:18<7:09:30,  2.15s/it]
Outer training loop:  76%|#######6  | 38019/50000 [24:57:20<6:52:15,  2.06s/it]
Outer training loop:  76%|#######6  | 38020/50000 [24:57:21<6:34:33,  1.98s/it]
Outer training loop:  76%|#######6  | 38021/50000 [24:57:23<6:34:36,  1.98s/it]
Outer training loop:  76%|#######6  | 38022/50000 [24:57:25<6:25:17,  1.93s/it]
Outer training loop:  76%|#######6  | 38023/50000 [24:57:27<6:17:37,  1.89s/it]
Outer training loop:  76%|#######6  | 38024/50000 [24:57:29<6:18:47,  1.90s/it]
Outer training loop:  76%|#######6  | 38025/50000 [24:57:31<6:23:03,  1.92s/it]
Outer training loop:  76%|#######6  | 38026/50000 [24:57:33<6:10:43,  1.86s/it]
Outer training loop:  76%|#######6  | 38027/50000 [24:57:35<6:10:52,  1.86s/it]
Outer training loop:  76%|#######6  | 38028/50000 [24:57:36<6:15:21,  1.88s/it]
Outer training loop:  76%|#######6  | 38029/50000 [24:57:38<6:14:47,  1.88s/it]
Outer training loop:  76%|#######6  | 38030/50000 [24:57:40<6:13:10,  1.87s/it]
Outer training loop:  76%|#######6  | 38031/50000 [24:57:42<6:33:54,  1.97s/it]
Outer training loop:  76%|#######6  | 38032/50000 [24:57:44<6:19:41,  1.90s/it]
Outer training loop:  76%|#######6  | 38033/50000 [24:57:46<6:12:07,  1.87s/it]
Outer training loop:  76%|#######6  | 38034/50000 [24:57:48<6:12:52,  1.87s/it]
Outer training loop:  76%|#######6  | 38035/50000 [24:57:50<6:20:23,  1.91s/it]
Outer training loop:  76%|#######6  | 38036/50000 [24:57:52<6:16:06,  1.89s/it]
Outer training loop:  76%|#######6  | 38037/50000 [24:57:53<6:09:30,  1.85s/it]
Outer training loop:  76%|#######6  | 38038/50000 [24:57:55<6:17:57,  1.90s/it]
Outer training loop:  76%|#######6  | 38039/50000 [24:57:57<6:13:11,  1.87s/it]
Outer training loop:  76%|#######6  | 38040/50000 [24:57:59<6:08:33,  1.85s/it]
Outer training loop:  76%|#######6  | 38041/50000 [24:58:01<6:08:46,  1.85s/it]
Outer training loop:  76%|#######6  | 38042/50000 [24:58:03<6:28:54,  1.95s/it]
Outer training loop:  76%|#######6  | 38043/50000 [24:58:05<6:27:47,  1.95s/it]
Outer training loop:  76%|#######6  | 38044/50000 [24:58:07<6:18:00,  1.90s/it]
Outer training loop:  76%|#######6  | 38045/50000 [24:58:09<6:22:31,  1.92s/it]
Outer training loop:  76%|#######6  | 38046/50000 [24:58:11<6:20:03,  1.91s/it]
Outer training loop:  76%|#######6  | 38047/50000 [24:58:13<6:25:03,  1.93s/it]
Outer training loop:  76%|#######6  | 38048/50000 [24:58:15<6:29:29,  1.96s/it]
Outer training loop:  76%|#######6  | 38049/50000 [24:58:16<6:14:59,  1.88s/it]
Outer training loop:  76%|#######6  | 38050/50000 [24:58:18<6:05:57,  1.84s/it]
Outer training loop:  76%|#######6  | 38051/50000 [24:58:20<6:16:16,  1.89s/it]
Outer training loop:  76%|#######6  | 38052/50000 [24:58:22<6:32:07,  1.97s/it]
Outer training loop:  76%|#######6  | 38053/50000 [24:58:24<6:28:23,  1.95s/it]
Outer training loop:  76%|#######6  | 38054/50000 [24:58:26<6:27:15,  1.95s/it]
Outer training loop:  76%|#######6  | 38055/50000 [24:58:28<6:33:41,  1.98s/it]
Outer training loop:  76%|#######6  | 38056/50000 [24:58:30<6:27:34,  1.95s/it]
Outer training loop:  76%|#######6  | 38057/50000 [24:58:32<6:21:47,  1.92s/it]
Outer training loop:  76%|#######6  | 38058/50000 [24:58:34<6:24:34,  1.93s/it]
Outer training loop:  76%|#######6  | 38059/50000 [24:58:36<6:23:24,  1.93s/it]
Outer training loop:  76%|#######6  | 38060/50000 [24:58:38<6:19:33,  1.91s/it]
Outer training loop:  76%|#######6  | 38061/50000 [24:58:40<6:26:50,  1.94s/it]
Outer training loop:  76%|#######6  | 38062/50000 [24:58:41<6:20:08,  1.91s/it]
Outer training loop:  76%|#######6  | 38063/50000 [24:58:43<6:21:41,  1.92s/it]
Outer training loop:  76%|#######6  | 38064/50000 [24:58:46<6:38:31,  2.00s/it]
Outer training loop:  76%|#######6  | 38065/50000 [24:58:47<6:28:16,  1.95s/it]
Outer training loop:  76%|#######6  | 38066/50000 [24:58:49<6:21:35,  1.92s/it]
Outer training loop:  76%|#######6  | 38067/50000 [24:58:51<6:19:10,  1.91s/it]
Outer training loop:  76%|#######6  | 38068/50000 [24:58:53<6:20:18,  1.91s/it]
Outer training loop:  76%|#######6  | 38069/50000 [24:58:55<6:21:28,  1.92s/it]
Outer training loop:  76%|#######6  | 38070/50000 [24:58:57<6:11:36,  1.87s/it]
Outer training loop:  76%|#######6  | 38071/50000 [24:58:59<6:20:47,  1.92s/it]
Outer training loop:  76%|#######6  | 38072/50000 [24:59:01<6:11:25,  1.87s/it]
Outer training loop:  76%|#######6  | 38073/50000 [24:59:02<6:06:17,  1.84s/it]
Outer training loop:  76%|#######6  | 38074/50000 [24:59:04<6:05:16,  1.84s/it]
Outer training loop:  76%|#######6  | 38075/50000 [24:59:06<6:17:31,  1.90s/it]
Outer training loop:  76%|#######6  | 38076/50000 [24:59:08<6:14:14,  1.88s/it]
Outer training loop:  76%|#######6  | 38077/50000 [24:59:10<6:19:13,  1.91s/it]
Outer training loop:  76%|#######6  | 38078/50000 [24:59:12<6:27:16,  1.95s/it]
Outer training loop:  76%|#######6  | 38079/50000 [24:59:14<6:22:23,  1.92s/it]
Outer training loop:  76%|#######6  | 38080/50000 [24:59:16<6:14:37,  1.89s/it]
Outer training loop:  76%|#######6  | 38081/50000 [24:59:18<6:20:56,  1.92s/it]
Outer training loop:  76%|#######6  | 38082/50000 [24:59:20<6:19:35,  1.91s/it]
Outer training loop:  76%|#######6  | 38083/50000 [24:59:21<6:13:48,  1.88s/it]
Outer training loop:  76%|#######6  | 38084/50000 [24:59:23<6:03:58,  1.83s/it]
Outer training loop:  76%|#######6  | 38085/50000 [24:59:25<6:14:57,  1.89s/it]
Outer training loop:  76%|#######6  | 38086/50000 [24:59:27<6:07:28,  1.85s/it]
Outer training loop:  76%|#######6  | 38087/50000 [24:59:29<6:07:37,  1.85s/it]
Outer training loop:  76%|#######6  | 38088/50000 [24:59:31<6:13:54,  1.88s/it]
Outer training loop:  76%|#######6  | 38089/50000 [24:59:33<6:10:45,  1.87s/it]
Outer training loop:  76%|#######6  | 38090/50000 [24:59:34<6:09:12,  1.86s/it]
Outer training loop:  76%|#######6  | 38091/50000 [24:59:36<6:18:07,  1.91s/it]
Outer training loop:  76%|#######6  | 38092/50000 [24:59:38<6:19:26,  1.91s/it]
Outer training loop:  76%|#######6  | 38093/50000 [24:59:40<6:14:31,  1.89s/it]
Outer training loop:  76%|#######6  | 38094/50000 [24:59:42<6:19:28,  1.91s/it]
Outer training loop:  76%|#######6  | 38095/50000 [24:59:44<6:29:12,  1.96s/it]
Outer training loop:  76%|#######6  | 38096/50000 [24:59:46<6:29:10,  1.96s/it]
Outer training loop:  76%|#######6  | 38097/50000 [24:59:48<6:23:53,  1.94s/it]
Outer training loop:  76%|#######6  | 38098/50000 [24:59:50<6:38:33,  2.01s/it]
Outer training loop:  76%|#######6  | 38099/50000 [24:59:52<6:30:33,  1.97s/it]
Outer training loop:  76%|#######6  | 38100/50000 [24:59:54<6:21:30,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.02s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.98s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  76%|#######6  | 38101/50000 [25:00:01<11:59:22,  3.63s/it]
Outer training loop:  76%|#######6  | 38102/50000 [25:00:04<10:24:37,  3.15s/it]
Outer training loop:  76%|#######6  | 38103/50000 [25:00:05<9:03:26,  2.74s/it] 
Outer training loop:  76%|#######6  | 38104/50000 [25:00:07<8:07:24,  2.46s/it]
Outer training loop:  76%|#######6  | 38105/50000 [25:00:09<7:37:19,  2.31s/it]
Outer training loop:  76%|#######6  | 38106/50000 [25:00:11<7:21:46,  2.23s/it]
Outer training loop:  76%|#######6  | 38107/50000 [25:00:13<7:09:31,  2.17s/it]
Outer training loop:  76%|#######6  | 38108/50000 [25:00:15<6:43:57,  2.04s/it]
Outer training loop:  76%|#######6  | 38109/50000 [25:00:17<6:44:35,  2.04s/it]
Outer training loop:  76%|#######6  | 38110/50000 [25:00:19<6:32:56,  1.98s/it]
Outer training loop:  76%|#######6  | 38111/50000 [25:00:21<6:30:29,  1.97s/it]
Outer training loop:  76%|#######6  | 38112/50000 [25:00:23<6:36:57,  2.00s/it]
Outer training loop:  76%|#######6  | 38113/50000 [25:00:25<6:30:21,  1.97s/it]
Outer training loop:  76%|#######6  | 38114/50000 [25:00:27<6:21:41,  1.93s/it]
Outer training loop:  76%|#######6  | 38115/50000 [25:00:28<6:16:31,  1.90s/it]
Outer training loop:  76%|#######6  | 38116/50000 [25:00:30<6:19:52,  1.92s/it]
Outer training loop:  76%|#######6  | 38117/50000 [25:00:32<6:15:49,  1.90s/it]
Outer training loop:  76%|#######6  | 38118/50000 [25:00:34<6:15:14,  1.89s/it]
Outer training loop:  76%|#######6  | 38119/50000 [25:00:36<6:27:06,  1.95s/it]
Outer training loop:  76%|#######6  | 38120/50000 [25:00:38<6:18:33,  1.91s/it]
Outer training loop:  76%|#######6  | 38121/50000 [25:00:40<6:21:45,  1.93s/it]
Outer training loop:  76%|#######6  | 38122/50000 [25:00:42<6:17:48,  1.91s/it]
Outer training loop:  76%|#######6  | 38123/50000 [25:00:44<6:30:43,  1.97s/it]
Outer training loop:  76%|#######6  | 38124/50000 [25:00:46<6:25:25,  1.95s/it]
Outer training loop:  76%|#######6  | 38125/50000 [25:00:48<6:20:42,  1.92s/it]
Outer training loop:  76%|#######6  | 38126/50000 [25:00:50<6:27:00,  1.96s/it]
Outer training loop:  76%|#######6  | 38127/50000 [25:00:52<6:18:36,  1.91s/it]
Outer training loop:  76%|#######6  | 38128/50000 [25:00:53<6:13:10,  1.89s/it]
Outer training loop:  76%|#######6  | 38129/50000 [25:00:55<6:10:51,  1.87s/it]
Outer training loop:  76%|#######6  | 38130/50000 [25:00:57<6:19:26,  1.92s/it]
Outer training loop:  76%|#######6  | 38131/50000 [25:00:59<6:14:19,  1.89s/it]
Outer training loop:  76%|#######6  | 38132/50000 [25:01:01<6:17:03,  1.91s/it]
Outer training loop:  76%|#######6  | 38133/50000 [25:01:03<6:26:39,  1.95s/it]
Outer training loop:  76%|#######6  | 38134/50000 [25:01:05<6:18:26,  1.91s/it]
Outer training loop:  76%|#######6  | 38135/50000 [25:01:07<6:13:09,  1.89s/it]
Outer training loop:  76%|#######6  | 38136/50000 [25:01:09<6:21:28,  1.93s/it]
Outer training loop:  76%|#######6  | 38137/50000 [25:01:11<6:23:31,  1.94s/it]
Outer training loop:  76%|#######6  | 38138/50000 [25:01:12<6:15:03,  1.90s/it]
Outer training loop:  76%|#######6  | 38139/50000 [25:01:14<6:22:32,  1.94s/it]
Outer training loop:  76%|#######6  | 38140/50000 [25:01:17<6:32:07,  1.98s/it]
Outer training loop:  76%|#######6  | 38141/50000 [25:01:18<6:22:49,  1.94s/it]
Outer training loop:  76%|#######6  | 38142/50000 [25:01:20<6:15:35,  1.90s/it]
Outer training loop:  76%|#######6  | 38143/50000 [25:01:22<6:21:43,  1.93s/it]
Outer training loop:  76%|#######6  | 38144/50000 [25:01:24<6:12:41,  1.89s/it]
Outer training loop:  76%|#######6  | 38145/50000 [25:01:26<6:14:43,  1.90s/it]
Outer training loop:  76%|#######6  | 38146/50000 [25:01:28<6:24:33,  1.95s/it]
Outer training loop:  76%|#######6  | 38147/50000 [25:01:30<6:18:20,  1.92s/it]
Outer training loop:  76%|#######6  | 38148/50000 [25:01:32<6:08:09,  1.86s/it]
Outer training loop:  76%|#######6  | 38149/50000 [25:01:33<6:06:59,  1.86s/it]
Outer training loop:  76%|#######6  | 38150/50000 [25:01:35<6:16:44,  1.91s/it]
Outer training loop:  76%|#######6  | 38151/50000 [25:01:37<6:09:39,  1.87s/it]
Outer training loop:  76%|#######6  | 38152/50000 [25:01:39<6:10:42,  1.88s/it]
Outer training loop:  76%|#######6  | 38153/50000 [25:01:41<6:21:17,  1.93s/it]
Outer training loop:  76%|#######6  | 38154/50000 [25:01:43<6:16:50,  1.91s/it]
Outer training loop:  76%|#######6  | 38155/50000 [25:01:45<6:11:05,  1.88s/it]
Outer training loop:  76%|#######6  | 38156/50000 [25:01:47<6:10:14,  1.88s/it]
Outer training loop:  76%|#######6  | 38157/50000 [25:01:49<6:12:26,  1.89s/it]
Outer training loop:  76%|#######6  | 38158/50000 [25:01:51<6:14:02,  1.90s/it]
Outer training loop:  76%|#######6  | 38159/50000 [25:01:52<6:15:17,  1.90s/it]
Outer training loop:  76%|#######6  | 38160/50000 [25:01:54<6:21:54,  1.94s/it]
Outer training loop:  76%|#######6  | 38161/50000 [25:01:56<6:19:34,  1.92s/it]
Outer training loop:  76%|#######6  | 38162/50000 [25:01:58<6:18:53,  1.92s/it]
Outer training loop:  76%|#######6  | 38163/50000 [25:02:00<6:24:14,  1.95s/it]
Outer training loop:  76%|#######6  | 38164/50000 [25:02:02<6:19:04,  1.92s/it]
Outer training loop:  76%|#######6  | 38165/50000 [25:02:04<6:11:32,  1.88s/it]
Outer training loop:  76%|#######6  | 38166/50000 [25:02:06<6:02:52,  1.84s/it]
Outer training loop:  76%|#######6  | 38167/50000 [25:02:08<6:09:50,  1.88s/it]
Outer training loop:  76%|#######6  | 38168/50000 [25:02:10<6:07:57,  1.87s/it]
Outer training loop:  76%|#######6  | 38169/50000 [25:02:11<6:04:25,  1.85s/it]
Outer training loop:  76%|#######6  | 38170/50000 [25:02:13<6:11:44,  1.89s/it]
Outer training loop:  76%|#######6  | 38171/50000 [25:02:15<6:04:15,  1.85s/it]
Outer training loop:  76%|#######6  | 38172/50000 [25:02:17<5:58:33,  1.82s/it]
Outer training loop:  76%|#######6  | 38173/50000 [25:02:19<5:52:52,  1.79s/it]
Outer training loop:  76%|#######6  | 38174/50000 [25:02:21<6:05:10,  1.85s/it]
Outer training loop:  76%|#######6  | 38175/50000 [25:02:22<6:02:57,  1.84s/it]
Outer training loop:  76%|#######6  | 38176/50000 [25:02:24<6:03:31,  1.84s/it]
Outer training loop:  76%|#######6  | 38177/50000 [25:02:26<6:15:52,  1.91s/it]
Outer training loop:  76%|#######6  | 38178/50000 [25:02:28<6:07:43,  1.87s/it]
Outer training loop:  76%|#######6  | 38179/50000 [25:02:30<6:02:29,  1.84s/it]
Outer training loop:  76%|#######6  | 38180/50000 [25:02:32<5:56:12,  1.81s/it]
Outer training loop:  76%|#######6  | 38181/50000 [25:02:34<6:10:24,  1.88s/it]
Outer training loop:  76%|#######6  | 38182/50000 [25:02:35<6:13:14,  1.89s/it]
Outer training loop:  76%|#######6  | 38183/50000 [25:02:37<6:03:49,  1.85s/it]
Outer training loop:  76%|#######6  | 38184/50000 [25:02:39<6:10:47,  1.88s/it]
Outer training loop:  76%|#######6  | 38185/50000 [25:02:41<6:17:19,  1.92s/it]
Outer training loop:  76%|#######6  | 38186/50000 [25:02:43<6:08:41,  1.87s/it]
Outer training loop:  76%|#######6  | 38187/50000 [25:02:45<6:21:41,  1.94s/it]
Outer training loop:  76%|#######6  | 38188/50000 [25:02:47<6:11:44,  1.89s/it]
Outer training loop:  76%|#######6  | 38189/50000 [25:02:49<6:18:54,  1.92s/it]
Outer training loop:  76%|#######6  | 38190/50000 [25:02:51<6:10:34,  1.88s/it]
Outer training loop:  76%|#######6  | 38191/50000 [25:02:53<6:21:28,  1.94s/it]
Outer training loop:  76%|#######6  | 38192/50000 [25:02:54<6:11:18,  1.89s/it]
Outer training loop:  76%|#######6  | 38193/50000 [25:02:56<6:10:10,  1.88s/it]
Outer training loop:  76%|#######6  | 38194/50000 [25:02:58<6:19:07,  1.93s/it]
Outer training loop:  76%|#######6  | 38195/50000 [25:03:00<6:15:17,  1.91s/it]
Outer training loop:  76%|#######6  | 38196/50000 [25:03:02<6:03:29,  1.85s/it]
Outer training loop:  76%|#######6  | 38197/50000 [25:03:04<6:23:03,  1.95s/it]
Outer training loop:  76%|#######6  | 38198/50000 [25:03:06<6:18:19,  1.92s/it]
Outer training loop:  76%|#######6  | 38199/50000 [25:03:08<6:12:22,  1.89s/it]
Outer training loop:  76%|#######6  | 38200/50000 [25:03:10<6:15:59,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  76%|#######6  | 38201/50000 [25:03:17<11:46:23,  3.59s/it]
Outer training loop:  76%|#######6  | 38202/50000 [25:03:19<10:04:39,  3.08s/it]
Outer training loop:  76%|#######6  | 38203/50000 [25:03:21<8:49:51,  2.69s/it] 
Outer training loop:  76%|#######6  | 38204/50000 [25:03:23<8:01:11,  2.45s/it]
Outer training loop:  76%|#######6  | 38205/50000 [25:03:25<7:39:30,  2.34s/it]
Outer training loop:  76%|#######6  | 38206/50000 [25:03:27<7:04:33,  2.16s/it]
Outer training loop:  76%|#######6  | 38207/50000 [25:03:28<6:43:41,  2.05s/it]
Outer training loop:  76%|#######6  | 38208/50000 [25:03:30<6:40:51,  2.04s/it]
Outer training loop:  76%|#######6  | 38209/50000 [25:03:32<6:28:05,  1.97s/it]
Outer training loop:  76%|#######6  | 38210/50000 [25:03:34<6:27:10,  1.97s/it]
Outer training loop:  76%|#######6  | 38211/50000 [25:03:36<6:21:03,  1.94s/it]
Outer training loop:  76%|#######6  | 38212/50000 [25:03:38<6:29:32,  1.98s/it]
Outer training loop:  76%|#######6  | 38213/50000 [25:03:40<6:21:43,  1.94s/it]
Outer training loop:  76%|#######6  | 38214/50000 [25:03:42<6:10:37,  1.89s/it]
Outer training loop:  76%|#######6  | 38215/50000 [25:03:44<6:18:34,  1.93s/it]
Outer training loop:  76%|#######6  | 38216/50000 [25:03:46<6:07:12,  1.87s/it]
Outer training loop:  76%|#######6  | 38217/50000 [25:03:47<6:05:54,  1.86s/it]
Outer training loop:  76%|#######6  | 38218/50000 [25:03:49<6:16:56,  1.92s/it]
Outer training loop:  76%|#######6  | 38219/50000 [25:03:51<6:14:29,  1.91s/it]
Outer training loop:  76%|#######6  | 38220/50000 [25:03:53<6:05:32,  1.86s/it]
Outer training loop:  76%|#######6  | 38221/50000 [25:03:55<6:20:27,  1.94s/it]
Outer training loop:  76%|#######6  | 38222/50000 [25:03:57<6:11:06,  1.89s/it]
Outer training loop:  76%|#######6  | 38223/50000 [25:03:59<6:10:35,  1.89s/it]
Outer training loop:  76%|#######6  | 38224/50000 [25:04:01<6:08:56,  1.88s/it]
Outer training loop:  76%|#######6  | 38225/50000 [25:04:03<6:18:02,  1.93s/it]
Outer training loop:  76%|#######6  | 38226/50000 [25:04:05<6:14:42,  1.91s/it]
Outer training loop:  76%|#######6  | 38227/50000 [25:04:06<6:06:48,  1.87s/it]
Outer training loop:  76%|#######6  | 38228/50000 [25:04:08<6:13:24,  1.90s/it]
Outer training loop:  76%|#######6  | 38229/50000 [25:04:10<6:11:12,  1.89s/it]
Outer training loop:  76%|#######6  | 38230/50000 [25:04:12<5:59:45,  1.83s/it]
Outer training loop:  76%|#######6  | 38231/50000 [25:04:14<5:54:58,  1.81s/it]
Outer training loop:  76%|#######6  | 38232/50000 [25:04:16<6:11:21,  1.89s/it]
Outer training loop:  76%|#######6  | 38233/50000 [25:04:18<6:01:33,  1.84s/it]
Outer training loop:  76%|#######6  | 38234/50000 [25:04:19<6:00:16,  1.84s/it]
Outer training loop:  76%|#######6  | 38235/50000 [25:04:21<5:54:53,  1.81s/it]
Outer training loop:  76%|#######6  | 38236/50000 [25:04:23<6:12:55,  1.90s/it]
Outer training loop:  76%|#######6  | 38237/50000 [25:04:25<6:11:50,  1.90s/it]
Outer training loop:  76%|#######6  | 38238/50000 [25:04:27<6:05:26,  1.86s/it]
Outer training loop:  76%|#######6  | 38239/50000 [25:04:29<6:11:39,  1.90s/it]
Outer training loop:  76%|#######6  | 38240/50000 [25:04:31<6:06:46,  1.87s/it]
Outer training loop:  76%|#######6  | 38241/50000 [25:04:32<6:00:40,  1.84s/it]
Outer training loop:  76%|#######6  | 38242/50000 [25:04:34<6:09:28,  1.89s/it]
Outer training loop:  76%|#######6  | 38243/50000 [25:04:36<6:06:49,  1.87s/it]
Outer training loop:  76%|#######6  | 38244/50000 [25:04:38<6:00:28,  1.84s/it]
Outer training loop:  76%|#######6  | 38245/50000 [25:04:40<5:58:35,  1.83s/it]
Outer training loop:  76%|#######6  | 38246/50000 [25:04:42<6:08:05,  1.88s/it]
Outer training loop:  76%|#######6  | 38247/50000 [25:04:44<6:02:34,  1.85s/it]
Outer training loop:  76%|#######6  | 38248/50000 [25:04:45<6:03:40,  1.86s/it]
Outer training loop:  76%|#######6  | 38249/50000 [25:04:47<6:11:23,  1.90s/it]
Outer training loop:  76%|#######6  | 38250/50000 [25:04:49<6:09:00,  1.88s/it]
Outer training loop:  77%|#######6  | 38251/50000 [25:04:51<6:12:43,  1.90s/it]
Outer training loop:  77%|#######6  | 38252/50000 [25:04:53<6:17:23,  1.93s/it]
Outer training loop:  77%|#######6  | 38253/50000 [25:04:55<6:12:40,  1.90s/it]
Outer training loop:  77%|#######6  | 38254/50000 [25:04:57<6:04:48,  1.86s/it]
Outer training loop:  77%|#######6  | 38255/50000 [25:04:59<6:02:01,  1.85s/it]
Outer training loop:  77%|#######6  | 38256/50000 [25:05:01<6:13:56,  1.91s/it]
Outer training loop:  77%|#######6  | 38257/50000 [25:05:03<6:04:51,  1.86s/it]
Outer training loop:  77%|#######6  | 38258/50000 [25:05:04<6:01:25,  1.85s/it]
Outer training loop:  77%|#######6  | 38259/50000 [25:05:06<6:14:17,  1.91s/it]
Outer training loop:  77%|#######6  | 38260/50000 [25:05:08<6:06:08,  1.87s/it]
Outer training loop:  77%|#######6  | 38261/50000 [25:05:10<6:05:19,  1.87s/it]
Outer training loop:  77%|#######6  | 38262/50000 [25:05:12<6:05:29,  1.87s/it]
Outer training loop:  77%|#######6  | 38263/50000 [25:05:14<6:13:48,  1.91s/it]
Outer training loop:  77%|#######6  | 38264/50000 [25:05:16<6:13:00,  1.91s/it]
Outer training loop:  77%|#######6  | 38265/50000 [25:05:18<6:04:55,  1.87s/it]
Outer training loop:  77%|#######6  | 38266/50000 [25:05:20<6:19:59,  1.94s/it]
Outer training loop:  77%|#######6  | 38267/50000 [25:05:22<6:14:13,  1.91s/it]
Outer training loop:  77%|#######6  | 38268/50000 [25:05:23<6:07:19,  1.88s/it]
Outer training loop:  77%|#######6  | 38269/50000 [25:05:25<6:02:05,  1.85s/it]
Outer training loop:  77%|#######6  | 38270/50000 [25:05:27<6:07:44,  1.88s/it]
Outer training loop:  77%|#######6  | 38271/50000 [25:05:29<6:10:45,  1.90s/it]
Outer training loop:  77%|#######6  | 38272/50000 [25:05:31<6:06:39,  1.88s/it]
Outer training loop:  77%|#######6  | 38273/50000 [25:05:33<6:11:31,  1.90s/it]
Outer training loop:  77%|#######6  | 38274/50000 [25:05:35<6:07:54,  1.88s/it]
Outer training loop:  77%|#######6  | 38275/50000 [25:05:36<6:01:06,  1.85s/it]
Outer training loop:  77%|#######6  | 38276/50000 [25:05:38<5:58:57,  1.84s/it]
Outer training loop:  77%|#######6  | 38277/50000 [25:05:40<6:04:51,  1.87s/it]
Outer training loop:  77%|#######6  | 38278/50000 [25:05:42<6:02:16,  1.85s/it]
Outer training loop:  77%|#######6  | 38279/50000 [25:05:44<5:57:22,  1.83s/it]
Outer training loop:  77%|#######6  | 38280/50000 [25:05:46<6:10:35,  1.90s/it]
Outer training loop:  77%|#######6  | 38281/50000 [25:05:48<6:07:30,  1.88s/it]
Outer training loop:  77%|#######6  | 38282/50000 [25:05:49<5:59:32,  1.84s/it]
Outer training loop:  77%|#######6  | 38283/50000 [25:05:51<5:57:42,  1.83s/it]
Outer training loop:  77%|#######6  | 38284/50000 [25:05:53<6:08:14,  1.89s/it]
Outer training loop:  77%|#######6  | 38285/50000 [25:05:55<6:05:34,  1.87s/it]
Outer training loop:  77%|#######6  | 38286/50000 [25:05:57<6:31:02,  2.00s/it]
Outer training loop:  77%|#######6  | 38287/50000 [25:05:59<6:14:54,  1.92s/it]
Outer training loop:  77%|#######6  | 38288/50000 [25:06:01<6:22:38,  1.96s/it]
Outer training loop:  77%|#######6  | 38289/50000 [25:06:03<6:08:11,  1.89s/it]
Outer training loop:  77%|#######6  | 38290/50000 [25:06:05<6:09:30,  1.89s/it]
Outer training loop:  77%|#######6  | 38291/50000 [25:06:07<6:24:34,  1.97s/it]
Outer training loop:  77%|#######6  | 38292/50000 [25:06:09<6:21:41,  1.96s/it]
Outer training loop:  77%|#######6  | 38293/50000 [25:06:11<6:15:35,  1.92s/it]
Outer training loop:  77%|#######6  | 38294/50000 [25:06:13<6:10:35,  1.90s/it]
Outer training loop:  77%|#######6  | 38295/50000 [25:06:14<6:11:09,  1.90s/it]
Outer training loop:  77%|#######6  | 38296/50000 [25:06:16<6:07:53,  1.89s/it]
Outer training loop:  77%|#######6  | 38297/50000 [25:06:18<6:00:39,  1.85s/it]
Outer training loop:  77%|#######6  | 38298/50000 [25:06:20<6:11:29,  1.90s/it]
Outer training loop:  77%|#######6  | 38299/50000 [25:06:22<6:02:45,  1.86s/it]
Outer training loop:  77%|#######6  | 38300/50000 [25:06:24<6:01:11,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.95s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  77%|#######6  | 38301/50000 [25:06:31<11:17:33,  3.47s/it]
Outer training loop:  77%|#######6  | 38302/50000 [25:06:33<9:50:51,  3.03s/it] 
Outer training loop:  77%|#######6  | 38303/50000 [25:06:35<8:38:41,  2.66s/it]
Outer training loop:  77%|#######6  | 38304/50000 [25:06:37<7:47:43,  2.40s/it]
Outer training loop:  77%|#######6  | 38305/50000 [25:06:39<7:28:24,  2.30s/it]
Outer training loop:  77%|#######6  | 38306/50000 [25:06:40<7:05:50,  2.18s/it]
Outer training loop:  77%|#######6  | 38307/50000 [25:06:42<6:46:45,  2.09s/it]
Outer training loop:  77%|#######6  | 38308/50000 [25:06:44<6:47:13,  2.09s/it]
Outer training loop:  77%|#######6  | 38309/50000 [25:06:46<6:26:36,  1.98s/it]
Outer training loop:  77%|#######6  | 38310/50000 [25:06:48<6:16:58,  1.93s/it]
Outer training loop:  77%|#######6  | 38311/50000 [25:06:50<6:04:38,  1.87s/it]
Outer training loop:  77%|#######6  | 38312/50000 [25:06:52<6:12:55,  1.91s/it]
Outer training loop:  77%|#######6  | 38313/50000 [25:06:54<6:05:57,  1.88s/it]
Outer training loop:  77%|#######6  | 38314/50000 [25:06:55<5:55:03,  1.82s/it]
Outer training loop:  77%|#######6  | 38315/50000 [25:06:57<5:54:23,  1.82s/it]
Outer training loop:  77%|#######6  | 38316/50000 [25:06:59<6:07:21,  1.89s/it]
Outer training loop:  77%|#######6  | 38317/50000 [25:07:01<5:56:46,  1.83s/it]
Outer training loop:  77%|#######6  | 38318/50000 [25:07:03<6:03:46,  1.87s/it]
Outer training loop:  77%|#######6  | 38319/50000 [25:07:05<6:15:31,  1.93s/it]
Outer training loop:  77%|#######6  | 38320/50000 [25:07:07<6:08:40,  1.89s/it]
Outer training loop:  77%|#######6  | 38321/50000 [25:07:08<6:05:04,  1.88s/it]
Outer training loop:  77%|#######6  | 38322/50000 [25:07:10<6:13:00,  1.92s/it]
Outer training loop:  77%|#######6  | 38323/50000 [25:07:12<6:09:19,  1.90s/it]
Outer training loop:  77%|#######6  | 38324/50000 [25:07:14<6:06:01,  1.88s/it]
Outer training loop:  77%|#######6  | 38325/50000 [25:07:16<6:14:36,  1.93s/it]
Outer training loop:  77%|#######6  | 38326/50000 [25:07:18<6:08:28,  1.89s/it]
Outer training loop:  77%|#######6  | 38327/50000 [25:07:20<6:04:15,  1.87s/it]
Outer training loop:  77%|#######6  | 38328/50000 [25:07:22<5:56:03,  1.83s/it]
Outer training loop:  77%|#######6  | 38329/50000 [25:07:24<6:07:45,  1.89s/it]
Outer training loop:  77%|#######6  | 38330/50000 [25:07:25<6:03:12,  1.87s/it]
Outer training loop:  77%|#######6  | 38331/50000 [25:07:27<6:03:11,  1.87s/it]
Outer training loop:  77%|#######6  | 38332/50000 [25:07:29<6:11:10,  1.91s/it]
Outer training loop:  77%|#######6  | 38333/50000 [25:07:31<6:13:29,  1.92s/it]
Outer training loop:  77%|#######6  | 38334/50000 [25:07:33<6:07:38,  1.89s/it]
Outer training loop:  77%|#######6  | 38335/50000 [25:07:35<6:10:16,  1.90s/it]
Outer training loop:  77%|#######6  | 38336/50000 [25:07:37<6:20:53,  1.96s/it]
Outer training loop:  77%|#######6  | 38337/50000 [25:07:39<6:16:17,  1.94s/it]
Outer training loop:  77%|#######6  | 38338/50000 [25:07:41<6:08:16,  1.89s/it]
Outer training loop:  77%|#######6  | 38339/50000 [25:07:43<6:19:44,  1.95s/it]
Outer training loop:  77%|#######6  | 38340/50000 [25:07:45<6:13:07,  1.92s/it]
Outer training loop:  77%|#######6  | 38341/50000 [25:07:47<6:07:08,  1.89s/it]
Outer training loop:  77%|#######6  | 38342/50000 [25:07:49<6:20:03,  1.96s/it]
Outer training loop:  77%|#######6  | 38343/50000 [25:07:51<6:17:51,  1.94s/it]
Outer training loop:  77%|#######6  | 38344/50000 [25:07:52<6:12:23,  1.92s/it]
Outer training loop:  77%|#######6  | 38345/50000 [25:07:54<6:09:40,  1.90s/it]
Outer training loop:  77%|#######6  | 38346/50000 [25:07:56<6:19:57,  1.96s/it]
Outer training loop:  77%|#######6  | 38347/50000 [25:07:58<6:18:33,  1.95s/it]
Outer training loop:  77%|#######6  | 38348/50000 [25:08:00<6:11:57,  1.92s/it]
Outer training loop:  77%|#######6  | 38349/50000 [25:08:02<6:20:36,  1.96s/it]
Outer training loop:  77%|#######6  | 38350/50000 [25:08:04<6:11:22,  1.91s/it]
Outer training loop:  77%|#######6  | 38351/50000 [25:08:06<6:09:33,  1.90s/it]
Outer training loop:  77%|#######6  | 38352/50000 [25:08:08<6:02:19,  1.87s/it]
Outer training loop:  77%|#######6  | 38353/50000 [25:08:10<6:09:47,  1.90s/it]
Outer training loop:  77%|#######6  | 38354/50000 [25:08:12<6:08:43,  1.90s/it]
Outer training loop:  77%|#######6  | 38355/50000 [25:08:13<6:09:58,  1.91s/it]
Outer training loop:  77%|#######6  | 38356/50000 [25:08:16<6:24:58,  1.98s/it]
Outer training loop:  77%|#######6  | 38357/50000 [25:08:17<6:14:03,  1.93s/it]
Outer training loop:  77%|#######6  | 38358/50000 [25:08:19<6:04:51,  1.88s/it]
Outer training loop:  77%|#######6  | 38359/50000 [25:08:21<6:12:44,  1.92s/it]
Outer training loop:  77%|#######6  | 38360/50000 [25:08:23<6:07:00,  1.89s/it]
Outer training loop:  77%|#######6  | 38361/50000 [25:08:25<6:00:40,  1.86s/it]
Outer training loop:  77%|#######6  | 38362/50000 [25:08:27<6:15:38,  1.94s/it]
Outer training loop:  77%|#######6  | 38363/50000 [25:08:29<6:04:00,  1.88s/it]
Outer training loop:  77%|#######6  | 38364/50000 [25:08:31<6:03:11,  1.87s/it]
Outer training loop:  77%|#######6  | 38365/50000 [25:08:33<6:09:59,  1.91s/it]
Outer training loop:  77%|#######6  | 38366/50000 [25:08:34<6:13:05,  1.92s/it]
Outer training loop:  77%|#######6  | 38367/50000 [25:08:36<6:05:23,  1.88s/it]
Outer training loop:  77%|#######6  | 38368/50000 [25:08:38<5:58:56,  1.85s/it]
Outer training loop:  77%|#######6  | 38369/50000 [25:08:40<6:13:17,  1.93s/it]
Outer training loop:  77%|#######6  | 38370/50000 [25:08:42<6:07:53,  1.90s/it]
Outer training loop:  77%|#######6  | 38371/50000 [25:08:44<6:01:00,  1.86s/it]
Outer training loop:  77%|#######6  | 38372/50000 [25:08:46<6:05:39,  1.89s/it]
Outer training loop:  77%|#######6  | 38373/50000 [25:08:47<5:59:48,  1.86s/it]
Outer training loop:  77%|#######6  | 38374/50000 [25:08:49<6:03:10,  1.87s/it]
Outer training loop:  77%|#######6  | 38375/50000 [25:08:51<5:53:18,  1.82s/it]
Outer training loop:  77%|#######6  | 38376/50000 [25:08:53<6:01:13,  1.86s/it]
Outer training loop:  77%|#######6  | 38377/50000 [25:08:55<6:00:34,  1.86s/it]
Outer training loop:  77%|#######6  | 38378/50000 [25:08:57<5:51:18,  1.81s/it]
Outer training loop:  77%|#######6  | 38379/50000 [25:08:59<6:03:50,  1.88s/it]
Outer training loop:  77%|#######6  | 38380/50000 [25:09:01<6:06:56,  1.89s/it]
Outer training loop:  77%|#######6  | 38381/50000 [25:09:02<5:58:48,  1.85s/it]
Outer training loop:  77%|#######6  | 38382/50000 [25:09:04<6:01:28,  1.87s/it]
Outer training loop:  77%|#######6  | 38383/50000 [25:09:06<6:15:59,  1.94s/it]
Outer training loop:  77%|#######6  | 38384/50000 [25:09:08<6:09:06,  1.91s/it]
Outer training loop:  77%|#######6  | 38385/50000 [25:09:10<6:00:28,  1.86s/it]
Outer training loop:  77%|#######6  | 38386/50000 [25:09:12<6:05:57,  1.89s/it]
Outer training loop:  77%|#######6  | 38387/50000 [25:09:14<6:07:26,  1.90s/it]
Outer training loop:  77%|#######6  | 38388/50000 [25:09:16<6:05:44,  1.89s/it]
Outer training loop:  77%|#######6  | 38389/50000 [25:09:17<6:00:57,  1.87s/it]
Outer training loop:  77%|#######6  | 38390/50000 [25:09:20<6:10:26,  1.91s/it]
Outer training loop:  77%|#######6  | 38391/50000 [25:09:21<6:04:08,  1.88s/it]
Outer training loop:  77%|#######6  | 38392/50000 [25:09:23<6:03:58,  1.88s/it]
Outer training loop:  77%|#######6  | 38393/50000 [25:09:25<5:59:30,  1.86s/it]
Outer training loop:  77%|#######6  | 38394/50000 [25:09:27<6:09:54,  1.91s/it]
Outer training loop:  77%|#######6  | 38395/50000 [25:09:29<6:06:54,  1.90s/it]
Outer training loop:  77%|#######6  | 38396/50000 [25:09:31<6:26:20,  2.00s/it]
Outer training loop:  77%|#######6  | 38397/50000 [25:09:33<6:14:07,  1.93s/it]
Outer training loop:  77%|#######6  | 38398/50000 [25:09:35<6:11:24,  1.92s/it]
Outer training loop:  77%|#######6  | 38399/50000 [25:09:37<6:00:31,  1.86s/it]
Outer training loop:  77%|#######6  | 38400/50000 [25:09:39<6:08:59,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  77%|#######6  | 38401/50000 [25:09:46<11:22:41,  3.53s/it]
Outer training loop:  77%|#######6  | 38402/50000 [25:09:48<9:37:48,  2.99s/it] 
Outer training loop:  77%|#######6  | 38403/50000 [25:09:49<8:32:09,  2.65s/it]
Outer training loop:  77%|#######6  | 38404/50000 [25:09:52<8:01:37,  2.49s/it]
Outer training loop:  77%|#######6  | 38405/50000 [25:09:53<7:19:51,  2.28s/it]
Outer training loop:  77%|#######6  | 38406/50000 [25:09:55<6:52:55,  2.14s/it]
Outer training loop:  77%|#######6  | 38407/50000 [25:09:57<6:45:52,  2.10s/it]
Outer training loop:  77%|#######6  | 38408/50000 [25:09:59<6:27:35,  2.01s/it]
Outer training loop:  77%|#######6  | 38409/50000 [25:10:01<6:18:01,  1.96s/it]
Outer training loop:  77%|#######6  | 38410/50000 [25:10:03<6:04:31,  1.89s/it]
Outer training loop:  77%|#######6  | 38411/50000 [25:10:05<6:09:50,  1.91s/it]
Outer training loop:  77%|#######6  | 38412/50000 [25:10:06<6:03:43,  1.88s/it]
Outer training loop:  77%|#######6  | 38413/50000 [25:10:08<6:04:19,  1.89s/it]
Outer training loop:  77%|#######6  | 38414/50000 [25:10:10<6:09:23,  1.91s/it]
Outer training loop:  77%|#######6  | 38415/50000 [25:10:12<6:15:51,  1.95s/it]
Outer training loop:  77%|#######6  | 38416/50000 [25:10:14<6:10:31,  1.92s/it]
Outer training loop:  77%|#######6  | 38417/50000 [25:10:16<5:57:31,  1.85s/it]
Outer training loop:  77%|#######6  | 38418/50000 [25:10:18<6:02:29,  1.88s/it]
Outer training loop:  77%|#######6  | 38419/50000 [25:10:20<6:03:11,  1.88s/it]
Outer training loop:  77%|#######6  | 38420/50000 [25:10:21<5:53:55,  1.83s/it]
Outer training loop:  77%|#######6  | 38421/50000 [25:10:23<5:58:45,  1.86s/it]
Outer training loop:  77%|#######6  | 38422/50000 [25:10:25<6:01:41,  1.87s/it]
Outer training loop:  77%|#######6  | 38423/50000 [25:10:27<5:57:17,  1.85s/it]
Outer training loop:  77%|#######6  | 38424/50000 [25:10:29<6:09:54,  1.92s/it]
Outer training loop:  77%|#######6  | 38425/50000 [25:10:31<6:09:20,  1.91s/it]
Outer training loop:  77%|#######6  | 38426/50000 [25:10:33<5:57:57,  1.86s/it]
Outer training loop:  77%|#######6  | 38427/50000 [25:10:34<5:57:53,  1.86s/it]
Outer training loop:  77%|#######6  | 38428/50000 [25:10:37<6:09:28,  1.92s/it]
Outer training loop:  77%|#######6  | 38429/50000 [25:10:38<5:59:53,  1.87s/it]
Outer training loop:  77%|#######6  | 38430/50000 [25:10:40<6:10:44,  1.92s/it]
Outer training loop:  77%|#######6  | 38431/50000 [25:10:42<6:15:51,  1.95s/it]
Outer training loop:  77%|#######6  | 38432/50000 [25:10:44<6:06:58,  1.90s/it]
Outer training loop:  77%|#######6  | 38433/50000 [25:10:46<6:00:35,  1.87s/it]
Outer training loop:  77%|#######6  | 38434/50000 [25:10:48<6:05:45,  1.90s/it]
Outer training loop:  77%|#######6  | 38435/50000 [25:10:50<6:20:18,  1.97s/it]
Outer training loop:  77%|#######6  | 38436/50000 [25:10:52<6:11:45,  1.93s/it]
Outer training loop:  77%|#######6  | 38437/50000 [25:10:54<6:00:29,  1.87s/it]
Outer training loop:  77%|#######6  | 38438/50000 [25:10:56<6:10:30,  1.92s/it]
Outer training loop:  77%|#######6  | 38439/50000 [25:10:57<6:04:47,  1.89s/it]
Outer training loop:  77%|#######6  | 38440/50000 [25:10:59<5:56:53,  1.85s/it]
Outer training loop:  77%|#######6  | 38441/50000 [25:11:01<5:54:26,  1.84s/it]
Outer training loop:  77%|#######6  | 38442/50000 [25:11:03<6:06:44,  1.90s/it]
Outer training loop:  77%|#######6  | 38443/50000 [25:11:05<6:07:55,  1.91s/it]
Outer training loop:  77%|#######6  | 38444/50000 [25:11:07<5:58:04,  1.86s/it]
Outer training loop:  77%|#######6  | 38445/50000 [25:11:09<6:02:35,  1.88s/it]
Outer training loop:  77%|#######6  | 38446/50000 [25:11:10<5:55:10,  1.84s/it]
Outer training loop:  77%|#######6  | 38447/50000 [25:11:12<5:46:45,  1.80s/it]
Outer training loop:  77%|#######6  | 38448/50000 [25:11:14<5:49:54,  1.82s/it]
Outer training loop:  77%|#######6  | 38449/50000 [25:11:16<5:59:10,  1.87s/it]
Outer training loop:  77%|#######6  | 38450/50000 [25:11:18<5:54:19,  1.84s/it]
Outer training loop:  77%|#######6  | 38451/50000 [25:11:20<5:55:24,  1.85s/it]
Outer training loop:  77%|#######6  | 38452/50000 [25:11:22<5:59:20,  1.87s/it]
Outer training loop:  77%|#######6  | 38453/50000 [25:11:23<5:56:32,  1.85s/it]
Outer training loop:  77%|#######6  | 38454/50000 [25:11:25<5:49:36,  1.82s/it]
Outer training loop:  77%|#######6  | 38455/50000 [25:11:27<6:03:45,  1.89s/it]
Outer training loop:  77%|#######6  | 38456/50000 [25:11:29<6:01:28,  1.88s/it]
Outer training loop:  77%|#######6  | 38457/50000 [25:11:31<5:53:12,  1.84s/it]
Outer training loop:  77%|#######6  | 38458/50000 [25:11:33<5:52:10,  1.83s/it]
Outer training loop:  77%|#######6  | 38459/50000 [25:11:35<6:04:09,  1.89s/it]
Outer training loop:  77%|#######6  | 38460/50000 [25:11:36<5:58:42,  1.87s/it]
Outer training loop:  77%|#######6  | 38461/50000 [25:11:38<5:55:19,  1.85s/it]
Outer training loop:  77%|#######6  | 38462/50000 [25:11:40<6:06:50,  1.91s/it]
Outer training loop:  77%|#######6  | 38463/50000 [25:11:42<5:59:07,  1.87s/it]
Outer training loop:  77%|#######6  | 38464/50000 [25:11:44<5:58:04,  1.86s/it]
Outer training loop:  77%|#######6  | 38465/50000 [25:11:46<5:55:01,  1.85s/it]
Outer training loop:  77%|#######6  | 38466/50000 [25:11:48<5:58:36,  1.87s/it]
Outer training loop:  77%|#######6  | 38467/50000 [25:11:49<5:50:58,  1.83s/it]
Outer training loop:  77%|#######6  | 38468/50000 [25:11:51<5:49:37,  1.82s/it]
Outer training loop:  77%|#######6  | 38469/50000 [25:11:53<5:58:15,  1.86s/it]
Outer training loop:  77%|#######6  | 38470/50000 [25:11:55<6:00:16,  1.87s/it]
Outer training loop:  77%|#######6  | 38471/50000 [25:11:57<5:57:04,  1.86s/it]
Outer training loop:  77%|#######6  | 38472/50000 [25:11:59<5:56:06,  1.85s/it]
Outer training loop:  77%|#######6  | 38473/50000 [25:12:01<6:01:38,  1.88s/it]
Outer training loop:  77%|#######6  | 38474/50000 [25:12:02<5:56:00,  1.85s/it]
Outer training loop:  77%|#######6  | 38475/50000 [25:12:04<6:00:21,  1.88s/it]
Outer training loop:  77%|#######6  | 38476/50000 [25:12:06<6:03:17,  1.89s/it]
Outer training loop:  77%|#######6  | 38477/50000 [25:12:08<5:58:46,  1.87s/it]
Outer training loop:  77%|#######6  | 38478/50000 [25:12:10<5:58:58,  1.87s/it]
Outer training loop:  77%|#######6  | 38479/50000 [25:12:12<6:01:06,  1.88s/it]
Outer training loop:  77%|#######6  | 38480/50000 [25:12:14<6:08:38,  1.92s/it]
Outer training loop:  77%|#######6  | 38481/50000 [25:12:16<6:03:48,  1.89s/it]
Outer training loop:  77%|#######6  | 38482/50000 [25:12:18<5:57:48,  1.86s/it]
Outer training loop:  77%|#######6  | 38483/50000 [25:12:20<6:16:48,  1.96s/it]
Outer training loop:  77%|#######6  | 38484/50000 [25:12:21<6:03:46,  1.90s/it]
Outer training loop:  77%|#######6  | 38485/50000 [25:12:23<6:05:03,  1.90s/it]
Outer training loop:  77%|#######6  | 38486/50000 [25:12:25<6:14:16,  1.95s/it]
Outer training loop:  77%|#######6  | 38487/50000 [25:12:27<6:04:26,  1.90s/it]
Outer training loop:  77%|#######6  | 38488/50000 [25:12:29<5:59:46,  1.88s/it]
Outer training loop:  77%|#######6  | 38489/50000 [25:12:31<5:54:28,  1.85s/it]
Outer training loop:  77%|#######6  | 38490/50000 [25:12:33<6:10:12,  1.93s/it]
Outer training loop:  77%|#######6  | 38491/50000 [25:12:35<6:06:36,  1.91s/it]
Outer training loop:  77%|#######6  | 38492/50000 [25:12:37<5:56:18,  1.86s/it]
Outer training loop:  77%|#######6  | 38493/50000 [25:12:38<6:02:22,  1.89s/it]
Outer training loop:  77%|#######6  | 38494/50000 [25:12:40<5:59:08,  1.87s/it]
Outer training loop:  77%|#######6  | 38495/50000 [25:12:42<5:48:25,  1.82s/it]
Outer training loop:  77%|#######6  | 38496/50000 [25:12:44<5:49:30,  1.82s/it]
Outer training loop:  77%|#######6  | 38497/50000 [25:12:46<6:00:05,  1.88s/it]
Outer training loop:  77%|#######6  | 38498/50000 [25:12:48<5:59:41,  1.88s/it]
Outer training loop:  77%|#######6  | 38499/50000 [25:12:50<5:59:45,  1.88s/it]
Outer training loop:  77%|#######7  | 38500/50000 [25:12:52<6:04:40,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.76s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:22,  1.76s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:27,  1.84s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:25,  1.84s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:23,  1.84s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:21,  1.84s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:22,  1.90s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:20,  1.90s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:15,  1.86s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:20,  1.95s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:15,  1.91s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.88s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.88s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.92s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:04,  1.86s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:01,  1.85s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:58,  1.82s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:56,  1.82s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:02,  1.95s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:47,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:51,  1.96s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:48,  1.94s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:45,  1.91s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.85s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:44,  2.00s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:39,  1.95s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:27,  1.85s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:28,  1.93s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.82s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:14,  1.82s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:12,  1.82s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:11,  1.87s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:08,  1.85s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.83s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:01,  1.80s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:59,  1.91s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:42<00:50,  1.80s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:47,  1.82s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:48,  1.94s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:57<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [01:59<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:30,  1.80s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:31,  1.94s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:28,  1.90s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:10<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:12<00:21,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:14,  1.87s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:14,  2.10s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:12,  2.02s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.98s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.96s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:19,  1.70s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:18,  1.71s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:33,  1.92s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.88s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:20,  1.90s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.90s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:22,  1.98s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:13,  1.96s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:11,  1.96s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:08,  1.95s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:04,  1.95s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:03,  1.96s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:57,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:53,  1.86s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:51,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:44,  1.86s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:41,  1.85s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.92s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:35,  1.87s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:32,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.86s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:33,  1.99s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:29,  1.95s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:25,  1.90s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:21,  1.88s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:21,  2.00s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:20,  2.02s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:16,  1.96s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:13,  1.93s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:08,  1.90s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.88s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:07,  1.99s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:03,  1.93s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:01,  1.92s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:55,  1.84s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:52,  1.80s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:50,  1.81s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:53,  1.99s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:50,  1.94s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:48,  1.95s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:43,  1.88s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:41,  1.87s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:37,  1.97s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:34,  1.91s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:32,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:21,  1.81s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:19,  1.79s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.95s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.91s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.90s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.87s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.85s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.81s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.86s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.85s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.81s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  2.02s/it]

Outer training loop:  77%|#######7  | 38501/50000 [25:18:13<312:03:26, 97.70s/it]
Outer training loop:  77%|#######7  | 38502/50000 [25:18:15<220:12:29, 68.95s/it]
Outer training loop:  77%|#######7  | 38503/50000 [25:18:16<155:53:59, 48.82s/it]
Outer training loop:  77%|#######7  | 38504/50000 [25:18:18<110:47:05, 34.69s/it]
Outer training loop:  77%|#######7  | 38505/50000 [25:18:20<79:30:11, 24.90s/it] 
Outer training loop:  77%|#######7  | 38506/50000 [25:18:22<57:21:39, 17.97s/it]
Outer training loop:  77%|#######7  | 38507/50000 [25:18:24<41:54:32, 13.13s/it]
Outer training loop:  77%|#######7  | 38508/50000 [25:18:26<31:02:13,  9.72s/it]
Outer training loop:  77%|#######7  | 38509/50000 [25:18:28<23:41:56,  7.42s/it]
Outer training loop:  77%|#######7  | 38510/50000 [25:18:29<18:14:23,  5.71s/it]
Outer training loop:  77%|#######7  | 38511/50000 [25:18:32<15:13:34,  4.77s/it]
Outer training loop:  77%|#######7  | 38512/50000 [25:18:34<12:24:29,  3.89s/it]
Outer training loop:  77%|#######7  | 38513/50000 [25:18:36<10:30:01,  3.29s/it]
Outer training loop:  77%|#######7  | 38514/50000 [25:18:38<9:05:37,  2.85s/it] 
Outer training loop:  77%|#######7  | 38515/50000 [25:18:40<8:22:44,  2.63s/it]
Outer training loop:  77%|#######7  | 38516/50000 [25:18:41<7:34:01,  2.37s/it]
Outer training loop:  77%|#######7  | 38517/50000 [25:18:43<7:07:52,  2.24s/it]
Outer training loop:  77%|#######7  | 38518/50000 [25:18:45<6:53:10,  2.16s/it]
Outer training loop:  77%|#######7  | 38519/50000 [25:18:47<6:39:15,  2.09s/it]
Outer training loop:  77%|#######7  | 38520/50000 [25:18:49<6:24:53,  2.01s/it]
Outer training loop:  77%|#######7  | 38521/50000 [25:18:51<6:33:16,  2.06s/it]
Outer training loop:  77%|#######7  | 38522/50000 [25:18:53<6:19:22,  1.98s/it]
Outer training loop:  77%|#######7  | 38523/50000 [25:18:55<6:08:56,  1.93s/it]
Outer training loop:  77%|#######7  | 38524/50000 [25:18:57<6:12:02,  1.95s/it]
Outer training loop:  77%|#######7  | 38525/50000 [25:18:59<6:09:46,  1.93s/it]
Outer training loop:  77%|#######7  | 38526/50000 [25:19:01<6:02:16,  1.89s/it]
Outer training loop:  77%|#######7  | 38527/50000 [25:19:02<5:58:39,  1.88s/it]
Outer training loop:  77%|#######7  | 38528/50000 [25:19:04<6:07:08,  1.92s/it]
Outer training loop:  77%|#######7  | 38529/50000 [25:19:06<6:02:42,  1.90s/it]
Outer training loop:  77%|#######7  | 38530/50000 [25:19:08<5:58:39,  1.88s/it]
Outer training loop:  77%|#######7  | 38531/50000 [25:19:10<6:04:15,  1.91s/it]
Outer training loop:  77%|#######7  | 38532/50000 [25:19:12<5:59:42,  1.88s/it]
Outer training loop:  77%|#######7  | 38533/50000 [25:19:14<5:56:24,  1.86s/it]
Outer training loop:  77%|#######7  | 38534/50000 [25:19:16<6:03:17,  1.90s/it]
Outer training loop:  77%|#######7  | 38535/50000 [25:19:18<5:55:55,  1.86s/it]
Outer training loop:  77%|#######7  | 38536/50000 [25:19:19<5:56:21,  1.87s/it]
Outer training loop:  77%|#######7  | 38537/50000 [25:19:21<6:09:38,  1.93s/it]
Outer training loop:  77%|#######7  | 38538/50000 [25:19:23<5:59:05,  1.88s/it]
Outer training loop:  77%|#######7  | 38539/50000 [25:19:25<6:05:08,  1.91s/it]
Outer training loop:  77%|#######7  | 38540/50000 [25:19:27<5:58:24,  1.88s/it]
Outer training loop:  77%|#######7  | 38541/50000 [25:19:29<6:17:08,  1.97s/it]
Outer training loop:  77%|#######7  | 38542/50000 [25:19:31<6:08:31,  1.93s/it]
Outer training loop:  77%|#######7  | 38543/50000 [25:19:33<5:55:13,  1.86s/it]
Outer training loop:  77%|#######7  | 38544/50000 [25:19:35<6:08:26,  1.93s/it]
Outer training loop:  77%|#######7  | 38545/50000 [25:19:37<6:06:39,  1.92s/it]
Outer training loop:  77%|#######7  | 38546/50000 [25:19:39<6:00:44,  1.89s/it]
Outer training loop:  77%|#######7  | 38547/50000 [25:19:40<5:53:39,  1.85s/it]
Outer training loop:  77%|#######7  | 38548/50000 [25:19:42<6:03:53,  1.91s/it]
Outer training loop:  77%|#######7  | 38549/50000 [25:19:44<5:54:33,  1.86s/it]
Outer training loop:  77%|#######7  | 38550/50000 [25:19:46<5:53:03,  1.85s/it]
Outer training loop:  77%|#######7  | 38551/50000 [25:19:48<5:46:46,  1.82s/it]
Outer training loop:  77%|#######7  | 38552/50000 [25:19:50<5:59:57,  1.89s/it]
Outer training loop:  77%|#######7  | 38553/50000 [25:19:51<5:53:24,  1.85s/it]
Outer training loop:  77%|#######7  | 38554/50000 [25:19:53<5:51:12,  1.84s/it]
Outer training loop:  77%|#######7  | 38555/50000 [25:19:55<5:59:11,  1.88s/it]
Outer training loop:  77%|#######7  | 38556/50000 [25:19:57<5:56:43,  1.87s/it]
Outer training loop:  77%|#######7  | 38557/50000 [25:19:59<6:03:09,  1.90s/it]
Outer training loop:  77%|#######7  | 38558/50000 [25:20:01<6:00:37,  1.89s/it]
Outer training loop:  77%|#######7  | 38559/50000 [25:20:03<6:08:13,  1.93s/it]
Outer training loop:  77%|#######7  | 38560/50000 [25:20:05<6:07:43,  1.93s/it]
Outer training loop:  77%|#######7  | 38561/50000 [25:20:07<6:07:59,  1.93s/it]
Outer training loop:  77%|#######7  | 38562/50000 [25:20:09<6:12:20,  1.95s/it]
Outer training loop:  77%|#######7  | 38563/50000 [25:20:11<6:02:38,  1.90s/it]
Outer training loop:  77%|#######7  | 38564/50000 [25:20:12<5:55:46,  1.87s/it]
Outer training loop:  77%|#######7  | 38565/50000 [25:20:14<6:07:48,  1.93s/it]
Outer training loop:  77%|#######7  | 38566/50000 [25:20:16<5:58:28,  1.88s/it]
Outer training loop:  77%|#######7  | 38567/50000 [25:20:18<5:56:25,  1.87s/it]
Outer training loop:  77%|#######7  | 38568/50000 [25:20:20<6:03:00,  1.91s/it]
Outer training loop:  77%|#######7  | 38569/50000 [25:20:22<5:54:20,  1.86s/it]
Outer training loop:  77%|#######7  | 38570/50000 [25:20:24<5:49:21,  1.83s/it]
Outer training loop:  77%|#######7  | 38571/50000 [25:20:25<5:48:11,  1.83s/it]
Outer training loop:  77%|#######7  | 38572/50000 [25:20:27<5:54:46,  1.86s/it]
Outer training loop:  77%|#######7  | 38573/50000 [25:20:29<5:49:42,  1.84s/it]
Outer training loop:  77%|#######7  | 38574/50000 [25:20:31<5:51:01,  1.84s/it]
Outer training loop:  77%|#######7  | 38575/50000 [25:20:33<5:52:21,  1.85s/it]
Outer training loop:  77%|#######7  | 38576/50000 [25:20:35<5:55:57,  1.87s/it]
Outer training loop:  77%|#######7  | 38577/50000 [25:20:37<5:51:25,  1.85s/it]
Outer training loop:  77%|#######7  | 38578/50000 [25:20:38<5:45:47,  1.82s/it]
Outer training loop:  77%|#######7  | 38579/50000 [25:20:40<5:55:48,  1.87s/it]
Outer training loop:  77%|#######7  | 38580/50000 [25:20:42<5:54:55,  1.86s/it]
Outer training loop:  77%|#######7  | 38581/50000 [25:20:44<5:52:50,  1.85s/it]
Outer training loop:  77%|#######7  | 38582/50000 [25:20:46<5:52:51,  1.85s/it]
Outer training loop:  77%|#######7  | 38583/50000 [25:20:48<6:02:29,  1.91s/it]
Outer training loop:  77%|#######7  | 38584/50000 [25:20:50<6:02:06,  1.90s/it]
Outer training loop:  77%|#######7  | 38585/50000 [25:20:52<5:54:06,  1.86s/it]
Outer training loop:  77%|#######7  | 38586/50000 [25:20:54<6:07:32,  1.93s/it]
Outer training loop:  77%|#######7  | 38587/50000 [25:20:55<6:01:27,  1.90s/it]
Outer training loop:  77%|#######7  | 38588/50000 [25:20:57<5:51:14,  1.85s/it]
Outer training loop:  77%|#######7  | 38589/50000 [25:20:59<5:54:39,  1.86s/it]
Outer training loop:  77%|#######7  | 38590/50000 [25:21:01<5:59:54,  1.89s/it]
Outer training loop:  77%|#######7  | 38591/50000 [25:21:03<5:56:29,  1.87s/it]
Outer training loop:  77%|#######7  | 38592/50000 [25:21:05<5:54:20,  1.86s/it]
Outer training loop:  77%|#######7  | 38593/50000 [25:21:07<5:57:39,  1.88s/it]
Outer training loop:  77%|#######7  | 38594/50000 [25:21:08<5:55:17,  1.87s/it]
Outer training loop:  77%|#######7  | 38595/50000 [25:21:10<6:00:38,  1.90s/it]
Outer training loop:  77%|#######7  | 38596/50000 [25:21:13<6:10:27,  1.95s/it]
Outer training loop:  77%|#######7  | 38597/50000 [25:21:14<6:04:55,  1.92s/it]
Outer training loop:  77%|#######7  | 38598/50000 [25:21:16<5:57:55,  1.88s/it]
Outer training loop:  77%|#######7  | 38599/50000 [25:21:18<5:48:12,  1.83s/it]
Outer training loop:  77%|#######7  | 38600/50000 [25:21:20<6:06:39,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  77%|#######7  | 38601/50000 [25:21:27<11:14:23,  3.55s/it]
Outer training loop:  77%|#######7  | 38602/50000 [25:21:29<9:28:47,  2.99s/it] 
Outer training loop:  77%|#######7  | 38603/50000 [25:21:31<8:15:10,  2.61s/it]
Outer training loop:  77%|#######7  | 38604/50000 [25:21:33<7:35:29,  2.40s/it]
Outer training loop:  77%|#######7  | 38605/50000 [25:21:35<7:04:02,  2.23s/it]
Outer training loop:  77%|#######7  | 38606/50000 [25:21:36<6:44:54,  2.13s/it]
Outer training loop:  77%|#######7  | 38607/50000 [25:21:38<6:37:47,  2.09s/it]
Outer training loop:  77%|#######7  | 38608/50000 [25:21:40<6:28:13,  2.04s/it]
Outer training loop:  77%|#######7  | 38609/50000 [25:21:42<6:15:03,  1.98s/it]
Outer training loop:  77%|#######7  | 38610/50000 [25:21:44<5:59:04,  1.89s/it]
Outer training loop:  77%|#######7  | 38611/50000 [25:21:46<6:01:16,  1.90s/it]
Outer training loop:  77%|#######7  | 38612/50000 [25:21:48<5:57:49,  1.89s/it]
Outer training loop:  77%|#######7  | 38613/50000 [25:21:50<5:57:23,  1.88s/it]
Outer training loop:  77%|#######7  | 38614/50000 [25:21:52<6:05:36,  1.93s/it]
Outer training loop:  77%|#######7  | 38615/50000 [25:21:53<6:05:25,  1.93s/it]
Outer training loop:  77%|#######7  | 38616/50000 [25:21:55<6:03:15,  1.91s/it]
Outer training loop:  77%|#######7  | 38617/50000 [25:21:57<6:10:08,  1.95s/it]
Outer training loop:  77%|#######7  | 38618/50000 [25:21:59<5:59:44,  1.90s/it]
Outer training loop:  77%|#######7  | 38619/50000 [25:22:01<5:55:28,  1.87s/it]
Outer training loop:  77%|#######7  | 38620/50000 [25:22:03<5:44:47,  1.82s/it]
Outer training loop:  77%|#######7  | 38621/50000 [25:22:05<5:53:14,  1.86s/it]
Outer training loop:  77%|#######7  | 38622/50000 [25:22:07<5:54:58,  1.87s/it]
Outer training loop:  77%|#######7  | 38623/50000 [25:22:08<5:57:41,  1.89s/it]
Outer training loop:  77%|#######7  | 38624/50000 [25:22:10<5:53:11,  1.86s/it]
Outer training loop:  77%|#######7  | 38625/50000 [25:22:12<6:05:53,  1.93s/it]
Outer training loop:  77%|#######7  | 38626/50000 [25:22:14<6:00:27,  1.90s/it]
Outer training loop:  77%|#######7  | 38627/50000 [25:22:16<5:53:10,  1.86s/it]
Outer training loop:  77%|#######7  | 38628/50000 [25:22:18<5:58:34,  1.89s/it]
Outer training loop:  77%|#######7  | 38629/50000 [25:22:20<5:55:38,  1.88s/it]
Outer training loop:  77%|#######7  | 38630/50000 [25:22:22<5:55:22,  1.88s/it]
Outer training loop:  77%|#######7  | 38631/50000 [25:22:24<6:07:11,  1.94s/it]
Outer training loop:  77%|#######7  | 38632/50000 [25:22:26<6:04:26,  1.92s/it]
Outer training loop:  77%|#######7  | 38633/50000 [25:22:27<6:01:31,  1.91s/it]
Outer training loop:  77%|#######7  | 38634/50000 [25:22:30<6:12:30,  1.97s/it]
Outer training loop:  77%|#######7  | 38635/50000 [25:22:31<6:04:13,  1.92s/it]
Outer training loop:  77%|#######7  | 38636/50000 [25:22:33<6:00:40,  1.90s/it]
Outer training loop:  77%|#######7  | 38637/50000 [25:22:35<5:58:26,  1.89s/it]
Outer training loop:  77%|#######7  | 38638/50000 [25:22:37<5:58:34,  1.89s/it]
Outer training loop:  77%|#######7  | 38639/50000 [25:22:39<5:58:22,  1.89s/it]
Outer training loop:  77%|#######7  | 38640/50000 [25:22:41<5:55:57,  1.88s/it]
Outer training loop:  77%|#######7  | 38641/50000 [25:22:43<6:04:35,  1.93s/it]
Outer training loop:  77%|#######7  | 38642/50000 [25:22:44<5:51:14,  1.86s/it]
Outer training loop:  77%|#######7  | 38643/50000 [25:22:46<5:45:03,  1.82s/it]
Outer training loop:  77%|#######7  | 38644/50000 [25:22:48<5:51:16,  1.86s/it]
Outer training loop:  77%|#######7  | 38645/50000 [25:22:50<5:47:32,  1.84s/it]
Outer training loop:  77%|#######7  | 38646/50000 [25:22:52<5:44:23,  1.82s/it]
Outer training loop:  77%|#######7  | 38647/50000 [25:22:54<5:47:53,  1.84s/it]
Outer training loop:  77%|#######7  | 38648/50000 [25:22:56<5:58:06,  1.89s/it]
Outer training loop:  77%|#######7  | 38649/50000 [25:22:58<6:01:24,  1.91s/it]
Outer training loop:  77%|#######7  | 38650/50000 [25:22:59<6:00:02,  1.90s/it]
Outer training loop:  77%|#######7  | 38651/50000 [25:23:01<6:03:56,  1.92s/it]
Outer training loop:  77%|#######7  | 38652/50000 [25:23:03<5:56:05,  1.88s/it]
Outer training loop:  77%|#######7  | 38653/50000 [25:23:05<5:54:15,  1.87s/it]
Outer training loop:  77%|#######7  | 38654/50000 [25:23:07<5:47:37,  1.84s/it]
Outer training loop:  77%|#######7  | 38655/50000 [25:23:09<5:58:01,  1.89s/it]
Outer training loop:  77%|#######7  | 38656/50000 [25:23:11<5:48:39,  1.84s/it]
Outer training loop:  77%|#######7  | 38657/50000 [25:23:12<5:51:28,  1.86s/it]
Outer training loop:  77%|#######7  | 38658/50000 [25:23:15<6:09:50,  1.96s/it]
Outer training loop:  77%|#######7  | 38659/50000 [25:23:17<6:03:22,  1.92s/it]
Outer training loop:  77%|#######7  | 38660/50000 [25:23:18<5:57:59,  1.89s/it]
Outer training loop:  77%|#######7  | 38661/50000 [25:23:21<6:12:53,  1.97s/it]
Outer training loop:  77%|#######7  | 38662/50000 [25:23:22<6:06:02,  1.94s/it]
Outer training loop:  77%|#######7  | 38663/50000 [25:23:24<6:00:43,  1.91s/it]
Outer training loop:  77%|#######7  | 38664/50000 [25:23:26<6:04:36,  1.93s/it]
Outer training loop:  77%|#######7  | 38665/50000 [25:23:28<6:03:39,  1.92s/it]
Outer training loop:  77%|#######7  | 38666/50000 [25:23:30<5:58:33,  1.90s/it]
Outer training loop:  77%|#######7  | 38667/50000 [25:23:32<5:51:18,  1.86s/it]
Outer training loop:  77%|#######7  | 38668/50000 [25:23:34<6:07:49,  1.95s/it]
Outer training loop:  77%|#######7  | 38669/50000 [25:23:36<6:02:51,  1.92s/it]
Outer training loop:  77%|#######7  | 38670/50000 [25:23:38<6:02:31,  1.92s/it]
Outer training loop:  77%|#######7  | 38671/50000 [25:23:40<6:11:30,  1.97s/it]
Outer training loop:  77%|#######7  | 38672/50000 [25:23:42<6:11:57,  1.97s/it]
Outer training loop:  77%|#######7  | 38673/50000 [25:23:44<6:05:59,  1.94s/it]
Outer training loop:  77%|#######7  | 38674/50000 [25:23:45<5:55:48,  1.88s/it]
Outer training loop:  77%|#######7  | 38675/50000 [25:23:47<5:56:08,  1.89s/it]
Outer training loop:  77%|#######7  | 38676/50000 [25:23:49<5:58:30,  1.90s/it]
Outer training loop:  77%|#######7  | 38677/50000 [25:23:51<5:53:45,  1.87s/it]
Outer training loop:  77%|#######7  | 38678/50000 [25:23:53<5:57:21,  1.89s/it]
Outer training loop:  77%|#######7  | 38679/50000 [25:23:55<5:51:51,  1.86s/it]
Outer training loop:  77%|#######7  | 38680/50000 [25:23:56<5:44:33,  1.83s/it]
Outer training loop:  77%|#######7  | 38681/50000 [25:23:58<5:58:15,  1.90s/it]
Outer training loop:  77%|#######7  | 38682/50000 [25:24:00<5:53:07,  1.87s/it]
Outer training loop:  77%|#######7  | 38683/50000 [25:24:02<5:44:31,  1.83s/it]
Outer training loop:  77%|#######7  | 38684/50000 [25:24:04<5:46:46,  1.84s/it]
Outer training loop:  77%|#######7  | 38685/50000 [25:24:06<6:01:30,  1.92s/it]
Outer training loop:  77%|#######7  | 38686/50000 [25:24:08<5:57:50,  1.90s/it]
Outer training loop:  77%|#######7  | 38687/50000 [25:24:10<5:54:23,  1.88s/it]
Outer training loop:  77%|#######7  | 38688/50000 [25:24:12<6:01:20,  1.92s/it]
Outer training loop:  77%|#######7  | 38689/50000 [25:24:14<6:07:10,  1.95s/it]
Outer training loop:  77%|#######7  | 38690/50000 [25:24:15<5:57:09,  1.89s/it]
Outer training loop:  77%|#######7  | 38691/50000 [25:24:17<6:01:55,  1.92s/it]
Outer training loop:  77%|#######7  | 38692/50000 [25:24:19<5:52:51,  1.87s/it]
Outer training loop:  77%|#######7  | 38693/50000 [25:24:21<5:43:04,  1.82s/it]
Outer training loop:  77%|#######7  | 38694/50000 [25:24:23<5:42:45,  1.82s/it]
Outer training loop:  77%|#######7  | 38695/50000 [25:24:25<5:56:40,  1.89s/it]
Outer training loop:  77%|#######7  | 38696/50000 [25:24:27<5:46:46,  1.84s/it]
Outer training loop:  77%|#######7  | 38697/50000 [25:24:28<5:39:36,  1.80s/it]
Outer training loop:  77%|#######7  | 38698/50000 [25:24:30<5:36:30,  1.79s/it]
Outer training loop:  77%|#######7  | 38699/50000 [25:24:32<5:43:56,  1.83s/it]
Outer training loop:  77%|#######7  | 38700/50000 [25:24:34<5:42:47,  1.82s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.93s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  77%|#######7  | 38701/50000 [25:24:41<10:56:49,  3.49s/it]
Outer training loop:  77%|#######7  | 38702/50000 [25:24:43<9:28:04,  3.02s/it] 
Outer training loop:  77%|#######7  | 38703/50000 [25:24:45<8:33:54,  2.73s/it]
Outer training loop:  77%|#######7  | 38704/50000 [25:24:47<7:38:58,  2.44s/it]
Outer training loop:  77%|#######7  | 38705/50000 [25:24:49<7:10:58,  2.29s/it]
Outer training loop:  77%|#######7  | 38706/50000 [25:24:51<6:54:57,  2.20s/it]
Outer training loop:  77%|#######7  | 38707/50000 [25:24:53<6:32:38,  2.09s/it]
Outer training loop:  77%|#######7  | 38708/50000 [25:24:55<6:26:45,  2.06s/it]
Outer training loop:  77%|#######7  | 38709/50000 [25:24:56<6:13:50,  1.99s/it]
Outer training loop:  77%|#######7  | 38710/50000 [25:24:58<6:11:17,  1.97s/it]
Outer training loop:  77%|#######7  | 38711/50000 [25:25:00<6:05:12,  1.94s/it]
Outer training loop:  77%|#######7  | 38712/50000 [25:25:02<5:57:34,  1.90s/it]
Outer training loop:  77%|#######7  | 38713/50000 [25:25:04<6:15:29,  2.00s/it]
Outer training loop:  77%|#######7  | 38714/50000 [25:25:06<6:02:35,  1.93s/it]
Outer training loop:  77%|#######7  | 38715/50000 [25:25:08<5:54:34,  1.89s/it]
Outer training loop:  77%|#######7  | 38716/50000 [25:25:10<6:03:27,  1.93s/it]
Outer training loop:  77%|#######7  | 38717/50000 [25:25:12<5:57:12,  1.90s/it]
Outer training loop:  77%|#######7  | 38718/50000 [25:25:13<5:51:19,  1.87s/it]
Outer training loop:  77%|#######7  | 38719/50000 [25:25:15<5:45:04,  1.84s/it]
Outer training loop:  77%|#######7  | 38720/50000 [25:25:17<5:53:38,  1.88s/it]
Outer training loop:  77%|#######7  | 38721/50000 [25:25:19<5:46:31,  1.84s/it]
Outer training loop:  77%|#######7  | 38722/50000 [25:25:21<5:44:15,  1.83s/it]
Outer training loop:  77%|#######7  | 38723/50000 [25:25:23<5:53:40,  1.88s/it]
Outer training loop:  77%|#######7  | 38724/50000 [25:25:25<5:47:15,  1.85s/it]
Outer training loop:  77%|#######7  | 38725/50000 [25:25:26<5:52:51,  1.88s/it]
Outer training loop:  77%|#######7  | 38726/50000 [25:25:29<6:05:59,  1.95s/it]
Outer training loop:  77%|#######7  | 38727/50000 [25:25:30<5:58:48,  1.91s/it]
Outer training loop:  77%|#######7  | 38728/50000 [25:25:32<5:56:32,  1.90s/it]
Outer training loop:  77%|#######7  | 38729/50000 [25:25:34<5:54:54,  1.89s/it]
Outer training loop:  77%|#######7  | 38730/50000 [25:25:36<5:58:07,  1.91s/it]
Outer training loop:  77%|#######7  | 38731/50000 [25:25:38<5:52:20,  1.88s/it]
Outer training loop:  77%|#######7  | 38732/50000 [25:25:40<5:48:22,  1.86s/it]
Outer training loop:  77%|#######7  | 38733/50000 [25:25:42<6:00:01,  1.92s/it]
Outer training loop:  77%|#######7  | 38734/50000 [25:25:44<5:53:38,  1.88s/it]
Outer training loop:  77%|#######7  | 38735/50000 [25:25:45<5:50:05,  1.86s/it]
Outer training loop:  77%|#######7  | 38736/50000 [25:25:47<5:43:33,  1.83s/it]
Outer training loop:  77%|#######7  | 38737/50000 [25:25:49<5:54:55,  1.89s/it]
Outer training loop:  77%|#######7  | 38738/50000 [25:25:51<5:45:15,  1.84s/it]
Outer training loop:  77%|#######7  | 38739/50000 [25:25:53<5:42:39,  1.83s/it]
Outer training loop:  77%|#######7  | 38740/50000 [25:25:55<5:54:48,  1.89s/it]
Outer training loop:  77%|#######7  | 38741/50000 [25:25:57<5:56:41,  1.90s/it]
Outer training loop:  77%|#######7  | 38742/50000 [25:25:59<6:01:29,  1.93s/it]
Outer training loop:  77%|#######7  | 38743/50000 [25:26:01<6:06:25,  1.95s/it]
Outer training loop:  77%|#######7  | 38744/50000 [25:26:02<5:58:17,  1.91s/it]
Outer training loop:  77%|#######7  | 38745/50000 [25:26:04<5:57:04,  1.90s/it]
Outer training loop:  77%|#######7  | 38746/50000 [25:26:06<5:56:31,  1.90s/it]
Outer training loop:  77%|#######7  | 38747/50000 [25:26:08<5:58:43,  1.91s/it]
Outer training loop:  77%|#######7  | 38748/50000 [25:26:10<5:54:12,  1.89s/it]
Outer training loop:  77%|#######7  | 38749/50000 [25:26:12<5:54:58,  1.89s/it]
Outer training loop:  78%|#######7  | 38750/50000 [25:26:14<6:07:29,  1.96s/it]
Outer training loop:  78%|#######7  | 38751/50000 [25:26:16<6:01:15,  1.93s/it]
Outer training loop:  78%|#######7  | 38752/50000 [25:26:18<5:49:45,  1.87s/it]
Outer training loop:  78%|#######7  | 38753/50000 [25:26:19<5:50:17,  1.87s/it]
Outer training loop:  78%|#######7  | 38754/50000 [25:26:22<6:00:26,  1.92s/it]
Outer training loop:  78%|#######7  | 38755/50000 [25:26:24<6:04:10,  1.94s/it]
Outer training loop:  78%|#######7  | 38756/50000 [25:26:25<6:02:45,  1.94s/it]
Outer training loop:  78%|#######7  | 38757/50000 [25:26:27<6:09:35,  1.97s/it]
Outer training loop:  78%|#######7  | 38758/50000 [25:26:29<5:57:40,  1.91s/it]
Outer training loop:  78%|#######7  | 38759/50000 [25:26:31<5:51:03,  1.87s/it]
Outer training loop:  78%|#######7  | 38760/50000 [25:26:33<5:53:14,  1.89s/it]
Outer training loop:  78%|#######7  | 38761/50000 [25:26:35<5:50:18,  1.87s/it]
Outer training loop:  78%|#######7  | 38762/50000 [25:26:37<5:49:25,  1.87s/it]
Outer training loop:  78%|#######7  | 38763/50000 [25:26:38<5:45:31,  1.84s/it]
Outer training loop:  78%|#######7  | 38764/50000 [25:26:40<5:55:34,  1.90s/it]
Outer training loop:  78%|#######7  | 38765/50000 [25:26:42<5:54:54,  1.90s/it]
Outer training loop:  78%|#######7  | 38766/50000 [25:26:44<5:52:16,  1.88s/it]
Outer training loop:  78%|#######7  | 38767/50000 [25:26:46<5:58:43,  1.92s/it]
Outer training loop:  78%|#######7  | 38768/50000 [25:26:48<5:52:36,  1.88s/it]
Outer training loop:  78%|#######7  | 38769/50000 [25:26:50<5:56:50,  1.91s/it]
Outer training loop:  78%|#######7  | 38770/50000 [25:26:52<6:07:20,  1.96s/it]
Outer training loop:  78%|#######7  | 38771/50000 [25:26:54<5:53:19,  1.89s/it]
Outer training loop:  78%|#######7  | 38772/50000 [25:26:56<5:49:53,  1.87s/it]
Outer training loop:  78%|#######7  | 38773/50000 [25:26:58<6:02:11,  1.94s/it]
Outer training loop:  78%|#######7  | 38774/50000 [25:26:59<5:53:53,  1.89s/it]
Outer training loop:  78%|#######7  | 38775/50000 [25:27:01<5:54:16,  1.89s/it]
Outer training loop:  78%|#######7  | 38776/50000 [25:27:04<6:07:51,  1.97s/it]
Outer training loop:  78%|#######7  | 38777/50000 [25:27:05<5:55:40,  1.90s/it]
Outer training loop:  78%|#######7  | 38778/50000 [25:27:07<5:50:07,  1.87s/it]
Outer training loop:  78%|#######7  | 38779/50000 [25:27:09<5:48:00,  1.86s/it]
Outer training loop:  78%|#######7  | 38780/50000 [25:27:11<5:58:14,  1.92s/it]
Outer training loop:  78%|#######7  | 38781/50000 [25:27:13<5:49:41,  1.87s/it]
Outer training loop:  78%|#######7  | 38782/50000 [25:27:14<5:44:11,  1.84s/it]
Outer training loop:  78%|#######7  | 38783/50000 [25:27:16<5:41:28,  1.83s/it]
Outer training loop:  78%|#######7  | 38784/50000 [25:27:18<5:49:49,  1.87s/it]
Outer training loop:  78%|#######7  | 38785/50000 [25:27:20<5:55:54,  1.90s/it]
Outer training loop:  78%|#######7  | 38786/50000 [25:27:22<5:56:24,  1.91s/it]
Outer training loop:  78%|#######7  | 38787/50000 [25:27:24<6:00:34,  1.93s/it]
Outer training loop:  78%|#######7  | 38788/50000 [25:27:26<5:56:11,  1.91s/it]
Outer training loop:  78%|#######7  | 38789/50000 [25:27:28<5:52:13,  1.89s/it]
Outer training loop:  78%|#######7  | 38790/50000 [25:27:30<5:57:56,  1.92s/it]
Outer training loop:  78%|#######7  | 38791/50000 [25:27:32<5:48:51,  1.87s/it]
Outer training loop:  78%|#######7  | 38792/50000 [25:27:33<5:44:09,  1.84s/it]
Outer training loop:  78%|#######7  | 38793/50000 [25:27:35<6:01:31,  1.94s/it]
Outer training loop:  78%|#######7  | 38794/50000 [25:27:37<5:52:51,  1.89s/it]
Outer training loop:  78%|#######7  | 38795/50000 [25:27:39<5:55:35,  1.90s/it]
Outer training loop:  78%|#######7  | 38796/50000 [25:27:41<5:43:07,  1.84s/it]
Outer training loop:  78%|#######7  | 38797/50000 [25:27:43<5:53:48,  1.89s/it]
Outer training loop:  78%|#######7  | 38798/50000 [25:27:45<5:44:19,  1.84s/it]
Outer training loop:  78%|#######7  | 38799/50000 [25:27:46<5:34:57,  1.79s/it]
Outer training loop:  78%|#######7  | 38800/50000 [25:27:48<5:35:46,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  78%|#######7  | 38801/50000 [25:27:56<10:53:49,  3.50s/it]
Outer training loop:  78%|#######7  | 38802/50000 [25:27:58<9:31:58,  3.06s/it] 
Outer training loop:  78%|#######7  | 38803/50000 [25:27:59<8:21:29,  2.69s/it]
Outer training loop:  78%|#######7  | 38804/50000 [25:28:02<7:48:15,  2.51s/it]
Outer training loop:  78%|#######7  | 38805/50000 [25:28:03<7:04:19,  2.27s/it]
Outer training loop:  78%|#######7  | 38806/50000 [25:28:05<6:48:39,  2.19s/it]
Outer training loop:  78%|#######7  | 38807/50000 [25:28:07<6:30:19,  2.09s/it]
Outer training loop:  78%|#######7  | 38808/50000 [25:28:09<6:30:54,  2.10s/it]
Outer training loop:  78%|#######7  | 38809/50000 [25:28:11<6:15:13,  2.01s/it]
Outer training loop:  78%|#######7  | 38810/50000 [25:28:13<6:04:37,  1.96s/it]
Outer training loop:  78%|#######7  | 38811/50000 [25:28:15<6:08:27,  1.98s/it]
Outer training loop:  78%|#######7  | 38812/50000 [25:28:17<6:00:47,  1.93s/it]
Outer training loop:  78%|#######7  | 38813/50000 [25:28:19<5:55:21,  1.91s/it]
Outer training loop:  78%|#######7  | 38814/50000 [25:28:20<5:47:28,  1.86s/it]
Outer training loop:  78%|#######7  | 38815/50000 [25:28:22<5:56:39,  1.91s/it]
Outer training loop:  78%|#######7  | 38816/50000 [25:28:24<5:50:09,  1.88s/it]
Outer training loop:  78%|#######7  | 38817/50000 [25:28:26<5:55:15,  1.91s/it]
Outer training loop:  78%|#######7  | 38818/50000 [25:28:28<6:01:51,  1.94s/it]
Outer training loop:  78%|#######7  | 38819/50000 [25:28:30<5:51:00,  1.88s/it]
Outer training loop:  78%|#######7  | 38820/50000 [25:28:32<5:48:04,  1.87s/it]
Outer training loop:  78%|#######7  | 38821/50000 [25:28:34<6:00:13,  1.93s/it]
Outer training loop:  78%|#######7  | 38822/50000 [25:28:36<5:55:27,  1.91s/it]
Outer training loop:  78%|#######7  | 38823/50000 [25:28:38<5:51:42,  1.89s/it]
Outer training loop:  78%|#######7  | 38824/50000 [25:28:39<5:41:47,  1.83s/it]
Outer training loop:  78%|#######7  | 38825/50000 [25:28:41<5:51:39,  1.89s/it]
Outer training loop:  78%|#######7  | 38826/50000 [25:28:43<5:48:00,  1.87s/it]
Outer training loop:  78%|#######7  | 38827/50000 [25:28:45<5:39:32,  1.82s/it]
Outer training loop:  78%|#######7  | 38828/50000 [25:28:47<5:46:51,  1.86s/it]
Outer training loop:  78%|#######7  | 38829/50000 [25:28:49<5:44:34,  1.85s/it]
Outer training loop:  78%|#######7  | 38830/50000 [25:28:50<5:45:37,  1.86s/it]
Outer training loop:  78%|#######7  | 38831/50000 [25:28:52<5:46:44,  1.86s/it]
Outer training loop:  78%|#######7  | 38832/50000 [25:28:54<5:55:13,  1.91s/it]
Outer training loop:  78%|#######7  | 38833/50000 [25:28:56<5:44:03,  1.85s/it]
Outer training loop:  78%|#######7  | 38834/50000 [25:28:58<5:44:49,  1.85s/it]
Outer training loop:  78%|#######7  | 38835/50000 [25:29:00<5:56:41,  1.92s/it]
Outer training loop:  78%|#######7  | 38836/50000 [25:29:02<5:52:25,  1.89s/it]
Outer training loop:  78%|#######7  | 38837/50000 [25:29:04<5:41:42,  1.84s/it]
Outer training loop:  78%|#######7  | 38838/50000 [25:29:05<5:45:23,  1.86s/it]
Outer training loop:  78%|#######7  | 38839/50000 [25:29:07<5:51:52,  1.89s/it]
Outer training loop:  78%|#######7  | 38840/50000 [25:29:09<5:39:49,  1.83s/it]
Outer training loop:  78%|#######7  | 38841/50000 [25:29:11<5:43:31,  1.85s/it]
Outer training loop:  78%|#######7  | 38842/50000 [25:29:13<5:51:04,  1.89s/it]
Outer training loop:  78%|#######7  | 38843/50000 [25:29:15<5:43:19,  1.85s/it]
Outer training loop:  78%|#######7  | 38844/50000 [25:29:17<5:42:52,  1.84s/it]
Outer training loop:  78%|#######7  | 38845/50000 [25:29:19<5:51:37,  1.89s/it]
Outer training loop:  78%|#######7  | 38846/50000 [25:29:20<5:41:32,  1.84s/it]
Outer training loop:  78%|#######7  | 38847/50000 [25:29:22<5:38:28,  1.82s/it]
Outer training loop:  78%|#######7  | 38848/50000 [25:29:24<5:43:17,  1.85s/it]
Outer training loop:  78%|#######7  | 38849/50000 [25:29:26<5:53:05,  1.90s/it]
Outer training loop:  78%|#######7  | 38850/50000 [25:29:28<5:51:29,  1.89s/it]
Outer training loop:  78%|#######7  | 38851/50000 [25:29:30<5:46:19,  1.86s/it]
Outer training loop:  78%|#######7  | 38852/50000 [25:29:32<6:08:05,  1.98s/it]
Outer training loop:  78%|#######7  | 38853/50000 [25:29:34<5:54:41,  1.91s/it]
Outer training loop:  78%|#######7  | 38854/50000 [25:29:35<5:51:28,  1.89s/it]
Outer training loop:  78%|#######7  | 38855/50000 [25:29:37<5:58:36,  1.93s/it]
Outer training loop:  78%|#######7  | 38856/50000 [25:29:39<5:50:06,  1.89s/it]
Outer training loop:  78%|#######7  | 38857/50000 [25:29:41<5:39:22,  1.83s/it]
Outer training loop:  78%|#######7  | 38858/50000 [25:29:43<5:41:07,  1.84s/it]
Outer training loop:  78%|#######7  | 38859/50000 [25:29:45<5:53:48,  1.91s/it]
Outer training loop:  78%|#######7  | 38860/50000 [25:29:47<5:44:53,  1.86s/it]
Outer training loop:  78%|#######7  | 38861/50000 [25:29:48<5:35:37,  1.81s/it]
Outer training loop:  78%|#######7  | 38862/50000 [25:29:50<5:43:29,  1.85s/it]
Outer training loop:  78%|#######7  | 38863/50000 [25:29:52<5:51:00,  1.89s/it]
Outer training loop:  78%|#######7  | 38864/50000 [25:29:54<5:41:43,  1.84s/it]
Outer training loop:  78%|#######7  | 38865/50000 [25:29:56<5:38:00,  1.82s/it]
Outer training loop:  78%|#######7  | 38866/50000 [25:29:58<5:53:25,  1.90s/it]
Outer training loop:  78%|#######7  | 38867/50000 [25:30:00<5:48:53,  1.88s/it]
Outer training loop:  78%|#######7  | 38868/50000 [25:30:01<5:44:17,  1.86s/it]
Outer training loop:  78%|#######7  | 38869/50000 [25:30:03<5:47:19,  1.87s/it]
Outer training loop:  78%|#######7  | 38870/50000 [25:30:05<5:45:13,  1.86s/it]
Outer training loop:  78%|#######7  | 38871/50000 [25:30:07<5:45:42,  1.86s/it]
Outer training loop:  78%|#######7  | 38872/50000 [25:30:09<5:43:34,  1.85s/it]
Outer training loop:  78%|#######7  | 38873/50000 [25:30:11<5:57:07,  1.93s/it]
Outer training loop:  78%|#######7  | 38874/50000 [25:30:13<5:58:10,  1.93s/it]
Outer training loop:  78%|#######7  | 38875/50000 [25:30:15<5:51:37,  1.90s/it]
Outer training loop:  78%|#######7  | 38876/50000 [25:30:17<6:01:09,  1.95s/it]
Outer training loop:  78%|#######7  | 38877/50000 [25:30:19<5:50:24,  1.89s/it]
Outer training loop:  78%|#######7  | 38878/50000 [25:30:21<5:51:14,  1.89s/it]
Outer training loop:  78%|#######7  | 38879/50000 [25:30:22<5:53:48,  1.91s/it]
Outer training loop:  78%|#######7  | 38880/50000 [25:30:24<5:55:34,  1.92s/it]
Outer training loop:  78%|#######7  | 38881/50000 [25:30:26<5:52:03,  1.90s/it]
Outer training loop:  78%|#######7  | 38882/50000 [25:30:28<6:04:56,  1.97s/it]
Outer training loop:  78%|#######7  | 38883/50000 [25:30:30<5:54:39,  1.91s/it]
Outer training loop:  78%|#######7  | 38884/50000 [25:30:32<5:50:56,  1.89s/it]
Outer training loop:  78%|#######7  | 38885/50000 [25:30:34<6:00:57,  1.95s/it]
Outer training loop:  78%|#######7  | 38886/50000 [25:30:36<5:51:49,  1.90s/it]
Outer training loop:  78%|#######7  | 38887/50000 [25:30:38<5:42:55,  1.85s/it]
Outer training loop:  78%|#######7  | 38888/50000 [25:30:39<5:36:42,  1.82s/it]
Outer training loop:  78%|#######7  | 38889/50000 [25:30:41<5:45:28,  1.87s/it]
Outer training loop:  78%|#######7  | 38890/50000 [25:30:43<5:43:48,  1.86s/it]
Outer training loop:  78%|#######7  | 38891/50000 [25:30:45<5:41:12,  1.84s/it]
Outer training loop:  78%|#######7  | 38892/50000 [25:30:47<5:47:59,  1.88s/it]
Outer training loop:  78%|#######7  | 38893/50000 [25:30:49<5:46:03,  1.87s/it]
Outer training loop:  78%|#######7  | 38894/50000 [25:30:51<5:45:32,  1.87s/it]
Outer training loop:  78%|#######7  | 38895/50000 [25:30:52<5:44:20,  1.86s/it]
Outer training loop:  78%|#######7  | 38896/50000 [25:30:55<5:55:42,  1.92s/it]
Outer training loop:  78%|#######7  | 38897/50000 [25:30:56<5:53:51,  1.91s/it]
Outer training loop:  78%|#######7  | 38898/50000 [25:30:58<5:47:11,  1.88s/it]
Outer training loop:  78%|#######7  | 38899/50000 [25:31:00<5:56:20,  1.93s/it]
Outer training loop:  78%|#######7  | 38900/50000 [25:31:02<5:46:31,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.27s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.15s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  2.00s/it]

Outer training loop:  78%|#######7  | 38901/50000 [25:31:10<10:59:36,  3.57s/it]
Outer training loop:  78%|#######7  | 38902/50000 [25:31:11<9:20:14,  3.03s/it] 
Outer training loop:  78%|#######7  | 38903/50000 [25:31:13<8:23:10,  2.72s/it]
Outer training loop:  78%|#######7  | 38904/50000 [25:31:15<7:32:06,  2.44s/it]
Outer training loop:  78%|#######7  | 38905/50000 [25:31:17<6:51:02,  2.22s/it]
Outer training loop:  78%|#######7  | 38906/50000 [25:31:19<6:42:05,  2.17s/it]
Outer training loop:  78%|#######7  | 38907/50000 [25:31:21<6:30:17,  2.11s/it]
Outer training loop:  78%|#######7  | 38908/50000 [25:31:23<6:17:00,  2.04s/it]
Outer training loop:  78%|#######7  | 38909/50000 [25:31:25<6:05:36,  1.98s/it]
Outer training loop:  78%|#######7  | 38910/50000 [25:31:27<6:06:41,  1.98s/it]
Outer training loop:  78%|#######7  | 38911/50000 [25:31:28<6:02:11,  1.96s/it]
Outer training loop:  78%|#######7  | 38912/50000 [25:31:30<5:55:12,  1.92s/it]
Outer training loop:  78%|#######7  | 38913/50000 [25:31:32<6:04:53,  1.97s/it]
Outer training loop:  78%|#######7  | 38914/50000 [25:31:34<5:57:39,  1.94s/it]
Outer training loop:  78%|#######7  | 38915/50000 [25:31:36<5:53:39,  1.91s/it]
Outer training loop:  78%|#######7  | 38916/50000 [25:31:38<6:04:39,  1.97s/it]
Outer training loop:  78%|#######7  | 38917/50000 [25:31:40<5:59:45,  1.95s/it]
Outer training loop:  78%|#######7  | 38918/50000 [25:31:42<5:48:21,  1.89s/it]
Outer training loop:  78%|#######7  | 38919/50000 [25:31:44<5:44:00,  1.86s/it]
Outer training loop:  78%|#######7  | 38920/50000 [25:31:46<5:48:51,  1.89s/it]
Outer training loop:  78%|#######7  | 38921/50000 [25:31:48<5:53:18,  1.91s/it]
Outer training loop:  78%|#######7  | 38922/50000 [25:31:49<5:48:02,  1.89s/it]
Outer training loop:  78%|#######7  | 38923/50000 [25:31:51<5:54:39,  1.92s/it]
Outer training loop:  78%|#######7  | 38924/50000 [25:31:53<5:53:27,  1.91s/it]
Outer training loop:  78%|#######7  | 38925/50000 [25:31:55<5:50:02,  1.90s/it]
Outer training loop:  78%|#######7  | 38926/50000 [25:31:57<5:42:25,  1.86s/it]
Outer training loop:  78%|#######7  | 38927/50000 [25:31:59<5:57:57,  1.94s/it]
Outer training loop:  78%|#######7  | 38928/50000 [25:32:01<5:52:47,  1.91s/it]
Outer training loop:  78%|#######7  | 38929/50000 [25:32:03<5:52:43,  1.91s/it]
Outer training loop:  78%|#######7  | 38930/50000 [25:32:05<6:06:38,  1.99s/it]
Outer training loop:  78%|#######7  | 38931/50000 [25:32:07<6:05:14,  1.98s/it]
Outer training loop:  78%|#######7  | 38932/50000 [25:32:09<5:58:49,  1.95s/it]
Outer training loop:  78%|#######7  | 38933/50000 [25:32:11<6:05:39,  1.98s/it]
Outer training loop:  78%|#######7  | 38934/50000 [25:32:13<6:01:38,  1.96s/it]
Outer training loop:  78%|#######7  | 38935/50000 [25:32:15<5:58:57,  1.95s/it]
Outer training loop:  78%|#######7  | 38936/50000 [25:32:17<5:59:26,  1.95s/it]
Outer training loop:  78%|#######7  | 38937/50000 [25:32:18<5:48:00,  1.89s/it]
Outer training loop:  78%|#######7  | 38938/50000 [25:32:20<5:48:25,  1.89s/it]
Outer training loop:  78%|#######7  | 38939/50000 [25:32:22<5:42:01,  1.86s/it]
Outer training loop:  78%|#######7  | 38940/50000 [25:32:24<5:52:34,  1.91s/it]
Outer training loop:  78%|#######7  | 38941/50000 [25:32:26<5:50:16,  1.90s/it]
Outer training loop:  78%|#######7  | 38942/50000 [25:32:28<5:43:24,  1.86s/it]
Outer training loop:  78%|#######7  | 38943/50000 [25:32:30<5:43:50,  1.87s/it]
Outer training loop:  78%|#######7  | 38944/50000 [25:32:31<5:39:20,  1.84s/it]
Outer training loop:  78%|#######7  | 38945/50000 [25:32:33<5:41:44,  1.85s/it]
Outer training loop:  78%|#######7  | 38946/50000 [25:32:35<5:41:55,  1.86s/it]
Outer training loop:  78%|#######7  | 38947/50000 [25:32:37<5:50:32,  1.90s/it]
Outer training loop:  78%|#######7  | 38948/50000 [25:32:39<5:41:15,  1.85s/it]
Outer training loop:  78%|#######7  | 38949/50000 [25:32:41<5:51:15,  1.91s/it]
Outer training loop:  78%|#######7  | 38950/50000 [25:32:43<5:59:53,  1.95s/it]
Outer training loop:  78%|#######7  | 38951/50000 [25:32:45<5:52:41,  1.92s/it]
Outer training loop:  78%|#######7  | 38952/50000 [25:32:47<5:40:23,  1.85s/it]
Outer training loop:  78%|#######7  | 38953/50000 [25:32:49<5:54:36,  1.93s/it]
Outer training loop:  78%|#######7  | 38954/50000 [25:32:50<5:49:28,  1.90s/it]
Outer training loop:  78%|#######7  | 38955/50000 [25:32:52<5:39:14,  1.84s/it]
Outer training loop:  78%|#######7  | 38956/50000 [25:32:54<5:36:25,  1.83s/it]
Outer training loop:  78%|#######7  | 38957/50000 [25:32:56<5:48:37,  1.89s/it]
Outer training loop:  78%|#######7  | 38958/50000 [25:32:58<5:48:47,  1.90s/it]
Outer training loop:  78%|#######7  | 38959/50000 [25:33:00<5:42:31,  1.86s/it]
Outer training loop:  78%|#######7  | 38960/50000 [25:33:02<5:48:33,  1.89s/it]
Outer training loop:  78%|#######7  | 38961/50000 [25:33:03<5:40:59,  1.85s/it]
Outer training loop:  78%|#######7  | 38962/50000 [25:33:05<5:38:27,  1.84s/it]
Outer training loop:  78%|#######7  | 38963/50000 [25:33:07<5:35:44,  1.83s/it]
Outer training loop:  78%|#######7  | 38964/50000 [25:33:09<5:50:37,  1.91s/it]
Outer training loop:  78%|#######7  | 38965/50000 [25:33:11<5:49:51,  1.90s/it]
Outer training loop:  78%|#######7  | 38966/50000 [25:33:13<5:44:55,  1.88s/it]
Outer training loop:  78%|#######7  | 38967/50000 [25:33:15<5:53:13,  1.92s/it]
Outer training loop:  78%|#######7  | 38968/50000 [25:33:17<5:42:55,  1.87s/it]
Outer training loop:  78%|#######7  | 38969/50000 [25:33:18<5:40:32,  1.85s/it]
Outer training loop:  78%|#######7  | 38970/50000 [25:33:20<5:44:28,  1.87s/it]
Outer training loop:  78%|#######7  | 38971/50000 [25:33:22<5:52:52,  1.92s/it]
Outer training loop:  78%|#######7  | 38972/50000 [25:33:24<5:47:58,  1.89s/it]
Outer training loop:  78%|#######7  | 38973/50000 [25:33:26<5:44:42,  1.88s/it]
Outer training loop:  78%|#######7  | 38974/50000 [25:33:28<5:50:37,  1.91s/it]
Outer training loop:  78%|#######7  | 38975/50000 [25:33:30<5:44:29,  1.87s/it]
Outer training loop:  78%|#######7  | 38976/50000 [25:33:32<5:37:36,  1.84s/it]
Outer training loop:  78%|#######7  | 38977/50000 [25:33:33<5:34:33,  1.82s/it]
Outer training loop:  78%|#######7  | 38978/50000 [25:33:35<5:46:21,  1.89s/it]
Outer training loop:  78%|#######7  | 38979/50000 [25:33:37<5:36:41,  1.83s/it]
Outer training loop:  78%|#######7  | 38980/50000 [25:33:39<5:36:37,  1.83s/it]
Outer training loop:  78%|#######7  | 38981/50000 [25:33:41<5:43:15,  1.87s/it]
Outer training loop:  78%|#######7  | 38982/50000 [25:33:43<5:42:41,  1.87s/it]
Outer training loop:  78%|#######7  | 38983/50000 [25:33:45<5:43:34,  1.87s/it]
Outer training loop:  78%|#######7  | 38984/50000 [25:33:47<5:52:49,  1.92s/it]
Outer training loop:  78%|#######7  | 38985/50000 [25:33:49<5:51:53,  1.92s/it]
Outer training loop:  78%|#######7  | 38986/50000 [25:33:50<5:50:55,  1.91s/it]
Outer training loop:  78%|#######7  | 38987/50000 [25:33:52<5:56:21,  1.94s/it]
Outer training loop:  78%|#######7  | 38988/50000 [25:33:54<5:44:21,  1.88s/it]
Outer training loop:  78%|#######7  | 38989/50000 [25:33:56<5:36:45,  1.84s/it]
Outer training loop:  78%|#######7  | 38990/50000 [25:33:58<5:38:18,  1.84s/it]
Outer training loop:  78%|#######7  | 38991/50000 [25:34:00<5:44:49,  1.88s/it]
Outer training loop:  78%|#######7  | 38992/50000 [25:34:01<5:33:20,  1.82s/it]
Outer training loop:  78%|#######7  | 38993/50000 [25:34:03<5:38:02,  1.84s/it]
Outer training loop:  78%|#######7  | 38994/50000 [25:34:05<5:54:46,  1.93s/it]
Outer training loop:  78%|#######7  | 38995/50000 [25:34:07<5:44:17,  1.88s/it]
Outer training loop:  78%|#######7  | 38996/50000 [25:34:09<5:38:11,  1.84s/it]
Outer training loop:  78%|#######7  | 38997/50000 [25:34:11<5:39:35,  1.85s/it]
Outer training loop:  78%|#######7  | 38998/50000 [25:34:13<5:56:15,  1.94s/it]
Outer training loop:  78%|#######7  | 38999/50000 [25:34:15<5:59:35,  1.96s/it]
Outer training loop:  78%|#######8  | 39000/50000 [25:34:17<5:49:31,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.86s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:47,  2.04s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:42,  2.00s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:38,  1.98s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:36,  1.98s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:32,  1.95s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.93s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:25,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:28,  1.98s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:27,  1.99s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:22,  1.95s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:21,  1.99s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:16,  1.94s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:07,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:03,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:03,  1.96s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:59,  1.93s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.89s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:42,  1.86s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:41,  1.96s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.93s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.90s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:26,  1.84s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:29,  1.94s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:22,  1.87s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.82s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:15,  1.83s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:12,  1.82s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.91s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:08,  1.85s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:01,  1.82s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:50,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.84s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.85s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.80s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.94s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.92s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:21,  1.81s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.80s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:14,  1.83s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.98s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.95s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.92s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.80s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:20,  1.72s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:20,  1.74s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:34,  1.93s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:32,  1.92s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.88s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:20,  1.83s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:20,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:21,  1.94s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:20,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:14,  1.92s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:17,  1.99s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:12,  1.95s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.94s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.92s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:05,  1.96s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.93s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:57,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:57,  1.95s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:49,  1.93s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:44,  1.87s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:42,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:45,  1.95s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:42,  1.94s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:34,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:32,  1.85s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:34,  1.97s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:32,  1.97s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:28,  1.91s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:21,  1.94s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:19,  1.93s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:16,  1.91s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:14,  1.90s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.91s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:07,  1.89s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:08,  1.96s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:06,  1.96s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:03,  1.92s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:59,  1.91s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:56,  1.88s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:54,  1.96s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:51,  1.92s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:42,  1.83s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:39,  1.81s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:42,  2.01s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:38,  1.95s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:36,  1.91s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:33,  1.88s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:27,  1.84s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:25,  1.84s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:23,  1.81s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:23,  1.97s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.89s/it]

Evaluating policy:  88%|########7 | 73/83 [02:19<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.90s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.89s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.86s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:04,  2.00s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.93s/it]

Outer training loop:  78%|#######8  | 39001/50000 [25:39:39<299:11:31, 97.93s/it]
Outer training loop:  78%|#######8  | 39002/50000 [25:39:41<211:09:37, 69.12s/it]
Outer training loop:  78%|#######8  | 39003/50000 [25:39:43<149:29:10, 48.94s/it]
Outer training loop:  78%|#######8  | 39004/50000 [25:39:44<106:15:32, 34.79s/it]
Outer training loop:  78%|#######8  | 39005/50000 [25:39:46<75:59:16, 24.88s/it] 
Outer training loop:  78%|#######8  | 39006/50000 [25:39:48<54:50:16, 17.96s/it]
Outer training loop:  78%|#######8  | 39007/50000 [25:39:50<40:03:20, 13.12s/it]
Outer training loop:  78%|#######8  | 39008/50000 [25:39:52<29:43:28,  9.74s/it]
Outer training loop:  78%|#######8  | 39009/50000 [25:39:54<22:58:42,  7.53s/it]
Outer training loop:  78%|#######8  | 39010/50000 [25:39:56<17:43:32,  5.81s/it]
Outer training loop:  78%|#######8  | 39011/50000 [25:39:58<14:10:11,  4.64s/it]
Outer training loop:  78%|#######8  | 39012/50000 [25:39:59<11:29:04,  3.76s/it]
Outer training loop:  78%|#######8  | 39013/50000 [25:40:01<9:55:58,  3.25s/it] 
Outer training loop:  78%|#######8  | 39014/50000 [25:40:03<8:24:30,  2.76s/it]
Outer training loop:  78%|#######8  | 39015/50000 [25:40:05<7:39:19,  2.51s/it]
Outer training loop:  78%|#######8  | 39016/50000 [25:40:07<7:01:41,  2.30s/it]
Outer training loop:  78%|#######8  | 39017/50000 [25:40:09<6:51:51,  2.25s/it]
Outer training loop:  78%|#######8  | 39018/50000 [25:40:11<6:30:14,  2.13s/it]
Outer training loop:  78%|#######8  | 39019/50000 [25:40:12<6:09:40,  2.02s/it]
Outer training loop:  78%|#######8  | 39020/50000 [25:40:15<6:13:04,  2.04s/it]
Outer training loop:  78%|#######8  | 39021/50000 [25:40:16<6:00:10,  1.97s/it]
Outer training loop:  78%|#######8  | 39022/50000 [25:40:18<5:52:23,  1.93s/it]
Outer training loop:  78%|#######8  | 39023/50000 [25:40:20<5:48:46,  1.91s/it]
Outer training loop:  78%|#######8  | 39024/50000 [25:40:22<5:50:15,  1.91s/it]
Outer training loop:  78%|#######8  | 39025/50000 [25:40:24<5:41:30,  1.87s/it]
Outer training loop:  78%|#######8  | 39026/50000 [25:40:26<5:37:44,  1.85s/it]
Outer training loop:  78%|#######8  | 39027/50000 [25:40:27<5:39:43,  1.86s/it]
Outer training loop:  78%|#######8  | 39028/50000 [25:40:29<5:48:08,  1.90s/it]
Outer training loop:  78%|#######8  | 39029/50000 [25:40:31<5:47:58,  1.90s/it]
Outer training loop:  78%|#######8  | 39030/50000 [25:40:33<5:49:20,  1.91s/it]
Outer training loop:  78%|#######8  | 39031/50000 [25:40:35<5:57:39,  1.96s/it]
Outer training loop:  78%|#######8  | 39032/50000 [25:40:37<5:48:01,  1.90s/it]
Outer training loop:  78%|#######8  | 39033/50000 [25:40:39<5:45:19,  1.89s/it]
Outer training loop:  78%|#######8  | 39034/50000 [25:40:41<5:58:55,  1.96s/it]
Outer training loop:  78%|#######8  | 39035/50000 [25:40:43<5:47:10,  1.90s/it]
Outer training loop:  78%|#######8  | 39036/50000 [25:40:45<5:45:16,  1.89s/it]
Outer training loop:  78%|#######8  | 39037/50000 [25:40:47<5:55:25,  1.95s/it]
Outer training loop:  78%|#######8  | 39038/50000 [25:40:49<5:48:32,  1.91s/it]
Outer training loop:  78%|#######8  | 39039/50000 [25:40:50<5:40:39,  1.86s/it]
Outer training loop:  78%|#######8  | 39040/50000 [25:40:52<5:29:11,  1.80s/it]
Outer training loop:  78%|#######8  | 39041/50000 [25:40:54<5:39:47,  1.86s/it]
Outer training loop:  78%|#######8  | 39042/50000 [25:40:56<5:36:29,  1.84s/it]
Outer training loop:  78%|#######8  | 39043/50000 [25:40:58<5:32:02,  1.82s/it]
Outer training loop:  78%|#######8  | 39044/50000 [25:41:00<5:47:20,  1.90s/it]
Outer training loop:  78%|#######8  | 39045/50000 [25:41:02<5:43:23,  1.88s/it]
Outer training loop:  78%|#######8  | 39046/50000 [25:41:03<5:32:50,  1.82s/it]
Outer training loop:  78%|#######8  | 39047/50000 [25:41:05<5:36:28,  1.84s/it]
Outer training loop:  78%|#######8  | 39048/50000 [25:41:07<5:45:24,  1.89s/it]
Outer training loop:  78%|#######8  | 39049/50000 [25:41:09<5:43:04,  1.88s/it]
Outer training loop:  78%|#######8  | 39050/50000 [25:41:11<5:41:18,  1.87s/it]
Outer training loop:  78%|#######8  | 39051/50000 [25:41:13<5:53:43,  1.94s/it]
Outer training loop:  78%|#######8  | 39052/50000 [25:41:15<5:49:49,  1.92s/it]
Outer training loop:  78%|#######8  | 39053/50000 [25:41:17<5:47:09,  1.90s/it]
Outer training loop:  78%|#######8  | 39054/50000 [25:41:19<5:50:59,  1.92s/it]
Outer training loop:  78%|#######8  | 39055/50000 [25:41:20<5:47:22,  1.90s/it]
Outer training loop:  78%|#######8  | 39056/50000 [25:41:22<5:41:19,  1.87s/it]
Outer training loop:  78%|#######8  | 39057/50000 [25:41:24<5:50:19,  1.92s/it]
Outer training loop:  78%|#######8  | 39058/50000 [25:41:26<5:46:59,  1.90s/it]
Outer training loop:  78%|#######8  | 39059/50000 [25:41:28<5:43:29,  1.88s/it]
Outer training loop:  78%|#######8  | 39060/50000 [25:41:30<5:39:09,  1.86s/it]
Outer training loop:  78%|#######8  | 39061/50000 [25:41:32<5:42:31,  1.88s/it]
Outer training loop:  78%|#######8  | 39062/50000 [25:41:34<5:42:02,  1.88s/it]
Outer training loop:  78%|#######8  | 39063/50000 [25:41:36<5:48:20,  1.91s/it]
Outer training loop:  78%|#######8  | 39064/50000 [25:41:38<5:49:31,  1.92s/it]
Outer training loop:  78%|#######8  | 39065/50000 [25:41:40<5:52:55,  1.94s/it]
Outer training loop:  78%|#######8  | 39066/50000 [25:41:41<5:44:59,  1.89s/it]
Outer training loop:  78%|#######8  | 39067/50000 [25:41:43<5:39:09,  1.86s/it]
Outer training loop:  78%|#######8  | 39068/50000 [25:41:45<5:53:29,  1.94s/it]
Outer training loop:  78%|#######8  | 39069/50000 [25:41:47<5:42:26,  1.88s/it]
Outer training loop:  78%|#######8  | 39070/50000 [25:41:49<5:37:33,  1.85s/it]
Outer training loop:  78%|#######8  | 39071/50000 [25:41:51<5:47:38,  1.91s/it]
Outer training loop:  78%|#######8  | 39072/50000 [25:41:53<5:43:50,  1.89s/it]
Outer training loop:  78%|#######8  | 39073/50000 [25:41:54<5:40:03,  1.87s/it]
Outer training loop:  78%|#######8  | 39074/50000 [25:41:56<5:35:16,  1.84s/it]
Outer training loop:  78%|#######8  | 39075/50000 [25:41:58<5:43:06,  1.88s/it]
Outer training loop:  78%|#######8  | 39076/50000 [25:42:00<5:39:39,  1.87s/it]
Outer training loop:  78%|#######8  | 39077/50000 [25:42:02<5:36:35,  1.85s/it]
Outer training loop:  78%|#######8  | 39078/50000 [25:42:04<5:39:31,  1.87s/it]
Outer training loop:  78%|#######8  | 39079/50000 [25:42:06<5:37:21,  1.85s/it]
Outer training loop:  78%|#######8  | 39080/50000 [25:42:08<5:45:53,  1.90s/it]
Outer training loop:  78%|#######8  | 39081/50000 [25:42:09<5:40:41,  1.87s/it]
Outer training loop:  78%|#######8  | 39082/50000 [25:42:11<5:45:15,  1.90s/it]
Outer training loop:  78%|#######8  | 39083/50000 [25:42:13<5:38:56,  1.86s/it]
Outer training loop:  78%|#######8  | 39084/50000 [25:42:15<5:38:11,  1.86s/it]
Outer training loop:  78%|#######8  | 39085/50000 [25:42:17<5:37:37,  1.86s/it]
Outer training loop:  78%|#######8  | 39086/50000 [25:42:19<5:36:36,  1.85s/it]
Outer training loop:  78%|#######8  | 39087/50000 [25:42:21<5:39:19,  1.87s/it]
Outer training loop:  78%|#######8  | 39088/50000 [25:42:22<5:41:10,  1.88s/it]
Outer training loop:  78%|#######8  | 39089/50000 [25:42:24<5:39:48,  1.87s/it]
Outer training loop:  78%|#######8  | 39090/50000 [25:42:26<5:48:06,  1.91s/it]
Outer training loop:  78%|#######8  | 39091/50000 [25:42:28<5:54:18,  1.95s/it]
Outer training loop:  78%|#######8  | 39092/50000 [25:42:30<5:54:54,  1.95s/it]
Outer training loop:  78%|#######8  | 39093/50000 [25:42:32<5:50:33,  1.93s/it]
Outer training loop:  78%|#######8  | 39094/50000 [25:42:34<5:49:02,  1.92s/it]
Outer training loop:  78%|#######8  | 39095/50000 [25:42:36<5:53:09,  1.94s/it]
Outer training loop:  78%|#######8  | 39096/50000 [25:42:38<5:45:38,  1.90s/it]
Outer training loop:  78%|#######8  | 39097/50000 [25:42:40<5:36:53,  1.85s/it]
Outer training loop:  78%|#######8  | 39098/50000 [25:42:42<5:40:23,  1.87s/it]
Outer training loop:  78%|#######8  | 39099/50000 [25:42:43<5:37:34,  1.86s/it]
Outer training loop:  78%|#######8  | 39100/50000 [25:42:45<5:35:42,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.88s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  78%|#######8  | 39101/50000 [25:42:53<10:45:21,  3.55s/it]
Outer training loop:  78%|#######8  | 39102/50000 [25:42:55<9:20:44,  3.09s/it] 
Outer training loop:  78%|#######8  | 39103/50000 [25:42:56<8:07:43,  2.69s/it]
Outer training loop:  78%|#######8  | 39104/50000 [25:42:58<7:21:01,  2.43s/it]
Outer training loop:  78%|#######8  | 39105/50000 [25:43:00<6:48:27,  2.25s/it]
Outer training loop:  78%|#######8  | 39106/50000 [25:43:02<6:33:45,  2.17s/it]
Outer training loop:  78%|#######8  | 39107/50000 [25:43:04<6:24:25,  2.12s/it]
Outer training loop:  78%|#######8  | 39108/50000 [25:43:06<6:15:14,  2.07s/it]
Outer training loop:  78%|#######8  | 39109/50000 [25:43:08<6:15:36,  2.07s/it]
Outer training loop:  78%|#######8  | 39110/50000 [25:43:10<5:59:01,  1.98s/it]
Outer training loop:  78%|#######8  | 39111/50000 [25:43:12<5:58:02,  1.97s/it]
Outer training loop:  78%|#######8  | 39112/50000 [25:43:14<6:05:07,  2.01s/it]
Outer training loop:  78%|#######8  | 39113/50000 [25:43:16<5:55:29,  1.96s/it]
Outer training loop:  78%|#######8  | 39114/50000 [25:43:18<5:47:53,  1.92s/it]
Outer training loop:  78%|#######8  | 39115/50000 [25:43:20<6:00:05,  1.98s/it]
Outer training loop:  78%|#######8  | 39116/50000 [25:43:22<5:50:54,  1.93s/it]
Outer training loop:  78%|#######8  | 39117/50000 [25:43:23<5:47:48,  1.92s/it]
Outer training loop:  78%|#######8  | 39118/50000 [25:43:25<5:43:50,  1.90s/it]
Outer training loop:  78%|#######8  | 39119/50000 [25:43:27<5:58:50,  1.98s/it]
Outer training loop:  78%|#######8  | 39120/50000 [25:43:29<5:53:05,  1.95s/it]
Outer training loop:  78%|#######8  | 39121/50000 [25:43:31<5:45:01,  1.90s/it]
Outer training loop:  78%|#######8  | 39122/50000 [25:43:33<5:46:34,  1.91s/it]
Outer training loop:  78%|#######8  | 39123/50000 [25:43:35<5:46:10,  1.91s/it]
Outer training loop:  78%|#######8  | 39124/50000 [25:43:37<5:42:37,  1.89s/it]
Outer training loop:  78%|#######8  | 39125/50000 [25:43:39<5:35:24,  1.85s/it]
Outer training loop:  78%|#######8  | 39126/50000 [25:43:41<5:41:13,  1.88s/it]
Outer training loop:  78%|#######8  | 39127/50000 [25:43:43<5:57:56,  1.98s/it]
Outer training loop:  78%|#######8  | 39128/50000 [25:43:45<5:48:18,  1.92s/it]
Outer training loop:  78%|#######8  | 39129/50000 [25:43:47<5:54:40,  1.96s/it]
Outer training loop:  78%|#######8  | 39130/50000 [25:43:48<5:43:20,  1.90s/it]
Outer training loop:  78%|#######8  | 39131/50000 [25:43:50<5:45:12,  1.91s/it]
Outer training loop:  78%|#######8  | 39132/50000 [25:43:52<5:43:45,  1.90s/it]
Outer training loop:  78%|#######8  | 39133/50000 [25:43:54<5:50:36,  1.94s/it]
Outer training loop:  78%|#######8  | 39134/50000 [25:43:56<5:45:58,  1.91s/it]
Outer training loop:  78%|#######8  | 39135/50000 [25:43:58<5:40:55,  1.88s/it]
Outer training loop:  78%|#######8  | 39136/50000 [25:44:00<5:48:47,  1.93s/it]
Outer training loop:  78%|#######8  | 39137/50000 [25:44:02<5:43:03,  1.89s/it]
Outer training loop:  78%|#######8  | 39138/50000 [25:44:04<5:38:58,  1.87s/it]
Outer training loop:  78%|#######8  | 39139/50000 [25:44:05<5:35:47,  1.86s/it]
Outer training loop:  78%|#######8  | 39140/50000 [25:44:07<5:50:05,  1.93s/it]
Outer training loop:  78%|#######8  | 39141/50000 [25:44:09<5:45:13,  1.91s/it]
Outer training loop:  78%|#######8  | 39142/50000 [25:44:11<5:39:07,  1.87s/it]
Outer training loop:  78%|#######8  | 39143/50000 [25:44:13<5:52:26,  1.95s/it]
Outer training loop:  78%|#######8  | 39144/50000 [25:44:15<5:41:11,  1.89s/it]
Outer training loop:  78%|#######8  | 39145/50000 [25:44:17<5:38:19,  1.87s/it]
Outer training loop:  78%|#######8  | 39146/50000 [25:44:19<5:32:50,  1.84s/it]
Outer training loop:  78%|#######8  | 39147/50000 [25:44:21<5:45:16,  1.91s/it]
Outer training loop:  78%|#######8  | 39148/50000 [25:44:22<5:41:57,  1.89s/it]
Outer training loop:  78%|#######8  | 39149/50000 [25:44:24<5:34:39,  1.85s/it]
Outer training loop:  78%|#######8  | 39150/50000 [25:44:26<5:49:01,  1.93s/it]
Outer training loop:  78%|#######8  | 39151/50000 [25:44:28<5:46:58,  1.92s/it]
Outer training loop:  78%|#######8  | 39152/50000 [25:44:30<5:39:36,  1.88s/it]
Outer training loop:  78%|#######8  | 39153/50000 [25:44:32<5:55:44,  1.97s/it]
Outer training loop:  78%|#######8  | 39154/50000 [25:44:34<5:47:13,  1.92s/it]
Outer training loop:  78%|#######8  | 39155/50000 [25:44:36<5:45:54,  1.91s/it]
Outer training loop:  78%|#######8  | 39156/50000 [25:44:38<5:41:47,  1.89s/it]
Outer training loop:  78%|#######8  | 39157/50000 [25:44:40<5:50:56,  1.94s/it]
Outer training loop:  78%|#######8  | 39158/50000 [25:44:42<5:38:31,  1.87s/it]
Outer training loop:  78%|#######8  | 39159/50000 [25:44:43<5:37:03,  1.87s/it]
Outer training loop:  78%|#######8  | 39160/50000 [25:44:45<5:48:06,  1.93s/it]
Outer training loop:  78%|#######8  | 39161/50000 [25:44:47<5:43:06,  1.90s/it]
Outer training loop:  78%|#######8  | 39162/50000 [25:44:49<5:35:53,  1.86s/it]
Outer training loop:  78%|#######8  | 39163/50000 [25:44:51<5:49:50,  1.94s/it]
Outer training loop:  78%|#######8  | 39164/50000 [25:44:53<5:44:01,  1.90s/it]
Outer training loop:  78%|#######8  | 39165/50000 [25:44:55<5:38:10,  1.87s/it]
Outer training loop:  78%|#######8  | 39166/50000 [25:44:57<5:43:01,  1.90s/it]
Outer training loop:  78%|#######8  | 39167/50000 [25:44:59<5:51:54,  1.95s/it]
Outer training loop:  78%|#######8  | 39168/50000 [25:45:01<5:43:12,  1.90s/it]
Outer training loop:  78%|#######8  | 39169/50000 [25:45:02<5:39:51,  1.88s/it]
Outer training loop:  78%|#######8  | 39170/50000 [25:45:04<5:46:30,  1.92s/it]
Outer training loop:  78%|#######8  | 39171/50000 [25:45:06<5:41:32,  1.89s/it]
Outer training loop:  78%|#######8  | 39172/50000 [25:45:08<5:34:50,  1.86s/it]
Outer training loop:  78%|#######8  | 39173/50000 [25:45:10<5:43:52,  1.91s/it]
Outer training loop:  78%|#######8  | 39174/50000 [25:45:12<5:32:59,  1.85s/it]
Outer training loop:  78%|#######8  | 39175/50000 [25:45:14<5:32:07,  1.84s/it]
Outer training loop:  78%|#######8  | 39176/50000 [25:45:15<5:33:52,  1.85s/it]
Outer training loop:  78%|#######8  | 39177/50000 [25:45:18<5:54:13,  1.96s/it]
Outer training loop:  78%|#######8  | 39178/50000 [25:45:20<5:45:30,  1.92s/it]
Outer training loop:  78%|#######8  | 39179/50000 [25:45:21<5:45:36,  1.92s/it]
Outer training loop:  78%|#######8  | 39180/50000 [25:45:23<5:53:40,  1.96s/it]
Outer training loop:  78%|#######8  | 39181/50000 [25:45:25<5:47:43,  1.93s/it]
Outer training loop:  78%|#######8  | 39182/50000 [25:45:27<5:42:04,  1.90s/it]
Outer training loop:  78%|#######8  | 39183/50000 [25:45:29<5:37:47,  1.87s/it]
Outer training loop:  78%|#######8  | 39184/50000 [25:45:31<5:49:25,  1.94s/it]
Outer training loop:  78%|#######8  | 39185/50000 [25:45:33<5:44:30,  1.91s/it]
Outer training loop:  78%|#######8  | 39186/50000 [25:45:35<5:39:27,  1.88s/it]
Outer training loop:  78%|#######8  | 39187/50000 [25:45:37<5:46:28,  1.92s/it]
Outer training loop:  78%|#######8  | 39188/50000 [25:45:38<5:36:18,  1.87s/it]
Outer training loop:  78%|#######8  | 39189/50000 [25:45:40<5:33:13,  1.85s/it]
Outer training loop:  78%|#######8  | 39190/50000 [25:45:42<5:42:19,  1.90s/it]
Outer training loop:  78%|#######8  | 39191/50000 [25:45:44<5:38:11,  1.88s/it]
Outer training loop:  78%|#######8  | 39192/50000 [25:45:46<5:35:56,  1.86s/it]
Outer training loop:  78%|#######8  | 39193/50000 [25:45:48<5:39:43,  1.89s/it]
Outer training loop:  78%|#######8  | 39194/50000 [25:45:50<5:42:27,  1.90s/it]
Outer training loop:  78%|#######8  | 39195/50000 [25:45:52<5:38:10,  1.88s/it]
Outer training loop:  78%|#######8  | 39196/50000 [25:45:53<5:29:15,  1.83s/it]
Outer training loop:  78%|#######8  | 39197/50000 [25:45:55<5:31:34,  1.84s/it]
Outer training loop:  78%|#######8  | 39198/50000 [25:45:57<5:31:13,  1.84s/it]
Outer training loop:  78%|#######8  | 39199/50000 [25:45:59<5:32:00,  1.84s/it]
Outer training loop:  78%|#######8  | 39200/50000 [25:46:01<5:40:06,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  78%|#######8  | 39201/50000 [25:46:08<10:36:01,  3.53s/it]
Outer training loop:  78%|#######8  | 39202/50000 [25:46:10<9:04:03,  3.02s/it] 
Outer training loop:  78%|#######8  | 39203/50000 [25:46:12<7:55:01,  2.64s/it]
Outer training loop:  78%|#######8  | 39204/50000 [25:46:14<7:22:16,  2.46s/it]
Outer training loop:  78%|#######8  | 39205/50000 [25:46:16<6:46:33,  2.26s/it]
Outer training loop:  78%|#######8  | 39206/50000 [25:46:18<6:22:09,  2.12s/it]
Outer training loop:  78%|#######8  | 39207/50000 [25:46:20<6:23:59,  2.13s/it]
Outer training loop:  78%|#######8  | 39208/50000 [25:46:22<6:06:45,  2.04s/it]
Outer training loop:  78%|#######8  | 39209/50000 [25:46:23<5:55:15,  1.98s/it]
Outer training loop:  78%|#######8  | 39210/50000 [25:46:25<5:51:36,  1.96s/it]
Outer training loop:  78%|#######8  | 39211/50000 [25:46:27<5:52:47,  1.96s/it]
Outer training loop:  78%|#######8  | 39212/50000 [25:46:29<5:56:04,  1.98s/it]
Outer training loop:  78%|#######8  | 39213/50000 [25:46:31<5:45:24,  1.92s/it]
Outer training loop:  78%|#######8  | 39214/50000 [25:46:33<5:56:24,  1.98s/it]
Outer training loop:  78%|#######8  | 39215/50000 [25:46:35<5:45:15,  1.92s/it]
Outer training loop:  78%|#######8  | 39216/50000 [25:46:37<5:38:08,  1.88s/it]
Outer training loop:  78%|#######8  | 39217/50000 [25:46:39<5:34:55,  1.86s/it]
Outer training loop:  78%|#######8  | 39218/50000 [25:46:41<5:49:30,  1.94s/it]
Outer training loop:  78%|#######8  | 39219/50000 [25:46:42<5:37:25,  1.88s/it]
Outer training loop:  78%|#######8  | 39220/50000 [25:46:44<5:34:31,  1.86s/it]
Outer training loop:  78%|#######8  | 39221/50000 [25:46:46<5:40:10,  1.89s/it]
Outer training loop:  78%|#######8  | 39222/50000 [25:46:48<5:37:04,  1.88s/it]
Outer training loop:  78%|#######8  | 39223/50000 [25:46:50<5:37:23,  1.88s/it]
Outer training loop:  78%|#######8  | 39224/50000 [25:46:52<5:54:49,  1.98s/it]
Outer training loop:  78%|#######8  | 39225/50000 [25:46:54<5:41:10,  1.90s/it]
Outer training loop:  78%|#######8  | 39226/50000 [25:46:56<5:41:13,  1.90s/it]
Outer training loop:  78%|#######8  | 39227/50000 [25:46:58<5:45:40,  1.93s/it]
Outer training loop:  78%|#######8  | 39228/50000 [25:47:00<5:41:59,  1.90s/it]
Outer training loop:  78%|#######8  | 39229/50000 [25:47:01<5:34:58,  1.87s/it]
Outer training loop:  78%|#######8  | 39230/50000 [25:47:03<5:27:52,  1.83s/it]
Outer training loop:  78%|#######8  | 39231/50000 [25:47:05<5:44:54,  1.92s/it]
Outer training loop:  78%|#######8  | 39232/50000 [25:47:07<5:43:16,  1.91s/it]
Outer training loop:  78%|#######8  | 39233/50000 [25:47:09<5:34:01,  1.86s/it]
Outer training loop:  78%|#######8  | 39234/50000 [25:47:11<5:46:21,  1.93s/it]
Outer training loop:  78%|#######8  | 39235/50000 [25:47:13<5:47:04,  1.93s/it]
Outer training loop:  78%|#######8  | 39236/50000 [25:47:15<5:38:14,  1.89s/it]
Outer training loop:  78%|#######8  | 39237/50000 [25:47:16<5:33:20,  1.86s/it]
Outer training loop:  78%|#######8  | 39238/50000 [25:47:19<5:44:45,  1.92s/it]
Outer training loop:  78%|#######8  | 39239/50000 [25:47:20<5:42:25,  1.91s/it]
Outer training loop:  78%|#######8  | 39240/50000 [25:47:22<5:39:39,  1.89s/it]
Outer training loop:  78%|#######8  | 39241/50000 [25:47:24<5:43:17,  1.91s/it]
Outer training loop:  78%|#######8  | 39242/50000 [25:47:26<5:49:00,  1.95s/it]
Outer training loop:  78%|#######8  | 39243/50000 [25:47:28<5:43:04,  1.91s/it]
Outer training loop:  78%|#######8  | 39244/50000 [25:47:30<5:42:11,  1.91s/it]
Outer training loop:  78%|#######8  | 39245/50000 [25:47:32<5:48:10,  1.94s/it]
Outer training loop:  78%|#######8  | 39246/50000 [25:47:34<5:35:57,  1.87s/it]
Outer training loop:  78%|#######8  | 39247/50000 [25:47:36<5:34:24,  1.87s/it]
Outer training loop:  78%|#######8  | 39248/50000 [25:47:38<5:41:22,  1.90s/it]
Outer training loop:  78%|#######8  | 39249/50000 [25:47:39<5:34:41,  1.87s/it]
Outer training loop:  78%|#######8  | 39250/50000 [25:47:41<5:34:43,  1.87s/it]
Outer training loop:  79%|#######8  | 39251/50000 [25:47:43<5:48:13,  1.94s/it]
Outer training loop:  79%|#######8  | 39252/50000 [25:47:45<5:36:57,  1.88s/it]
Outer training loop:  79%|#######8  | 39253/50000 [25:47:47<5:33:24,  1.86s/it]
Outer training loop:  79%|#######8  | 39254/50000 [25:47:49<5:37:15,  1.88s/it]
Outer training loop:  79%|#######8  | 39255/50000 [25:47:51<5:46:10,  1.93s/it]
Outer training loop:  79%|#######8  | 39256/50000 [25:47:53<5:37:34,  1.89s/it]
Outer training loop:  79%|#######8  | 39257/50000 [25:47:54<5:29:10,  1.84s/it]
Outer training loop:  79%|#######8  | 39258/50000 [25:47:56<5:37:17,  1.88s/it]
Outer training loop:  79%|#######8  | 39259/50000 [25:47:58<5:33:13,  1.86s/it]
Outer training loop:  79%|#######8  | 39260/50000 [25:48:00<5:33:16,  1.86s/it]
Outer training loop:  79%|#######8  | 39261/50000 [25:48:02<5:29:44,  1.84s/it]
Outer training loop:  79%|#######8  | 39262/50000 [25:48:04<5:38:16,  1.89s/it]
Outer training loop:  79%|#######8  | 39263/50000 [25:48:06<5:34:42,  1.87s/it]
Outer training loop:  79%|#######8  | 39264/50000 [25:48:07<5:27:27,  1.83s/it]
Outer training loop:  79%|#######8  | 39265/50000 [25:48:09<5:40:35,  1.90s/it]
Outer training loop:  79%|#######8  | 39266/50000 [25:48:11<5:36:50,  1.88s/it]
Outer training loop:  79%|#######8  | 39267/50000 [25:48:13<5:26:09,  1.82s/it]
Outer training loop:  79%|#######8  | 39268/50000 [25:48:15<5:24:29,  1.81s/it]
Outer training loop:  79%|#######8  | 39269/50000 [25:48:17<5:35:24,  1.88s/it]
Outer training loop:  79%|#######8  | 39270/50000 [25:48:19<5:28:10,  1.84s/it]
Outer training loop:  79%|#######8  | 39271/50000 [25:48:20<5:27:32,  1.83s/it]
Outer training loop:  79%|#######8  | 39272/50000 [25:48:22<5:30:25,  1.85s/it]
Outer training loop:  79%|#######8  | 39273/50000 [25:48:24<5:29:42,  1.84s/it]
Outer training loop:  79%|#######8  | 39274/50000 [25:48:26<5:37:30,  1.89s/it]
Outer training loop:  79%|#######8  | 39275/50000 [25:48:28<5:46:30,  1.94s/it]
Outer training loop:  79%|#######8  | 39276/50000 [25:48:30<5:39:15,  1.90s/it]
Outer training loop:  79%|#######8  | 39277/50000 [25:48:32<5:28:46,  1.84s/it]
Outer training loop:  79%|#######8  | 39278/50000 [25:48:33<5:26:37,  1.83s/it]
Outer training loop:  79%|#######8  | 39279/50000 [25:48:35<5:36:35,  1.88s/it]
Outer training loop:  79%|#######8  | 39280/50000 [25:48:37<5:29:30,  1.84s/it]
Outer training loop:  79%|#######8  | 39281/50000 [25:48:39<5:33:11,  1.87s/it]
Outer training loop:  79%|#######8  | 39282/50000 [25:48:41<5:28:14,  1.84s/it]
Outer training loop:  79%|#######8  | 39283/50000 [25:48:43<5:39:57,  1.90s/it]
Outer training loop:  79%|#######8  | 39284/50000 [25:48:45<5:36:32,  1.88s/it]
Outer training loop:  79%|#######8  | 39285/50000 [25:48:47<5:27:39,  1.83s/it]
Outer training loop:  79%|#######8  | 39286/50000 [25:48:49<5:38:36,  1.90s/it]
Outer training loop:  79%|#######8  | 39287/50000 [25:48:50<5:35:23,  1.88s/it]
Outer training loop:  79%|#######8  | 39288/50000 [25:48:52<5:33:12,  1.87s/it]
Outer training loop:  79%|#######8  | 39289/50000 [25:48:54<5:50:39,  1.96s/it]
Outer training loop:  79%|#######8  | 39290/50000 [25:48:56<5:42:37,  1.92s/it]
Outer training loop:  79%|#######8  | 39291/50000 [25:48:58<5:36:47,  1.89s/it]
Outer training loop:  79%|#######8  | 39292/50000 [25:49:00<5:32:10,  1.86s/it]
Outer training loop:  79%|#######8  | 39293/50000 [25:49:02<5:43:06,  1.92s/it]
Outer training loop:  79%|#######8  | 39294/50000 [25:49:04<5:36:49,  1.89s/it]
Outer training loop:  79%|#######8  | 39295/50000 [25:49:06<5:37:43,  1.89s/it]
Outer training loop:  79%|#######8  | 39296/50000 [25:49:08<5:43:18,  1.92s/it]
Outer training loop:  79%|#######8  | 39297/50000 [25:49:09<5:38:27,  1.90s/it]
Outer training loop:  79%|#######8  | 39298/50000 [25:49:11<5:40:28,  1.91s/it]
Outer training loop:  79%|#######8  | 39299/50000 [25:49:13<5:41:31,  1.91s/it]
Outer training loop:  79%|#######8  | 39300/50000 [25:49:15<5:49:48,  1.96s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  79%|#######8  | 39301/50000 [25:49:23<10:43:50,  3.61s/it]
Outer training loop:  79%|#######8  | 39302/50000 [25:49:25<9:06:39,  3.07s/it] 
Outer training loop:  79%|#######8  | 39303/50000 [25:49:26<7:58:40,  2.68s/it]
Outer training loop:  79%|#######8  | 39304/50000 [25:49:28<7:07:38,  2.40s/it]
Outer training loop:  79%|#######8  | 39305/50000 [25:49:30<6:45:12,  2.27s/it]
Outer training loop:  79%|#######8  | 39306/50000 [25:49:32<6:21:18,  2.14s/it]
Outer training loop:  79%|#######8  | 39307/50000 [25:49:34<6:02:21,  2.03s/it]
Outer training loop:  79%|#######8  | 39308/50000 [25:49:36<5:55:45,  2.00s/it]
Outer training loop:  79%|#######8  | 39309/50000 [25:49:38<5:48:51,  1.96s/it]
Outer training loop:  79%|#######8  | 39310/50000 [25:49:39<5:41:26,  1.92s/it]
Outer training loop:  79%|#######8  | 39311/50000 [25:49:41<5:36:17,  1.89s/it]
Outer training loop:  79%|#######8  | 39312/50000 [25:49:43<5:44:24,  1.93s/it]
Outer training loop:  79%|#######8  | 39313/50000 [25:49:45<5:39:06,  1.90s/it]
Outer training loop:  79%|#######8  | 39314/50000 [25:49:47<5:27:52,  1.84s/it]
Outer training loop:  79%|#######8  | 39315/50000 [25:49:49<5:42:17,  1.92s/it]
Outer training loop:  79%|#######8  | 39316/50000 [25:49:51<5:39:28,  1.91s/it]
Outer training loop:  79%|#######8  | 39317/50000 [25:49:52<5:30:03,  1.85s/it]
Outer training loop:  79%|#######8  | 39318/50000 [25:49:54<5:37:15,  1.89s/it]
Outer training loop:  79%|#######8  | 39319/50000 [25:49:56<5:30:44,  1.86s/it]
Outer training loop:  79%|#######8  | 39320/50000 [25:49:58<5:26:21,  1.83s/it]
Outer training loop:  79%|#######8  | 39321/50000 [25:50:00<5:35:34,  1.89s/it]
Outer training loop:  79%|#######8  | 39322/50000 [25:50:02<5:38:53,  1.90s/it]
Outer training loop:  79%|#######8  | 39323/50000 [25:50:04<5:36:26,  1.89s/it]
Outer training loop:  79%|#######8  | 39324/50000 [25:50:06<5:32:26,  1.87s/it]
Outer training loop:  79%|#######8  | 39325/50000 [25:50:08<5:38:24,  1.90s/it]
Outer training loop:  79%|#######8  | 39326/50000 [25:50:09<5:32:48,  1.87s/it]
Outer training loop:  79%|#######8  | 39327/50000 [25:50:11<5:35:23,  1.89s/it]
Outer training loop:  79%|#######8  | 39328/50000 [25:50:13<5:47:12,  1.95s/it]
Outer training loop:  79%|#######8  | 39329/50000 [25:50:15<5:40:49,  1.92s/it]
Outer training loop:  79%|#######8  | 39330/50000 [25:50:17<5:30:54,  1.86s/it]
Outer training loop:  79%|#######8  | 39331/50000 [25:50:19<5:28:55,  1.85s/it]
Outer training loop:  79%|#######8  | 39332/50000 [25:50:21<5:34:42,  1.88s/it]
Outer training loop:  79%|#######8  | 39333/50000 [25:50:23<5:37:14,  1.90s/it]
Outer training loop:  79%|#######8  | 39334/50000 [25:50:25<5:40:59,  1.92s/it]
Outer training loop:  79%|#######8  | 39335/50000 [25:50:27<5:42:02,  1.92s/it]
Outer training loop:  79%|#######8  | 39336/50000 [25:50:29<5:48:14,  1.96s/it]
Outer training loop:  79%|#######8  | 39337/50000 [25:50:30<5:35:57,  1.89s/it]
Outer training loop:  79%|#######8  | 39338/50000 [25:50:32<5:34:10,  1.88s/it]
Outer training loop:  79%|#######8  | 39339/50000 [25:50:34<5:38:27,  1.90s/it]
Outer training loop:  79%|#######8  | 39340/50000 [25:50:36<5:35:47,  1.89s/it]
Outer training loop:  79%|#######8  | 39341/50000 [25:50:38<5:29:49,  1.86s/it]
Outer training loop:  79%|#######8  | 39342/50000 [25:50:40<5:42:10,  1.93s/it]
Outer training loop:  79%|#######8  | 39343/50000 [25:50:42<5:33:37,  1.88s/it]
Outer training loop:  79%|#######8  | 39344/50000 [25:50:44<5:33:44,  1.88s/it]
Outer training loop:  79%|#######8  | 39345/50000 [25:50:45<5:27:03,  1.84s/it]
Outer training loop:  79%|#######8  | 39346/50000 [25:50:47<5:33:08,  1.88s/it]
Outer training loop:  79%|#######8  | 39347/50000 [25:50:49<5:31:37,  1.87s/it]
Outer training loop:  79%|#######8  | 39348/50000 [25:50:51<5:31:23,  1.87s/it]
Outer training loop:  79%|#######8  | 39349/50000 [25:50:53<5:36:08,  1.89s/it]
Outer training loop:  79%|#######8  | 39350/50000 [25:50:55<5:34:03,  1.88s/it]
Outer training loop:  79%|#######8  | 39351/50000 [25:50:57<5:30:58,  1.86s/it]
Outer training loop:  79%|#######8  | 39352/50000 [25:50:59<5:35:34,  1.89s/it]
Outer training loop:  79%|#######8  | 39353/50000 [25:51:00<5:30:50,  1.86s/it]
Outer training loop:  79%|#######8  | 39354/50000 [25:51:02<5:33:27,  1.88s/it]
Outer training loop:  79%|#######8  | 39355/50000 [25:51:04<5:45:49,  1.95s/it]
Outer training loop:  79%|#######8  | 39356/50000 [25:51:06<5:43:06,  1.93s/it]
Outer training loop:  79%|#######8  | 39357/50000 [25:51:08<5:32:28,  1.87s/it]
Outer training loop:  79%|#######8  | 39358/50000 [25:51:10<5:27:57,  1.85s/it]
Outer training loop:  79%|#######8  | 39359/50000 [25:51:12<5:33:01,  1.88s/it]
Outer training loop:  79%|#######8  | 39360/50000 [25:51:14<5:34:57,  1.89s/it]
Outer training loop:  79%|#######8  | 39361/50000 [25:51:16<5:30:40,  1.86s/it]
Outer training loop:  79%|#######8  | 39362/50000 [25:51:18<5:38:54,  1.91s/it]
Outer training loop:  79%|#######8  | 39363/50000 [25:51:19<5:33:31,  1.88s/it]
Outer training loop:  79%|#######8  | 39364/50000 [25:51:21<5:27:01,  1.84s/it]
Outer training loop:  79%|#######8  | 39365/50000 [25:51:23<5:32:41,  1.88s/it]
Outer training loop:  79%|#######8  | 39366/50000 [25:51:25<5:32:45,  1.88s/it]
Outer training loop:  79%|#######8  | 39367/50000 [25:51:27<5:26:15,  1.84s/it]
Outer training loop:  79%|#######8  | 39368/50000 [25:51:29<5:35:03,  1.89s/it]
Outer training loop:  79%|#######8  | 39369/50000 [25:51:30<5:25:37,  1.84s/it]
Outer training loop:  79%|#######8  | 39370/50000 [25:51:32<5:25:50,  1.84s/it]
Outer training loop:  79%|#######8  | 39371/50000 [25:51:34<5:45:17,  1.95s/it]
Outer training loop:  79%|#######8  | 39372/50000 [25:51:36<5:41:21,  1.93s/it]
Outer training loop:  79%|#######8  | 39373/50000 [25:51:38<5:37:46,  1.91s/it]
Outer training loop:  79%|#######8  | 39374/50000 [25:51:40<5:31:57,  1.87s/it]
Outer training loop:  79%|#######8  | 39375/50000 [25:51:42<5:37:21,  1.91s/it]
Outer training loop:  79%|#######8  | 39376/50000 [25:51:44<5:32:41,  1.88s/it]
Outer training loop:  79%|#######8  | 39377/50000 [25:51:46<5:33:01,  1.88s/it]
Outer training loop:  79%|#######8  | 39378/50000 [25:51:48<5:38:21,  1.91s/it]
Outer training loop:  79%|#######8  | 39379/50000 [25:51:50<5:35:04,  1.89s/it]
Outer training loop:  79%|#######8  | 39380/50000 [25:51:51<5:32:47,  1.88s/it]
Outer training loop:  79%|#######8  | 39381/50000 [25:51:53<5:30:41,  1.87s/it]
Outer training loop:  79%|#######8  | 39382/50000 [25:51:55<5:42:41,  1.94s/it]
Outer training loop:  79%|#######8  | 39383/50000 [25:51:57<5:38:01,  1.91s/it]
Outer training loop:  79%|#######8  | 39384/50000 [25:51:59<5:36:07,  1.90s/it]
Outer training loop:  79%|#######8  | 39385/50000 [25:52:01<5:45:28,  1.95s/it]
Outer training loop:  79%|#######8  | 39386/50000 [25:52:03<5:36:18,  1.90s/it]
Outer training loop:  79%|#######8  | 39387/50000 [25:52:05<5:34:57,  1.89s/it]
Outer training loop:  79%|#######8  | 39388/50000 [25:52:07<5:47:04,  1.96s/it]
Outer training loop:  79%|#######8  | 39389/50000 [25:52:09<5:40:39,  1.93s/it]
Outer training loop:  79%|#######8  | 39390/50000 [25:52:11<5:36:09,  1.90s/it]
Outer training loop:  79%|#######8  | 39391/50000 [25:52:13<5:45:20,  1.95s/it]
Outer training loop:  79%|#######8  | 39392/50000 [25:52:14<5:35:03,  1.90s/it]
Outer training loop:  79%|#######8  | 39393/50000 [25:52:16<5:28:07,  1.86s/it]
Outer training loop:  79%|#######8  | 39394/50000 [25:52:18<5:27:43,  1.85s/it]
Outer training loop:  79%|#######8  | 39395/50000 [25:52:20<5:36:30,  1.90s/it]
Outer training loop:  79%|#######8  | 39396/50000 [25:52:22<5:32:30,  1.88s/it]
Outer training loop:  79%|#######8  | 39397/50000 [25:52:24<5:30:58,  1.87s/it]
Outer training loop:  79%|#######8  | 39398/50000 [25:52:26<5:36:17,  1.90s/it]
Outer training loop:  79%|#######8  | 39399/50000 [25:52:28<5:34:56,  1.90s/it]
Outer training loop:  79%|#######8  | 39400/50000 [25:52:29<5:26:34,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  79%|#######8  | 39401/50000 [25:52:37<10:17:17,  3.49s/it]
Outer training loop:  79%|#######8  | 39402/50000 [25:52:39<8:58:54,  3.05s/it] 
Outer training loop:  79%|#######8  | 39403/50000 [25:52:41<7:57:16,  2.70s/it]
Outer training loop:  79%|#######8  | 39404/50000 [25:52:42<7:06:55,  2.42s/it]
Outer training loop:  79%|#######8  | 39405/50000 [25:52:44<6:42:48,  2.28s/it]
Outer training loop:  79%|#######8  | 39406/50000 [25:52:46<6:22:18,  2.17s/it]
Outer training loop:  79%|#######8  | 39407/50000 [25:52:48<6:01:04,  2.05s/it]
Outer training loop:  79%|#######8  | 39408/50000 [25:52:50<5:49:00,  1.98s/it]
Outer training loop:  79%|#######8  | 39409/50000 [25:52:52<5:47:45,  1.97s/it]
Outer training loop:  79%|#######8  | 39410/50000 [25:52:53<5:35:00,  1.90s/it]
Outer training loop:  79%|#######8  | 39411/50000 [25:52:55<5:29:09,  1.87s/it]
Outer training loop:  79%|#######8  | 39412/50000 [25:52:57<5:29:06,  1.86s/it]
Outer training loop:  79%|#######8  | 39413/50000 [25:52:59<5:26:15,  1.85s/it]
Outer training loop:  79%|#######8  | 39414/50000 [25:53:01<5:23:16,  1.83s/it]
Outer training loop:  79%|#######8  | 39415/50000 [25:53:03<5:30:11,  1.87s/it]
Outer training loop:  79%|#######8  | 39416/50000 [25:53:04<5:28:14,  1.86s/it]
Outer training loop:  79%|#######8  | 39417/50000 [25:53:06<5:29:36,  1.87s/it]
Outer training loop:  79%|#######8  | 39418/50000 [25:53:08<5:39:22,  1.92s/it]
Outer training loop:  79%|#######8  | 39419/50000 [25:53:10<5:45:19,  1.96s/it]
Outer training loop:  79%|#######8  | 39420/50000 [25:53:12<5:38:27,  1.92s/it]
Outer training loop:  79%|#######8  | 39421/50000 [25:53:14<5:39:11,  1.92s/it]
Outer training loop:  79%|#######8  | 39422/50000 [25:53:16<5:38:55,  1.92s/it]
Outer training loop:  79%|#######8  | 39423/50000 [25:53:18<5:31:27,  1.88s/it]
Outer training loop:  79%|#######8  | 39424/50000 [25:53:20<5:31:26,  1.88s/it]
Outer training loop:  79%|#######8  | 39425/50000 [25:53:22<5:29:34,  1.87s/it]
Outer training loop:  79%|#######8  | 39426/50000 [25:53:24<5:35:02,  1.90s/it]
Outer training loop:  79%|#######8  | 39427/50000 [25:53:25<5:31:38,  1.88s/it]
Outer training loop:  79%|#######8  | 39428/50000 [25:53:27<5:31:29,  1.88s/it]
Outer training loop:  79%|#######8  | 39429/50000 [25:53:29<5:38:11,  1.92s/it]
Outer training loop:  79%|#######8  | 39430/50000 [25:53:31<5:31:41,  1.88s/it]
Outer training loop:  79%|#######8  | 39431/50000 [25:53:33<5:27:16,  1.86s/it]
Outer training loop:  79%|#######8  | 39432/50000 [25:53:35<5:24:12,  1.84s/it]
Outer training loop:  79%|#######8  | 39433/50000 [25:53:37<5:32:44,  1.89s/it]
Outer training loop:  79%|#######8  | 39434/50000 [25:53:39<5:32:25,  1.89s/it]
Outer training loop:  79%|#######8  | 39435/50000 [25:53:40<5:30:04,  1.87s/it]
Outer training loop:  79%|#######8  | 39436/50000 [25:53:42<5:34:56,  1.90s/it]
Outer training loop:  79%|#######8  | 39437/50000 [25:53:44<5:28:33,  1.87s/it]
Outer training loop:  79%|#######8  | 39438/50000 [25:53:46<5:28:01,  1.86s/it]
Outer training loop:  79%|#######8  | 39439/50000 [25:53:48<5:20:57,  1.82s/it]
Outer training loop:  79%|#######8  | 39440/50000 [25:53:50<5:26:55,  1.86s/it]
Outer training loop:  79%|#######8  | 39441/50000 [25:53:52<5:24:56,  1.85s/it]
Outer training loop:  79%|#######8  | 39442/50000 [25:53:53<5:19:15,  1.81s/it]
Outer training loop:  79%|#######8  | 39443/50000 [25:53:55<5:34:55,  1.90s/it]
Outer training loop:  79%|#######8  | 39444/50000 [25:53:57<5:31:05,  1.88s/it]
Outer training loop:  79%|#######8  | 39445/50000 [25:53:59<5:27:12,  1.86s/it]
Outer training loop:  79%|#######8  | 39446/50000 [25:54:01<5:26:25,  1.86s/it]
Outer training loop:  79%|#######8  | 39447/50000 [25:54:03<5:33:03,  1.89s/it]
Outer training loop:  79%|#######8  | 39448/50000 [25:54:05<5:30:36,  1.88s/it]
Outer training loop:  79%|#######8  | 39449/50000 [25:54:07<5:26:47,  1.86s/it]
Outer training loop:  79%|#######8  | 39450/50000 [25:54:08<5:32:20,  1.89s/it]
Outer training loop:  79%|#######8  | 39451/50000 [25:54:10<5:28:30,  1.87s/it]
Outer training loop:  79%|#######8  | 39452/50000 [25:54:12<5:26:49,  1.86s/it]
Outer training loop:  79%|#######8  | 39453/50000 [25:54:14<5:42:43,  1.95s/it]
Outer training loop:  79%|#######8  | 39454/50000 [25:54:16<5:35:33,  1.91s/it]
Outer training loop:  79%|#######8  | 39455/50000 [25:54:18<5:30:41,  1.88s/it]
Outer training loop:  79%|#######8  | 39456/50000 [25:54:20<5:38:28,  1.93s/it]
Outer training loop:  79%|#######8  | 39457/50000 [25:54:22<5:46:17,  1.97s/it]
Outer training loop:  79%|#######8  | 39458/50000 [25:54:24<5:34:23,  1.90s/it]
Outer training loop:  79%|#######8  | 39459/50000 [25:54:26<5:25:40,  1.85s/it]
Outer training loop:  79%|#######8  | 39460/50000 [25:54:27<5:25:47,  1.85s/it]
Outer training loop:  79%|#######8  | 39461/50000 [25:54:29<5:36:33,  1.92s/it]
Outer training loop:  79%|#######8  | 39462/50000 [25:54:31<5:33:30,  1.90s/it]
Outer training loop:  79%|#######8  | 39463/50000 [25:54:33<5:28:57,  1.87s/it]
Outer training loop:  79%|#######8  | 39464/50000 [25:54:35<5:38:43,  1.93s/it]
Outer training loop:  79%|#######8  | 39465/50000 [25:54:37<5:30:15,  1.88s/it]
Outer training loop:  79%|#######8  | 39466/50000 [25:54:39<5:27:20,  1.86s/it]
Outer training loop:  79%|#######8  | 39467/50000 [25:54:41<5:35:19,  1.91s/it]
Outer training loop:  79%|#######8  | 39468/50000 [25:54:43<5:48:17,  1.98s/it]
Outer training loop:  79%|#######8  | 39469/50000 [25:54:45<5:41:29,  1.95s/it]
Outer training loop:  79%|#######8  | 39470/50000 [25:54:47<5:36:24,  1.92s/it]
Outer training loop:  79%|#######8  | 39471/50000 [25:54:49<5:44:39,  1.96s/it]
Outer training loop:  79%|#######8  | 39472/50000 [25:54:51<5:45:08,  1.97s/it]
Outer training loop:  79%|#######8  | 39473/50000 [25:54:52<5:33:52,  1.90s/it]
Outer training loop:  79%|#######8  | 39474/50000 [25:54:54<5:40:04,  1.94s/it]
Outer training loop:  79%|#######8  | 39475/50000 [25:54:56<5:32:26,  1.90s/it]
Outer training loop:  79%|#######8  | 39476/50000 [25:54:58<5:29:07,  1.88s/it]
Outer training loop:  79%|#######8  | 39477/50000 [25:55:00<5:29:50,  1.88s/it]
Outer training loop:  79%|#######8  | 39478/50000 [25:55:02<5:38:53,  1.93s/it]
Outer training loop:  79%|#######8  | 39479/50000 [25:55:04<5:33:55,  1.90s/it]
Outer training loop:  79%|#######8  | 39480/50000 [25:55:06<5:30:53,  1.89s/it]
Outer training loop:  79%|#######8  | 39481/50000 [25:55:08<5:35:05,  1.91s/it]
Outer training loop:  79%|#######8  | 39482/50000 [25:55:10<5:37:35,  1.93s/it]
Outer training loop:  79%|#######8  | 39483/50000 [25:55:11<5:29:52,  1.88s/it]
Outer training loop:  79%|#######8  | 39484/50000 [25:55:14<5:42:09,  1.95s/it]
Outer training loop:  79%|#######8  | 39485/50000 [25:55:15<5:38:02,  1.93s/it]
Outer training loop:  79%|#######8  | 39486/50000 [25:55:17<5:27:13,  1.87s/it]
Outer training loop:  79%|#######8  | 39487/50000 [25:55:19<5:22:45,  1.84s/it]
Outer training loop:  79%|#######8  | 39488/50000 [25:55:21<5:28:31,  1.88s/it]
Outer training loop:  79%|#######8  | 39489/50000 [25:55:23<5:22:44,  1.84s/it]
Outer training loop:  79%|#######8  | 39490/50000 [25:55:25<5:30:52,  1.89s/it]
Outer training loop:  79%|#######8  | 39491/50000 [25:55:27<5:42:10,  1.95s/it]
Outer training loop:  79%|#######8  | 39492/50000 [25:55:29<5:38:52,  1.93s/it]
Outer training loop:  79%|#######8  | 39493/50000 [25:55:31<5:38:15,  1.93s/it]
Outer training loop:  79%|#######8  | 39494/50000 [25:55:33<5:40:02,  1.94s/it]
Outer training loop:  79%|#######8  | 39495/50000 [25:55:35<5:47:50,  1.99s/it]
Outer training loop:  79%|#######8  | 39496/50000 [25:55:36<5:37:47,  1.93s/it]
Outer training loop:  79%|#######8  | 39497/50000 [25:55:38<5:36:59,  1.93s/it]
Outer training loop:  79%|#######8  | 39498/50000 [25:55:40<5:42:37,  1.96s/it]
Outer training loop:  79%|#######8  | 39499/50000 [25:55:42<5:43:15,  1.96s/it]
Outer training loop:  79%|#######9  | 39500/50000 [25:55:44<5:44:44,  1.97s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.98s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.01s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:28,  1.84s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:27,  1.84s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.82s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.90s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.90s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:26,  1.98s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:21,  1.94s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:17,  1.96s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:16,  1.97s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:12,  1.94s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:10,  1.98s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.91s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.90s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:58,  1.95s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:58,  1.97s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:53,  1.93s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:53,  1.96s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:49,  1.92s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:53,  2.02s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:47,  1.96s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:42,  1.89s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:41,  1.91s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:39,  1.91s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:39,  1.99s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:34,  1.94s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:28,  1.89s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:25,  1.86s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:25,  1.94s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:21,  1.90s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:18,  1.88s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:17,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:11,  1.82s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:08,  1.80s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:10,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:07,  1.86s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:02,  1.85s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<00:59,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.85s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:58,  1.95s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:55,  1.93s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:53,  1.90s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.86s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.88s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:43,  1.82s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:41,  1.82s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:43,  1.97s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:40,  1.93s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.89s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.93s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.88s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.84s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:27,  1.95s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:25,  1.95s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.85s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.86s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.88s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:10,  2.04s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:08,  2.03s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.91s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:21,  1.73s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:18,  1.72s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:19,  1.75s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.79s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:18,  1.78s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:31,  1.97s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.94s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:20,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:17,  1.86s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:11,  1.86s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:06,  1.81s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.92s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:02,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:03,  1.93s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:59,  1.90s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:57,  1.95s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:44,  1.86s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:42,  1.90s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:41,  1.91s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:40,  1.94s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.89s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:36,  1.97s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:32,  1.93s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.92s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.91s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:20,  1.83s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:23,  1.95s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:16,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.91s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:12,  1.96s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:08,  1.90s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.85s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:56,  1.83s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:55,  1.92s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:53,  1.92s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:51,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.88s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:43,  2.00s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.94s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:38,  1.91s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.90s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.85s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.86s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:26,  2.00s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.97s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.89s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.88s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:17,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.88s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.86s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  2.00s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.97s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.91s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.90s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.89s/it]

Outer training loop:  79%|#######9  | 39501/50000 [26:01:07<286:26:14, 98.22s/it]
Outer training loop:  79%|#######9  | 39502/50000 [26:01:09<202:02:08, 69.28s/it]
Outer training loop:  79%|#######9  | 39503/50000 [26:01:11<143:02:48, 49.06s/it]
Outer training loop:  79%|#######9  | 39504/50000 [26:01:13<101:45:16, 34.90s/it]
Outer training loop:  79%|#######9  | 39505/50000 [26:01:14<72:50:11, 24.98s/it] 
Outer training loop:  79%|#######9  | 39506/50000 [26:01:16<52:35:13, 18.04s/it]
Outer training loop:  79%|#######9  | 39507/50000 [26:01:19<38:48:51, 13.32s/it]
Outer training loop:  79%|#######9  | 39508/50000 [26:01:20<28:48:59,  9.89s/it]
Outer training loop:  79%|#######9  | 39509/50000 [26:01:22<21:45:03,  7.46s/it]
Outer training loop:  79%|#######9  | 39510/50000 [26:01:24<17:07:40,  5.88s/it]
Outer training loop:  79%|#######9  | 39511/50000 [26:01:26<13:42:31,  4.71s/it]
Outer training loop:  79%|#######9  | 39512/50000 [26:01:28<11:12:20,  3.85s/it]
Outer training loop:  79%|#######9  | 39513/50000 [26:01:30<9:27:00,  3.24s/it] 
Outer training loop:  79%|#######9  | 39514/50000 [26:01:32<8:25:00,  2.89s/it]
Outer training loop:  79%|#######9  | 39515/50000 [26:01:34<7:32:35,  2.59s/it]
Outer training loop:  79%|#######9  | 39516/50000 [26:01:36<6:54:39,  2.37s/it]
Outer training loop:  79%|#######9  | 39517/50000 [26:01:38<6:34:53,  2.26s/it]
Outer training loop:  79%|#######9  | 39518/50000 [26:01:40<6:10:52,  2.12s/it]
Outer training loop:  79%|#######9  | 39519/50000 [26:01:42<5:54:25,  2.03s/it]
Outer training loop:  79%|#######9  | 39520/50000 [26:01:44<5:57:06,  2.04s/it]
Outer training loop:  79%|#######9  | 39521/50000 [26:01:45<5:42:33,  1.96s/it]
Outer training loop:  79%|#######9  | 39522/50000 [26:01:47<5:34:38,  1.92s/it]
Outer training loop:  79%|#######9  | 39523/50000 [26:01:49<5:28:34,  1.88s/it]
Outer training loop:  79%|#######9  | 39524/50000 [26:01:51<5:39:12,  1.94s/it]
Outer training loop:  79%|#######9  | 39525/50000 [26:01:53<5:32:57,  1.91s/it]
Outer training loop:  79%|#######9  | 39526/50000 [26:01:55<5:26:57,  1.87s/it]
Outer training loop:  79%|#######9  | 39527/50000 [26:01:57<5:36:34,  1.93s/it]
Outer training loop:  79%|#######9  | 39528/50000 [26:01:59<5:29:29,  1.89s/it]
Outer training loop:  79%|#######9  | 39529/50000 [26:02:00<5:26:44,  1.87s/it]
Outer training loop:  79%|#######9  | 39530/50000 [26:02:02<5:30:00,  1.89s/it]
Outer training loop:  79%|#######9  | 39531/50000 [26:02:04<5:36:17,  1.93s/it]
Outer training loop:  79%|#######9  | 39532/50000 [26:02:06<5:39:30,  1.95s/it]
Outer training loop:  79%|#######9  | 39533/50000 [26:02:08<5:31:08,  1.90s/it]
Outer training loop:  79%|#######9  | 39534/50000 [26:02:10<5:36:38,  1.93s/it]
Outer training loop:  79%|#######9  | 39535/50000 [26:02:12<5:29:04,  1.89s/it]
Outer training loop:  79%|#######9  | 39536/50000 [26:02:14<5:23:23,  1.85s/it]
Outer training loop:  79%|#######9  | 39537/50000 [26:02:16<5:27:41,  1.88s/it]
Outer training loop:  79%|#######9  | 39538/50000 [26:02:17<5:25:39,  1.87s/it]
Outer training loop:  79%|#######9  | 39539/50000 [26:02:19<5:24:04,  1.86s/it]
Outer training loop:  79%|#######9  | 39540/50000 [26:02:21<5:20:10,  1.84s/it]
Outer training loop:  79%|#######9  | 39541/50000 [26:02:23<5:29:18,  1.89s/it]
Outer training loop:  79%|#######9  | 39542/50000 [26:02:25<5:26:34,  1.87s/it]
Outer training loop:  79%|#######9  | 39543/50000 [26:02:27<5:29:09,  1.89s/it]
Outer training loop:  79%|#######9  | 39544/50000 [26:02:29<5:46:57,  1.99s/it]
Outer training loop:  79%|#######9  | 39545/50000 [26:02:31<5:37:16,  1.94s/it]
Outer training loop:  79%|#######9  | 39546/50000 [26:02:33<5:30:26,  1.90s/it]
Outer training loop:  79%|#######9  | 39547/50000 [26:02:35<5:35:20,  1.92s/it]
Outer training loop:  79%|#######9  | 39548/50000 [26:02:36<5:27:11,  1.88s/it]
Outer training loop:  79%|#######9  | 39549/50000 [26:02:39<5:37:20,  1.94s/it]
Outer training loop:  79%|#######9  | 39550/50000 [26:02:40<5:34:06,  1.92s/it]
Outer training loop:  79%|#######9  | 39551/50000 [26:02:42<5:35:19,  1.93s/it]
Outer training loop:  79%|#######9  | 39552/50000 [26:02:44<5:29:52,  1.89s/it]
Outer training loop:  79%|#######9  | 39553/50000 [26:02:46<5:28:27,  1.89s/it]
Outer training loop:  79%|#######9  | 39554/50000 [26:02:48<5:24:36,  1.86s/it]
Outer training loop:  79%|#######9  | 39555/50000 [26:02:50<5:39:35,  1.95s/it]
Outer training loop:  79%|#######9  | 39556/50000 [26:02:52<5:28:35,  1.89s/it]
Outer training loop:  79%|#######9  | 39557/50000 [26:02:54<5:27:17,  1.88s/it]
Outer training loop:  79%|#######9  | 39558/50000 [26:02:56<5:33:39,  1.92s/it]
Outer training loop:  79%|#######9  | 39559/50000 [26:02:58<5:36:25,  1.93s/it]
Outer training loop:  79%|#######9  | 39560/50000 [26:02:59<5:33:50,  1.92s/it]
Outer training loop:  79%|#######9  | 39561/50000 [26:03:01<5:38:34,  1.95s/it]
Outer training loop:  79%|#######9  | 39562/50000 [26:03:03<5:33:59,  1.92s/it]
Outer training loop:  79%|#######9  | 39563/50000 [26:03:05<5:26:43,  1.88s/it]
Outer training loop:  79%|#######9  | 39564/50000 [26:03:07<5:22:49,  1.86s/it]
Outer training loop:  79%|#######9  | 39565/50000 [26:03:09<5:29:35,  1.90s/it]
Outer training loop:  79%|#######9  | 39566/50000 [26:03:11<5:28:53,  1.89s/it]
Outer training loop:  79%|#######9  | 39567/50000 [26:03:13<5:28:53,  1.89s/it]
Outer training loop:  79%|#######9  | 39568/50000 [26:03:15<5:36:54,  1.94s/it]
Outer training loop:  79%|#######9  | 39569/50000 [26:03:17<5:30:06,  1.90s/it]
Outer training loop:  79%|#######9  | 39570/50000 [26:03:18<5:25:45,  1.87s/it]
Outer training loop:  79%|#######9  | 39571/50000 [26:03:21<5:40:05,  1.96s/it]
Outer training loop:  79%|#######9  | 39572/50000 [26:03:22<5:32:25,  1.91s/it]
Outer training loop:  79%|#######9  | 39573/50000 [26:03:24<5:26:41,  1.88s/it]
Outer training loop:  79%|#######9  | 39574/50000 [26:03:26<5:24:47,  1.87s/it]
Outer training loop:  79%|#######9  | 39575/50000 [26:03:28<5:36:17,  1.94s/it]
Outer training loop:  79%|#######9  | 39576/50000 [26:03:30<5:35:36,  1.93s/it]
Outer training loop:  79%|#######9  | 39577/50000 [26:03:32<5:35:42,  1.93s/it]
Outer training loop:  79%|#######9  | 39578/50000 [26:03:34<5:43:06,  1.98s/it]
Outer training loop:  79%|#######9  | 39579/50000 [26:03:36<5:32:01,  1.91s/it]
Outer training loop:  79%|#######9  | 39580/50000 [26:03:38<5:29:47,  1.90s/it]
Outer training loop:  79%|#######9  | 39581/50000 [26:03:40<5:36:56,  1.94s/it]
Outer training loop:  79%|#######9  | 39582/50000 [26:03:42<5:31:15,  1.91s/it]
Outer training loop:  79%|#######9  | 39583/50000 [26:03:44<5:35:58,  1.94s/it]
Outer training loop:  79%|#######9  | 39584/50000 [26:03:45<5:29:51,  1.90s/it]
Outer training loop:  79%|#######9  | 39585/50000 [26:03:47<5:34:17,  1.93s/it]
Outer training loop:  79%|#######9  | 39586/50000 [26:03:49<5:25:17,  1.87s/it]
Outer training loop:  79%|#######9  | 39587/50000 [26:03:51<5:29:18,  1.90s/it]
Outer training loop:  79%|#######9  | 39588/50000 [26:03:53<5:36:16,  1.94s/it]
Outer training loop:  79%|#######9  | 39589/50000 [26:03:55<5:29:18,  1.90s/it]
Outer training loop:  79%|#######9  | 39590/50000 [26:03:57<5:30:01,  1.90s/it]
Outer training loop:  79%|#######9  | 39591/50000 [26:03:59<5:36:12,  1.94s/it]
Outer training loop:  79%|#######9  | 39592/50000 [26:04:01<5:29:36,  1.90s/it]
Outer training loop:  79%|#######9  | 39593/50000 [26:04:02<5:25:46,  1.88s/it]
Outer training loop:  79%|#######9  | 39594/50000 [26:04:04<5:31:23,  1.91s/it]
Outer training loop:  79%|#######9  | 39595/50000 [26:04:06<5:24:13,  1.87s/it]
Outer training loop:  79%|#######9  | 39596/50000 [26:04:08<5:20:36,  1.85s/it]
Outer training loop:  79%|#######9  | 39597/50000 [26:04:10<5:23:10,  1.86s/it]
Outer training loop:  79%|#######9  | 39598/50000 [26:04:12<5:24:32,  1.87s/it]
Outer training loop:  79%|#######9  | 39599/50000 [26:04:14<5:31:09,  1.91s/it]
Outer training loop:  79%|#######9  | 39600/50000 [26:04:16<5:22:56,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]

Outer training loop:  79%|#######9  | 39601/50000 [26:04:23<10:08:04,  3.51s/it]
Outer training loop:  79%|#######9  | 39602/50000 [26:04:25<8:56:26,  3.10s/it] 
Outer training loop:  79%|#######9  | 39603/50000 [26:04:27<7:47:25,  2.70s/it]
Outer training loop:  79%|#######9  | 39604/50000 [26:04:29<7:08:19,  2.47s/it]
Outer training loop:  79%|#######9  | 39605/50000 [26:04:31<6:44:57,  2.34s/it]
Outer training loop:  79%|#######9  | 39606/50000 [26:04:33<6:18:42,  2.19s/it]
Outer training loop:  79%|#######9  | 39607/50000 [26:04:34<6:04:38,  2.11s/it]
Outer training loop:  79%|#######9  | 39608/50000 [26:04:37<6:00:17,  2.08s/it]
Outer training loop:  79%|#######9  | 39609/50000 [26:04:38<5:48:37,  2.01s/it]
Outer training loop:  79%|#######9  | 39610/50000 [26:04:40<5:40:58,  1.97s/it]
Outer training loop:  79%|#######9  | 39611/50000 [26:04:42<5:44:10,  1.99s/it]
Outer training loop:  79%|#######9  | 39612/50000 [26:04:44<5:38:14,  1.95s/it]
Outer training loop:  79%|#######9  | 39613/50000 [26:04:46<5:27:33,  1.89s/it]
Outer training loop:  79%|#######9  | 39614/50000 [26:04:48<5:26:11,  1.88s/it]
Outer training loop:  79%|#######9  | 39615/50000 [26:04:50<5:40:10,  1.97s/it]
Outer training loop:  79%|#######9  | 39616/50000 [26:04:52<5:40:07,  1.97s/it]
Outer training loop:  79%|#######9  | 39617/50000 [26:04:54<5:29:26,  1.90s/it]
Outer training loop:  79%|#######9  | 39618/50000 [26:04:56<5:34:47,  1.93s/it]
Outer training loop:  79%|#######9  | 39619/50000 [26:04:58<5:31:37,  1.92s/it]
Outer training loop:  79%|#######9  | 39620/50000 [26:04:59<5:25:05,  1.88s/it]
Outer training loop:  79%|#######9  | 39621/50000 [26:05:01<5:28:51,  1.90s/it]
Outer training loop:  79%|#######9  | 39622/50000 [26:05:03<5:30:31,  1.91s/it]
Outer training loop:  79%|#######9  | 39623/50000 [26:05:05<5:25:58,  1.88s/it]
Outer training loop:  79%|#######9  | 39624/50000 [26:05:07<5:25:37,  1.88s/it]
Outer training loop:  79%|#######9  | 39625/50000 [26:05:09<5:32:36,  1.92s/it]
Outer training loop:  79%|#######9  | 39626/50000 [26:05:11<5:32:18,  1.92s/it]
Outer training loop:  79%|#######9  | 39627/50000 [26:05:13<5:29:43,  1.91s/it]
Outer training loop:  79%|#######9  | 39628/50000 [26:05:15<5:24:08,  1.88s/it]
Outer training loop:  79%|#######9  | 39629/50000 [26:05:17<5:35:06,  1.94s/it]
Outer training loop:  79%|#######9  | 39630/50000 [26:05:18<5:23:13,  1.87s/it]
Outer training loop:  79%|#######9  | 39631/50000 [26:05:20<5:29:32,  1.91s/it]
Outer training loop:  79%|#######9  | 39632/50000 [26:05:22<5:35:45,  1.94s/it]
Outer training loop:  79%|#######9  | 39633/50000 [26:05:24<5:27:08,  1.89s/it]
Outer training loop:  79%|#######9  | 39634/50000 [26:05:26<5:26:57,  1.89s/it]
Outer training loop:  79%|#######9  | 39635/50000 [26:05:28<5:30:44,  1.91s/it]
Outer training loop:  79%|#######9  | 39636/50000 [26:05:30<5:24:46,  1.88s/it]
Outer training loop:  79%|#######9  | 39637/50000 [26:05:32<5:18:57,  1.85s/it]
Outer training loop:  79%|#######9  | 39638/50000 [26:05:33<5:18:09,  1.84s/it]
Outer training loop:  79%|#######9  | 39639/50000 [26:05:35<5:25:41,  1.89s/it]
Outer training loop:  79%|#######9  | 39640/50000 [26:05:37<5:16:27,  1.83s/it]
Outer training loop:  79%|#######9  | 39641/50000 [26:05:39<5:15:06,  1.83s/it]
Outer training loop:  79%|#######9  | 39642/50000 [26:05:41<5:34:06,  1.94s/it]
Outer training loop:  79%|#######9  | 39643/50000 [26:05:43<5:28:06,  1.90s/it]
Outer training loop:  79%|#######9  | 39644/50000 [26:05:45<5:23:50,  1.88s/it]
Outer training loop:  79%|#######9  | 39645/50000 [26:05:47<5:28:50,  1.91s/it]
Outer training loop:  79%|#######9  | 39646/50000 [26:05:49<5:31:34,  1.92s/it]
Outer training loop:  79%|#######9  | 39647/50000 [26:05:51<5:36:24,  1.95s/it]
Outer training loop:  79%|#######9  | 39648/50000 [26:05:53<5:33:57,  1.94s/it]
Outer training loop:  79%|#######9  | 39649/50000 [26:05:55<5:43:40,  1.99s/it]
Outer training loop:  79%|#######9  | 39650/50000 [26:05:56<5:34:43,  1.94s/it]
Outer training loop:  79%|#######9  | 39651/50000 [26:05:58<5:31:37,  1.92s/it]
Outer training loop:  79%|#######9  | 39652/50000 [26:06:00<5:33:09,  1.93s/it]
Outer training loop:  79%|#######9  | 39653/50000 [26:06:02<5:32:12,  1.93s/it]
Outer training loop:  79%|#######9  | 39654/50000 [26:06:04<5:25:00,  1.88s/it]
Outer training loop:  79%|#######9  | 39655/50000 [26:06:06<5:32:28,  1.93s/it]
Outer training loop:  79%|#######9  | 39656/50000 [26:06:08<5:44:13,  2.00s/it]
Outer training loop:  79%|#######9  | 39657/50000 [26:06:10<5:34:04,  1.94s/it]
Outer training loop:  79%|#######9  | 39658/50000 [26:06:12<5:25:40,  1.89s/it]
Outer training loop:  79%|#######9  | 39659/50000 [26:06:14<5:33:07,  1.93s/it]
Outer training loop:  79%|#######9  | 39660/50000 [26:06:16<5:30:24,  1.92s/it]
Outer training loop:  79%|#######9  | 39661/50000 [26:06:18<5:28:33,  1.91s/it]
Outer training loop:  79%|#######9  | 39662/50000 [26:06:19<5:22:10,  1.87s/it]
Outer training loop:  79%|#######9  | 39663/50000 [26:06:21<5:29:44,  1.91s/it]
Outer training loop:  79%|#######9  | 39664/50000 [26:06:23<5:25:20,  1.89s/it]
Outer training loop:  79%|#######9  | 39665/50000 [26:06:25<5:17:14,  1.84s/it]
Outer training loop:  79%|#######9  | 39666/50000 [26:06:27<5:37:09,  1.96s/it]
Outer training loop:  79%|#######9  | 39667/50000 [26:06:29<5:31:58,  1.93s/it]
Outer training loop:  79%|#######9  | 39668/50000 [26:06:31<5:28:39,  1.91s/it]
Outer training loop:  79%|#######9  | 39669/50000 [26:06:33<5:27:32,  1.90s/it]
Outer training loop:  79%|#######9  | 39670/50000 [26:06:35<5:47:37,  2.02s/it]
Outer training loop:  79%|#######9  | 39671/50000 [26:06:37<5:39:01,  1.97s/it]
Outer training loop:  79%|#######9  | 39672/50000 [26:06:39<5:35:23,  1.95s/it]
Outer training loop:  79%|#######9  | 39673/50000 [26:06:41<5:47:13,  2.02s/it]
Outer training loop:  79%|#######9  | 39674/50000 [26:06:43<5:35:55,  1.95s/it]
Outer training loop:  79%|#######9  | 39675/50000 [26:06:45<5:34:00,  1.94s/it]
Outer training loop:  79%|#######9  | 39676/50000 [26:06:47<5:41:02,  1.98s/it]
Outer training loop:  79%|#######9  | 39677/50000 [26:06:49<5:27:02,  1.90s/it]
Outer training loop:  79%|#######9  | 39678/50000 [26:06:50<5:28:11,  1.91s/it]
Outer training loop:  79%|#######9  | 39679/50000 [26:06:52<5:33:45,  1.94s/it]
Outer training loop:  79%|#######9  | 39680/50000 [26:06:54<5:33:00,  1.94s/it]
Outer training loop:  79%|#######9  | 39681/50000 [26:06:56<5:24:02,  1.88s/it]
Outer training loop:  79%|#######9  | 39682/50000 [26:06:58<5:19:03,  1.86s/it]
Outer training loop:  79%|#######9  | 39683/50000 [26:07:00<5:28:56,  1.91s/it]
Outer training loop:  79%|#######9  | 39684/50000 [26:07:02<5:30:52,  1.92s/it]
Outer training loop:  79%|#######9  | 39685/50000 [26:07:04<5:28:29,  1.91s/it]
Outer training loop:  79%|#######9  | 39686/50000 [26:07:06<5:32:07,  1.93s/it]
Outer training loop:  79%|#######9  | 39687/50000 [26:07:08<5:24:46,  1.89s/it]
Outer training loop:  79%|#######9  | 39688/50000 [26:07:09<5:20:40,  1.87s/it]
Outer training loop:  79%|#######9  | 39689/50000 [26:07:11<5:18:55,  1.86s/it]
Outer training loop:  79%|#######9  | 39690/50000 [26:07:13<5:27:41,  1.91s/it]
Outer training loop:  79%|#######9  | 39691/50000 [26:07:15<5:19:08,  1.86s/it]
Outer training loop:  79%|#######9  | 39692/50000 [26:07:17<5:11:53,  1.82s/it]
Outer training loop:  79%|#######9  | 39693/50000 [26:07:19<5:21:22,  1.87s/it]
Outer training loop:  79%|#######9  | 39694/50000 [26:07:21<5:21:17,  1.87s/it]
Outer training loop:  79%|#######9  | 39695/50000 [26:07:22<5:17:11,  1.85s/it]
Outer training loop:  79%|#######9  | 39696/50000 [26:07:24<5:24:24,  1.89s/it]
Outer training loop:  79%|#######9  | 39697/50000 [26:07:26<5:30:22,  1.92s/it]
Outer training loop:  79%|#######9  | 39698/50000 [26:07:28<5:24:28,  1.89s/it]
Outer training loop:  79%|#######9  | 39699/50000 [26:07:30<5:15:58,  1.84s/it]
Outer training loop:  79%|#######9  | 39700/50000 [26:07:32<5:14:56,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  79%|#######9  | 39701/50000 [26:07:39<10:18:31,  3.60s/it]
Outer training loop:  79%|#######9  | 39702/50000 [26:07:41<8:43:54,  3.05s/it] 
Outer training loop:  79%|#######9  | 39703/50000 [26:07:43<7:39:00,  2.67s/it]
Outer training loop:  79%|#######9  | 39704/50000 [26:07:45<6:50:15,  2.39s/it]
Outer training loop:  79%|#######9  | 39705/50000 [26:07:47<6:34:38,  2.30s/it]
Outer training loop:  79%|#######9  | 39706/50000 [26:07:49<6:09:47,  2.16s/it]
Outer training loop:  79%|#######9  | 39707/50000 [26:07:51<5:54:23,  2.07s/it]
Outer training loop:  79%|#######9  | 39708/50000 [26:07:53<5:51:25,  2.05s/it]
Outer training loop:  79%|#######9  | 39709/50000 [26:07:54<5:33:02,  1.94s/it]
Outer training loop:  79%|#######9  | 39710/50000 [26:07:56<5:23:49,  1.89s/it]
Outer training loop:  79%|#######9  | 39711/50000 [26:07:58<5:32:26,  1.94s/it]
Outer training loop:  79%|#######9  | 39712/50000 [26:08:00<5:30:24,  1.93s/it]
Outer training loop:  79%|#######9  | 39713/50000 [26:08:02<5:28:43,  1.92s/it]
Outer training loop:  79%|#######9  | 39714/50000 [26:08:04<5:19:58,  1.87s/it]
Outer training loop:  79%|#######9  | 39715/50000 [26:08:06<5:27:10,  1.91s/it]
Outer training loop:  79%|#######9  | 39716/50000 [26:08:07<5:25:41,  1.90s/it]
Outer training loop:  79%|#######9  | 39717/50000 [26:08:09<5:20:43,  1.87s/it]
Outer training loop:  79%|#######9  | 39718/50000 [26:08:11<5:29:16,  1.92s/it]
Outer training loop:  79%|#######9  | 39719/50000 [26:08:13<5:22:27,  1.88s/it]
Outer training loop:  79%|#######9  | 39720/50000 [26:08:15<5:18:21,  1.86s/it]
Outer training loop:  79%|#######9  | 39721/50000 [26:08:17<5:16:18,  1.85s/it]
Outer training loop:  79%|#######9  | 39722/50000 [26:08:19<5:21:22,  1.88s/it]
Outer training loop:  79%|#######9  | 39723/50000 [26:08:21<5:20:21,  1.87s/it]
Outer training loop:  79%|#######9  | 39724/50000 [26:08:22<5:19:34,  1.87s/it]
Outer training loop:  79%|#######9  | 39725/50000 [26:08:25<5:35:30,  1.96s/it]
Outer training loop:  79%|#######9  | 39726/50000 [26:08:26<5:33:38,  1.95s/it]
Outer training loop:  79%|#######9  | 39727/50000 [26:08:28<5:31:25,  1.94s/it]
Outer training loop:  79%|#######9  | 39728/50000 [26:08:30<5:31:04,  1.93s/it]
Outer training loop:  79%|#######9  | 39729/50000 [26:08:32<5:37:46,  1.97s/it]
Outer training loop:  79%|#######9  | 39730/50000 [26:08:34<5:29:24,  1.92s/it]
Outer training loop:  79%|#######9  | 39731/50000 [26:08:36<5:25:45,  1.90s/it]
Outer training loop:  79%|#######9  | 39732/50000 [26:08:38<5:30:04,  1.93s/it]
Outer training loop:  79%|#######9  | 39733/50000 [26:08:40<5:31:20,  1.94s/it]
Outer training loop:  79%|#######9  | 39734/50000 [26:08:42<5:19:20,  1.87s/it]
Outer training loop:  79%|#######9  | 39735/50000 [26:08:44<5:19:14,  1.87s/it]
Outer training loop:  79%|#######9  | 39736/50000 [26:08:46<5:26:36,  1.91s/it]
Outer training loop:  79%|#######9  | 39737/50000 [26:08:47<5:19:20,  1.87s/it]
Outer training loop:  79%|#######9  | 39738/50000 [26:08:49<5:21:54,  1.88s/it]
Outer training loop:  79%|#######9  | 39739/50000 [26:08:51<5:33:19,  1.95s/it]
Outer training loop:  79%|#######9  | 39740/50000 [26:08:53<5:27:13,  1.91s/it]
Outer training loop:  79%|#######9  | 39741/50000 [26:08:55<5:23:46,  1.89s/it]
Outer training loop:  79%|#######9  | 39742/50000 [26:08:57<5:24:54,  1.90s/it]
Outer training loop:  79%|#######9  | 39743/50000 [26:08:59<5:29:16,  1.93s/it]
Outer training loop:  79%|#######9  | 39744/50000 [26:09:01<5:21:43,  1.88s/it]
Outer training loop:  79%|#######9  | 39745/50000 [26:09:03<5:27:35,  1.92s/it]
Outer training loop:  79%|#######9  | 39746/50000 [26:09:05<5:22:58,  1.89s/it]
Outer training loop:  79%|#######9  | 39747/50000 [26:09:06<5:20:33,  1.88s/it]
Outer training loop:  79%|#######9  | 39748/50000 [26:09:08<5:28:24,  1.92s/it]
Outer training loop:  79%|#######9  | 39749/50000 [26:09:10<5:27:29,  1.92s/it]
Outer training loop:  80%|#######9  | 39750/50000 [26:09:12<5:20:47,  1.88s/it]
Outer training loop:  80%|#######9  | 39751/50000 [26:09:14<5:22:32,  1.89s/it]
Outer training loop:  80%|#######9  | 39752/50000 [26:09:16<5:21:30,  1.88s/it]
Outer training loop:  80%|#######9  | 39753/50000 [26:09:18<5:21:28,  1.88s/it]
Outer training loop:  80%|#######9  | 39754/50000 [26:09:20<5:24:07,  1.90s/it]
Outer training loop:  80%|#######9  | 39755/50000 [26:09:21<5:17:54,  1.86s/it]
Outer training loop:  80%|#######9  | 39756/50000 [26:09:24<5:31:59,  1.94s/it]
Outer training loop:  80%|#######9  | 39757/50000 [26:09:26<5:29:32,  1.93s/it]
Outer training loop:  80%|#######9  | 39758/50000 [26:09:27<5:23:12,  1.89s/it]
Outer training loop:  80%|#######9  | 39759/50000 [26:09:29<5:33:38,  1.95s/it]
Outer training loop:  80%|#######9  | 39760/50000 [26:09:31<5:31:46,  1.94s/it]
Outer training loop:  80%|#######9  | 39761/50000 [26:09:33<5:29:17,  1.93s/it]
Outer training loop:  80%|#######9  | 39762/50000 [26:09:35<5:39:37,  1.99s/it]
Outer training loop:  80%|#######9  | 39763/50000 [26:09:37<5:32:33,  1.95s/it]
Outer training loop:  80%|#######9  | 39764/50000 [26:09:39<5:28:03,  1.92s/it]
Outer training loop:  80%|#######9  | 39765/50000 [26:09:41<5:32:57,  1.95s/it]
Outer training loop:  80%|#######9  | 39766/50000 [26:09:43<5:30:00,  1.93s/it]
Outer training loop:  80%|#######9  | 39767/50000 [26:09:45<5:22:50,  1.89s/it]
Outer training loop:  80%|#######9  | 39768/50000 [26:09:47<5:21:04,  1.88s/it]
Outer training loop:  80%|#######9  | 39769/50000 [26:09:49<5:23:00,  1.89s/it]
Outer training loop:  80%|#######9  | 39770/50000 [26:09:50<5:23:35,  1.90s/it]
Outer training loop:  80%|#######9  | 39771/50000 [26:09:52<5:17:59,  1.87s/it]
Outer training loop:  80%|#######9  | 39772/50000 [26:09:54<5:27:05,  1.92s/it]
Outer training loop:  80%|#######9  | 39773/50000 [26:09:56<5:24:51,  1.91s/it]
Outer training loop:  80%|#######9  | 39774/50000 [26:09:58<5:26:05,  1.91s/it]
Outer training loop:  80%|#######9  | 39775/50000 [26:10:00<5:16:43,  1.86s/it]
Outer training loop:  80%|#######9  | 39776/50000 [26:10:02<5:24:06,  1.90s/it]
Outer training loop:  80%|#######9  | 39777/50000 [26:10:04<5:13:36,  1.84s/it]
Outer training loop:  80%|#######9  | 39778/50000 [26:10:05<5:13:33,  1.84s/it]
Outer training loop:  80%|#######9  | 39779/50000 [26:10:07<5:23:38,  1.90s/it]
Outer training loop:  80%|#######9  | 39780/50000 [26:10:09<5:13:28,  1.84s/it]
Outer training loop:  80%|#######9  | 39781/50000 [26:10:11<5:08:07,  1.81s/it]
Outer training loop:  80%|#######9  | 39782/50000 [26:10:13<5:07:09,  1.80s/it]
Outer training loop:  80%|#######9  | 39783/50000 [26:10:15<5:19:08,  1.87s/it]
Outer training loop:  80%|#######9  | 39784/50000 [26:10:17<5:17:32,  1.86s/it]
Outer training loop:  80%|#######9  | 39785/50000 [26:10:18<5:10:06,  1.82s/it]
Outer training loop:  80%|#######9  | 39786/50000 [26:10:20<5:24:51,  1.91s/it]
Outer training loop:  80%|#######9  | 39787/50000 [26:10:22<5:13:56,  1.84s/it]
Outer training loop:  80%|#######9  | 39788/50000 [26:10:24<5:03:25,  1.78s/it]
Outer training loop:  80%|#######9  | 39789/50000 [26:10:26<5:22:35,  1.90s/it]
Outer training loop:  80%|#######9  | 39790/50000 [26:10:28<5:18:35,  1.87s/it]
Outer training loop:  80%|#######9  | 39791/50000 [26:10:29<5:12:16,  1.84s/it]
Outer training loop:  80%|#######9  | 39792/50000 [26:10:31<5:03:18,  1.78s/it]
Outer training loop:  80%|#######9  | 39793/50000 [26:10:33<5:26:31,  1.92s/it]
Outer training loop:  80%|#######9  | 39794/50000 [26:10:35<5:18:50,  1.87s/it]
Outer training loop:  80%|#######9  | 39795/50000 [26:10:37<5:16:31,  1.86s/it]
Outer training loop:  80%|#######9  | 39796/50000 [26:10:39<5:23:50,  1.90s/it]
Outer training loop:  80%|#######9  | 39797/50000 [26:10:41<5:21:08,  1.89s/it]
Outer training loop:  80%|#######9  | 39798/50000 [26:10:43<5:21:45,  1.89s/it]
Outer training loop:  80%|#######9  | 39799/50000 [26:10:45<5:28:09,  1.93s/it]
Outer training loop:  80%|#######9  | 39800/50000 [26:10:47<5:26:14,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.95s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  80%|#######9  | 39801/50000 [26:10:54<10:16:15,  3.63s/it]
Outer training loop:  80%|#######9  | 39802/50000 [26:10:56<8:47:30,  3.10s/it] 
Outer training loop:  80%|#######9  | 39803/50000 [26:10:58<7:59:20,  2.82s/it]
Outer training loop:  80%|#######9  | 39804/50000 [26:11:00<7:06:18,  2.51s/it]
Outer training loop:  80%|#######9  | 39805/50000 [26:11:02<6:31:18,  2.30s/it]
Outer training loop:  80%|#######9  | 39806/50000 [26:11:04<6:20:56,  2.24s/it]
Outer training loop:  80%|#######9  | 39807/50000 [26:11:06<5:58:54,  2.11s/it]
Outer training loop:  80%|#######9  | 39808/50000 [26:11:07<5:38:38,  1.99s/it]
Outer training loop:  80%|#######9  | 39809/50000 [26:11:10<5:46:00,  2.04s/it]
Outer training loop:  80%|#######9  | 39810/50000 [26:11:11<5:31:42,  1.95s/it]
Outer training loop:  80%|#######9  | 39811/50000 [26:11:13<5:31:05,  1.95s/it]
Outer training loop:  80%|#######9  | 39812/50000 [26:11:15<5:29:14,  1.94s/it]
Outer training loop:  80%|#######9  | 39813/50000 [26:11:17<5:26:46,  1.92s/it]
Outer training loop:  80%|#######9  | 39814/50000 [26:11:19<5:23:07,  1.90s/it]
Outer training loop:  80%|#######9  | 39815/50000 [26:11:21<5:32:15,  1.96s/it]
Outer training loop:  80%|#######9  | 39816/50000 [26:11:23<5:20:18,  1.89s/it]
Outer training loop:  80%|#######9  | 39817/50000 [26:11:25<5:21:54,  1.90s/it]
Outer training loop:  80%|#######9  | 39818/50000 [26:11:27<5:20:42,  1.89s/it]
Outer training loop:  80%|#######9  | 39819/50000 [26:11:29<5:32:32,  1.96s/it]
Outer training loop:  80%|#######9  | 39820/50000 [26:11:30<5:21:15,  1.89s/it]
Outer training loop:  80%|#######9  | 39821/50000 [26:11:32<5:16:39,  1.87s/it]
Outer training loop:  80%|#######9  | 39822/50000 [26:11:34<5:20:36,  1.89s/it]
Outer training loop:  80%|#######9  | 39823/50000 [26:11:36<5:20:55,  1.89s/it]
Outer training loop:  80%|#######9  | 39824/50000 [26:11:38<5:16:41,  1.87s/it]
Outer training loop:  80%|#######9  | 39825/50000 [26:11:40<5:11:34,  1.84s/it]
Outer training loop:  80%|#######9  | 39826/50000 [26:11:42<5:17:36,  1.87s/it]
Outer training loop:  80%|#######9  | 39827/50000 [26:11:43<5:16:17,  1.87s/it]
Outer training loop:  80%|#######9  | 39828/50000 [26:11:45<5:08:13,  1.82s/it]
Outer training loop:  80%|#######9  | 39829/50000 [26:11:47<5:17:58,  1.88s/it]
Outer training loop:  80%|#######9  | 39830/50000 [26:11:49<5:18:29,  1.88s/it]
Outer training loop:  80%|#######9  | 39831/50000 [26:11:51<5:12:38,  1.84s/it]
Outer training loop:  80%|#######9  | 39832/50000 [26:11:52<5:03:52,  1.79s/it]
Outer training loop:  80%|#######9  | 39833/50000 [26:11:55<5:15:40,  1.86s/it]
Outer training loop:  80%|#######9  | 39834/50000 [26:11:56<5:06:07,  1.81s/it]
Outer training loop:  80%|#######9  | 39835/50000 [26:11:58<5:10:11,  1.83s/it]
Outer training loop:  80%|#######9  | 39836/50000 [26:12:00<5:16:45,  1.87s/it]
Outer training loop:  80%|#######9  | 39837/50000 [26:12:02<5:14:15,  1.86s/it]
Outer training loop:  80%|#######9  | 39838/50000 [26:12:04<5:14:00,  1.85s/it]
Outer training loop:  80%|#######9  | 39839/50000 [26:12:06<5:13:40,  1.85s/it]
Outer training loop:  80%|#######9  | 39840/50000 [26:12:08<5:30:05,  1.95s/it]
Outer training loop:  80%|#######9  | 39841/50000 [26:12:10<5:26:04,  1.93s/it]
Outer training loop:  80%|#######9  | 39842/50000 [26:12:11<5:17:00,  1.87s/it]
Outer training loop:  80%|#######9  | 39843/50000 [26:12:13<5:13:09,  1.85s/it]
Outer training loop:  80%|#######9  | 39844/50000 [26:12:15<5:25:36,  1.92s/it]
Outer training loop:  80%|#######9  | 39845/50000 [26:12:17<5:21:14,  1.90s/it]
Outer training loop:  80%|#######9  | 39846/50000 [26:12:19<5:22:37,  1.91s/it]
Outer training loop:  80%|#######9  | 39847/50000 [26:12:21<5:28:28,  1.94s/it]
Outer training loop:  80%|#######9  | 39848/50000 [26:12:23<5:20:11,  1.89s/it]
Outer training loop:  80%|#######9  | 39849/50000 [26:12:25<5:20:40,  1.90s/it]
Outer training loop:  80%|#######9  | 39850/50000 [26:12:27<5:27:15,  1.93s/it]
Outer training loop:  80%|#######9  | 39851/50000 [26:12:29<5:27:12,  1.93s/it]
Outer training loop:  80%|#######9  | 39852/50000 [26:12:30<5:19:31,  1.89s/it]
Outer training loop:  80%|#######9  | 39853/50000 [26:12:33<5:34:38,  1.98s/it]
Outer training loop:  80%|#######9  | 39854/50000 [26:12:35<5:36:18,  1.99s/it]
Outer training loop:  80%|#######9  | 39855/50000 [26:12:36<5:26:30,  1.93s/it]
Outer training loop:  80%|#######9  | 39856/50000 [26:12:38<5:24:10,  1.92s/it]
Outer training loop:  80%|#######9  | 39857/50000 [26:12:40<5:30:23,  1.95s/it]
Outer training loop:  80%|#######9  | 39858/50000 [26:12:42<5:19:57,  1.89s/it]
Outer training loop:  80%|#######9  | 39859/50000 [26:12:44<5:14:24,  1.86s/it]
Outer training loop:  80%|#######9  | 39860/50000 [26:12:46<5:20:07,  1.89s/it]
Outer training loop:  80%|#######9  | 39861/50000 [26:12:48<5:15:32,  1.87s/it]
Outer training loop:  80%|#######9  | 39862/50000 [26:12:50<5:18:50,  1.89s/it]
Outer training loop:  80%|#######9  | 39863/50000 [26:12:52<5:19:26,  1.89s/it]
Outer training loop:  80%|#######9  | 39864/50000 [26:12:54<5:25:59,  1.93s/it]
Outer training loop:  80%|#######9  | 39865/50000 [26:12:55<5:16:51,  1.88s/it]
Outer training loop:  80%|#######9  | 39866/50000 [26:12:57<5:17:01,  1.88s/it]
Outer training loop:  80%|#######9  | 39867/50000 [26:12:59<5:14:29,  1.86s/it]
Outer training loop:  80%|#######9  | 39868/50000 [26:13:01<5:26:29,  1.93s/it]
Outer training loop:  80%|#######9  | 39869/50000 [26:13:03<5:22:40,  1.91s/it]
Outer training loop:  80%|#######9  | 39870/50000 [26:13:05<5:24:03,  1.92s/it]
Outer training loop:  80%|#######9  | 39871/50000 [26:13:07<5:27:39,  1.94s/it]
Outer training loop:  80%|#######9  | 39872/50000 [26:13:09<5:13:17,  1.86s/it]
Outer training loop:  80%|#######9  | 39873/50000 [26:13:10<5:16:30,  1.88s/it]
Outer training loop:  80%|#######9  | 39874/50000 [26:13:13<5:24:36,  1.92s/it]
Outer training loop:  80%|#######9  | 39875/50000 [26:13:14<5:15:53,  1.87s/it]
Outer training loop:  80%|#######9  | 39876/50000 [26:13:16<5:05:40,  1.81s/it]
Outer training loop:  80%|#######9  | 39877/50000 [26:13:18<5:10:17,  1.84s/it]
Outer training loop:  80%|#######9  | 39878/50000 [26:13:20<5:25:07,  1.93s/it]
Outer training loop:  80%|#######9  | 39879/50000 [26:13:22<5:21:41,  1.91s/it]
Outer training loop:  80%|#######9  | 39880/50000 [26:13:24<5:13:34,  1.86s/it]
Outer training loop:  80%|#######9  | 39881/50000 [26:13:25<5:15:56,  1.87s/it]
Outer training loop:  80%|#######9  | 39882/50000 [26:13:27<5:21:31,  1.91s/it]
Outer training loop:  80%|#######9  | 39883/50000 [26:13:29<5:18:20,  1.89s/it]
Outer training loop:  80%|#######9  | 39884/50000 [26:13:31<5:23:00,  1.92s/it]
Outer training loop:  80%|#######9  | 39885/50000 [26:13:33<5:27:23,  1.94s/it]
Outer training loop:  80%|#######9  | 39886/50000 [26:13:35<5:26:40,  1.94s/it]
Outer training loop:  80%|#######9  | 39887/50000 [26:13:37<5:21:24,  1.91s/it]
Outer training loop:  80%|#######9  | 39888/50000 [26:13:39<5:26:44,  1.94s/it]
Outer training loop:  80%|#######9  | 39889/50000 [26:13:41<5:21:10,  1.91s/it]
Outer training loop:  80%|#######9  | 39890/50000 [26:13:43<5:30:05,  1.96s/it]
Outer training loop:  80%|#######9  | 39891/50000 [26:13:45<5:23:22,  1.92s/it]
Outer training loop:  80%|#######9  | 39892/50000 [26:13:47<5:29:03,  1.95s/it]
Outer training loop:  80%|#######9  | 39893/50000 [26:13:49<5:18:17,  1.89s/it]
Outer training loop:  80%|#######9  | 39894/50000 [26:13:50<5:19:19,  1.90s/it]
Outer training loop:  80%|#######9  | 39895/50000 [26:13:53<5:31:26,  1.97s/it]
Outer training loop:  80%|#######9  | 39896/50000 [26:13:55<5:29:17,  1.96s/it]
Outer training loop:  80%|#######9  | 39897/50000 [26:13:56<5:20:18,  1.90s/it]
Outer training loop:  80%|#######9  | 39898/50000 [26:13:58<5:25:54,  1.94s/it]
Outer training loop:  80%|#######9  | 39899/50000 [26:14:00<5:16:18,  1.88s/it]
Outer training loop:  80%|#######9  | 39900/50000 [26:14:02<5:12:44,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.05s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:02,  2.01s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]

Outer training loop:  80%|#######9  | 39901/50000 [26:14:09<9:55:24,  3.54s/it]
Outer training loop:  80%|#######9  | 39902/50000 [26:14:11<8:36:58,  3.07s/it]
Outer training loop:  80%|#######9  | 39903/50000 [26:14:13<7:32:09,  2.69s/it]
Outer training loop:  80%|#######9  | 39904/50000 [26:14:15<6:44:32,  2.40s/it]
Outer training loop:  80%|#######9  | 39905/50000 [26:14:17<6:17:08,  2.24s/it]
Outer training loop:  80%|#######9  | 39906/50000 [26:14:19<6:00:42,  2.14s/it]
Outer training loop:  80%|#######9  | 39907/50000 [26:14:20<5:45:12,  2.05s/it]
Outer training loop:  80%|#######9  | 39908/50000 [26:14:22<5:32:54,  1.98s/it]
Outer training loop:  80%|#######9  | 39909/50000 [26:14:24<5:36:02,  2.00s/it]
Outer training loop:  80%|#######9  | 39910/50000 [26:14:26<5:43:50,  2.04s/it]
Outer training loop:  80%|#######9  | 39911/50000 [26:14:28<5:38:42,  2.01s/it]
Outer training loop:  80%|#######9  | 39912/50000 [26:14:30<5:31:23,  1.97s/it]
Outer training loop:  80%|#######9  | 39913/50000 [26:14:32<5:38:28,  2.01s/it]
Outer training loop:  80%|#######9  | 39914/50000 [26:14:34<5:32:18,  1.98s/it]
Outer training loop:  80%|#######9  | 39915/50000 [26:14:36<5:33:27,  1.98s/it]
Outer training loop:  80%|#######9  | 39916/50000 [26:14:38<5:42:35,  2.04s/it]
Outer training loop:  80%|#######9  | 39917/50000 [26:14:40<5:34:56,  1.99s/it]
Outer training loop:  80%|#######9  | 39918/50000 [26:14:42<5:22:21,  1.92s/it]
Outer training loop:  80%|#######9  | 39919/50000 [26:14:44<5:23:30,  1.93s/it]
Outer training loop:  80%|#######9  | 39920/50000 [26:14:46<5:33:07,  1.98s/it]
Outer training loop:  80%|#######9  | 39921/50000 [26:14:48<5:24:44,  1.93s/it]
Outer training loop:  80%|#######9  | 39922/50000 [26:14:50<5:21:41,  1.92s/it]
Outer training loop:  80%|#######9  | 39923/50000 [26:14:52<5:24:39,  1.93s/it]
Outer training loop:  80%|#######9  | 39924/50000 [26:14:54<5:26:41,  1.95s/it]
Outer training loop:  80%|#######9  | 39925/50000 [26:14:56<5:30:40,  1.97s/it]
Outer training loop:  80%|#######9  | 39926/50000 [26:14:58<5:35:19,  2.00s/it]
Outer training loop:  80%|#######9  | 39927/50000 [26:15:00<5:26:36,  1.95s/it]
Outer training loop:  80%|#######9  | 39928/50000 [26:15:01<5:18:07,  1.90s/it]
Outer training loop:  80%|#######9  | 39929/50000 [26:15:03<5:13:00,  1.86s/it]
Outer training loop:  80%|#######9  | 39930/50000 [26:15:05<5:29:04,  1.96s/it]
Outer training loop:  80%|#######9  | 39931/50000 [26:15:07<5:24:49,  1.94s/it]
Outer training loop:  80%|#######9  | 39932/50000 [26:15:09<5:30:13,  1.97s/it]
Outer training loop:  80%|#######9  | 39933/50000 [26:15:11<5:32:49,  1.98s/it]
Outer training loop:  80%|#######9  | 39934/50000 [26:15:13<5:28:19,  1.96s/it]
Outer training loop:  80%|#######9  | 39935/50000 [26:15:15<5:24:15,  1.93s/it]
Outer training loop:  80%|#######9  | 39936/50000 [26:15:17<5:17:32,  1.89s/it]
Outer training loop:  80%|#######9  | 39937/50000 [26:15:19<5:39:08,  2.02s/it]
Outer training loop:  80%|#######9  | 39938/50000 [26:15:21<5:30:09,  1.97s/it]
Outer training loop:  80%|#######9  | 39939/50000 [26:15:23<5:22:58,  1.93s/it]
Outer training loop:  80%|#######9  | 39940/50000 [26:15:25<5:30:08,  1.97s/it]
Outer training loop:  80%|#######9  | 39941/50000 [26:15:27<5:26:47,  1.95s/it]
Outer training loop:  80%|#######9  | 39942/50000 [26:15:29<5:29:30,  1.97s/it]
Outer training loop:  80%|#######9  | 39943/50000 [26:15:31<5:25:02,  1.94s/it]
Outer training loop:  80%|#######9  | 39944/50000 [26:15:33<5:30:30,  1.97s/it]
Outer training loop:  80%|#######9  | 39945/50000 [26:15:35<5:36:56,  2.01s/it]
Outer training loop:  80%|#######9  | 39946/50000 [26:15:37<5:27:03,  1.95s/it]
Outer training loop:  80%|#######9  | 39947/50000 [26:15:39<5:17:01,  1.89s/it]
Outer training loop:  80%|#######9  | 39948/50000 [26:15:41<5:21:23,  1.92s/it]
Outer training loop:  80%|#######9  | 39949/50000 [26:15:42<5:10:50,  1.86s/it]
Outer training loop:  80%|#######9  | 39950/50000 [26:15:44<5:09:41,  1.85s/it]
Outer training loop:  80%|#######9  | 39951/50000 [26:15:46<5:16:21,  1.89s/it]
Outer training loop:  80%|#######9  | 39952/50000 [26:15:48<5:12:43,  1.87s/it]
Outer training loop:  80%|#######9  | 39953/50000 [26:15:50<5:18:22,  1.90s/it]
Outer training loop:  80%|#######9  | 39954/50000 [26:15:52<5:10:28,  1.85s/it]
Outer training loop:  80%|#######9  | 39955/50000 [26:15:54<5:18:33,  1.90s/it]
Outer training loop:  80%|#######9  | 39956/50000 [26:15:56<5:20:41,  1.92s/it]
Outer training loop:  80%|#######9  | 39957/50000 [26:15:57<5:15:29,  1.88s/it]
Outer training loop:  80%|#######9  | 39958/50000 [26:15:59<5:17:56,  1.90s/it]
Outer training loop:  80%|#######9  | 39959/50000 [26:16:01<5:28:45,  1.96s/it]
Outer training loop:  80%|#######9  | 39960/50000 [26:16:03<5:24:30,  1.94s/it]
Outer training loop:  80%|#######9  | 39961/50000 [26:16:05<5:20:39,  1.92s/it]
Outer training loop:  80%|#######9  | 39962/50000 [26:16:07<5:26:32,  1.95s/it]
Outer training loop:  80%|#######9  | 39963/50000 [26:16:09<5:16:51,  1.89s/it]
Outer training loop:  80%|#######9  | 39964/50000 [26:16:11<5:11:57,  1.87s/it]
Outer training loop:  80%|#######9  | 39965/50000 [26:16:13<5:26:43,  1.95s/it]
Outer training loop:  80%|#######9  | 39966/50000 [26:16:15<5:16:57,  1.90s/it]
Outer training loop:  80%|#######9  | 39967/50000 [26:16:16<5:08:21,  1.84s/it]
Outer training loop:  80%|#######9  | 39968/50000 [26:16:18<5:09:30,  1.85s/it]
Outer training loop:  80%|#######9  | 39969/50000 [26:16:20<5:18:16,  1.90s/it]
Outer training loop:  80%|#######9  | 39970/50000 [26:16:22<5:16:03,  1.89s/it]
Outer training loop:  80%|#######9  | 39971/50000 [26:16:24<5:14:43,  1.88s/it]
Outer training loop:  80%|#######9  | 39972/50000 [26:16:26<5:23:32,  1.94s/it]
Outer training loop:  80%|#######9  | 39973/50000 [26:16:28<5:22:11,  1.93s/it]
Outer training loop:  80%|#######9  | 39974/50000 [26:16:30<5:18:24,  1.91s/it]
Outer training loop:  80%|#######9  | 39975/50000 [26:16:32<5:10:30,  1.86s/it]
Outer training loop:  80%|#######9  | 39976/50000 [26:16:34<5:14:40,  1.88s/it]
Outer training loop:  80%|#######9  | 39977/50000 [26:16:35<5:13:22,  1.88s/it]
Outer training loop:  80%|#######9  | 39978/50000 [26:16:37<5:09:08,  1.85s/it]
Outer training loop:  80%|#######9  | 39979/50000 [26:16:39<5:12:55,  1.87s/it]
Outer training loop:  80%|#######9  | 39980/50000 [26:16:41<5:13:32,  1.88s/it]
Outer training loop:  80%|#######9  | 39981/50000 [26:16:43<5:10:11,  1.86s/it]
Outer training loop:  80%|#######9  | 39982/50000 [26:16:45<5:11:06,  1.86s/it]
Outer training loop:  80%|#######9  | 39983/50000 [26:16:47<5:20:22,  1.92s/it]
Outer training loop:  80%|#######9  | 39984/50000 [26:16:49<5:17:53,  1.90s/it]
Outer training loop:  80%|#######9  | 39985/50000 [26:16:51<5:19:58,  1.92s/it]
Outer training loop:  80%|#######9  | 39986/50000 [26:16:53<5:24:55,  1.95s/it]
Outer training loop:  80%|#######9  | 39987/50000 [26:16:55<5:31:20,  1.99s/it]
Outer training loop:  80%|#######9  | 39988/50000 [26:16:56<5:24:20,  1.94s/it]
Outer training loop:  80%|#######9  | 39989/50000 [26:16:58<5:13:59,  1.88s/it]
Outer training loop:  80%|#######9  | 39990/50000 [26:17:00<5:16:19,  1.90s/it]
Outer training loop:  80%|#######9  | 39991/50000 [26:17:02<5:18:15,  1.91s/it]
Outer training loop:  80%|#######9  | 39992/50000 [26:17:04<5:20:49,  1.92s/it]
Outer training loop:  80%|#######9  | 39993/50000 [26:17:06<5:28:22,  1.97s/it]
Outer training loop:  80%|#######9  | 39994/50000 [26:17:08<5:18:15,  1.91s/it]
Outer training loop:  80%|#######9  | 39995/50000 [26:17:10<5:16:47,  1.90s/it]
Outer training loop:  80%|#######9  | 39996/50000 [26:17:12<5:21:50,  1.93s/it]
Outer training loop:  80%|#######9  | 39997/50000 [26:17:14<5:16:20,  1.90s/it]
Outer training loop:  80%|#######9  | 39998/50000 [26:17:15<5:12:16,  1.87s/it]
Outer training loop:  80%|#######9  | 39999/50000 [26:17:17<5:18:25,  1.91s/it]
Outer training loop:  80%|########  | 40000/50000 [26:17:19<5:20:33,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.77s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:26,  1.78s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:28,  1.84s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:26,  1.83s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.81s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:23,  1.84s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.90s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:20,  1.90s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:20,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:10,  1.87s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.91s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:03,  1.84s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:58,  1.82s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:02,  1.91s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:01,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.86s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:56,  1.95s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:45,  1.85s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:43,  1.85s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:47,  1.95s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:41,  1.89s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:39,  1.88s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:39,  1.91s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:35,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:42,  2.04s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:35,  1.95s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:25,  1.86s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:21,  1.80s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:20,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:21,  1.90s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.88s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:12,  1.85s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:09,  1.83s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:11,  1.94s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:08,  1.89s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:05,  1.86s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.81s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:58,  1.81s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:54,  1.82s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:56,  1.95s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:53,  1.92s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:51,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:42,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:41,  1.97s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:38,  1.94s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.87s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:29,  1.82s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.82s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:25,  1.98s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:23,  1.95s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.90s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:12,  1.85s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.85s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.83s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:08,  2.07s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:06,  2.04s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.96s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.90s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:26,  1.79s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.77s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.78s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:24,  1.83s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:21,  1.82s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:34,  2.01s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:30,  1.98s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:22,  1.95s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:14,  1.96s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.93s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:07,  1.99s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:02,  1.94s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:59,  1.93s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:52,  1.88s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:55,  1.96s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:53,  1.95s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.92s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:43,  1.88s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:46,  1.97s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:41,  1.92s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:39,  1.91s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:34,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:31,  1.88s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:36,  2.02s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:35,  2.02s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:31,  1.99s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:26,  1.93s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:22,  1.91s/it]

Evaluating policy:  49%|####9     | 41/83 [01:19<01:23,  1.99s/it]

Evaluating policy:  51%|#####     | 42/83 [01:21<01:21,  1.98s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:18,  1.97s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:15,  1.95s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:09,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:13,  2.04s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:09,  2.00s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:07,  1.97s/it]

Evaluating policy:  60%|######    | 50/83 [01:36<01:05,  1.99s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<01:03,  1.98s/it]

Evaluating policy:  63%|######2   | 52/83 [01:40<01:00,  1.96s/it]

Evaluating policy:  64%|######3   | 53/83 [01:42<00:57,  1.90s/it]

Evaluating policy:  65%|######5   | 54/83 [01:44<00:54,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:46<00:56,  2.02s/it]

Evaluating policy:  67%|######7   | 56/83 [01:48<00:53,  1.99s/it]

Evaluating policy:  69%|######8   | 57/83 [01:50<00:50,  1.93s/it]

Evaluating policy:  70%|######9   | 58/83 [01:52<00:48,  1.94s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:54<00:45,  1.90s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:43,  1.90s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:42,  1.93s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:39,  1.90s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:02<00:41,  2.07s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:04<00:37,  1.99s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:05<00:34,  1.93s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:07<00:32,  1.92s/it]

Evaluating policy:  81%|########  | 67/83 [02:09<00:30,  1.93s/it]

Evaluating policy:  82%|########1 | 68/83 [02:11<00:28,  1.90s/it]

Evaluating policy:  83%|########3 | 69/83 [02:13<00:26,  1.92s/it]

Evaluating policy:  84%|########4 | 70/83 [02:15<00:24,  1.90s/it]

Evaluating policy:  86%|########5 | 71/83 [02:17<00:24,  2.03s/it]

Evaluating policy:  87%|########6 | 72/83 [02:19<00:21,  1.96s/it]

Evaluating policy:  88%|########7 | 73/83 [02:21<00:19,  1.93s/it]

Evaluating policy:  89%|########9 | 74/83 [02:23<00:17,  1.92s/it]

Evaluating policy:  90%|######### | 75/83 [02:25<00:15,  1.95s/it]

Evaluating policy:  92%|#########1| 76/83 [02:27<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:28<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:30<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:32<00:07,  1.87s/it]

Evaluating policy:  96%|#########6| 80/83 [02:34<00:06,  2.01s/it]

Evaluating policy:  98%|#########7| 81/83 [02:36<00:03,  1.95s/it]

Evaluating policy:  99%|#########8| 82/83 [02:38<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 83/83 [02:40<00:00,  1.92s/it]

Outer training loop:  80%|########  | 40001/50000 [26:22:44<274:09:04, 98.70s/it]
Outer training loop:  80%|########  | 40002/50000 [26:22:46<193:23:58, 69.64s/it]
Outer training loop:  80%|########  | 40003/50000 [26:22:47<136:48:34, 49.27s/it]
Outer training loop:  80%|########  | 40004/50000 [26:22:49<97:15:46, 35.03s/it] 
Outer training loop:  80%|########  | 40005/50000 [26:22:51<69:32:57, 25.05s/it]
Outer training loop:  80%|########  | 40006/50000 [26:22:53<50:16:19, 18.11s/it]
Outer training loop:  80%|########  | 40007/50000 [26:22:55<36:46:42, 13.25s/it]
Outer training loop:  80%|########  | 40008/50000 [26:22:57<27:49:33, 10.03s/it]
Outer training loop:  80%|########  | 40009/50000 [26:22:59<21:00:05,  7.57s/it]
Outer training loop:  80%|########  | 40010/50000 [26:23:01<16:12:25,  5.84s/it]
Outer training loop:  80%|########  | 40011/50000 [26:23:03<12:50:16,  4.63s/it]
Outer training loop:  80%|########  | 40012/50000 [26:23:05<10:39:53,  3.84s/it]
Outer training loop:  80%|########  | 40013/50000 [26:23:06<8:52:47,  3.20s/it] 
Outer training loop:  80%|########  | 40014/50000 [26:23:08<7:40:29,  2.77s/it]
Outer training loop:  80%|########  | 40015/50000 [26:23:10<7:04:33,  2.55s/it]
Outer training loop:  80%|########  | 40016/50000 [26:23:12<6:31:08,  2.35s/it]
Outer training loop:  80%|########  | 40017/50000 [26:23:14<6:05:14,  2.20s/it]
Outer training loop:  80%|########  | 40018/50000 [26:23:16<6:04:18,  2.19s/it]
Outer training loop:  80%|########  | 40019/50000 [26:23:18<5:42:29,  2.06s/it]
Outer training loop:  80%|########  | 40020/50000 [26:23:20<5:35:33,  2.02s/it]
Outer training loop:  80%|########  | 40021/50000 [26:23:22<5:25:08,  1.95s/it]
Outer training loop:  80%|########  | 40022/50000 [26:23:24<5:30:48,  1.99s/it]
Outer training loop:  80%|########  | 40023/50000 [26:23:26<5:22:14,  1.94s/it]
Outer training loop:  80%|########  | 40024/50000 [26:23:27<5:20:04,  1.93s/it]
Outer training loop:  80%|########  | 40025/50000 [26:23:29<5:18:10,  1.91s/it]
Outer training loop:  80%|########  | 40026/50000 [26:23:31<5:19:29,  1.92s/it]
Outer training loop:  80%|########  | 40027/50000 [26:23:33<5:19:07,  1.92s/it]
Outer training loop:  80%|########  | 40028/50000 [26:23:35<5:09:37,  1.86s/it]
Outer training loop:  80%|########  | 40029/50000 [26:23:37<5:15:08,  1.90s/it]
Outer training loop:  80%|########  | 40030/50000 [26:23:39<5:09:09,  1.86s/it]
Outer training loop:  80%|########  | 40031/50000 [26:23:41<5:10:49,  1.87s/it]
Outer training loop:  80%|########  | 40032/50000 [26:23:43<5:13:59,  1.89s/it]
Outer training loop:  80%|########  | 40033/50000 [26:23:44<5:18:37,  1.92s/it]
Outer training loop:  80%|########  | 40034/50000 [26:23:46<5:14:10,  1.89s/it]
Outer training loop:  80%|########  | 40035/50000 [26:23:48<5:07:23,  1.85s/it]
Outer training loop:  80%|########  | 40036/50000 [26:23:50<5:18:00,  1.91s/it]
Outer training loop:  80%|########  | 40037/50000 [26:23:52<5:15:41,  1.90s/it]
Outer training loop:  80%|########  | 40038/50000 [26:23:54<5:11:39,  1.88s/it]
Outer training loop:  80%|########  | 40039/50000 [26:23:56<5:07:40,  1.85s/it]
Outer training loop:  80%|########  | 40040/50000 [26:23:58<5:15:38,  1.90s/it]
Outer training loop:  80%|########  | 40041/50000 [26:24:00<5:19:30,  1.92s/it]
Outer training loop:  80%|########  | 40042/50000 [26:24:01<5:08:53,  1.86s/it]
Outer training loop:  80%|########  | 40043/50000 [26:24:03<5:10:36,  1.87s/it]
Outer training loop:  80%|########  | 40044/50000 [26:24:05<5:14:51,  1.90s/it]
Outer training loop:  80%|########  | 40045/50000 [26:24:07<5:11:53,  1.88s/it]
Outer training loop:  80%|########  | 40046/50000 [26:24:09<5:19:16,  1.92s/it]
Outer training loop:  80%|########  | 40047/50000 [26:24:11<5:14:45,  1.90s/it]
Outer training loop:  80%|########  | 40048/50000 [26:24:13<5:09:02,  1.86s/it]
Outer training loop:  80%|########  | 40049/50000 [26:24:15<5:13:17,  1.89s/it]
Outer training loop:  80%|########  | 40050/50000 [26:24:16<5:05:57,  1.84s/it]
Outer training loop:  80%|########  | 40051/50000 [26:24:18<5:03:21,  1.83s/it]
Outer training loop:  80%|########  | 40052/50000 [26:24:20<5:07:45,  1.86s/it]
Outer training loop:  80%|########  | 40053/50000 [26:24:22<5:11:34,  1.88s/it]
Outer training loop:  80%|########  | 40054/50000 [26:24:24<5:09:02,  1.86s/it]
Outer training loop:  80%|########  | 40055/50000 [26:24:26<5:11:42,  1.88s/it]
Outer training loop:  80%|########  | 40056/50000 [26:24:28<5:15:43,  1.90s/it]
Outer training loop:  80%|########  | 40057/50000 [26:24:30<5:13:30,  1.89s/it]
Outer training loop:  80%|########  | 40058/50000 [26:24:32<5:14:38,  1.90s/it]
Outer training loop:  80%|########  | 40059/50000 [26:24:34<5:21:45,  1.94s/it]
Outer training loop:  80%|########  | 40060/50000 [26:24:35<5:20:58,  1.94s/it]
Outer training loop:  80%|########  | 40061/50000 [26:24:37<5:12:41,  1.89s/it]
Outer training loop:  80%|########  | 40062/50000 [26:24:39<5:11:42,  1.88s/it]
Outer training loop:  80%|########  | 40063/50000 [26:24:41<5:22:26,  1.95s/it]
Outer training loop:  80%|########  | 40064/50000 [26:24:43<5:14:04,  1.90s/it]
Outer training loop:  80%|########  | 40065/50000 [26:24:45<5:13:57,  1.90s/it]
Outer training loop:  80%|########  | 40066/50000 [26:24:47<5:18:51,  1.93s/it]
Outer training loop:  80%|########  | 40067/50000 [26:24:49<5:15:24,  1.91s/it]
Outer training loop:  80%|########  | 40068/50000 [26:24:51<5:22:50,  1.95s/it]
Outer training loop:  80%|########  | 40069/50000 [26:24:53<5:24:09,  1.96s/it]
Outer training loop:  80%|########  | 40070/50000 [26:24:55<5:19:20,  1.93s/it]
Outer training loop:  80%|########  | 40071/50000 [26:24:56<5:09:07,  1.87s/it]
Outer training loop:  80%|########  | 40072/50000 [26:24:58<5:13:58,  1.90s/it]
Outer training loop:  80%|########  | 40073/50000 [26:25:00<5:17:05,  1.92s/it]
Outer training loop:  80%|########  | 40074/50000 [26:25:02<5:14:09,  1.90s/it]
Outer training loop:  80%|########  | 40075/50000 [26:25:04<5:12:03,  1.89s/it]
Outer training loop:  80%|########  | 40076/50000 [26:25:06<5:21:20,  1.94s/it]
Outer training loop:  80%|########  | 40077/50000 [26:25:08<5:10:07,  1.88s/it]
Outer training loop:  80%|########  | 40078/50000 [26:25:10<5:09:11,  1.87s/it]
Outer training loop:  80%|########  | 40079/50000 [26:25:11<5:08:13,  1.86s/it]
Outer training loop:  80%|########  | 40080/50000 [26:25:13<5:08:41,  1.87s/it]
Outer training loop:  80%|########  | 40081/50000 [26:25:15<5:07:17,  1.86s/it]
Outer training loop:  80%|########  | 40082/50000 [26:25:17<5:06:54,  1.86s/it]
Outer training loop:  80%|########  | 40083/50000 [26:25:19<5:15:31,  1.91s/it]
Outer training loop:  80%|########  | 40084/50000 [26:25:21<5:14:16,  1.90s/it]
Outer training loop:  80%|########  | 40085/50000 [26:25:23<5:06:58,  1.86s/it]
Outer training loop:  80%|########  | 40086/50000 [26:25:25<5:02:39,  1.83s/it]
Outer training loop:  80%|########  | 40087/50000 [26:25:26<5:06:16,  1.85s/it]
Outer training loop:  80%|########  | 40088/50000 [26:25:28<5:04:09,  1.84s/it]
Outer training loop:  80%|########  | 40089/50000 [26:25:30<4:55:07,  1.79s/it]
Outer training loop:  80%|########  | 40090/50000 [26:25:32<4:59:04,  1.81s/it]
Outer training loop:  80%|########  | 40091/50000 [26:25:34<5:03:21,  1.84s/it]
Outer training loop:  80%|########  | 40092/50000 [26:25:36<5:05:33,  1.85s/it]
Outer training loop:  80%|########  | 40093/50000 [26:25:37<5:11:14,  1.89s/it]
Outer training loop:  80%|########  | 40094/50000 [26:25:39<5:10:49,  1.88s/it]
Outer training loop:  80%|########  | 40095/50000 [26:25:41<5:03:44,  1.84s/it]
Outer training loop:  80%|########  | 40096/50000 [26:25:43<4:58:51,  1.81s/it]
Outer training loop:  80%|########  | 40097/50000 [26:25:45<5:04:33,  1.85s/it]
Outer training loop:  80%|########  | 40098/50000 [26:25:47<5:03:47,  1.84s/it]
Outer training loop:  80%|########  | 40099/50000 [26:25:48<5:05:17,  1.85s/it]
Outer training loop:  80%|########  | 40100/50000 [26:25:51<5:16:29,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  80%|########  | 40101/50000 [26:25:58<9:44:31,  3.54s/it]
Outer training loop:  80%|########  | 40102/50000 [26:26:00<8:14:29,  3.00s/it]
Outer training loop:  80%|########  | 40103/50000 [26:26:02<7:21:35,  2.68s/it]
Outer training loop:  80%|########  | 40104/50000 [26:26:04<6:51:56,  2.50s/it]
Outer training loop:  80%|########  | 40105/50000 [26:26:05<6:21:01,  2.31s/it]
Outer training loop:  80%|########  | 40106/50000 [26:26:07<5:55:13,  2.15s/it]
Outer training loop:  80%|########  | 40107/50000 [26:26:09<5:47:00,  2.10s/it]
Outer training loop:  80%|########  | 40108/50000 [26:26:11<5:31:15,  2.01s/it]
Outer training loop:  80%|########  | 40109/50000 [26:26:13<5:21:27,  1.95s/it]
Outer training loop:  80%|########  | 40110/50000 [26:26:15<5:14:30,  1.91s/it]
Outer training loop:  80%|########  | 40111/50000 [26:26:17<5:15:16,  1.91s/it]
Outer training loop:  80%|########  | 40112/50000 [26:26:18<5:12:09,  1.89s/it]
Outer training loop:  80%|########  | 40113/50000 [26:26:20<5:11:48,  1.89s/it]
Outer training loop:  80%|########  | 40114/50000 [26:26:22<5:10:35,  1.89s/it]
Outer training loop:  80%|########  | 40115/50000 [26:26:24<5:09:15,  1.88s/it]
Outer training loop:  80%|########  | 40116/50000 [26:26:26<5:07:35,  1.87s/it]
Outer training loop:  80%|########  | 40117/50000 [26:26:28<5:05:30,  1.85s/it]
Outer training loop:  80%|########  | 40118/50000 [26:26:30<5:14:04,  1.91s/it]
Outer training loop:  80%|########  | 40119/50000 [26:26:32<5:07:43,  1.87s/it]
Outer training loop:  80%|########  | 40120/50000 [26:26:33<5:04:54,  1.85s/it]
Outer training loop:  80%|########  | 40121/50000 [26:26:35<5:14:21,  1.91s/it]
Outer training loop:  80%|########  | 40122/50000 [26:26:37<5:04:25,  1.85s/it]
Outer training loop:  80%|########  | 40123/50000 [26:26:39<4:59:10,  1.82s/it]
Outer training loop:  80%|########  | 40124/50000 [26:26:41<5:07:50,  1.87s/it]
Outer training loop:  80%|########  | 40125/50000 [26:26:43<5:11:07,  1.89s/it]
Outer training loop:  80%|########  | 40126/50000 [26:26:45<5:13:19,  1.90s/it]
Outer training loop:  80%|########  | 40127/50000 [26:26:47<5:14:10,  1.91s/it]
Outer training loop:  80%|########  | 40128/50000 [26:26:49<5:19:59,  1.94s/it]
Outer training loop:  80%|########  | 40129/50000 [26:26:51<5:18:02,  1.93s/it]
Outer training loop:  80%|########  | 40130/50000 [26:26:52<5:09:45,  1.88s/it]
Outer training loop:  80%|########  | 40131/50000 [26:26:54<5:14:13,  1.91s/it]
Outer training loop:  80%|########  | 40132/50000 [26:26:56<5:14:34,  1.91s/it]
Outer training loop:  80%|########  | 40133/50000 [26:26:58<5:12:42,  1.90s/it]
Outer training loop:  80%|########  | 40134/50000 [26:27:00<5:04:00,  1.85s/it]
Outer training loop:  80%|########  | 40135/50000 [26:27:02<5:12:41,  1.90s/it]
Outer training loop:  80%|########  | 40136/50000 [26:27:04<5:10:41,  1.89s/it]
Outer training loop:  80%|########  | 40137/50000 [26:27:06<5:12:51,  1.90s/it]
Outer training loop:  80%|########  | 40138/50000 [26:27:08<5:26:49,  1.99s/it]
Outer training loop:  80%|########  | 40139/50000 [26:27:10<5:23:36,  1.97s/it]
Outer training loop:  80%|########  | 40140/50000 [26:27:12<5:17:31,  1.93s/it]
Outer training loop:  80%|########  | 40141/50000 [26:27:14<5:17:41,  1.93s/it]
Outer training loop:  80%|########  | 40142/50000 [26:27:16<5:26:28,  1.99s/it]
Outer training loop:  80%|########  | 40143/50000 [26:27:18<5:21:25,  1.96s/it]
Outer training loop:  80%|########  | 40144/50000 [26:27:19<5:11:16,  1.89s/it]
Outer training loop:  80%|########  | 40145/50000 [26:27:21<5:09:26,  1.88s/it]
Outer training loop:  80%|########  | 40146/50000 [26:27:23<5:22:52,  1.97s/it]
Outer training loop:  80%|########  | 40147/50000 [26:27:25<5:10:30,  1.89s/it]
Outer training loop:  80%|########  | 40148/50000 [26:27:27<5:07:14,  1.87s/it]
Outer training loop:  80%|########  | 40149/50000 [26:27:29<5:13:08,  1.91s/it]
Outer training loop:  80%|########  | 40150/50000 [26:27:31<5:05:51,  1.86s/it]
Outer training loop:  80%|########  | 40151/50000 [26:27:32<4:58:47,  1.82s/it]
Outer training loop:  80%|########  | 40152/50000 [26:27:34<5:03:12,  1.85s/it]
Outer training loop:  80%|########  | 40153/50000 [26:27:36<5:09:36,  1.89s/it]
Outer training loop:  80%|########  | 40154/50000 [26:27:38<5:02:44,  1.84s/it]
Outer training loop:  80%|########  | 40155/50000 [26:27:40<4:59:00,  1.82s/it]
Outer training loop:  80%|########  | 40156/50000 [26:27:42<5:12:52,  1.91s/it]
Outer training loop:  80%|########  | 40157/50000 [26:27:44<5:08:59,  1.88s/it]
Outer training loop:  80%|########  | 40158/50000 [26:27:45<5:04:10,  1.85s/it]
Outer training loop:  80%|########  | 40159/50000 [26:27:47<5:07:05,  1.87s/it]
Outer training loop:  80%|########  | 40160/50000 [26:27:49<5:02:48,  1.85s/it]
Outer training loop:  80%|########  | 40161/50000 [26:27:51<4:59:27,  1.83s/it]
Outer training loop:  80%|########  | 40162/50000 [26:27:53<5:00:04,  1.83s/it]
Outer training loop:  80%|########  | 40163/50000 [26:27:55<5:06:50,  1.87s/it]
Outer training loop:  80%|########  | 40164/50000 [26:27:56<4:59:18,  1.83s/it]
Outer training loop:  80%|########  | 40165/50000 [26:27:59<5:15:00,  1.92s/it]
Outer training loop:  80%|########  | 40166/50000 [26:28:01<5:18:51,  1.95s/it]
Outer training loop:  80%|########  | 40167/50000 [26:28:03<5:16:19,  1.93s/it]
Outer training loop:  80%|########  | 40168/50000 [26:28:04<5:14:33,  1.92s/it]
Outer training loop:  80%|########  | 40169/50000 [26:28:07<5:24:57,  1.98s/it]
Outer training loop:  80%|########  | 40170/50000 [26:28:08<5:11:43,  1.90s/it]
Outer training loop:  80%|########  | 40171/50000 [26:28:10<5:06:38,  1.87s/it]
Outer training loop:  80%|########  | 40172/50000 [26:28:12<5:07:21,  1.88s/it]
Outer training loop:  80%|########  | 40173/50000 [26:28:14<5:13:00,  1.91s/it]
Outer training loop:  80%|########  | 40174/50000 [26:28:16<5:05:56,  1.87s/it]
Outer training loop:  80%|########  | 40175/50000 [26:28:17<5:01:34,  1.84s/it]
Outer training loop:  80%|########  | 40176/50000 [26:28:19<5:04:23,  1.86s/it]
Outer training loop:  80%|########  | 40177/50000 [26:28:22<5:18:01,  1.94s/it]
Outer training loop:  80%|########  | 40178/50000 [26:28:23<5:11:02,  1.90s/it]
Outer training loop:  80%|########  | 40179/50000 [26:28:25<5:05:11,  1.86s/it]
Outer training loop:  80%|########  | 40180/50000 [26:28:27<5:11:18,  1.90s/it]
Outer training loop:  80%|########  | 40181/50000 [26:28:29<5:04:33,  1.86s/it]
Outer training loop:  80%|########  | 40182/50000 [26:28:31<5:02:00,  1.85s/it]
Outer training loop:  80%|########  | 40183/50000 [26:28:33<5:11:03,  1.90s/it]
Outer training loop:  80%|########  | 40184/50000 [26:28:35<5:14:30,  1.92s/it]
Outer training loop:  80%|########  | 40185/50000 [26:28:37<5:11:36,  1.90s/it]
Outer training loop:  80%|########  | 40186/50000 [26:28:39<5:20:03,  1.96s/it]
Outer training loop:  80%|########  | 40187/50000 [26:28:40<5:16:12,  1.93s/it]
Outer training loop:  80%|########  | 40188/50000 [26:28:42<5:11:08,  1.90s/it]
Outer training loop:  80%|########  | 40189/50000 [26:28:44<5:19:19,  1.95s/it]
Outer training loop:  80%|########  | 40190/50000 [26:28:46<5:07:27,  1.88s/it]
Outer training loop:  80%|########  | 40191/50000 [26:28:48<5:02:50,  1.85s/it]
Outer training loop:  80%|########  | 40192/50000 [26:28:50<5:07:17,  1.88s/it]
Outer training loop:  80%|########  | 40193/50000 [26:28:52<5:09:57,  1.90s/it]
Outer training loop:  80%|########  | 40194/50000 [26:28:54<5:04:50,  1.87s/it]
Outer training loop:  80%|########  | 40195/50000 [26:28:55<5:04:48,  1.87s/it]
Outer training loop:  80%|########  | 40196/50000 [26:28:57<5:00:48,  1.84s/it]
Outer training loop:  80%|########  | 40197/50000 [26:28:59<5:11:55,  1.91s/it]
Outer training loop:  80%|########  | 40198/50000 [26:29:01<5:06:24,  1.88s/it]
Outer training loop:  80%|########  | 40199/50000 [26:29:03<5:03:40,  1.86s/it]
Outer training loop:  80%|########  | 40200/50000 [26:29:05<5:17:25,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  80%|########  | 40201/50000 [26:29:12<9:40:12,  3.55s/it]
Outer training loop:  80%|########  | 40202/50000 [26:29:14<8:13:38,  3.02s/it]
Outer training loop:  80%|########  | 40203/50000 [26:29:16<7:24:47,  2.72s/it]
Outer training loop:  80%|########  | 40204/50000 [26:29:18<6:35:14,  2.42s/it]
Outer training loop:  80%|########  | 40205/50000 [26:29:20<6:08:56,  2.26s/it]
Outer training loop:  80%|########  | 40206/50000 [26:29:22<5:48:26,  2.13s/it]
Outer training loop:  80%|########  | 40207/50000 [26:29:24<5:41:37,  2.09s/it]
Outer training loop:  80%|########  | 40208/50000 [26:29:26<5:33:02,  2.04s/it]
Outer training loop:  80%|########  | 40209/50000 [26:29:27<5:24:43,  1.99s/it]
Outer training loop:  80%|########  | 40210/50000 [26:29:29<5:25:03,  1.99s/it]
Outer training loop:  80%|########  | 40211/50000 [26:29:31<5:18:43,  1.95s/it]
Outer training loop:  80%|########  | 40212/50000 [26:29:33<5:14:51,  1.93s/it]
Outer training loop:  80%|########  | 40213/50000 [26:29:35<5:13:21,  1.92s/it]
Outer training loop:  80%|########  | 40214/50000 [26:29:37<5:16:28,  1.94s/it]
Outer training loop:  80%|########  | 40215/50000 [26:29:39<5:05:48,  1.88s/it]
Outer training loop:  80%|########  | 40216/50000 [26:29:41<5:04:46,  1.87s/it]
Outer training loop:  80%|########  | 40217/50000 [26:29:43<5:13:47,  1.92s/it]
Outer training loop:  80%|########  | 40218/50000 [26:29:44<5:08:29,  1.89s/it]
Outer training loop:  80%|########  | 40219/50000 [26:29:46<5:00:56,  1.85s/it]
Outer training loop:  80%|########  | 40220/50000 [26:29:48<5:06:40,  1.88s/it]
Outer training loop:  80%|########  | 40221/50000 [26:29:50<5:07:21,  1.89s/it]
Outer training loop:  80%|########  | 40222/50000 [26:29:52<5:03:12,  1.86s/it]
Outer training loop:  80%|########  | 40223/50000 [26:29:54<5:04:07,  1.87s/it]
Outer training loop:  80%|########  | 40224/50000 [26:29:56<5:09:40,  1.90s/it]
Outer training loop:  80%|########  | 40225/50000 [26:29:58<5:08:33,  1.89s/it]
Outer training loop:  80%|########  | 40226/50000 [26:29:59<4:59:56,  1.84s/it]
Outer training loop:  80%|########  | 40227/50000 [26:30:01<5:08:42,  1.90s/it]
Outer training loop:  80%|########  | 40228/50000 [26:30:03<5:06:28,  1.88s/it]
Outer training loop:  80%|########  | 40229/50000 [26:30:05<5:07:37,  1.89s/it]
Outer training loop:  80%|########  | 40230/50000 [26:30:07<5:01:34,  1.85s/it]
Outer training loop:  80%|########  | 40231/50000 [26:30:09<5:07:09,  1.89s/it]
Outer training loop:  80%|########  | 40232/50000 [26:30:11<5:05:24,  1.88s/it]
Outer training loop:  80%|########  | 40233/50000 [26:30:12<5:02:10,  1.86s/it]
Outer training loop:  80%|########  | 40234/50000 [26:30:14<5:07:02,  1.89s/it]
Outer training loop:  80%|########  | 40235/50000 [26:30:16<4:59:05,  1.84s/it]
Outer training loop:  80%|########  | 40236/50000 [26:30:18<4:56:48,  1.82s/it]
Outer training loop:  80%|########  | 40237/50000 [26:30:20<5:09:26,  1.90s/it]
Outer training loop:  80%|########  | 40238/50000 [26:30:22<5:08:28,  1.90s/it]
Outer training loop:  80%|########  | 40239/50000 [26:30:24<5:05:44,  1.88s/it]
Outer training loop:  80%|########  | 40240/50000 [26:30:26<5:06:28,  1.88s/it]
Outer training loop:  80%|########  | 40241/50000 [26:30:28<5:18:01,  1.96s/it]
Outer training loop:  80%|########  | 40242/50000 [26:30:30<5:13:03,  1.92s/it]
Outer training loop:  80%|########  | 40243/50000 [26:30:31<5:08:44,  1.90s/it]
Outer training loop:  80%|########  | 40244/50000 [26:30:33<5:12:05,  1.92s/it]
Outer training loop:  80%|########  | 40245/50000 [26:30:35<5:10:53,  1.91s/it]
Outer training loop:  80%|########  | 40246/50000 [26:30:37<5:06:21,  1.88s/it]
Outer training loop:  80%|########  | 40247/50000 [26:30:39<5:03:11,  1.87s/it]
Outer training loop:  80%|########  | 40248/50000 [26:30:41<5:14:12,  1.93s/it]
Outer training loop:  80%|########  | 40249/50000 [26:30:43<5:10:43,  1.91s/it]
Outer training loop:  80%|########  | 40250/50000 [26:30:45<5:03:02,  1.86s/it]
Outer training loop:  81%|########  | 40251/50000 [26:30:47<5:10:51,  1.91s/it]
Outer training loop:  81%|########  | 40252/50000 [26:30:49<5:10:35,  1.91s/it]
Outer training loop:  81%|########  | 40253/50000 [26:30:50<5:01:46,  1.86s/it]
Outer training loop:  81%|########  | 40254/50000 [26:30:52<5:08:34,  1.90s/it]
Outer training loop:  81%|########  | 40255/50000 [26:30:54<5:03:45,  1.87s/it]
Outer training loop:  81%|########  | 40256/50000 [26:30:56<5:06:15,  1.89s/it]
Outer training loop:  81%|########  | 40257/50000 [26:30:58<5:01:24,  1.86s/it]
Outer training loop:  81%|########  | 40258/50000 [26:31:00<5:07:56,  1.90s/it]
Outer training loop:  81%|########  | 40259/50000 [26:31:02<5:04:40,  1.88s/it]
Outer training loop:  81%|########  | 40260/50000 [26:31:03<4:58:05,  1.84s/it]
Outer training loop:  81%|########  | 40261/50000 [26:31:05<5:05:34,  1.88s/it]
Outer training loop:  81%|########  | 40262/50000 [26:31:07<5:13:34,  1.93s/it]
Outer training loop:  81%|########  | 40263/50000 [26:31:09<5:07:10,  1.89s/it]
Outer training loop:  81%|########  | 40264/50000 [26:31:11<5:03:27,  1.87s/it]
Outer training loop:  81%|########  | 40265/50000 [26:31:13<5:16:35,  1.95s/it]
Outer training loop:  81%|########  | 40266/50000 [26:31:15<5:06:59,  1.89s/it]
Outer training loop:  81%|########  | 40267/50000 [26:31:17<5:03:30,  1.87s/it]
Outer training loop:  81%|########  | 40268/50000 [26:31:19<5:15:41,  1.95s/it]
Outer training loop:  81%|########  | 40269/50000 [26:31:21<5:09:02,  1.91s/it]
Outer training loop:  81%|########  | 40270/50000 [26:31:23<5:05:53,  1.89s/it]
Outer training loop:  81%|########  | 40271/50000 [26:31:24<5:02:02,  1.86s/it]
Outer training loop:  81%|########  | 40272/50000 [26:31:26<5:08:16,  1.90s/it]
Outer training loop:  81%|########  | 40273/50000 [26:31:28<5:02:40,  1.87s/it]
Outer training loop:  81%|########  | 40274/50000 [26:31:30<4:58:33,  1.84s/it]
Outer training loop:  81%|########  | 40275/50000 [26:31:32<5:02:14,  1.86s/it]
Outer training loop:  81%|########  | 40276/50000 [26:31:34<5:03:13,  1.87s/it]
Outer training loop:  81%|########  | 40277/50000 [26:31:36<5:03:23,  1.87s/it]
Outer training loop:  81%|########  | 40278/50000 [26:31:37<5:01:53,  1.86s/it]
Outer training loop:  81%|########  | 40279/50000 [26:31:39<5:08:43,  1.91s/it]
Outer training loop:  81%|########  | 40280/50000 [26:31:41<5:14:04,  1.94s/it]
Outer training loop:  81%|########  | 40281/50000 [26:31:43<5:08:49,  1.91s/it]
Outer training loop:  81%|########  | 40282/50000 [26:31:45<5:08:16,  1.90s/it]
Outer training loop:  81%|########  | 40283/50000 [26:31:47<5:03:30,  1.87s/it]
Outer training loop:  81%|########  | 40284/50000 [26:31:49<5:00:36,  1.86s/it]
Outer training loop:  81%|########  | 40285/50000 [26:31:51<5:09:27,  1.91s/it]
Outer training loop:  81%|########  | 40286/50000 [26:31:53<5:02:47,  1.87s/it]
Outer training loop:  81%|########  | 40287/50000 [26:31:55<5:05:02,  1.88s/it]
Outer training loop:  81%|########  | 40288/50000 [26:31:56<5:04:02,  1.88s/it]
Outer training loop:  81%|########  | 40289/50000 [26:31:58<5:12:54,  1.93s/it]
Outer training loop:  81%|########  | 40290/50000 [26:32:00<5:05:21,  1.89s/it]
Outer training loop:  81%|########  | 40291/50000 [26:32:02<5:07:03,  1.90s/it]
Outer training loop:  81%|########  | 40292/50000 [26:32:04<5:11:13,  1.92s/it]
Outer training loop:  81%|########  | 40293/50000 [26:32:06<5:11:11,  1.92s/it]
Outer training loop:  81%|########  | 40294/50000 [26:32:08<5:01:48,  1.87s/it]
Outer training loop:  81%|########  | 40295/50000 [26:32:10<4:55:50,  1.83s/it]
Outer training loop:  81%|########  | 40296/50000 [26:32:12<5:04:50,  1.88s/it]
Outer training loop:  81%|########  | 40297/50000 [26:32:13<5:01:58,  1.87s/it]
Outer training loop:  81%|########  | 40298/50000 [26:32:15<4:59:09,  1.85s/it]
Outer training loop:  81%|########  | 40299/50000 [26:32:17<5:06:55,  1.90s/it]
Outer training loop:  81%|########  | 40300/50000 [26:32:19<5:08:40,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  81%|########  | 40301/50000 [26:32:27<9:33:05,  3.55s/it]
Outer training loop:  81%|########  | 40302/50000 [26:32:28<8:12:14,  3.05s/it]
Outer training loop:  81%|########  | 40303/50000 [26:32:30<7:16:36,  2.70s/it]
Outer training loop:  81%|########  | 40304/50000 [26:32:32<6:27:38,  2.40s/it]
Outer training loop:  81%|########  | 40305/50000 [26:32:34<5:58:14,  2.22s/it]
Outer training loop:  81%|########  | 40306/50000 [26:32:36<5:47:53,  2.15s/it]
Outer training loop:  81%|########  | 40307/50000 [26:32:38<5:35:07,  2.07s/it]
Outer training loop:  81%|########  | 40308/50000 [26:32:39<5:22:05,  1.99s/it]
Outer training loop:  81%|########  | 40309/50000 [26:32:41<5:21:17,  1.99s/it]
Outer training loop:  81%|########  | 40310/50000 [26:32:44<5:28:33,  2.03s/it]
Outer training loop:  81%|########  | 40311/50000 [26:32:45<5:15:42,  1.96s/it]
Outer training loop:  81%|########  | 40312/50000 [26:32:47<5:04:27,  1.89s/it]
Outer training loop:  81%|########  | 40313/50000 [26:32:49<5:13:54,  1.94s/it]
Outer training loop:  81%|########  | 40314/50000 [26:32:51<5:12:59,  1.94s/it]
Outer training loop:  81%|########  | 40315/50000 [26:32:53<5:06:04,  1.90s/it]
Outer training loop:  81%|########  | 40316/50000 [26:32:55<5:02:30,  1.87s/it]
Outer training loop:  81%|########  | 40317/50000 [26:32:57<5:07:40,  1.91s/it]
Outer training loop:  81%|########  | 40318/50000 [26:32:59<5:05:17,  1.89s/it]
Outer training loop:  81%|########  | 40319/50000 [26:33:00<4:57:07,  1.84s/it]
Outer training loop:  81%|########  | 40320/50000 [26:33:02<5:09:01,  1.92s/it]
Outer training loop:  81%|########  | 40321/50000 [26:33:04<5:10:31,  1.92s/it]
Outer training loop:  81%|########  | 40322/50000 [26:33:06<5:09:20,  1.92s/it]
Outer training loop:  81%|########  | 40323/50000 [26:33:08<5:11:36,  1.93s/it]
Outer training loop:  81%|########  | 40324/50000 [26:33:10<5:03:50,  1.88s/it]
Outer training loop:  81%|########  | 40325/50000 [26:33:12<5:00:19,  1.86s/it]
Outer training loop:  81%|########  | 40326/50000 [26:33:14<5:13:21,  1.94s/it]
Outer training loop:  81%|########  | 40327/50000 [26:33:16<5:05:47,  1.90s/it]
Outer training loop:  81%|########  | 40328/50000 [26:33:17<5:01:13,  1.87s/it]
Outer training loop:  81%|########  | 40329/50000 [26:33:19<5:05:52,  1.90s/it]
Outer training loop:  81%|########  | 40330/50000 [26:33:21<5:06:48,  1.90s/it]
Outer training loop:  81%|########  | 40331/50000 [26:33:23<4:58:48,  1.85s/it]
Outer training loop:  81%|########  | 40332/50000 [26:33:25<5:00:35,  1.87s/it]
Outer training loop:  81%|########  | 40333/50000 [26:33:27<5:06:53,  1.90s/it]
Outer training loop:  81%|########  | 40334/50000 [26:33:29<5:06:01,  1.90s/it]
Outer training loop:  81%|########  | 40335/50000 [26:33:31<5:02:19,  1.88s/it]
Outer training loop:  81%|########  | 40336/50000 [26:33:33<5:10:17,  1.93s/it]
Outer training loop:  81%|########  | 40337/50000 [26:33:35<5:12:13,  1.94s/it]
Outer training loop:  81%|########  | 40338/50000 [26:33:37<5:08:06,  1.91s/it]
Outer training loop:  81%|########  | 40339/50000 [26:33:39<5:13:46,  1.95s/it]
Outer training loop:  81%|########  | 40340/50000 [26:33:40<5:05:37,  1.90s/it]
Outer training loop:  81%|########  | 40341/50000 [26:33:42<5:02:42,  1.88s/it]
Outer training loop:  81%|########  | 40342/50000 [26:33:44<5:13:33,  1.95s/it]
Outer training loop:  81%|########  | 40343/50000 [26:33:46<5:02:37,  1.88s/it]
Outer training loop:  81%|########  | 40344/50000 [26:33:48<5:01:16,  1.87s/it]
Outer training loop:  81%|########  | 40345/50000 [26:33:50<5:02:02,  1.88s/it]
Outer training loop:  81%|########  | 40346/50000 [26:33:52<5:17:57,  1.98s/it]
Outer training loop:  81%|########  | 40347/50000 [26:33:54<5:12:34,  1.94s/it]
Outer training loop:  81%|########  | 40348/50000 [26:33:56<5:03:14,  1.89s/it]
Outer training loop:  81%|########  | 40349/50000 [26:33:58<5:11:35,  1.94s/it]
Outer training loop:  81%|########  | 40350/50000 [26:33:59<5:02:03,  1.88s/it]
Outer training loop:  81%|########  | 40351/50000 [26:34:01<4:58:00,  1.85s/it]
Outer training loop:  81%|########  | 40352/50000 [26:34:03<4:55:03,  1.83s/it]
Outer training loop:  81%|########  | 40353/50000 [26:34:05<5:01:28,  1.88s/it]
Outer training loop:  81%|########  | 40354/50000 [26:34:07<5:01:26,  1.87s/it]
Outer training loop:  81%|########  | 40355/50000 [26:34:09<4:55:35,  1.84s/it]
Outer training loop:  81%|########  | 40356/50000 [26:34:11<5:03:15,  1.89s/it]
Outer training loop:  81%|########  | 40357/50000 [26:34:12<4:57:01,  1.85s/it]
Outer training loop:  81%|########  | 40358/50000 [26:34:14<4:57:38,  1.85s/it]
Outer training loop:  81%|########  | 40359/50000 [26:34:16<5:06:42,  1.91s/it]
Outer training loop:  81%|########  | 40360/50000 [26:34:18<5:08:58,  1.92s/it]
Outer training loop:  81%|########  | 40361/50000 [26:34:20<5:09:43,  1.93s/it]
Outer training loop:  81%|########  | 40362/50000 [26:34:22<5:05:30,  1.90s/it]
Outer training loop:  81%|########  | 40363/50000 [26:34:24<5:10:19,  1.93s/it]
Outer training loop:  81%|########  | 40364/50000 [26:34:26<5:11:13,  1.94s/it]
Outer training loop:  81%|########  | 40365/50000 [26:34:28<5:03:12,  1.89s/it]
Outer training loop:  81%|########  | 40366/50000 [26:34:30<5:18:47,  1.99s/it]
Outer training loop:  81%|########  | 40367/50000 [26:34:32<5:07:32,  1.92s/it]
Outer training loop:  81%|########  | 40368/50000 [26:34:33<4:59:21,  1.86s/it]
Outer training loop:  81%|########  | 40369/50000 [26:34:35<4:55:35,  1.84s/it]
Outer training loop:  81%|########  | 40370/50000 [26:34:37<4:57:52,  1.86s/it]
Outer training loop:  81%|########  | 40371/50000 [26:34:39<4:56:34,  1.85s/it]
Outer training loop:  81%|########  | 40372/50000 [26:34:41<4:59:07,  1.86s/it]
Outer training loop:  81%|########  | 40373/50000 [26:34:43<4:54:55,  1.84s/it]
Outer training loop:  81%|########  | 40374/50000 [26:34:45<4:58:34,  1.86s/it]
Outer training loop:  81%|########  | 40375/50000 [26:34:46<4:55:53,  1.84s/it]
Outer training loop:  81%|########  | 40376/50000 [26:34:48<4:51:57,  1.82s/it]
Outer training loop:  81%|########  | 40377/50000 [26:34:50<4:59:22,  1.87s/it]
Outer training loop:  81%|########  | 40378/50000 [26:34:52<4:52:12,  1.82s/it]
Outer training loop:  81%|########  | 40379/50000 [26:34:54<4:46:29,  1.79s/it]
Outer training loop:  81%|########  | 40380/50000 [26:34:55<4:46:10,  1.78s/it]
Outer training loop:  81%|########  | 40381/50000 [26:34:57<4:57:57,  1.86s/it]
Outer training loop:  81%|########  | 40382/50000 [26:34:59<4:58:27,  1.86s/it]
Outer training loop:  81%|########  | 40383/50000 [26:35:01<4:52:38,  1.83s/it]
Outer training loop:  81%|########  | 40384/50000 [26:35:03<4:54:02,  1.83s/it]
Outer training loop:  81%|########  | 40385/50000 [26:35:05<4:59:52,  1.87s/it]
Outer training loop:  81%|########  | 40386/50000 [26:35:07<5:01:26,  1.88s/it]
Outer training loop:  81%|########  | 40387/50000 [26:35:09<5:11:08,  1.94s/it]
Outer training loop:  81%|########  | 40388/50000 [26:35:11<5:04:09,  1.90s/it]
Outer training loop:  81%|########  | 40389/50000 [26:35:12<4:58:57,  1.87s/it]
Outer training loop:  81%|########  | 40390/50000 [26:35:14<4:57:28,  1.86s/it]
Outer training loop:  81%|########  | 40391/50000 [26:35:16<5:02:59,  1.89s/it]
Outer training loop:  81%|########  | 40392/50000 [26:35:18<5:00:31,  1.88s/it]
Outer training loop:  81%|########  | 40393/50000 [26:35:20<5:05:31,  1.91s/it]
Outer training loop:  81%|########  | 40394/50000 [26:35:22<5:08:41,  1.93s/it]
Outer training loop:  81%|########  | 40395/50000 [26:35:24<5:03:42,  1.90s/it]
Outer training loop:  81%|########  | 40396/50000 [26:35:25<4:54:44,  1.84s/it]
Outer training loop:  81%|########  | 40397/50000 [26:35:27<5:01:11,  1.88s/it]
Outer training loop:  81%|########  | 40398/50000 [26:35:29<5:03:06,  1.89s/it]
Outer training loop:  81%|########  | 40399/50000 [26:35:31<5:01:10,  1.88s/it]
Outer training loop:  81%|########  | 40400/50000 [26:35:33<5:05:31,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  81%|########  | 40401/50000 [26:35:41<9:27:07,  3.54s/it]
Outer training loop:  81%|########  | 40402/50000 [26:35:42<8:08:02,  3.05s/it]
Outer training loop:  81%|########  | 40403/50000 [26:35:44<7:05:42,  2.66s/it]
Outer training loop:  81%|########  | 40404/50000 [26:35:46<6:39:06,  2.50s/it]
Outer training loop:  81%|########  | 40405/50000 [26:35:48<6:02:06,  2.26s/it]
Outer training loop:  81%|########  | 40406/50000 [26:35:50<5:54:55,  2.22s/it]
Outer training loop:  81%|########  | 40407/50000 [26:35:52<5:33:12,  2.08s/it]
Outer training loop:  81%|########  | 40408/50000 [26:35:54<5:30:15,  2.07s/it]
Outer training loop:  81%|########  | 40409/50000 [26:35:56<5:23:01,  2.02s/it]
Outer training loop:  81%|########  | 40410/50000 [26:35:58<5:17:10,  1.98s/it]
Outer training loop:  81%|########  | 40411/50000 [26:36:00<5:14:49,  1.97s/it]
Outer training loop:  81%|########  | 40412/50000 [26:36:01<5:04:42,  1.91s/it]
Outer training loop:  81%|########  | 40413/50000 [26:36:03<4:59:31,  1.87s/it]
Outer training loop:  81%|########  | 40414/50000 [26:36:05<5:13:16,  1.96s/it]
Outer training loop:  81%|########  | 40415/50000 [26:36:07<5:06:06,  1.92s/it]
Outer training loop:  81%|########  | 40416/50000 [26:36:09<4:59:23,  1.87s/it]
Outer training loop:  81%|########  | 40417/50000 [26:36:11<4:56:28,  1.86s/it]
Outer training loop:  81%|########  | 40418/50000 [26:36:13<5:05:36,  1.91s/it]
Outer training loop:  81%|########  | 40419/50000 [26:36:15<5:01:39,  1.89s/it]
Outer training loop:  81%|########  | 40420/50000 [26:36:17<4:58:33,  1.87s/it]
Outer training loop:  81%|########  | 40421/50000 [26:36:19<5:05:01,  1.91s/it]
Outer training loop:  81%|########  | 40422/50000 [26:36:20<4:57:47,  1.87s/it]
Outer training loop:  81%|########  | 40423/50000 [26:36:22<4:54:07,  1.84s/it]
Outer training loop:  81%|########  | 40424/50000 [26:36:24<5:01:13,  1.89s/it]
Outer training loop:  81%|########  | 40425/50000 [26:36:26<4:57:43,  1.87s/it]
Outer training loop:  81%|########  | 40426/50000 [26:36:28<4:58:46,  1.87s/it]
Outer training loop:  81%|########  | 40427/50000 [26:36:30<4:58:31,  1.87s/it]
Outer training loop:  81%|########  | 40428/50000 [26:36:31<4:52:29,  1.83s/it]
Outer training loop:  81%|########  | 40429/50000 [26:36:33<4:51:33,  1.83s/it]
Outer training loop:  81%|########  | 40430/50000 [26:36:35<4:59:35,  1.88s/it]
Outer training loop:  81%|########  | 40431/50000 [26:36:37<5:09:52,  1.94s/it]
Outer training loop:  81%|########  | 40432/50000 [26:36:39<5:04:40,  1.91s/it]
Outer training loop:  81%|########  | 40433/50000 [26:36:41<5:01:21,  1.89s/it]
Outer training loop:  81%|########  | 40434/50000 [26:36:43<5:07:34,  1.93s/it]
Outer training loop:  81%|########  | 40435/50000 [26:36:45<5:00:26,  1.88s/it]
Outer training loop:  81%|########  | 40436/50000 [26:36:46<4:52:25,  1.83s/it]
Outer training loop:  81%|########  | 40437/50000 [26:36:48<5:00:23,  1.88s/it]
Outer training loop:  81%|########  | 40438/50000 [26:36:50<4:59:11,  1.88s/it]
Outer training loop:  81%|########  | 40439/50000 [26:36:52<4:58:38,  1.87s/it]
Outer training loop:  81%|########  | 40440/50000 [26:36:54<4:55:00,  1.85s/it]
Outer training loop:  81%|########  | 40441/50000 [26:36:56<5:06:09,  1.92s/it]
Outer training loop:  81%|########  | 40442/50000 [26:36:58<5:07:25,  1.93s/it]
Outer training loop:  81%|########  | 40443/50000 [26:37:00<5:09:07,  1.94s/it]
Outer training loop:  81%|########  | 40444/50000 [26:37:02<5:10:03,  1.95s/it]
Outer training loop:  81%|########  | 40445/50000 [26:37:04<5:07:40,  1.93s/it]
Outer training loop:  81%|########  | 40446/50000 [26:37:06<5:08:27,  1.94s/it]
Outer training loop:  81%|########  | 40447/50000 [26:37:08<5:12:44,  1.96s/it]
Outer training loop:  81%|########  | 40448/50000 [26:37:10<5:04:01,  1.91s/it]
Outer training loop:  81%|########  | 40449/50000 [26:37:11<4:55:22,  1.86s/it]
Outer training loop:  81%|########  | 40450/50000 [26:37:13<4:58:42,  1.88s/it]
Outer training loop:  81%|########  | 40451/50000 [26:37:15<5:02:50,  1.90s/it]
Outer training loop:  81%|########  | 40452/50000 [26:37:17<4:56:45,  1.86s/it]
Outer training loop:  81%|########  | 40453/50000 [26:37:19<4:54:40,  1.85s/it]
Outer training loop:  81%|########  | 40454/50000 [26:37:21<5:04:06,  1.91s/it]
Outer training loop:  81%|########  | 40455/50000 [26:37:23<4:56:23,  1.86s/it]
Outer training loop:  81%|########  | 40456/50000 [26:37:25<5:04:04,  1.91s/it]
Outer training loop:  81%|########  | 40457/50000 [26:37:27<5:06:54,  1.93s/it]
Outer training loop:  81%|########  | 40458/50000 [26:37:28<4:57:51,  1.87s/it]
Outer training loop:  81%|########  | 40459/50000 [26:37:30<4:51:41,  1.83s/it]
Outer training loop:  81%|########  | 40460/50000 [26:37:32<4:47:53,  1.81s/it]
Outer training loop:  81%|########  | 40461/50000 [26:37:34<4:57:40,  1.87s/it]
Outer training loop:  81%|########  | 40462/50000 [26:37:36<4:57:42,  1.87s/it]
Outer training loop:  81%|########  | 40463/50000 [26:37:38<4:57:50,  1.87s/it]
Outer training loop:  81%|########  | 40464/50000 [26:37:40<5:01:53,  1.90s/it]
Outer training loop:  81%|########  | 40465/50000 [26:37:42<5:06:47,  1.93s/it]
Outer training loop:  81%|########  | 40466/50000 [26:37:43<5:04:43,  1.92s/it]
Outer training loop:  81%|########  | 40467/50000 [26:37:45<5:02:12,  1.90s/it]
Outer training loop:  81%|########  | 40468/50000 [26:37:47<5:08:29,  1.94s/it]
Outer training loop:  81%|########  | 40469/50000 [26:37:49<5:01:21,  1.90s/it]
Outer training loop:  81%|########  | 40470/50000 [26:37:51<4:57:15,  1.87s/it]
Outer training loop:  81%|########  | 40471/50000 [26:37:53<4:53:10,  1.85s/it]
Outer training loop:  81%|########  | 40472/50000 [26:37:55<5:01:31,  1.90s/it]
Outer training loop:  81%|########  | 40473/50000 [26:37:57<4:54:36,  1.86s/it]
Outer training loop:  81%|########  | 40474/50000 [26:37:58<4:51:33,  1.84s/it]
Outer training loop:  81%|########  | 40475/50000 [26:38:00<4:57:34,  1.87s/it]
Outer training loop:  81%|########  | 40476/50000 [26:38:02<4:52:54,  1.85s/it]
Outer training loop:  81%|########  | 40477/50000 [26:38:04<4:56:53,  1.87s/it]
Outer training loop:  81%|########  | 40478/50000 [26:38:06<5:06:44,  1.93s/it]
Outer training loop:  81%|########  | 40479/50000 [26:38:08<4:59:15,  1.89s/it]
Outer training loop:  81%|########  | 40480/50000 [26:38:10<4:58:24,  1.88s/it]
Outer training loop:  81%|########  | 40481/50000 [26:38:12<4:58:25,  1.88s/it]
Outer training loop:  81%|########  | 40482/50000 [26:38:14<5:06:32,  1.93s/it]
Outer training loop:  81%|########  | 40483/50000 [26:38:15<5:00:14,  1.89s/it]
Outer training loop:  81%|########  | 40484/50000 [26:38:17<4:57:04,  1.87s/it]
Outer training loop:  81%|########  | 40485/50000 [26:38:19<4:53:43,  1.85s/it]
Outer training loop:  81%|########  | 40486/50000 [26:38:21<5:12:10,  1.97s/it]
Outer training loop:  81%|########  | 40487/50000 [26:38:23<5:07:21,  1.94s/it]
Outer training loop:  81%|########  | 40488/50000 [26:38:25<5:06:40,  1.93s/it]
Outer training loop:  81%|########  | 40489/50000 [26:38:27<5:07:22,  1.94s/it]
Outer training loop:  81%|########  | 40490/50000 [26:38:29<5:05:56,  1.93s/it]
Outer training loop:  81%|########  | 40491/50000 [26:38:31<4:59:16,  1.89s/it]
Outer training loop:  81%|########  | 40492/50000 [26:38:33<5:03:27,  1.91s/it]
Outer training loop:  81%|########  | 40493/50000 [26:38:35<5:06:33,  1.93s/it]
Outer training loop:  81%|########  | 40494/50000 [26:38:37<4:58:18,  1.88s/it]
Outer training loop:  81%|########  | 40495/50000 [26:38:39<5:08:05,  1.94s/it]
Outer training loop:  81%|########  | 40496/50000 [26:38:40<5:02:31,  1.91s/it]
Outer training loop:  81%|########  | 40497/50000 [26:38:42<5:00:26,  1.90s/it]
Outer training loop:  81%|########  | 40498/50000 [26:38:44<5:01:29,  1.90s/it]
Outer training loop:  81%|########  | 40499/50000 [26:38:46<5:04:18,  1.92s/it]
Outer training loop:  81%|########1 | 40500/50000 [26:38:48<5:02:43,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.01s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.01s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.90s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:31,  1.87s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:29,  1.87s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.86s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.94s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.89s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:29,  2.02s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:21,  1.94s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:18,  1.98s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:13,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:09,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:03,  1.85s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:08,  1.95s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.90s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.92s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<02:00,  1.97s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:56,  1.94s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:50,  1.91s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:46,  1.87s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.89s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:49,  2.00s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.92s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:41,  1.91s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:41,  1.95s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:37,  1.92s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.90s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:35,  1.96s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:30,  1.93s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:26,  1.88s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:22,  1.93s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:19,  1.90s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:17,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:13,  1.84s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:11,  1.82s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:05,  1.78s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:08,  1.91s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.88s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:02,  1.85s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:57,  1.81s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:59,  1.91s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.87s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:54,  1.94s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.91s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.91s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.86s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.98s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.93s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.88s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.84s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.81s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:24,  1.79s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:23,  1.80s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:21,  1.78s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.78s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.92s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.90s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.90s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.87s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.85s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.83s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.89s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.88s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.85s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:02,  2.03s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.91s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.75s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:20,  1.75s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:20,  1.78s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.76s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:17,  1.79s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:16,  1.80s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:13,  1.78s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:25,  1.97s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.87s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:25<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:27<02:06,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:29<02:04,  1.85s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:06,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:03,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:01,  1.90s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:56,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.92s/it]

Evaluating policy:  29%|##8       | 24/83 [00:44<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:46,  1.87s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:48,  1.94s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.86s/it]

Evaluating policy:  37%|###7      | 31/83 [00:57<01:35,  1.84s/it]

Evaluating policy:  39%|###8      | 32/83 [00:59<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:37,  1.95s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:34,  1.92s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:31,  1.90s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:31,  1.95s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:28,  1.92s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:25,  1.95s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:22,  1.92s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:16,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:14,  1.91s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:17,  2.05s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:12,  1.96s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:10,  1.95s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:06,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:04,  1.91s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.85s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<01:00,  1.96s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:57,  1.92s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.85s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.85s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:43,  1.83s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:45,  1.99s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:42,  1.92s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.91s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.88s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.85s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.86s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:29,  2.00s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:28,  2.03s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.96s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.93s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.87s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.86s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:14,  2.02s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.97s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.86s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.83s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.88s/it]

Outer training loop:  81%|########1 | 40501/50000 [26:44:10<258:21:08, 97.91s/it]
Outer training loop:  81%|########1 | 40502/50000 [26:44:12<182:17:27, 69.09s/it]
Outer training loop:  81%|########1 | 40503/50000 [26:44:14<128:59:34, 48.90s/it]
Outer training loop:  81%|########1 | 40504/50000 [26:44:16<92:10:10, 34.94s/it] 
Outer training loop:  81%|########1 | 40505/50000 [26:44:18<65:57:58, 25.01s/it]
Outer training loop:  81%|########1 | 40506/50000 [26:44:20<47:38:10, 18.06s/it]
Outer training loop:  81%|########1 | 40507/50000 [26:44:21<34:45:38, 13.18s/it]
Outer training loop:  81%|########1 | 40508/50000 [26:44:24<25:56:25,  9.84s/it]
Outer training loop:  81%|########1 | 40509/50000 [26:44:25<19:40:36,  7.46s/it]
Outer training loop:  81%|########1 | 40510/50000 [26:44:27<15:13:39,  5.78s/it]
Outer training loop:  81%|########1 | 40511/50000 [26:44:29<12:13:48,  4.64s/it]
Outer training loop:  81%|########1 | 40512/50000 [26:44:31<9:55:38,  3.77s/it] 
Outer training loop:  81%|########1 | 40513/50000 [26:44:33<8:27:17,  3.21s/it]
Outer training loop:  81%|########1 | 40514/50000 [26:44:35<7:39:33,  2.91s/it]
Outer training loop:  81%|########1 | 40515/50000 [26:44:37<6:50:49,  2.60s/it]
Outer training loop:  81%|########1 | 40516/50000 [26:44:39<6:10:29,  2.34s/it]
Outer training loop:  81%|########1 | 40517/50000 [26:44:41<5:57:24,  2.26s/it]
Outer training loop:  81%|########1 | 40518/50000 [26:44:43<5:36:37,  2.13s/it]
Outer training loop:  81%|########1 | 40519/50000 [26:44:44<5:23:27,  2.05s/it]
Outer training loop:  81%|########1 | 40520/50000 [26:44:46<5:19:55,  2.02s/it]
Outer training loop:  81%|########1 | 40521/50000 [26:44:49<5:26:14,  2.07s/it]
Outer training loop:  81%|########1 | 40522/50000 [26:44:50<5:15:43,  2.00s/it]
Outer training loop:  81%|########1 | 40523/50000 [26:44:52<5:07:18,  1.95s/it]
Outer training loop:  81%|########1 | 40524/50000 [26:44:54<5:15:44,  2.00s/it]
Outer training loop:  81%|########1 | 40525/50000 [26:44:56<5:07:34,  1.95s/it]
Outer training loop:  81%|########1 | 40526/50000 [26:44:58<5:07:52,  1.95s/it]
Outer training loop:  81%|########1 | 40527/50000 [26:45:00<5:10:17,  1.97s/it]
Outer training loop:  81%|########1 | 40528/50000 [26:45:02<5:02:01,  1.91s/it]
Outer training loop:  81%|########1 | 40529/50000 [26:45:04<4:51:52,  1.85s/it]
Outer training loop:  81%|########1 | 40530/50000 [26:45:06<4:54:48,  1.87s/it]
Outer training loop:  81%|########1 | 40531/50000 [26:45:08<4:59:53,  1.90s/it]
Outer training loop:  81%|########1 | 40532/50000 [26:45:09<4:52:55,  1.86s/it]
Outer training loop:  81%|########1 | 40533/50000 [26:45:11<4:50:50,  1.84s/it]
Outer training loop:  81%|########1 | 40534/50000 [26:45:13<4:49:52,  1.84s/it]
Outer training loop:  81%|########1 | 40535/50000 [26:45:15<4:57:14,  1.88s/it]
Outer training loop:  81%|########1 | 40536/50000 [26:45:17<4:49:24,  1.83s/it]
Outer training loop:  81%|########1 | 40537/50000 [26:45:19<4:58:13,  1.89s/it]
Outer training loop:  81%|########1 | 40538/50000 [26:45:21<5:11:15,  1.97s/it]
Outer training loop:  81%|########1 | 40539/50000 [26:45:23<5:04:16,  1.93s/it]
Outer training loop:  81%|########1 | 40540/50000 [26:45:25<5:02:14,  1.92s/it]
Outer training loop:  81%|########1 | 40541/50000 [26:45:27<5:08:41,  1.96s/it]
Outer training loop:  81%|########1 | 40542/50000 [26:45:29<5:09:55,  1.97s/it]
Outer training loop:  81%|########1 | 40543/50000 [26:45:30<5:03:56,  1.93s/it]
Outer training loop:  81%|########1 | 40544/50000 [26:45:33<5:15:52,  2.00s/it]
Outer training loop:  81%|########1 | 40545/50000 [26:45:35<5:10:26,  1.97s/it]
Outer training loop:  81%|########1 | 40546/50000 [26:45:36<5:03:18,  1.92s/it]
Outer training loop:  81%|########1 | 40547/50000 [26:45:38<4:56:38,  1.88s/it]
Outer training loop:  81%|########1 | 40548/50000 [26:45:40<5:04:33,  1.93s/it]
Outer training loop:  81%|########1 | 40549/50000 [26:45:42<4:58:30,  1.90s/it]
Outer training loop:  81%|########1 | 40550/50000 [26:45:44<4:52:19,  1.86s/it]
Outer training loop:  81%|########1 | 40551/50000 [26:45:46<5:00:26,  1.91s/it]
Outer training loop:  81%|########1 | 40552/50000 [26:45:48<5:00:08,  1.91s/it]
Outer training loop:  81%|########1 | 40553/50000 [26:45:50<5:03:07,  1.93s/it]
Outer training loop:  81%|########1 | 40554/50000 [26:45:52<5:06:44,  1.95s/it]
Outer training loop:  81%|########1 | 40555/50000 [26:45:53<4:57:30,  1.89s/it]
Outer training loop:  81%|########1 | 40556/50000 [26:45:55<4:49:48,  1.84s/it]
Outer training loop:  81%|########1 | 40557/50000 [26:45:57<4:46:30,  1.82s/it]
Outer training loop:  81%|########1 | 40558/50000 [26:45:59<4:58:01,  1.89s/it]
Outer training loop:  81%|########1 | 40559/50000 [26:46:01<4:52:00,  1.86s/it]
Outer training loop:  81%|########1 | 40560/50000 [26:46:03<4:50:18,  1.85s/it]
Outer training loop:  81%|########1 | 40561/50000 [26:46:05<5:06:55,  1.95s/it]
Outer training loop:  81%|########1 | 40562/50000 [26:46:07<4:57:39,  1.89s/it]
Outer training loop:  81%|########1 | 40563/50000 [26:46:08<4:51:12,  1.85s/it]
Outer training loop:  81%|########1 | 40564/50000 [26:46:10<4:58:36,  1.90s/it]
Outer training loop:  81%|########1 | 40565/50000 [26:46:12<5:03:24,  1.93s/it]
Outer training loop:  81%|########1 | 40566/50000 [26:46:14<4:57:51,  1.89s/it]
Outer training loop:  81%|########1 | 40567/50000 [26:46:16<4:50:36,  1.85s/it]
Outer training loop:  81%|########1 | 40568/50000 [26:46:18<4:48:21,  1.83s/it]
Outer training loop:  81%|########1 | 40569/50000 [26:46:20<5:00:48,  1.91s/it]
Outer training loop:  81%|########1 | 40570/50000 [26:46:21<4:51:58,  1.86s/it]
Outer training loop:  81%|########1 | 40571/50000 [26:46:23<4:49:54,  1.84s/it]
Outer training loop:  81%|########1 | 40572/50000 [26:46:25<4:56:26,  1.89s/it]
Outer training loop:  81%|########1 | 40573/50000 [26:46:27<4:49:43,  1.84s/it]
Outer training loop:  81%|########1 | 40574/50000 [26:46:29<4:48:32,  1.84s/it]
Outer training loop:  81%|########1 | 40575/50000 [26:46:31<4:47:36,  1.83s/it]
Outer training loop:  81%|########1 | 40576/50000 [26:46:32<4:48:22,  1.84s/it]
Outer training loop:  81%|########1 | 40577/50000 [26:46:34<4:41:41,  1.79s/it]
Outer training loop:  81%|########1 | 40578/50000 [26:46:36<4:44:15,  1.81s/it]
Outer training loop:  81%|########1 | 40579/50000 [26:46:38<4:54:21,  1.87s/it]
Outer training loop:  81%|########1 | 40580/50000 [26:46:40<4:50:04,  1.85s/it]
Outer training loop:  81%|########1 | 40581/50000 [26:46:42<4:52:31,  1.86s/it]
Outer training loop:  81%|########1 | 40582/50000 [26:46:44<5:03:54,  1.94s/it]
Outer training loop:  81%|########1 | 40583/50000 [26:46:46<4:54:50,  1.88s/it]
Outer training loop:  81%|########1 | 40584/50000 [26:46:47<4:53:02,  1.87s/it]
Outer training loop:  81%|########1 | 40585/50000 [26:46:49<4:55:03,  1.88s/it]
Outer training loop:  81%|########1 | 40586/50000 [26:46:51<5:00:20,  1.91s/it]
Outer training loop:  81%|########1 | 40587/50000 [26:46:53<5:05:50,  1.95s/it]
Outer training loop:  81%|########1 | 40588/50000 [26:46:55<4:53:39,  1.87s/it]
Outer training loop:  81%|########1 | 40589/50000 [26:46:57<4:52:50,  1.87s/it]
Outer training loop:  81%|########1 | 40590/50000 [26:46:59<4:58:20,  1.90s/it]
Outer training loop:  81%|########1 | 40591/50000 [26:47:01<4:48:56,  1.84s/it]
Outer training loop:  81%|########1 | 40592/50000 [26:47:02<4:46:27,  1.83s/it]
Outer training loop:  81%|########1 | 40593/50000 [26:47:04<4:56:29,  1.89s/it]
Outer training loop:  81%|########1 | 40594/50000 [26:47:06<4:51:53,  1.86s/it]
Outer training loop:  81%|########1 | 40595/50000 [26:47:08<4:50:36,  1.85s/it]
Outer training loop:  81%|########1 | 40596/50000 [26:47:10<4:58:40,  1.91s/it]
Outer training loop:  81%|########1 | 40597/50000 [26:47:12<4:56:35,  1.89s/it]
Outer training loop:  81%|########1 | 40598/50000 [26:47:14<4:59:45,  1.91s/it]
Outer training loop:  81%|########1 | 40599/50000 [26:47:16<4:56:25,  1.89s/it]
Outer training loop:  81%|########1 | 40600/50000 [26:47:18<5:00:23,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  81%|########1 | 40601/50000 [26:47:25<9:13:12,  3.53s/it]
Outer training loop:  81%|########1 | 40602/50000 [26:47:27<7:51:28,  3.01s/it]
Outer training loop:  81%|########1 | 40603/50000 [26:47:29<6:55:16,  2.65s/it]
Outer training loop:  81%|########1 | 40604/50000 [26:47:31<6:20:04,  2.43s/it]
Outer training loop:  81%|########1 | 40605/50000 [26:47:32<5:49:05,  2.23s/it]
Outer training loop:  81%|########1 | 40606/50000 [26:47:34<5:28:34,  2.10s/it]
Outer training loop:  81%|########1 | 40607/50000 [26:47:36<5:23:58,  2.07s/it]
Outer training loop:  81%|########1 | 40608/50000 [26:47:38<5:12:56,  2.00s/it]
Outer training loop:  81%|########1 | 40609/50000 [26:47:40<5:12:04,  1.99s/it]
Outer training loop:  81%|########1 | 40610/50000 [26:47:42<5:16:32,  2.02s/it]
Outer training loop:  81%|########1 | 40611/50000 [26:47:44<5:07:11,  1.96s/it]
Outer training loop:  81%|########1 | 40612/50000 [26:47:46<5:01:42,  1.93s/it]
Outer training loop:  81%|########1 | 40613/50000 [26:47:47<4:52:36,  1.87s/it]
Outer training loop:  81%|########1 | 40614/50000 [26:47:49<4:59:02,  1.91s/it]
Outer training loop:  81%|########1 | 40615/50000 [26:47:51<4:58:41,  1.91s/it]
Outer training loop:  81%|########1 | 40616/50000 [26:47:53<4:52:27,  1.87s/it]
Outer training loop:  81%|########1 | 40617/50000 [26:47:55<4:59:24,  1.91s/it]
Outer training loop:  81%|########1 | 40618/50000 [26:47:57<4:55:21,  1.89s/it]
Outer training loop:  81%|########1 | 40619/50000 [26:47:59<4:52:53,  1.87s/it]
Outer training loop:  81%|########1 | 40620/50000 [26:48:01<4:57:12,  1.90s/it]
Outer training loop:  81%|########1 | 40621/50000 [26:48:02<4:49:10,  1.85s/it]
Outer training loop:  81%|########1 | 40622/50000 [26:48:04<4:48:50,  1.85s/it]
Outer training loop:  81%|########1 | 40623/50000 [26:48:06<4:48:05,  1.84s/it]
Outer training loop:  81%|########1 | 40624/50000 [26:48:08<4:52:05,  1.87s/it]
Outer training loop:  81%|########1 | 40625/50000 [26:48:10<4:48:11,  1.84s/it]
Outer training loop:  81%|########1 | 40626/50000 [26:48:12<4:49:12,  1.85s/it]
Outer training loop:  81%|########1 | 40627/50000 [26:48:14<4:58:13,  1.91s/it]
Outer training loop:  81%|########1 | 40628/50000 [26:48:16<4:53:27,  1.88s/it]
Outer training loop:  81%|########1 | 40629/50000 [26:48:17<4:47:54,  1.84s/it]
Outer training loop:  81%|########1 | 40630/50000 [26:48:19<4:52:24,  1.87s/it]
Outer training loop:  81%|########1 | 40631/50000 [26:48:21<4:50:55,  1.86s/it]
Outer training loop:  81%|########1 | 40632/50000 [26:48:23<4:48:43,  1.85s/it]
Outer training loop:  81%|########1 | 40633/50000 [26:48:25<4:54:19,  1.89s/it]
Outer training loop:  81%|########1 | 40634/50000 [26:48:27<5:01:43,  1.93s/it]
Outer training loop:  81%|########1 | 40635/50000 [26:48:29<4:54:32,  1.89s/it]
Outer training loop:  81%|########1 | 40636/50000 [26:48:31<4:53:22,  1.88s/it]
Outer training loop:  81%|########1 | 40637/50000 [26:48:33<5:00:24,  1.93s/it]
Outer training loop:  81%|########1 | 40638/50000 [26:48:34<4:57:24,  1.91s/it]
Outer training loop:  81%|########1 | 40639/50000 [26:48:36<4:54:13,  1.89s/it]
Outer training loop:  81%|########1 | 40640/50000 [26:48:38<5:03:14,  1.94s/it]
Outer training loop:  81%|########1 | 40641/50000 [26:48:40<4:57:43,  1.91s/it]
Outer training loop:  81%|########1 | 40642/50000 [26:48:42<4:54:02,  1.89s/it]
Outer training loop:  81%|########1 | 40643/50000 [26:48:44<5:04:30,  1.95s/it]
Outer training loop:  81%|########1 | 40644/50000 [26:48:46<5:03:47,  1.95s/it]
Outer training loop:  81%|########1 | 40645/50000 [26:48:48<5:00:01,  1.92s/it]
Outer training loop:  81%|########1 | 40646/50000 [26:48:50<5:07:07,  1.97s/it]
Outer training loop:  81%|########1 | 40647/50000 [26:48:52<5:05:02,  1.96s/it]
Outer training loop:  81%|########1 | 40648/50000 [26:48:54<4:57:37,  1.91s/it]
Outer training loop:  81%|########1 | 40649/50000 [26:48:56<4:51:52,  1.87s/it]
Outer training loop:  81%|########1 | 40650/50000 [26:48:58<4:56:15,  1.90s/it]
Outer training loop:  81%|########1 | 40651/50000 [26:48:59<4:54:11,  1.89s/it]
Outer training loop:  81%|########1 | 40652/50000 [26:49:01<4:47:24,  1.84s/it]
Outer training loop:  81%|########1 | 40653/50000 [26:49:03<4:48:34,  1.85s/it]
Outer training loop:  81%|########1 | 40654/50000 [26:49:05<4:52:48,  1.88s/it]
Outer training loop:  81%|########1 | 40655/50000 [26:49:07<4:48:42,  1.85s/it]
Outer training loop:  81%|########1 | 40656/50000 [26:49:09<4:46:21,  1.84s/it]
Outer training loop:  81%|########1 | 40657/50000 [26:49:10<4:50:16,  1.86s/it]
Outer training loop:  81%|########1 | 40658/50000 [26:49:12<4:50:52,  1.87s/it]
Outer training loop:  81%|########1 | 40659/50000 [26:49:14<4:51:29,  1.87s/it]
Outer training loop:  81%|########1 | 40660/50000 [26:49:16<4:56:56,  1.91s/it]
Outer training loop:  81%|########1 | 40661/50000 [26:49:18<4:50:18,  1.87s/it]
Outer training loop:  81%|########1 | 40662/50000 [26:49:20<4:47:34,  1.85s/it]
Outer training loop:  81%|########1 | 40663/50000 [26:49:22<4:53:27,  1.89s/it]
Outer training loop:  81%|########1 | 40664/50000 [26:49:24<4:49:59,  1.86s/it]
Outer training loop:  81%|########1 | 40665/50000 [26:49:25<4:43:48,  1.82s/it]
Outer training loop:  81%|########1 | 40666/50000 [26:49:27<4:36:11,  1.78s/it]
Outer training loop:  81%|########1 | 40667/50000 [26:49:29<4:43:35,  1.82s/it]
Outer training loop:  81%|########1 | 40668/50000 [26:49:31<4:46:28,  1.84s/it]
Outer training loop:  81%|########1 | 40669/50000 [26:49:33<4:47:06,  1.85s/it]
Outer training loop:  81%|########1 | 40670/50000 [26:49:35<5:02:30,  1.95s/it]
Outer training loop:  81%|########1 | 40671/50000 [26:49:37<4:59:17,  1.92s/it]
Outer training loop:  81%|########1 | 40672/50000 [26:49:39<4:53:06,  1.89s/it]
Outer training loop:  81%|########1 | 40673/50000 [26:49:40<4:55:36,  1.90s/it]
Outer training loop:  81%|########1 | 40674/50000 [26:49:42<5:00:18,  1.93s/it]
Outer training loop:  81%|########1 | 40675/50000 [26:49:44<4:52:06,  1.88s/it]
Outer training loop:  81%|########1 | 40676/50000 [26:49:46<4:47:27,  1.85s/it]
Outer training loop:  81%|########1 | 40677/50000 [26:49:48<4:52:48,  1.88s/it]
Outer training loop:  81%|########1 | 40678/50000 [26:49:50<4:59:09,  1.93s/it]
Outer training loop:  81%|########1 | 40679/50000 [26:49:52<4:54:52,  1.90s/it]
Outer training loop:  81%|########1 | 40680/50000 [26:49:54<5:00:51,  1.94s/it]
Outer training loop:  81%|########1 | 40681/50000 [26:49:56<4:53:40,  1.89s/it]
Outer training loop:  81%|########1 | 40682/50000 [26:49:57<4:48:51,  1.86s/it]
Outer training loop:  81%|########1 | 40683/50000 [26:49:59<4:55:31,  1.90s/it]
Outer training loop:  81%|########1 | 40684/50000 [26:50:01<4:51:16,  1.88s/it]
Outer training loop:  81%|########1 | 40685/50000 [26:50:03<4:50:50,  1.87s/it]
Outer training loop:  81%|########1 | 40686/50000 [26:50:05<4:48:40,  1.86s/it]
Outer training loop:  81%|########1 | 40687/50000 [26:50:07<4:56:00,  1.91s/it]
Outer training loop:  81%|########1 | 40688/50000 [26:50:09<4:52:52,  1.89s/it]
Outer training loop:  81%|########1 | 40689/50000 [26:50:11<4:48:39,  1.86s/it]
Outer training loop:  81%|########1 | 40690/50000 [26:50:13<5:03:30,  1.96s/it]
Outer training loop:  81%|########1 | 40691/50000 [26:50:15<4:59:37,  1.93s/it]
Outer training loop:  81%|########1 | 40692/50000 [26:50:17<4:59:25,  1.93s/it]
Outer training loop:  81%|########1 | 40693/50000 [26:50:18<4:56:45,  1.91s/it]
Outer training loop:  81%|########1 | 40694/50000 [26:50:21<5:07:27,  1.98s/it]
Outer training loop:  81%|########1 | 40695/50000 [26:50:22<4:56:06,  1.91s/it]
Outer training loop:  81%|########1 | 40696/50000 [26:50:24<4:52:40,  1.89s/it]
Outer training loop:  81%|########1 | 40697/50000 [26:50:26<5:05:47,  1.97s/it]
Outer training loop:  81%|########1 | 40698/50000 [26:50:28<5:04:32,  1.96s/it]
Outer training loop:  81%|########1 | 40699/50000 [26:50:30<4:53:42,  1.89s/it]
Outer training loop:  81%|########1 | 40700/50000 [26:50:32<4:50:33,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  81%|########1 | 40701/50000 [26:50:40<9:29:27,  3.67s/it]
Outer training loop:  81%|########1 | 40702/50000 [26:50:41<8:00:28,  3.10s/it]
Outer training loop:  81%|########1 | 40703/50000 [26:50:43<7:08:23,  2.76s/it]
Outer training loop:  81%|########1 | 40704/50000 [26:50:45<6:21:49,  2.46s/it]
Outer training loop:  81%|########1 | 40705/50000 [26:50:47<6:02:52,  2.34s/it]
Outer training loop:  81%|########1 | 40706/50000 [26:50:49<5:42:40,  2.21s/it]
Outer training loop:  81%|########1 | 40707/50000 [26:50:51<5:24:41,  2.10s/it]
Outer training loop:  81%|########1 | 40708/50000 [26:50:53<5:24:18,  2.09s/it]
Outer training loop:  81%|########1 | 40709/50000 [26:50:55<5:10:19,  2.00s/it]
Outer training loop:  81%|########1 | 40710/50000 [26:50:57<5:02:38,  1.95s/it]
Outer training loop:  81%|########1 | 40711/50000 [26:50:59<4:57:32,  1.92s/it]
Outer training loop:  81%|########1 | 40712/50000 [26:51:01<5:09:18,  2.00s/it]
Outer training loop:  81%|########1 | 40713/50000 [26:51:03<4:58:35,  1.93s/it]
Outer training loop:  81%|########1 | 40714/50000 [26:51:04<4:50:36,  1.88s/it]
Outer training loop:  81%|########1 | 40715/50000 [26:51:06<5:03:14,  1.96s/it]
Outer training loop:  81%|########1 | 40716/50000 [26:51:08<5:02:46,  1.96s/it]
Outer training loop:  81%|########1 | 40717/50000 [26:51:10<4:59:41,  1.94s/it]
Outer training loop:  81%|########1 | 40718/50000 [26:51:12<5:03:32,  1.96s/it]
Outer training loop:  81%|########1 | 40719/50000 [26:51:14<4:57:37,  1.92s/it]
Outer training loop:  81%|########1 | 40720/50000 [26:51:16<4:53:50,  1.90s/it]
Outer training loop:  81%|########1 | 40721/50000 [26:51:18<4:45:08,  1.84s/it]
Outer training loop:  81%|########1 | 40722/50000 [26:51:20<4:58:34,  1.93s/it]
Outer training loop:  81%|########1 | 40723/50000 [26:51:22<4:53:02,  1.90s/it]
Outer training loop:  81%|########1 | 40724/50000 [26:51:23<4:49:21,  1.87s/it]
Outer training loop:  81%|########1 | 40725/50000 [26:51:25<4:50:57,  1.88s/it]
Outer training loop:  81%|########1 | 40726/50000 [26:51:27<4:49:12,  1.87s/it]
Outer training loop:  81%|########1 | 40727/50000 [26:51:29<4:49:20,  1.87s/it]
Outer training loop:  81%|########1 | 40728/50000 [26:51:31<5:01:37,  1.95s/it]
Outer training loop:  81%|########1 | 40729/50000 [26:51:33<4:54:09,  1.90s/it]
Outer training loop:  81%|########1 | 40730/50000 [26:51:35<4:53:27,  1.90s/it]
Outer training loop:  81%|########1 | 40731/50000 [26:51:37<5:10:04,  2.01s/it]
Outer training loop:  81%|########1 | 40732/50000 [26:51:39<4:52:12,  1.89s/it]
Outer training loop:  81%|########1 | 40733/50000 [26:51:41<4:51:07,  1.88s/it]
Outer training loop:  81%|########1 | 40734/50000 [26:51:43<4:53:26,  1.90s/it]
Outer training loop:  81%|########1 | 40735/50000 [26:51:45<4:55:23,  1.91s/it]
Outer training loop:  81%|########1 | 40736/50000 [26:51:46<4:52:16,  1.89s/it]
Outer training loop:  81%|########1 | 40737/50000 [26:51:48<4:45:26,  1.85s/it]
Outer training loop:  81%|########1 | 40738/50000 [26:51:50<5:04:36,  1.97s/it]
Outer training loop:  81%|########1 | 40739/50000 [26:51:52<4:55:27,  1.91s/it]
Outer training loop:  81%|########1 | 40740/50000 [26:51:54<4:48:16,  1.87s/it]
Outer training loop:  81%|########1 | 40741/50000 [26:51:56<4:45:42,  1.85s/it]
Outer training loop:  81%|########1 | 40742/50000 [26:51:58<4:56:18,  1.92s/it]
Outer training loop:  81%|########1 | 40743/50000 [26:52:00<4:52:27,  1.90s/it]
Outer training loop:  81%|########1 | 40744/50000 [26:52:01<4:47:50,  1.87s/it]
Outer training loop:  81%|########1 | 40745/50000 [26:52:03<4:54:35,  1.91s/it]
Outer training loop:  81%|########1 | 40746/50000 [26:52:05<4:51:56,  1.89s/it]
Outer training loop:  81%|########1 | 40747/50000 [26:52:07<4:52:36,  1.90s/it]
Outer training loop:  81%|########1 | 40748/50000 [26:52:09<4:46:11,  1.86s/it]
Outer training loop:  81%|########1 | 40749/50000 [26:52:11<4:54:51,  1.91s/it]
Outer training loop:  82%|########1 | 40750/50000 [26:52:13<4:53:56,  1.91s/it]
Outer training loop:  82%|########1 | 40751/50000 [26:52:15<4:56:29,  1.92s/it]
Outer training loop:  82%|########1 | 40752/50000 [26:52:17<5:00:49,  1.95s/it]
Outer training loop:  82%|########1 | 40753/50000 [26:52:19<4:54:28,  1.91s/it]
Outer training loop:  82%|########1 | 40754/50000 [26:52:21<4:53:39,  1.91s/it]
Outer training loop:  82%|########1 | 40755/50000 [26:52:22<4:47:48,  1.87s/it]
Outer training loop:  82%|########1 | 40756/50000 [26:52:24<4:50:31,  1.89s/it]
Outer training loop:  82%|########1 | 40757/50000 [26:52:26<4:49:00,  1.88s/it]
Outer training loop:  82%|########1 | 40758/50000 [26:52:28<4:50:35,  1.89s/it]
Outer training loop:  82%|########1 | 40759/50000 [26:52:30<5:03:17,  1.97s/it]
Outer training loop:  82%|########1 | 40760/50000 [26:52:32<4:53:00,  1.90s/it]
Outer training loop:  82%|########1 | 40761/50000 [26:52:34<4:47:30,  1.87s/it]
Outer training loop:  82%|########1 | 40762/50000 [26:52:36<4:54:40,  1.91s/it]
Outer training loop:  82%|########1 | 40763/50000 [26:52:38<4:46:54,  1.86s/it]
Outer training loop:  82%|########1 | 40764/50000 [26:52:39<4:41:42,  1.83s/it]
Outer training loop:  82%|########1 | 40765/50000 [26:52:41<4:45:22,  1.85s/it]
Outer training loop:  82%|########1 | 40766/50000 [26:52:43<4:57:55,  1.94s/it]
Outer training loop:  82%|########1 | 40767/50000 [26:52:45<4:55:01,  1.92s/it]
Outer training loop:  82%|########1 | 40768/50000 [26:52:47<4:47:14,  1.87s/it]
Outer training loop:  82%|########1 | 40769/50000 [26:52:49<4:56:12,  1.93s/it]
Outer training loop:  82%|########1 | 40770/50000 [26:52:51<4:50:15,  1.89s/it]
Outer training loop:  82%|########1 | 40771/50000 [26:52:53<4:47:12,  1.87s/it]
Outer training loop:  82%|########1 | 40772/50000 [26:52:54<4:42:31,  1.84s/it]
Outer training loop:  82%|########1 | 40773/50000 [26:52:56<4:45:54,  1.86s/it]
Outer training loop:  82%|########1 | 40774/50000 [26:52:58<4:49:01,  1.88s/it]
Outer training loop:  82%|########1 | 40775/50000 [26:53:00<4:40:44,  1.83s/it]
Outer training loop:  82%|########1 | 40776/50000 [26:53:02<4:49:25,  1.88s/it]
Outer training loop:  82%|########1 | 40777/50000 [26:53:04<4:45:30,  1.86s/it]
Outer training loop:  82%|########1 | 40778/50000 [26:53:06<4:47:06,  1.87s/it]
Outer training loop:  82%|########1 | 40779/50000 [26:53:07<4:44:12,  1.85s/it]
Outer training loop:  82%|########1 | 40780/50000 [26:53:09<4:48:35,  1.88s/it]
Outer training loop:  82%|########1 | 40781/50000 [26:53:11<4:51:45,  1.90s/it]
Outer training loop:  82%|########1 | 40782/50000 [26:53:13<4:54:23,  1.92s/it]
Outer training loop:  82%|########1 | 40783/50000 [26:53:15<4:45:25,  1.86s/it]
Outer training loop:  82%|########1 | 40784/50000 [26:53:17<4:49:46,  1.89s/it]
Outer training loop:  82%|########1 | 40785/50000 [26:53:19<4:49:56,  1.89s/it]
Outer training loop:  82%|########1 | 40786/50000 [26:53:21<4:50:33,  1.89s/it]
Outer training loop:  82%|########1 | 40787/50000 [26:53:23<4:53:51,  1.91s/it]
Outer training loop:  82%|########1 | 40788/50000 [26:53:25<4:50:31,  1.89s/it]
Outer training loop:  82%|########1 | 40789/50000 [26:53:26<4:47:24,  1.87s/it]
Outer training loop:  82%|########1 | 40790/50000 [26:53:28<4:50:12,  1.89s/it]
Outer training loop:  82%|########1 | 40791/50000 [26:53:30<4:50:21,  1.89s/it]
Outer training loop:  82%|########1 | 40792/50000 [26:53:32<4:48:57,  1.88s/it]
Outer training loop:  82%|########1 | 40793/50000 [26:53:34<4:40:31,  1.83s/it]
Outer training loop:  82%|########1 | 40794/50000 [26:53:36<4:46:56,  1.87s/it]
Outer training loop:  82%|########1 | 40795/50000 [26:53:38<4:44:54,  1.86s/it]
Outer training loop:  82%|########1 | 40796/50000 [26:53:40<4:55:56,  1.93s/it]
Outer training loop:  82%|########1 | 40797/50000 [26:53:41<4:50:13,  1.89s/it]
Outer training loop:  82%|########1 | 40798/50000 [26:53:43<4:48:30,  1.88s/it]
Outer training loop:  82%|########1 | 40799/50000 [26:53:45<4:44:56,  1.86s/it]
Outer training loop:  82%|########1 | 40800/50000 [26:53:47<4:52:49,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  82%|########1 | 40801/50000 [26:53:54<9:02:20,  3.54s/it]
Outer training loop:  82%|########1 | 40802/50000 [26:53:56<7:40:11,  3.00s/it]
Outer training loop:  82%|########1 | 40803/50000 [26:53:58<6:49:08,  2.67s/it]
Outer training loop:  82%|########1 | 40804/50000 [26:54:00<6:14:55,  2.45s/it]
Outer training loop:  82%|########1 | 40805/50000 [26:54:02<5:47:04,  2.26s/it]
Outer training loop:  82%|########1 | 40806/50000 [26:54:04<5:30:41,  2.16s/it]
Outer training loop:  82%|########1 | 40807/50000 [26:54:06<5:24:44,  2.12s/it]
Outer training loop:  82%|########1 | 40808/50000 [26:54:08<5:06:18,  2.00s/it]
Outer training loop:  82%|########1 | 40809/50000 [26:54:09<4:56:57,  1.94s/it]
Outer training loop:  82%|########1 | 40810/50000 [26:54:11<4:48:54,  1.89s/it]
Outer training loop:  82%|########1 | 40811/50000 [26:54:13<4:56:21,  1.94s/it]
Outer training loop:  82%|########1 | 40812/50000 [26:54:15<4:54:35,  1.92s/it]
Outer training loop:  82%|########1 | 40813/50000 [26:54:17<4:42:49,  1.85s/it]
Outer training loop:  82%|########1 | 40814/50000 [26:54:19<4:48:58,  1.89s/it]
Outer training loop:  82%|########1 | 40815/50000 [26:54:21<4:47:17,  1.88s/it]
Outer training loop:  82%|########1 | 40816/50000 [26:54:22<4:43:48,  1.85s/it]
Outer training loop:  82%|########1 | 40817/50000 [26:54:24<4:41:40,  1.84s/it]
Outer training loop:  82%|########1 | 40818/50000 [26:54:26<4:54:01,  1.92s/it]
Outer training loop:  82%|########1 | 40819/50000 [26:54:28<4:51:13,  1.90s/it]
Outer training loop:  82%|########1 | 40820/50000 [26:54:30<4:49:22,  1.89s/it]
Outer training loop:  82%|########1 | 40821/50000 [26:54:32<4:53:14,  1.92s/it]
Outer training loop:  82%|########1 | 40822/50000 [26:54:34<4:46:04,  1.87s/it]
Outer training loop:  82%|########1 | 40823/50000 [26:54:36<4:42:39,  1.85s/it]
Outer training loop:  82%|########1 | 40824/50000 [26:54:38<4:47:31,  1.88s/it]
Outer training loop:  82%|########1 | 40825/50000 [26:54:39<4:44:29,  1.86s/it]
Outer training loop:  82%|########1 | 40826/50000 [26:54:41<4:44:27,  1.86s/it]
Outer training loop:  82%|########1 | 40827/50000 [26:54:43<4:41:34,  1.84s/it]
Outer training loop:  82%|########1 | 40828/50000 [26:54:45<4:58:23,  1.95s/it]
Outer training loop:  82%|########1 | 40829/50000 [26:54:47<4:53:39,  1.92s/it]
Outer training loop:  82%|########1 | 40830/50000 [26:54:49<4:51:10,  1.91s/it]
Outer training loop:  82%|########1 | 40831/50000 [26:54:51<4:59:04,  1.96s/it]
Outer training loop:  82%|########1 | 40832/50000 [26:54:53<4:52:51,  1.92s/it]
Outer training loop:  82%|########1 | 40833/50000 [26:54:55<4:51:06,  1.91s/it]
Outer training loop:  82%|########1 | 40834/50000 [26:54:57<4:57:03,  1.94s/it]
Outer training loop:  82%|########1 | 40835/50000 [26:54:59<4:51:15,  1.91s/it]
Outer training loop:  82%|########1 | 40836/50000 [26:55:00<4:46:13,  1.87s/it]
Outer training loop:  82%|########1 | 40837/50000 [26:55:02<4:36:08,  1.81s/it]
Outer training loop:  82%|########1 | 40838/50000 [26:55:04<4:46:49,  1.88s/it]
Outer training loop:  82%|########1 | 40839/50000 [26:55:06<4:48:08,  1.89s/it]
Outer training loop:  82%|########1 | 40840/50000 [26:55:08<4:48:15,  1.89s/it]
Outer training loop:  82%|########1 | 40841/50000 [26:55:10<5:01:08,  1.97s/it]
Outer training loop:  82%|########1 | 40842/50000 [26:55:12<4:50:51,  1.91s/it]
Outer training loop:  82%|########1 | 40843/50000 [26:55:14<4:47:37,  1.88s/it]
Outer training loop:  82%|########1 | 40844/50000 [26:55:15<4:46:09,  1.88s/it]
Outer training loop:  82%|########1 | 40845/50000 [26:55:17<4:50:21,  1.90s/it]
Outer training loop:  82%|########1 | 40846/50000 [26:55:19<4:45:12,  1.87s/it]
Outer training loop:  82%|########1 | 40847/50000 [26:55:21<4:40:02,  1.84s/it]
Outer training loop:  82%|########1 | 40848/50000 [26:55:23<4:36:57,  1.82s/it]
Outer training loop:  82%|########1 | 40849/50000 [26:55:25<4:50:36,  1.91s/it]
Outer training loop:  82%|########1 | 40850/50000 [26:55:27<4:43:06,  1.86s/it]
Outer training loop:  82%|########1 | 40851/50000 [26:55:29<4:56:10,  1.94s/it]
Outer training loop:  82%|########1 | 40852/50000 [26:55:31<4:59:21,  1.96s/it]
Outer training loop:  82%|########1 | 40853/50000 [26:55:32<4:49:14,  1.90s/it]
Outer training loop:  82%|########1 | 40854/50000 [26:55:34<4:46:28,  1.88s/it]
Outer training loop:  82%|########1 | 40855/50000 [26:55:36<4:51:59,  1.92s/it]
Outer training loop:  82%|########1 | 40856/50000 [26:55:38<4:47:39,  1.89s/it]
Outer training loop:  82%|########1 | 40857/50000 [26:55:40<4:44:05,  1.86s/it]
Outer training loop:  82%|########1 | 40858/50000 [26:55:42<4:49:44,  1.90s/it]
Outer training loop:  82%|########1 | 40859/50000 [26:55:44<4:45:14,  1.87s/it]
Outer training loop:  82%|########1 | 40860/50000 [26:55:46<4:45:13,  1.87s/it]
Outer training loop:  82%|########1 | 40861/50000 [26:55:47<4:38:01,  1.83s/it]
Outer training loop:  82%|########1 | 40862/50000 [26:55:49<4:49:50,  1.90s/it]
Outer training loop:  82%|########1 | 40863/50000 [26:55:51<4:53:23,  1.93s/it]
Outer training loop:  82%|########1 | 40864/50000 [26:55:53<4:46:44,  1.88s/it]
Outer training loop:  82%|########1 | 40865/50000 [26:55:55<4:51:19,  1.91s/it]
Outer training loop:  82%|########1 | 40866/50000 [26:55:57<4:48:10,  1.89s/it]
Outer training loop:  82%|########1 | 40867/50000 [26:55:59<4:42:27,  1.86s/it]
Outer training loop:  82%|########1 | 40868/50000 [26:56:01<4:46:08,  1.88s/it]
Outer training loop:  82%|########1 | 40869/50000 [26:56:03<4:41:59,  1.85s/it]
Outer training loop:  82%|########1 | 40870/50000 [26:56:04<4:48:19,  1.89s/it]
Outer training loop:  82%|########1 | 40871/50000 [26:56:07<4:53:47,  1.93s/it]
Outer training loop:  82%|########1 | 40872/50000 [26:56:09<4:59:22,  1.97s/it]
Outer training loop:  82%|########1 | 40873/50000 [26:56:10<4:55:00,  1.94s/it]
Outer training loop:  82%|########1 | 40874/50000 [26:56:12<4:51:56,  1.92s/it]
Outer training loop:  82%|########1 | 40875/50000 [26:56:14<4:57:03,  1.95s/it]
Outer training loop:  82%|########1 | 40876/50000 [26:56:16<4:52:01,  1.92s/it]
Outer training loop:  82%|########1 | 40877/50000 [26:56:18<4:47:32,  1.89s/it]
Outer training loop:  82%|########1 | 40878/50000 [26:56:20<4:59:52,  1.97s/it]
Outer training loop:  82%|########1 | 40879/50000 [26:56:22<4:54:52,  1.94s/it]
Outer training loop:  82%|########1 | 40880/50000 [26:56:24<4:50:52,  1.91s/it]
Outer training loop:  82%|########1 | 40881/50000 [26:56:26<4:48:01,  1.90s/it]
Outer training loop:  82%|########1 | 40882/50000 [26:56:28<4:59:43,  1.97s/it]
Outer training loop:  82%|########1 | 40883/50000 [26:56:30<4:57:17,  1.96s/it]
Outer training loop:  82%|########1 | 40884/50000 [26:56:32<4:47:16,  1.89s/it]
Outer training loop:  82%|########1 | 40885/50000 [26:56:33<4:49:22,  1.90s/it]
Outer training loop:  82%|########1 | 40886/50000 [26:56:35<4:46:52,  1.89s/it]
Outer training loop:  82%|########1 | 40887/50000 [26:56:37<4:44:09,  1.87s/it]
Outer training loop:  82%|########1 | 40888/50000 [26:56:39<4:41:18,  1.85s/it]
Outer training loop:  82%|########1 | 40889/50000 [26:56:41<4:50:38,  1.91s/it]
Outer training loop:  82%|########1 | 40890/50000 [26:56:43<4:51:21,  1.92s/it]
Outer training loop:  82%|########1 | 40891/50000 [26:56:45<4:45:01,  1.88s/it]
Outer training loop:  82%|########1 | 40892/50000 [26:56:47<4:48:25,  1.90s/it]
Outer training loop:  82%|########1 | 40893/50000 [26:56:48<4:39:37,  1.84s/it]
Outer training loop:  82%|########1 | 40894/50000 [26:56:50<4:43:37,  1.87s/it]
Outer training loop:  82%|########1 | 40895/50000 [26:56:52<4:46:07,  1.89s/it]
Outer training loop:  82%|########1 | 40896/50000 [26:56:54<4:47:43,  1.90s/it]
Outer training loop:  82%|########1 | 40897/50000 [26:56:56<4:51:08,  1.92s/it]
Outer training loop:  82%|########1 | 40898/50000 [26:56:58<4:49:22,  1.91s/it]
Outer training loop:  82%|########1 | 40899/50000 [26:57:00<4:48:18,  1.90s/it]
Outer training loop:  82%|########1 | 40900/50000 [26:57:02<4:45:31,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.97s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  82%|########1 | 40901/50000 [26:57:09<8:48:58,  3.49s/it]
Outer training loop:  82%|########1 | 40902/50000 [26:57:11<7:40:54,  3.04s/it]
Outer training loop:  82%|########1 | 40903/50000 [26:57:13<6:59:43,  2.77s/it]
Outer training loop:  82%|########1 | 40904/50000 [26:57:15<6:15:15,  2.48s/it]
Outer training loop:  82%|########1 | 40905/50000 [26:57:17<5:42:12,  2.26s/it]
Outer training loop:  82%|########1 | 40906/50000 [26:57:19<5:37:41,  2.23s/it]
Outer training loop:  82%|########1 | 40907/50000 [26:57:21<5:20:53,  2.12s/it]
Outer training loop:  82%|########1 | 40908/50000 [26:57:23<5:09:23,  2.04s/it]
Outer training loop:  82%|########1 | 40909/50000 [26:57:24<4:56:13,  1.96s/it]
Outer training loop:  82%|########1 | 40910/50000 [26:57:26<4:58:28,  1.97s/it]
Outer training loop:  82%|########1 | 40911/50000 [26:57:28<4:51:01,  1.92s/it]
Outer training loop:  82%|########1 | 40912/50000 [26:57:30<4:45:38,  1.89s/it]
Outer training loop:  82%|########1 | 40913/50000 [26:57:32<4:51:35,  1.93s/it]
Outer training loop:  82%|########1 | 40914/50000 [26:57:34<4:42:55,  1.87s/it]
Outer training loop:  82%|########1 | 40915/50000 [26:57:35<4:38:34,  1.84s/it]
Outer training loop:  82%|########1 | 40916/50000 [26:57:37<4:39:02,  1.84s/it]
Outer training loop:  82%|########1 | 40917/50000 [26:57:39<4:49:18,  1.91s/it]
Outer training loop:  82%|########1 | 40918/50000 [26:57:41<4:46:17,  1.89s/it]
Outer training loop:  82%|########1 | 40919/50000 [26:57:43<4:41:02,  1.86s/it]
Outer training loop:  82%|########1 | 40920/50000 [26:57:45<4:34:33,  1.81s/it]
Outer training loop:  82%|########1 | 40921/50000 [26:57:47<4:40:34,  1.85s/it]
Outer training loop:  82%|########1 | 40922/50000 [26:57:48<4:38:34,  1.84s/it]
Outer training loop:  82%|########1 | 40923/50000 [26:57:50<4:39:39,  1.85s/it]
Outer training loop:  82%|########1 | 40924/50000 [26:57:52<4:36:20,  1.83s/it]
Outer training loop:  82%|########1 | 40925/50000 [26:57:54<4:55:56,  1.96s/it]
Outer training loop:  82%|########1 | 40926/50000 [26:57:56<4:52:48,  1.94s/it]
Outer training loop:  82%|########1 | 40927/50000 [26:57:58<4:49:45,  1.92s/it]
Outer training loop:  82%|########1 | 40928/50000 [26:58:00<4:52:00,  1.93s/it]
Outer training loop:  82%|########1 | 40929/50000 [26:58:02<4:51:51,  1.93s/it]
Outer training loop:  82%|########1 | 40930/50000 [26:58:04<4:53:31,  1.94s/it]
Outer training loop:  82%|########1 | 40931/50000 [26:58:06<4:58:52,  1.98s/it]
Outer training loop:  82%|########1 | 40932/50000 [26:58:08<4:50:22,  1.92s/it]
Outer training loop:  82%|########1 | 40933/50000 [26:58:10<4:46:26,  1.90s/it]
Outer training loop:  82%|########1 | 40934/50000 [26:58:12<4:48:24,  1.91s/it]
Outer training loop:  82%|########1 | 40935/50000 [26:58:13<4:47:28,  1.90s/it]
Outer training loop:  82%|########1 | 40936/50000 [26:58:15<4:42:13,  1.87s/it]
Outer training loop:  82%|########1 | 40937/50000 [26:58:17<4:50:12,  1.92s/it]
Outer training loop:  82%|########1 | 40938/50000 [26:58:19<4:42:17,  1.87s/it]
Outer training loop:  82%|########1 | 40939/50000 [26:58:21<4:47:12,  1.90s/it]
Outer training loop:  82%|########1 | 40940/50000 [26:58:23<4:44:08,  1.88s/it]
Outer training loop:  82%|########1 | 40941/50000 [26:58:25<4:54:10,  1.95s/it]
Outer training loop:  82%|########1 | 40942/50000 [26:58:27<4:49:59,  1.92s/it]
Outer training loop:  82%|########1 | 40943/50000 [26:58:29<4:46:39,  1.90s/it]
Outer training loop:  82%|########1 | 40944/50000 [26:58:31<4:55:53,  1.96s/it]
Outer training loop:  82%|########1 | 40945/50000 [26:58:33<4:46:34,  1.90s/it]
Outer training loop:  82%|########1 | 40946/50000 [26:58:34<4:44:05,  1.88s/it]
Outer training loop:  82%|########1 | 40947/50000 [26:58:36<4:48:56,  1.91s/it]
Outer training loop:  82%|########1 | 40948/50000 [26:58:38<4:49:14,  1.92s/it]
Outer training loop:  82%|########1 | 40949/50000 [26:58:40<4:47:05,  1.90s/it]
Outer training loop:  82%|########1 | 40950/50000 [26:58:42<4:40:56,  1.86s/it]
Outer training loop:  82%|########1 | 40951/50000 [26:58:44<4:47:31,  1.91s/it]
Outer training loop:  82%|########1 | 40952/50000 [26:58:46<4:39:32,  1.85s/it]
Outer training loop:  82%|########1 | 40953/50000 [26:58:48<4:37:47,  1.84s/it]
Outer training loop:  82%|########1 | 40954/50000 [26:58:49<4:44:32,  1.89s/it]
Outer training loop:  82%|########1 | 40955/50000 [26:58:51<4:45:25,  1.89s/it]
Outer training loop:  82%|########1 | 40956/50000 [26:58:53<4:42:20,  1.87s/it]
Outer training loop:  82%|########1 | 40957/50000 [26:58:55<4:43:01,  1.88s/it]
Outer training loop:  82%|########1 | 40958/50000 [26:58:57<4:49:03,  1.92s/it]
Outer training loop:  82%|########1 | 40959/50000 [26:58:59<4:42:52,  1.88s/it]
Outer training loop:  82%|########1 | 40960/50000 [26:59:01<4:31:14,  1.80s/it]
Outer training loop:  82%|########1 | 40961/50000 [26:59:03<4:40:59,  1.87s/it]
Outer training loop:  82%|########1 | 40962/50000 [26:59:04<4:40:55,  1.87s/it]
Outer training loop:  82%|########1 | 40963/50000 [26:59:06<4:42:52,  1.88s/it]
Outer training loop:  82%|########1 | 40964/50000 [26:59:08<4:50:32,  1.93s/it]
Outer training loop:  82%|########1 | 40965/50000 [26:59:10<4:47:42,  1.91s/it]
Outer training loop:  82%|########1 | 40966/50000 [26:59:12<4:45:17,  1.89s/it]
Outer training loop:  82%|########1 | 40967/50000 [26:59:14<4:44:36,  1.89s/it]
Outer training loop:  82%|########1 | 40968/50000 [26:59:16<4:52:30,  1.94s/it]
Outer training loop:  82%|########1 | 40969/50000 [26:59:18<4:47:05,  1.91s/it]
Outer training loop:  82%|########1 | 40970/50000 [26:59:20<4:51:48,  1.94s/it]
Outer training loop:  82%|########1 | 40971/50000 [26:59:22<4:55:18,  1.96s/it]
Outer training loop:  82%|########1 | 40972/50000 [26:59:24<4:51:03,  1.93s/it]
Outer training loop:  82%|########1 | 40973/50000 [26:59:26<4:45:02,  1.89s/it]
Outer training loop:  82%|########1 | 40974/50000 [26:59:28<4:52:20,  1.94s/it]
Outer training loop:  82%|########1 | 40975/50000 [26:59:30<5:03:23,  2.02s/it]
Outer training loop:  82%|########1 | 40976/50000 [26:59:32<4:48:40,  1.92s/it]
Outer training loop:  82%|########1 | 40977/50000 [26:59:34<4:59:34,  1.99s/it]
Outer training loop:  82%|########1 | 40978/50000 [26:59:36<4:54:28,  1.96s/it]
Outer training loop:  82%|########1 | 40979/50000 [26:59:37<4:53:06,  1.95s/it]
Outer training loop:  82%|########1 | 40980/50000 [26:59:40<4:59:16,  1.99s/it]
Outer training loop:  82%|########1 | 40981/50000 [26:59:41<4:53:45,  1.95s/it]
Outer training loop:  82%|########1 | 40982/50000 [26:59:43<4:49:04,  1.92s/it]
Outer training loop:  82%|########1 | 40983/50000 [26:59:45<4:38:36,  1.85s/it]
Outer training loop:  82%|########1 | 40984/50000 [26:59:47<4:46:07,  1.90s/it]
Outer training loop:  82%|########1 | 40985/50000 [26:59:49<4:43:16,  1.89s/it]
Outer training loop:  82%|########1 | 40986/50000 [26:59:51<4:41:35,  1.87s/it]
Outer training loop:  82%|########1 | 40987/50000 [26:59:53<4:46:09,  1.90s/it]
Outer training loop:  82%|########1 | 40988/50000 [26:59:54<4:36:20,  1.84s/it]
Outer training loop:  82%|########1 | 40989/50000 [26:59:56<4:35:15,  1.83s/it]
Outer training loop:  82%|########1 | 40990/50000 [26:59:58<4:40:00,  1.86s/it]
Outer training loop:  82%|########1 | 40991/50000 [27:00:00<4:44:05,  1.89s/it]
Outer training loop:  82%|########1 | 40992/50000 [27:00:02<4:40:11,  1.87s/it]
Outer training loop:  82%|########1 | 40993/50000 [27:00:04<4:40:26,  1.87s/it]
Outer training loop:  82%|########1 | 40994/50000 [27:00:06<4:52:21,  1.95s/it]
Outer training loop:  82%|########1 | 40995/50000 [27:00:08<4:48:01,  1.92s/it]
Outer training loop:  82%|########1 | 40996/50000 [27:00:10<4:42:16,  1.88s/it]
Outer training loop:  82%|########1 | 40997/50000 [27:00:11<4:43:00,  1.89s/it]
Outer training loop:  82%|########1 | 40998/50000 [27:00:14<4:58:35,  1.99s/it]
Outer training loop:  82%|########1 | 40999/50000 [27:00:16<4:54:38,  1.96s/it]
Outer training loop:  82%|########2 | 41000/50000 [27:00:17<4:48:15,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.78s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:26,  1.81s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:25,  1.82s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:26,  1.85s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.87s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:24,  1.88s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:20,  1.90s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:20,  1.95s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:15,  1.94s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:13,  1.94s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.90s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.93s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:03,  1.98s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.92s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:55,  2.00s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:51,  1.96s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.94s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.89s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:43,  1.96s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:40,  1.94s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.90s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:31,  1.94s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.88s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.86s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:18,  1.96s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:08,  1.86s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:04,  1.94s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.90s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.88s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:52,  2.01s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:48,  1.95s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.90s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:33,  1.96s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.89s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.89s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.86s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:16,  2.01s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.97s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.93s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.81s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:19,  1.70s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:33,  1.90s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.86s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.87s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:29,  1.91s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.90s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.89s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:18,  1.85s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:17,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:17,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:12,  1.87s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.90s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:08,  1.86s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:06,  1.86s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:03,  1.85s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:08,  1.94s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.91s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:54,  1.85s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.90s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:47,  1.85s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:46,  1.87s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.92s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:43,  1.89s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:34,  1.82s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.89s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:28,  1.89s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:25,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:24,  1.93s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:21,  1.90s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:19,  1.93s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.90s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:17,  2.04s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:12,  1.96s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:09,  1.94s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:06,  1.89s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:05,  1.92s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<01:02,  2.00s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:59,  1.97s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.90s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.90s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.90s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.90s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.88s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:46,  2.02s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:43,  1.96s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.93s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:38,  1.91s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.88s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:34,  1.90s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:32,  1.89s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:28,  2.02s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:25,  1.97s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:23,  1.94s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.91s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.91s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.88s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.92s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.89s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.89s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:10,  2.00s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.94s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.89s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.90s/it]

Outer training loop:  82%|########2 | 41001/50000 [27:05:40<245:20:58, 98.15s/it]
Outer training loop:  82%|########2 | 41002/50000 [27:05:42<173:02:06, 69.23s/it]
Outer training loop:  82%|########2 | 41003/50000 [27:05:44<122:37:35, 49.07s/it]
Outer training loop:  82%|########2 | 41004/50000 [27:05:46<87:10:47, 34.89s/it] 
Outer training loop:  82%|########2 | 41005/50000 [27:05:48<62:27:31, 25.00s/it]
Outer training loop:  82%|########2 | 41006/50000 [27:05:50<45:40:21, 18.28s/it]
Outer training loop:  82%|########2 | 41007/50000 [27:05:52<33:22:31, 13.36s/it]
Outer training loop:  82%|########2 | 41008/50000 [27:05:54<24:39:51,  9.87s/it]
Outer training loop:  82%|########2 | 41009/50000 [27:05:56<18:34:03,  7.43s/it]
Outer training loop:  82%|########2 | 41010/50000 [27:05:58<14:33:34,  5.83s/it]
Outer training loop:  82%|########2 | 41011/50000 [27:05:59<11:32:05,  4.62s/it]
Outer training loop:  82%|########2 | 41012/50000 [27:06:01<9:25:38,  3.78s/it] 
Outer training loop:  82%|########2 | 41013/50000 [27:06:03<8:02:24,  3.22s/it]
Outer training loop:  82%|########2 | 41014/50000 [27:06:05<7:03:32,  2.83s/it]
Outer training loop:  82%|########2 | 41015/50000 [27:06:07<6:22:05,  2.55s/it]
Outer training loop:  82%|########2 | 41016/50000 [27:06:09<5:58:28,  2.39s/it]
Outer training loop:  82%|########2 | 41017/50000 [27:06:11<5:32:56,  2.22s/it]
Outer training loop:  82%|########2 | 41018/50000 [27:06:13<5:14:22,  2.10s/it]
Outer training loop:  82%|########2 | 41019/50000 [27:06:15<5:07:20,  2.05s/it]
Outer training loop:  82%|########2 | 41020/50000 [27:06:17<5:09:59,  2.07s/it]
Outer training loop:  82%|########2 | 41021/50000 [27:06:18<4:56:54,  1.98s/it]
Outer training loop:  82%|########2 | 41022/50000 [27:06:20<4:51:33,  1.95s/it]
Outer training loop:  82%|########2 | 41023/50000 [27:06:22<4:53:41,  1.96s/it]
Outer training loop:  82%|########2 | 41024/50000 [27:06:24<4:49:07,  1.93s/it]
Outer training loop:  82%|########2 | 41025/50000 [27:06:26<4:48:14,  1.93s/it]
Outer training loop:  82%|########2 | 41026/50000 [27:06:28<4:41:57,  1.89s/it]
Outer training loop:  82%|########2 | 41027/50000 [27:06:30<4:43:16,  1.89s/it]
Outer training loop:  82%|########2 | 41028/50000 [27:06:32<4:38:25,  1.86s/it]
Outer training loop:  82%|########2 | 41029/50000 [27:06:33<4:35:30,  1.84s/it]
Outer training loop:  82%|########2 | 41030/50000 [27:06:35<4:43:38,  1.90s/it]
Outer training loop:  82%|########2 | 41031/50000 [27:06:37<4:43:12,  1.89s/it]
Outer training loop:  82%|########2 | 41032/50000 [27:06:39<4:35:45,  1.84s/it]
Outer training loop:  82%|########2 | 41033/50000 [27:06:41<4:40:10,  1.87s/it]
Outer training loop:  82%|########2 | 41034/50000 [27:06:43<4:43:48,  1.90s/it]
Outer training loop:  82%|########2 | 41035/50000 [27:06:45<4:40:35,  1.88s/it]
Outer training loop:  82%|########2 | 41036/50000 [27:06:46<4:34:07,  1.83s/it]
Outer training loop:  82%|########2 | 41037/50000 [27:06:49<4:42:35,  1.89s/it]
Outer training loop:  82%|########2 | 41038/50000 [27:06:50<4:41:42,  1.89s/it]
Outer training loop:  82%|########2 | 41039/50000 [27:06:52<4:38:46,  1.87s/it]
Outer training loop:  82%|########2 | 41040/50000 [27:06:54<4:45:05,  1.91s/it]
Outer training loop:  82%|########2 | 41041/50000 [27:06:56<4:38:52,  1.87s/it]
Outer training loop:  82%|########2 | 41042/50000 [27:06:58<4:38:06,  1.86s/it]
Outer training loop:  82%|########2 | 41043/50000 [27:07:00<4:29:11,  1.80s/it]
Outer training loop:  82%|########2 | 41044/50000 [27:07:01<4:35:52,  1.85s/it]
Outer training loop:  82%|########2 | 41045/50000 [27:07:03<4:29:09,  1.80s/it]
Outer training loop:  82%|########2 | 41046/50000 [27:07:05<4:31:39,  1.82s/it]
Outer training loop:  82%|########2 | 41047/50000 [27:07:07<4:40:51,  1.88s/it]
Outer training loop:  82%|########2 | 41048/50000 [27:07:09<4:33:57,  1.84s/it]
Outer training loop:  82%|########2 | 41049/50000 [27:07:11<4:32:53,  1.83s/it]
Outer training loop:  82%|########2 | 41050/50000 [27:07:12<4:32:47,  1.83s/it]
Outer training loop:  82%|########2 | 41051/50000 [27:07:14<4:38:48,  1.87s/it]
Outer training loop:  82%|########2 | 41052/50000 [27:07:16<4:32:58,  1.83s/it]
Outer training loop:  82%|########2 | 41053/50000 [27:07:18<4:37:19,  1.86s/it]
Outer training loop:  82%|########2 | 41054/50000 [27:07:20<4:48:29,  1.93s/it]
Outer training loop:  82%|########2 | 41055/50000 [27:07:22<4:43:12,  1.90s/it]
Outer training loop:  82%|########2 | 41056/50000 [27:07:24<4:48:04,  1.93s/it]
Outer training loop:  82%|########2 | 41057/50000 [27:07:26<4:51:31,  1.96s/it]
Outer training loop:  82%|########2 | 41058/50000 [27:07:28<4:48:49,  1.94s/it]
Outer training loop:  82%|########2 | 41059/50000 [27:07:30<4:37:33,  1.86s/it]
Outer training loop:  82%|########2 | 41060/50000 [27:07:32<4:46:55,  1.93s/it]
Outer training loop:  82%|########2 | 41061/50000 [27:07:33<4:41:38,  1.89s/it]
Outer training loop:  82%|########2 | 41062/50000 [27:07:35<4:39:18,  1.87s/it]
Outer training loop:  82%|########2 | 41063/50000 [27:07:37<4:42:38,  1.90s/it]
Outer training loop:  82%|########2 | 41064/50000 [27:07:39<4:43:59,  1.91s/it]
Outer training loop:  82%|########2 | 41065/50000 [27:07:41<4:43:29,  1.90s/it]
Outer training loop:  82%|########2 | 41066/50000 [27:07:43<4:38:46,  1.87s/it]
Outer training loop:  82%|########2 | 41067/50000 [27:07:45<4:45:21,  1.92s/it]
Outer training loop:  82%|########2 | 41068/50000 [27:07:47<4:34:18,  1.84s/it]
Outer training loop:  82%|########2 | 41069/50000 [27:07:48<4:31:28,  1.82s/it]
Outer training loop:  82%|########2 | 41070/50000 [27:07:50<4:44:50,  1.91s/it]
Outer training loop:  82%|########2 | 41071/50000 [27:07:52<4:43:30,  1.91s/it]
Outer training loop:  82%|########2 | 41072/50000 [27:07:54<4:33:59,  1.84s/it]
Outer training loop:  82%|########2 | 41073/50000 [27:07:56<4:33:01,  1.84s/it]
Outer training loop:  82%|########2 | 41074/50000 [27:07:58<4:39:25,  1.88s/it]
Outer training loop:  82%|########2 | 41075/50000 [27:08:00<4:36:41,  1.86s/it]
Outer training loop:  82%|########2 | 41076/50000 [27:08:01<4:32:39,  1.83s/it]
Outer training loop:  82%|########2 | 41077/50000 [27:08:03<4:42:45,  1.90s/it]
Outer training loop:  82%|########2 | 41078/50000 [27:08:05<4:46:43,  1.93s/it]
Outer training loop:  82%|########2 | 41079/50000 [27:08:07<4:41:25,  1.89s/it]
Outer training loop:  82%|########2 | 41080/50000 [27:08:09<4:37:04,  1.86s/it]
Outer training loop:  82%|########2 | 41081/50000 [27:08:11<4:43:17,  1.91s/it]
Outer training loop:  82%|########2 | 41082/50000 [27:08:13<4:40:51,  1.89s/it]
Outer training loop:  82%|########2 | 41083/50000 [27:08:15<4:34:08,  1.84s/it]
Outer training loop:  82%|########2 | 41084/50000 [27:08:17<4:38:58,  1.88s/it]
Outer training loop:  82%|########2 | 41085/50000 [27:08:18<4:35:02,  1.85s/it]
Outer training loop:  82%|########2 | 41086/50000 [27:08:20<4:39:30,  1.88s/it]
Outer training loop:  82%|########2 | 41087/50000 [27:08:22<4:48:26,  1.94s/it]
Outer training loop:  82%|########2 | 41088/50000 [27:08:24<4:46:53,  1.93s/it]
Outer training loop:  82%|########2 | 41089/50000 [27:08:26<4:49:07,  1.95s/it]
Outer training loop:  82%|########2 | 41090/50000 [27:08:28<4:48:31,  1.94s/it]
Outer training loop:  82%|########2 | 41091/50000 [27:08:30<4:38:18,  1.87s/it]
Outer training loop:  82%|########2 | 41092/50000 [27:08:32<4:33:37,  1.84s/it]
Outer training loop:  82%|########2 | 41093/50000 [27:08:34<4:34:30,  1.85s/it]
Outer training loop:  82%|########2 | 41094/50000 [27:08:36<4:42:00,  1.90s/it]
Outer training loop:  82%|########2 | 41095/50000 [27:08:37<4:35:00,  1.85s/it]
Outer training loop:  82%|########2 | 41096/50000 [27:08:39<4:32:49,  1.84s/it]
Outer training loop:  82%|########2 | 41097/50000 [27:08:41<4:38:04,  1.87s/it]
Outer training loop:  82%|########2 | 41098/50000 [27:08:43<4:36:30,  1.86s/it]
Outer training loop:  82%|########2 | 41099/50000 [27:08:45<4:36:02,  1.86s/it]
Outer training loop:  82%|########2 | 41100/50000 [27:08:47<4:38:07,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  82%|########2 | 41101/50000 [27:08:54<8:38:50,  3.50s/it]
Outer training loop:  82%|########2 | 41102/50000 [27:08:56<7:28:59,  3.03s/it]
Outer training loop:  82%|########2 | 41103/50000 [27:08:58<6:45:36,  2.74s/it]
Outer training loop:  82%|########2 | 41104/50000 [27:09:00<6:00:58,  2.43s/it]
Outer training loop:  82%|########2 | 41105/50000 [27:09:02<5:42:32,  2.31s/it]
Outer training loop:  82%|########2 | 41106/50000 [27:09:04<5:17:37,  2.14s/it]
Outer training loop:  82%|########2 | 41107/50000 [27:09:05<5:08:09,  2.08s/it]
Outer training loop:  82%|########2 | 41108/50000 [27:09:08<5:09:04,  2.09s/it]
Outer training loop:  82%|########2 | 41109/50000 [27:09:10<5:03:46,  2.05s/it]
Outer training loop:  82%|########2 | 41110/50000 [27:09:11<4:49:28,  1.95s/it]
Outer training loop:  82%|########2 | 41111/50000 [27:09:13<4:52:51,  1.98s/it]
Outer training loop:  82%|########2 | 41112/50000 [27:09:15<4:40:54,  1.90s/it]
Outer training loop:  82%|########2 | 41113/50000 [27:09:17<4:35:21,  1.86s/it]
Outer training loop:  82%|########2 | 41114/50000 [27:09:19<4:31:08,  1.83s/it]
Outer training loop:  82%|########2 | 41115/50000 [27:09:21<4:47:18,  1.94s/it]
Outer training loop:  82%|########2 | 41116/50000 [27:09:23<4:40:28,  1.89s/it]
Outer training loop:  82%|########2 | 41117/50000 [27:09:24<4:39:55,  1.89s/it]
Outer training loop:  82%|########2 | 41118/50000 [27:09:26<4:47:44,  1.94s/it]
Outer training loop:  82%|########2 | 41119/50000 [27:09:28<4:42:30,  1.91s/it]
Outer training loop:  82%|########2 | 41120/50000 [27:09:30<4:37:20,  1.87s/it]
Outer training loop:  82%|########2 | 41121/50000 [27:09:32<4:41:11,  1.90s/it]
Outer training loop:  82%|########2 | 41122/50000 [27:09:34<4:49:26,  1.96s/it]
Outer training loop:  82%|########2 | 41123/50000 [27:09:36<4:42:15,  1.91s/it]
Outer training loop:  82%|########2 | 41124/50000 [27:09:38<4:36:48,  1.87s/it]
Outer training loop:  82%|########2 | 41125/50000 [27:09:40<4:41:09,  1.90s/it]
Outer training loop:  82%|########2 | 41126/50000 [27:09:41<4:33:25,  1.85s/it]
Outer training loop:  82%|########2 | 41127/50000 [27:09:43<4:34:20,  1.86s/it]
Outer training loop:  82%|########2 | 41128/50000 [27:09:45<4:32:50,  1.85s/it]
Outer training loop:  82%|########2 | 41129/50000 [27:09:47<4:38:42,  1.89s/it]
Outer training loop:  82%|########2 | 41130/50000 [27:09:49<4:35:42,  1.86s/it]
Outer training loop:  82%|########2 | 41131/50000 [27:09:51<4:39:52,  1.89s/it]
Outer training loop:  82%|########2 | 41132/50000 [27:09:53<4:44:48,  1.93s/it]
Outer training loop:  82%|########2 | 41133/50000 [27:09:55<4:40:08,  1.90s/it]
Outer training loop:  82%|########2 | 41134/50000 [27:09:56<4:32:50,  1.85s/it]
Outer training loop:  82%|########2 | 41135/50000 [27:09:58<4:39:49,  1.89s/it]
Outer training loop:  82%|########2 | 41136/50000 [27:10:00<4:36:59,  1.87s/it]
Outer training loop:  82%|########2 | 41137/50000 [27:10:02<4:36:18,  1.87s/it]
Outer training loop:  82%|########2 | 41138/50000 [27:10:04<4:39:37,  1.89s/it]
Outer training loop:  82%|########2 | 41139/50000 [27:10:06<4:41:26,  1.91s/it]
Outer training loop:  82%|########2 | 41140/50000 [27:10:08<4:37:56,  1.88s/it]
Outer training loop:  82%|########2 | 41141/50000 [27:10:10<4:32:37,  1.85s/it]
Outer training loop:  82%|########2 | 41142/50000 [27:10:12<4:38:33,  1.89s/it]
Outer training loop:  82%|########2 | 41143/50000 [27:10:13<4:34:19,  1.86s/it]
Outer training loop:  82%|########2 | 41144/50000 [27:10:15<4:30:50,  1.83s/it]
Outer training loop:  82%|########2 | 41145/50000 [27:10:17<4:37:28,  1.88s/it]
Outer training loop:  82%|########2 | 41146/50000 [27:10:19<4:37:23,  1.88s/it]
Outer training loop:  82%|########2 | 41147/50000 [27:10:21<4:37:52,  1.88s/it]
Outer training loop:  82%|########2 | 41148/50000 [27:10:23<4:31:48,  1.84s/it]
Outer training loop:  82%|########2 | 41149/50000 [27:10:25<4:43:06,  1.92s/it]
Outer training loop:  82%|########2 | 41150/50000 [27:10:27<4:42:37,  1.92s/it]
Outer training loop:  82%|########2 | 41151/50000 [27:10:28<4:39:04,  1.89s/it]
Outer training loop:  82%|########2 | 41152/50000 [27:10:30<4:36:19,  1.87s/it]
Outer training loop:  82%|########2 | 41153/50000 [27:10:32<4:49:04,  1.96s/it]
Outer training loop:  82%|########2 | 41154/50000 [27:10:34<4:43:19,  1.92s/it]
Outer training loop:  82%|########2 | 41155/50000 [27:10:36<4:38:12,  1.89s/it]
Outer training loop:  82%|########2 | 41156/50000 [27:10:38<4:43:33,  1.92s/it]
Outer training loop:  82%|########2 | 41157/50000 [27:10:40<4:44:36,  1.93s/it]
Outer training loop:  82%|########2 | 41158/50000 [27:10:42<4:44:15,  1.93s/it]
Outer training loop:  82%|########2 | 41159/50000 [27:10:44<4:54:31,  2.00s/it]
Outer training loop:  82%|########2 | 41160/50000 [27:10:46<4:42:05,  1.91s/it]
Outer training loop:  82%|########2 | 41161/50000 [27:10:48<4:39:27,  1.90s/it]
Outer training loop:  82%|########2 | 41162/50000 [27:10:50<4:48:59,  1.96s/it]
Outer training loop:  82%|########2 | 41163/50000 [27:10:52<4:38:24,  1.89s/it]
Outer training loop:  82%|########2 | 41164/50000 [27:10:53<4:36:36,  1.88s/it]
Outer training loop:  82%|########2 | 41165/50000 [27:10:55<4:35:01,  1.87s/it]
Outer training loop:  82%|########2 | 41166/50000 [27:10:57<4:39:40,  1.90s/it]
Outer training loop:  82%|########2 | 41167/50000 [27:10:59<4:35:43,  1.87s/it]
Outer training loop:  82%|########2 | 41168/50000 [27:11:01<4:29:43,  1.83s/it]
Outer training loop:  82%|########2 | 41169/50000 [27:11:03<4:36:08,  1.88s/it]
Outer training loop:  82%|########2 | 41170/50000 [27:11:05<4:32:42,  1.85s/it]
Outer training loop:  82%|########2 | 41171/50000 [27:11:06<4:32:42,  1.85s/it]
Outer training loop:  82%|########2 | 41172/50000 [27:11:08<4:31:21,  1.84s/it]
Outer training loop:  82%|########2 | 41173/50000 [27:11:10<4:38:47,  1.90s/it]
Outer training loop:  82%|########2 | 41174/50000 [27:11:12<4:37:09,  1.88s/it]
Outer training loop:  82%|########2 | 41175/50000 [27:11:14<4:36:07,  1.88s/it]
Outer training loop:  82%|########2 | 41176/50000 [27:11:16<4:38:14,  1.89s/it]
Outer training loop:  82%|########2 | 41177/50000 [27:11:18<4:34:13,  1.86s/it]
Outer training loop:  82%|########2 | 41178/50000 [27:11:20<4:31:38,  1.85s/it]
Outer training loop:  82%|########2 | 41179/50000 [27:11:21<4:34:16,  1.87s/it]
Outer training loop:  82%|########2 | 41180/50000 [27:11:23<4:36:21,  1.88s/it]
Outer training loop:  82%|########2 | 41181/50000 [27:11:25<4:34:47,  1.87s/it]
Outer training loop:  82%|########2 | 41182/50000 [27:11:27<4:31:13,  1.85s/it]
Outer training loop:  82%|########2 | 41183/50000 [27:11:29<4:33:23,  1.86s/it]
Outer training loop:  82%|########2 | 41184/50000 [27:11:31<4:34:02,  1.87s/it]
Outer training loop:  82%|########2 | 41185/50000 [27:11:32<4:29:29,  1.83s/it]
Outer training loop:  82%|########2 | 41186/50000 [27:11:35<4:41:02,  1.91s/it]
Outer training loop:  82%|########2 | 41187/50000 [27:11:36<4:36:57,  1.89s/it]
Outer training loop:  82%|########2 | 41188/50000 [27:11:38<4:35:08,  1.87s/it]
Outer training loop:  82%|########2 | 41189/50000 [27:11:40<4:34:03,  1.87s/it]
Outer training loop:  82%|########2 | 41190/50000 [27:11:42<4:36:50,  1.89s/it]
Outer training loop:  82%|########2 | 41191/50000 [27:11:44<4:42:45,  1.93s/it]
Outer training loop:  82%|########2 | 41192/50000 [27:11:46<4:36:00,  1.88s/it]
Outer training loop:  82%|########2 | 41193/50000 [27:11:48<4:40:18,  1.91s/it]
Outer training loop:  82%|########2 | 41194/50000 [27:11:50<4:37:29,  1.89s/it]
Outer training loop:  82%|########2 | 41195/50000 [27:11:51<4:33:08,  1.86s/it]
Outer training loop:  82%|########2 | 41196/50000 [27:11:53<4:31:54,  1.85s/it]
Outer training loop:  82%|########2 | 41197/50000 [27:11:55<4:40:41,  1.91s/it]
Outer training loop:  82%|########2 | 41198/50000 [27:11:57<4:33:33,  1.86s/it]
Outer training loop:  82%|########2 | 41199/50000 [27:11:59<4:30:43,  1.85s/it]
Outer training loop:  82%|########2 | 41200/50000 [27:12:01<4:41:38,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  82%|########2 | 41201/50000 [27:12:09<8:53:05,  3.64s/it]
Outer training loop:  82%|########2 | 41202/50000 [27:12:11<7:37:39,  3.12s/it]
Outer training loop:  82%|########2 | 41203/50000 [27:12:12<6:43:49,  2.75s/it]
Outer training loop:  82%|########2 | 41204/50000 [27:12:15<6:15:14,  2.56s/it]
Outer training loop:  82%|########2 | 41205/50000 [27:12:16<5:43:58,  2.35s/it]
Outer training loop:  82%|########2 | 41206/50000 [27:12:18<5:20:30,  2.19s/it]
Outer training loop:  82%|########2 | 41207/50000 [27:12:20<5:20:38,  2.19s/it]
Outer training loop:  82%|########2 | 41208/50000 [27:12:22<5:02:46,  2.07s/it]
Outer training loop:  82%|########2 | 41209/50000 [27:12:24<4:50:39,  1.98s/it]
Outer training loop:  82%|########2 | 41210/50000 [27:12:26<4:53:41,  2.00s/it]
Outer training loop:  82%|########2 | 41211/50000 [27:12:28<4:56:21,  2.02s/it]
Outer training loop:  82%|########2 | 41212/50000 [27:12:30<4:44:16,  1.94s/it]
Outer training loop:  82%|########2 | 41213/50000 [27:12:32<4:41:40,  1.92s/it]
Outer training loop:  82%|########2 | 41214/50000 [27:12:34<4:53:29,  2.00s/it]
Outer training loop:  82%|########2 | 41215/50000 [27:12:36<4:49:44,  1.98s/it]
Outer training loop:  82%|########2 | 41216/50000 [27:12:38<4:41:54,  1.93s/it]
Outer training loop:  82%|########2 | 41217/50000 [27:12:39<4:37:08,  1.89s/it]
Outer training loop:  82%|########2 | 41218/50000 [27:12:42<4:48:17,  1.97s/it]
Outer training loop:  82%|########2 | 41219/50000 [27:12:44<4:51:21,  1.99s/it]
Outer training loop:  82%|########2 | 41220/50000 [27:12:45<4:41:03,  1.92s/it]
Outer training loop:  82%|########2 | 41221/50000 [27:12:47<4:42:58,  1.93s/it]
Outer training loop:  82%|########2 | 41222/50000 [27:12:49<4:39:40,  1.91s/it]
Outer training loop:  82%|########2 | 41223/50000 [27:12:51<4:39:19,  1.91s/it]
Outer training loop:  82%|########2 | 41224/50000 [27:12:53<4:34:29,  1.88s/it]
Outer training loop:  82%|########2 | 41225/50000 [27:12:55<4:41:24,  1.92s/it]
Outer training loop:  82%|########2 | 41226/50000 [27:12:57<4:33:54,  1.87s/it]
Outer training loop:  82%|########2 | 41227/50000 [27:12:59<4:38:16,  1.90s/it]
Outer training loop:  82%|########2 | 41228/50000 [27:13:01<4:48:37,  1.97s/it]
Outer training loop:  82%|########2 | 41229/50000 [27:13:03<4:42:56,  1.94s/it]
Outer training loop:  82%|########2 | 41230/50000 [27:13:05<4:43:13,  1.94s/it]
Outer training loop:  82%|########2 | 41231/50000 [27:13:07<4:47:37,  1.97s/it]
Outer training loop:  82%|########2 | 41232/50000 [27:13:09<4:54:25,  2.01s/it]
Outer training loop:  82%|########2 | 41233/50000 [27:13:11<4:53:18,  2.01s/it]
Outer training loop:  82%|########2 | 41234/50000 [27:13:13<4:47:18,  1.97s/it]
Outer training loop:  82%|########2 | 41235/50000 [27:13:15<4:47:15,  1.97s/it]
Outer training loop:  82%|########2 | 41236/50000 [27:13:16<4:42:04,  1.93s/it]
Outer training loop:  82%|########2 | 41237/50000 [27:13:18<4:36:45,  1.89s/it]
Outer training loop:  82%|########2 | 41238/50000 [27:13:20<4:37:22,  1.90s/it]
Outer training loop:  82%|########2 | 41239/50000 [27:13:22<4:45:10,  1.95s/it]
Outer training loop:  82%|########2 | 41240/50000 [27:13:24<4:42:27,  1.93s/it]
Outer training loop:  82%|########2 | 41241/50000 [27:13:26<4:46:15,  1.96s/it]
Outer training loop:  82%|########2 | 41242/50000 [27:13:28<4:49:50,  1.99s/it]
Outer training loop:  82%|########2 | 41243/50000 [27:13:30<4:47:10,  1.97s/it]
Outer training loop:  82%|########2 | 41244/50000 [27:13:32<4:39:59,  1.92s/it]
Outer training loop:  82%|########2 | 41245/50000 [27:13:34<4:49:34,  1.98s/it]
Outer training loop:  82%|########2 | 41246/50000 [27:13:36<4:43:42,  1.94s/it]
Outer training loop:  82%|########2 | 41247/50000 [27:13:38<4:40:03,  1.92s/it]
Outer training loop:  82%|########2 | 41248/50000 [27:13:40<4:32:04,  1.87s/it]
Outer training loop:  82%|########2 | 41249/50000 [27:13:42<4:43:44,  1.95s/it]
Outer training loop:  82%|########2 | 41250/50000 [27:13:44<4:49:29,  1.99s/it]
Outer training loop:  83%|########2 | 41251/50000 [27:13:46<4:41:20,  1.93s/it]
Outer training loop:  83%|########2 | 41252/50000 [27:13:47<4:42:10,  1.94s/it]
Outer training loop:  83%|########2 | 41253/50000 [27:13:49<4:38:02,  1.91s/it]
Outer training loop:  83%|########2 | 41254/50000 [27:13:51<4:39:25,  1.92s/it]
Outer training loop:  83%|########2 | 41255/50000 [27:13:53<4:36:27,  1.90s/it]
Outer training loop:  83%|########2 | 41256/50000 [27:13:55<4:49:24,  1.99s/it]
Outer training loop:  83%|########2 | 41257/50000 [27:13:57<4:42:21,  1.94s/it]
Outer training loop:  83%|########2 | 41258/50000 [27:13:59<4:41:15,  1.93s/it]
Outer training loop:  83%|########2 | 41259/50000 [27:14:01<4:43:10,  1.94s/it]
Outer training loop:  83%|########2 | 41260/50000 [27:14:03<4:38:00,  1.91s/it]
Outer training loop:  83%|########2 | 41261/50000 [27:14:05<4:36:14,  1.90s/it]
Outer training loop:  83%|########2 | 41262/50000 [27:14:07<4:41:45,  1.93s/it]
Outer training loop:  83%|########2 | 41263/50000 [27:14:09<4:39:21,  1.92s/it]
Outer training loop:  83%|########2 | 41264/50000 [27:14:10<4:35:03,  1.89s/it]
Outer training loop:  83%|########2 | 41265/50000 [27:14:13<4:42:26,  1.94s/it]
Outer training loop:  83%|########2 | 41266/50000 [27:14:14<4:34:33,  1.89s/it]
Outer training loop:  83%|########2 | 41267/50000 [27:14:16<4:31:29,  1.87s/it]
Outer training loop:  83%|########2 | 41268/50000 [27:14:18<4:31:39,  1.87s/it]
Outer training loop:  83%|########2 | 41269/50000 [27:14:20<4:38:54,  1.92s/it]
Outer training loop:  83%|########2 | 41270/50000 [27:14:22<4:42:46,  1.94s/it]
Outer training loop:  83%|########2 | 41271/50000 [27:14:24<4:35:51,  1.90s/it]
Outer training loop:  83%|########2 | 41272/50000 [27:14:26<4:45:25,  1.96s/it]
Outer training loop:  83%|########2 | 41273/50000 [27:14:28<4:43:47,  1.95s/it]
Outer training loop:  83%|########2 | 41274/50000 [27:14:30<4:39:05,  1.92s/it]
Outer training loop:  83%|########2 | 41275/50000 [27:14:32<4:43:27,  1.95s/it]
Outer training loop:  83%|########2 | 41276/50000 [27:14:34<4:43:48,  1.95s/it]
Outer training loop:  83%|########2 | 41277/50000 [27:14:35<4:38:02,  1.91s/it]
Outer training loop:  83%|########2 | 41278/50000 [27:14:37<4:40:09,  1.93s/it]
Outer training loop:  83%|########2 | 41279/50000 [27:14:39<4:30:45,  1.86s/it]
Outer training loop:  83%|########2 | 41280/50000 [27:14:41<4:33:15,  1.88s/it]
Outer training loop:  83%|########2 | 41281/50000 [27:14:43<4:43:06,  1.95s/it]
Outer training loop:  83%|########2 | 41282/50000 [27:14:45<4:35:18,  1.89s/it]
Outer training loop:  83%|########2 | 41283/50000 [27:14:47<4:33:50,  1.88s/it]
Outer training loop:  83%|########2 | 41284/50000 [27:14:49<4:27:42,  1.84s/it]
Outer training loop:  83%|########2 | 41285/50000 [27:14:51<4:35:14,  1.89s/it]
Outer training loop:  83%|########2 | 41286/50000 [27:14:52<4:30:11,  1.86s/it]
Outer training loop:  83%|########2 | 41287/50000 [27:14:54<4:29:48,  1.86s/it]
Outer training loop:  83%|########2 | 41288/50000 [27:14:56<4:27:00,  1.84s/it]
Outer training loop:  83%|########2 | 41289/50000 [27:14:58<4:40:16,  1.93s/it]
Outer training loop:  83%|########2 | 41290/50000 [27:15:00<4:38:29,  1.92s/it]
Outer training loop:  83%|########2 | 41291/50000 [27:15:02<4:35:51,  1.90s/it]
Outer training loop:  83%|########2 | 41292/50000 [27:15:04<4:38:43,  1.92s/it]
Outer training loop:  83%|########2 | 41293/50000 [27:15:06<4:36:13,  1.90s/it]
Outer training loop:  83%|########2 | 41294/50000 [27:15:08<4:31:46,  1.87s/it]
Outer training loop:  83%|########2 | 41295/50000 [27:15:10<4:42:42,  1.95s/it]
Outer training loop:  83%|########2 | 41296/50000 [27:15:11<4:35:32,  1.90s/it]
Outer training loop:  83%|########2 | 41297/50000 [27:15:13<4:34:00,  1.89s/it]
Outer training loop:  83%|########2 | 41298/50000 [27:15:15<4:36:46,  1.91s/it]
Outer training loop:  83%|########2 | 41299/50000 [27:15:17<4:28:34,  1.85s/it]
Outer training loop:  83%|########2 | 41300/50000 [27:15:19<4:32:10,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  83%|########2 | 41301/50000 [27:15:26<8:28:04,  3.50s/it]
Outer training loop:  83%|########2 | 41302/50000 [27:15:28<7:20:04,  3.04s/it]
Outer training loop:  83%|########2 | 41303/50000 [27:15:30<6:34:08,  2.72s/it]
Outer training loop:  83%|########2 | 41304/50000 [27:15:32<5:58:33,  2.47s/it]
Outer training loop:  83%|########2 | 41305/50000 [27:15:34<5:36:09,  2.32s/it]
Outer training loop:  83%|########2 | 41306/50000 [27:15:36<5:22:18,  2.22s/it]
Outer training loop:  83%|########2 | 41307/50000 [27:15:38<5:07:28,  2.12s/it]
Outer training loop:  83%|########2 | 41308/50000 [27:15:40<4:55:23,  2.04s/it]
Outer training loop:  83%|########2 | 41309/50000 [27:15:42<4:46:31,  1.98s/it]
Outer training loop:  83%|########2 | 41310/50000 [27:15:44<4:48:48,  1.99s/it]
Outer training loop:  83%|########2 | 41311/50000 [27:15:45<4:38:50,  1.93s/it]
Outer training loop:  83%|########2 | 41312/50000 [27:15:47<4:32:22,  1.88s/it]
Outer training loop:  83%|########2 | 41313/50000 [27:15:49<4:39:48,  1.93s/it]
Outer training loop:  83%|########2 | 41314/50000 [27:15:51<4:36:29,  1.91s/it]
Outer training loop:  83%|########2 | 41315/50000 [27:15:53<4:29:53,  1.86s/it]
Outer training loop:  83%|########2 | 41316/50000 [27:15:55<4:33:13,  1.89s/it]
Outer training loop:  83%|########2 | 41317/50000 [27:15:57<4:33:44,  1.89s/it]
Outer training loop:  83%|########2 | 41318/50000 [27:15:59<4:35:34,  1.90s/it]
Outer training loop:  83%|########2 | 41319/50000 [27:16:00<4:31:58,  1.88s/it]
Outer training loop:  83%|########2 | 41320/50000 [27:16:02<4:36:59,  1.91s/it]
Outer training loop:  83%|########2 | 41321/50000 [27:16:04<4:34:52,  1.90s/it]
Outer training loop:  83%|########2 | 41322/50000 [27:16:06<4:33:31,  1.89s/it]
Outer training loop:  83%|########2 | 41323/50000 [27:16:08<4:30:32,  1.87s/it]
Outer training loop:  83%|########2 | 41324/50000 [27:16:10<4:38:15,  1.92s/it]
Outer training loop:  83%|########2 | 41325/50000 [27:16:12<4:33:02,  1.89s/it]
Outer training loop:  83%|########2 | 41326/50000 [27:16:14<4:36:47,  1.91s/it]
Outer training loop:  83%|########2 | 41327/50000 [27:16:16<4:44:06,  1.97s/it]
Outer training loop:  83%|########2 | 41328/50000 [27:16:18<4:35:54,  1.91s/it]
Outer training loop:  83%|########2 | 41329/50000 [27:16:19<4:27:08,  1.85s/it]
Outer training loop:  83%|########2 | 41330/50000 [27:16:22<4:41:35,  1.95s/it]
Outer training loop:  83%|########2 | 41331/50000 [27:16:23<4:36:47,  1.92s/it]
Outer training loop:  83%|########2 | 41332/50000 [27:16:25<4:36:42,  1.92s/it]
Outer training loop:  83%|########2 | 41333/50000 [27:16:27<4:30:05,  1.87s/it]
Outer training loop:  83%|########2 | 41334/50000 [27:16:29<4:35:40,  1.91s/it]
Outer training loop:  83%|########2 | 41335/50000 [27:16:31<4:32:55,  1.89s/it]
Outer training loop:  83%|########2 | 41336/50000 [27:16:33<4:29:07,  1.86s/it]
Outer training loop:  83%|########2 | 41337/50000 [27:16:34<4:25:55,  1.84s/it]
Outer training loop:  83%|########2 | 41338/50000 [27:16:37<4:38:47,  1.93s/it]
Outer training loop:  83%|########2 | 41339/50000 [27:16:38<4:33:02,  1.89s/it]
Outer training loop:  83%|########2 | 41340/50000 [27:16:40<4:29:00,  1.86s/it]
Outer training loop:  83%|########2 | 41341/50000 [27:16:42<4:32:32,  1.89s/it]
Outer training loop:  83%|########2 | 41342/50000 [27:16:44<4:26:25,  1.85s/it]
Outer training loop:  83%|########2 | 41343/50000 [27:16:46<4:27:53,  1.86s/it]
Outer training loop:  83%|########2 | 41344/50000 [27:16:48<4:24:20,  1.83s/it]
Outer training loop:  83%|########2 | 41345/50000 [27:16:50<4:31:20,  1.88s/it]
Outer training loop:  83%|########2 | 41346/50000 [27:16:51<4:27:29,  1.85s/it]
Outer training loop:  83%|########2 | 41347/50000 [27:16:53<4:39:32,  1.94s/it]
Outer training loop:  83%|########2 | 41348/50000 [27:16:56<4:44:03,  1.97s/it]
Outer training loop:  83%|########2 | 41349/50000 [27:16:57<4:42:01,  1.96s/it]
Outer training loop:  83%|########2 | 41350/50000 [27:16:59<4:33:47,  1.90s/it]
Outer training loop:  83%|########2 | 41351/50000 [27:17:01<4:36:41,  1.92s/it]
Outer training loop:  83%|########2 | 41352/50000 [27:17:03<4:27:49,  1.86s/it]
Outer training loop:  83%|########2 | 41353/50000 [27:17:05<4:29:16,  1.87s/it]
Outer training loop:  83%|########2 | 41354/50000 [27:17:07<4:32:20,  1.89s/it]
Outer training loop:  83%|########2 | 41355/50000 [27:17:09<4:38:17,  1.93s/it]
Outer training loop:  83%|########2 | 41356/50000 [27:17:11<4:33:08,  1.90s/it]
Outer training loop:  83%|########2 | 41357/50000 [27:17:12<4:27:49,  1.86s/it]
Outer training loop:  83%|########2 | 41358/50000 [27:17:14<4:36:48,  1.92s/it]
Outer training loop:  83%|########2 | 41359/50000 [27:17:16<4:35:22,  1.91s/it]
Outer training loop:  83%|########2 | 41360/50000 [27:17:18<4:28:42,  1.87s/it]
Outer training loop:  83%|########2 | 41361/50000 [27:17:20<4:34:21,  1.91s/it]
Outer training loop:  83%|########2 | 41362/50000 [27:17:22<4:35:15,  1.91s/it]
Outer training loop:  83%|########2 | 41363/50000 [27:17:24<4:31:58,  1.89s/it]
Outer training loop:  83%|########2 | 41364/50000 [27:17:26<4:28:31,  1.87s/it]
Outer training loop:  83%|########2 | 41365/50000 [27:17:28<4:32:45,  1.90s/it]
Outer training loop:  83%|########2 | 41366/50000 [27:17:29<4:29:14,  1.87s/it]
Outer training loop:  83%|########2 | 41367/50000 [27:17:31<4:27:04,  1.86s/it]
Outer training loop:  83%|########2 | 41368/50000 [27:17:33<4:35:16,  1.91s/it]
Outer training loop:  83%|########2 | 41369/50000 [27:17:35<4:32:09,  1.89s/it]
Outer training loop:  83%|########2 | 41370/50000 [27:17:37<4:31:39,  1.89s/it]
Outer training loop:  83%|########2 | 41371/50000 [27:17:39<4:28:19,  1.87s/it]
Outer training loop:  83%|########2 | 41372/50000 [27:17:41<4:36:11,  1.92s/it]
Outer training loop:  83%|########2 | 41373/50000 [27:17:43<4:32:22,  1.89s/it]
Outer training loop:  83%|########2 | 41374/50000 [27:17:45<4:30:35,  1.88s/it]
Outer training loop:  83%|########2 | 41375/50000 [27:17:47<4:35:15,  1.91s/it]
Outer training loop:  83%|########2 | 41376/50000 [27:17:48<4:31:10,  1.89s/it]
Outer training loop:  83%|########2 | 41377/50000 [27:17:50<4:32:25,  1.90s/it]
Outer training loop:  83%|########2 | 41378/50000 [27:17:52<4:36:43,  1.93s/it]
Outer training loop:  83%|########2 | 41379/50000 [27:17:54<4:30:40,  1.88s/it]
Outer training loop:  83%|########2 | 41380/50000 [27:17:56<4:28:19,  1.87s/it]
Outer training loop:  83%|########2 | 41381/50000 [27:17:58<4:27:19,  1.86s/it]
Outer training loop:  83%|########2 | 41382/50000 [27:18:00<4:37:57,  1.94s/it]
Outer training loop:  83%|########2 | 41383/50000 [27:18:02<4:32:56,  1.90s/it]
Outer training loop:  83%|########2 | 41384/50000 [27:18:04<4:35:11,  1.92s/it]
Outer training loop:  83%|########2 | 41385/50000 [27:18:06<4:44:43,  1.98s/it]
Outer training loop:  83%|########2 | 41386/50000 [27:18:08<4:34:41,  1.91s/it]
Outer training loop:  83%|########2 | 41387/50000 [27:18:09<4:31:58,  1.89s/it]
Outer training loop:  83%|########2 | 41388/50000 [27:18:11<4:38:05,  1.94s/it]
Outer training loop:  83%|########2 | 41389/50000 [27:18:13<4:31:10,  1.89s/it]
Outer training loop:  83%|########2 | 41390/50000 [27:18:15<4:29:16,  1.88s/it]
Outer training loop:  83%|########2 | 41391/50000 [27:18:17<4:25:33,  1.85s/it]
Outer training loop:  83%|########2 | 41392/50000 [27:18:19<4:37:46,  1.94s/it]
Outer training loop:  83%|########2 | 41393/50000 [27:18:21<4:37:30,  1.93s/it]
Outer training loop:  83%|########2 | 41394/50000 [27:18:23<4:28:41,  1.87s/it]
Outer training loop:  83%|########2 | 41395/50000 [27:18:24<4:26:10,  1.86s/it]
Outer training loop:  83%|########2 | 41396/50000 [27:18:26<4:33:55,  1.91s/it]
Outer training loop:  83%|########2 | 41397/50000 [27:18:28<4:35:23,  1.92s/it]
Outer training loop:  83%|########2 | 41398/50000 [27:18:30<4:33:52,  1.91s/it]
Outer training loop:  83%|########2 | 41399/50000 [27:18:32<4:41:30,  1.96s/it]
Outer training loop:  83%|########2 | 41400/50000 [27:18:34<4:39:35,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  83%|########2 | 41401/50000 [27:18:42<8:40:24,  3.63s/it]
Outer training loop:  83%|########2 | 41402/50000 [27:18:44<7:27:58,  3.13s/it]
Outer training loop:  83%|########2 | 41403/50000 [27:18:46<6:36:20,  2.77s/it]
Outer training loop:  83%|########2 | 41404/50000 [27:18:48<5:57:27,  2.50s/it]
Outer training loop:  83%|########2 | 41405/50000 [27:18:49<5:30:03,  2.30s/it]
Outer training loop:  83%|########2 | 41406/50000 [27:18:52<5:21:09,  2.24s/it]
Outer training loop:  83%|########2 | 41407/50000 [27:18:53<5:03:43,  2.12s/it]
Outer training loop:  83%|########2 | 41408/50000 [27:18:55<4:51:26,  2.04s/it]
Outer training loop:  83%|########2 | 41409/50000 [27:18:57<4:43:49,  1.98s/it]
Outer training loop:  83%|########2 | 41410/50000 [27:18:59<4:49:50,  2.02s/it]
Outer training loop:  83%|########2 | 41411/50000 [27:19:01<4:39:33,  1.95s/it]
Outer training loop:  83%|########2 | 41412/50000 [27:19:03<4:32:22,  1.90s/it]
Outer training loop:  83%|########2 | 41413/50000 [27:19:05<4:39:45,  1.95s/it]
Outer training loop:  83%|########2 | 41414/50000 [27:19:07<4:32:33,  1.90s/it]
Outer training loop:  83%|########2 | 41415/50000 [27:19:08<4:26:33,  1.86s/it]
Outer training loop:  83%|########2 | 41416/50000 [27:19:10<4:30:54,  1.89s/it]
Outer training loop:  83%|########2 | 41417/50000 [27:19:13<4:42:25,  1.97s/it]
Outer training loop:  83%|########2 | 41418/50000 [27:19:14<4:39:14,  1.95s/it]
Outer training loop:  83%|########2 | 41419/50000 [27:19:16<4:29:53,  1.89s/it]
Outer training loop:  83%|########2 | 41420/50000 [27:19:18<4:33:42,  1.91s/it]
Outer training loop:  83%|########2 | 41421/50000 [27:19:20<4:28:14,  1.88s/it]
Outer training loop:  83%|########2 | 41422/50000 [27:19:22<4:22:48,  1.84s/it]
Outer training loop:  83%|########2 | 41423/50000 [27:19:24<4:32:37,  1.91s/it]
Outer training loop:  83%|########2 | 41424/50000 [27:19:26<4:27:21,  1.87s/it]
Outer training loop:  83%|########2 | 41425/50000 [27:19:27<4:22:22,  1.84s/it]
Outer training loop:  83%|########2 | 41426/50000 [27:19:29<4:24:57,  1.85s/it]
Outer training loop:  83%|########2 | 41427/50000 [27:19:31<4:29:13,  1.88s/it]
Outer training loop:  83%|########2 | 41428/50000 [27:19:33<4:22:16,  1.84s/it]
Outer training loop:  83%|########2 | 41429/50000 [27:19:35<4:27:41,  1.87s/it]
Outer training loop:  83%|########2 | 41430/50000 [27:19:37<4:23:51,  1.85s/it]
Outer training loop:  83%|########2 | 41431/50000 [27:19:39<4:30:51,  1.90s/it]
Outer training loop:  83%|########2 | 41432/50000 [27:19:40<4:26:21,  1.87s/it]
Outer training loop:  83%|########2 | 41433/50000 [27:19:42<4:20:46,  1.83s/it]
Outer training loop:  83%|########2 | 41434/50000 [27:19:44<4:28:12,  1.88s/it]
Outer training loop:  83%|########2 | 41435/50000 [27:19:46<4:22:25,  1.84s/it]
Outer training loop:  83%|########2 | 41436/50000 [27:19:48<4:21:44,  1.83s/it]
Outer training loop:  83%|########2 | 41437/50000 [27:19:50<4:29:16,  1.89s/it]
Outer training loop:  83%|########2 | 41438/50000 [27:19:51<4:22:03,  1.84s/it]
Outer training loop:  83%|########2 | 41439/50000 [27:19:53<4:23:59,  1.85s/it]
Outer training loop:  83%|########2 | 41440/50000 [27:19:55<4:24:32,  1.85s/it]
Outer training loop:  83%|########2 | 41441/50000 [27:19:57<4:33:41,  1.92s/it]
Outer training loop:  83%|########2 | 41442/50000 [27:19:59<4:30:28,  1.90s/it]
Outer training loop:  83%|########2 | 41443/50000 [27:20:01<4:30:21,  1.90s/it]
Outer training loop:  83%|########2 | 41444/50000 [27:20:03<4:44:03,  1.99s/it]
Outer training loop:  83%|########2 | 41445/50000 [27:20:05<4:40:31,  1.97s/it]
Outer training loop:  83%|########2 | 41446/50000 [27:20:07<4:34:50,  1.93s/it]
Outer training loop:  83%|########2 | 41447/50000 [27:20:09<4:31:54,  1.91s/it]
Outer training loop:  83%|########2 | 41448/50000 [27:20:11<4:36:52,  1.94s/it]
Outer training loop:  83%|########2 | 41449/50000 [27:20:13<4:37:18,  1.95s/it]
Outer training loop:  83%|########2 | 41450/50000 [27:20:15<4:33:42,  1.92s/it]
Outer training loop:  83%|########2 | 41451/50000 [27:20:17<4:39:02,  1.96s/it]
Outer training loop:  83%|########2 | 41452/50000 [27:20:19<4:32:38,  1.91s/it]
Outer training loop:  83%|########2 | 41453/50000 [27:20:20<4:31:04,  1.90s/it]
Outer training loop:  83%|########2 | 41454/50000 [27:20:22<4:33:25,  1.92s/it]
Outer training loop:  83%|########2 | 41455/50000 [27:20:24<4:25:35,  1.86s/it]
Outer training loop:  83%|########2 | 41456/50000 [27:20:26<4:26:50,  1.87s/it]
Outer training loop:  83%|########2 | 41457/50000 [27:20:28<4:30:51,  1.90s/it]
Outer training loop:  83%|########2 | 41458/50000 [27:20:30<4:33:04,  1.92s/it]
Outer training loop:  83%|########2 | 41459/50000 [27:20:32<4:28:59,  1.89s/it]
Outer training loop:  83%|########2 | 41460/50000 [27:20:34<4:26:11,  1.87s/it]
Outer training loop:  83%|########2 | 41461/50000 [27:20:36<4:37:04,  1.95s/it]
Outer training loop:  83%|########2 | 41462/50000 [27:20:38<4:31:45,  1.91s/it]
Outer training loop:  83%|########2 | 41463/50000 [27:20:39<4:26:20,  1.87s/it]
Outer training loop:  83%|########2 | 41464/50000 [27:20:41<4:21:09,  1.84s/it]
Outer training loop:  83%|########2 | 41465/50000 [27:20:43<4:33:01,  1.92s/it]
Outer training loop:  83%|########2 | 41466/50000 [27:20:45<4:27:02,  1.88s/it]
Outer training loop:  83%|########2 | 41467/50000 [27:20:47<4:23:37,  1.85s/it]
Outer training loop:  83%|########2 | 41468/50000 [27:20:49<4:27:24,  1.88s/it]
Outer training loop:  83%|########2 | 41469/50000 [27:20:51<4:29:48,  1.90s/it]
Outer training loop:  83%|########2 | 41470/50000 [27:20:52<4:27:26,  1.88s/it]
Outer training loop:  83%|########2 | 41471/50000 [27:20:54<4:30:06,  1.90s/it]
Outer training loop:  83%|########2 | 41472/50000 [27:20:56<4:22:35,  1.85s/it]
Outer training loop:  83%|########2 | 41473/50000 [27:20:58<4:23:35,  1.85s/it]
Outer training loop:  83%|########2 | 41474/50000 [27:21:00<4:21:10,  1.84s/it]
Outer training loop:  83%|########2 | 41475/50000 [27:21:02<4:25:15,  1.87s/it]
Outer training loop:  83%|########2 | 41476/50000 [27:21:04<4:24:21,  1.86s/it]
Outer training loop:  83%|########2 | 41477/50000 [27:21:05<4:23:16,  1.85s/it]
Outer training loop:  83%|########2 | 41478/50000 [27:21:07<4:27:53,  1.89s/it]
Outer training loop:  83%|########2 | 41479/50000 [27:21:09<4:21:42,  1.84s/it]
Outer training loop:  83%|########2 | 41480/50000 [27:21:11<4:22:47,  1.85s/it]
Outer training loop:  83%|########2 | 41481/50000 [27:21:13<4:24:07,  1.86s/it]
Outer training loop:  83%|########2 | 41482/50000 [27:21:15<4:30:33,  1.91s/it]
Outer training loop:  83%|########2 | 41483/50000 [27:21:17<4:32:03,  1.92s/it]
Outer training loop:  83%|########2 | 41484/50000 [27:21:19<4:30:42,  1.91s/it]
Outer training loop:  83%|########2 | 41485/50000 [27:21:21<4:38:34,  1.96s/it]
Outer training loop:  83%|########2 | 41486/50000 [27:21:23<4:33:47,  1.93s/it]
Outer training loop:  83%|########2 | 41487/50000 [27:21:24<4:24:36,  1.87s/it]
Outer training loop:  83%|########2 | 41488/50000 [27:21:26<4:35:14,  1.94s/it]
Outer training loop:  83%|########2 | 41489/50000 [27:21:28<4:33:21,  1.93s/it]
Outer training loop:  83%|########2 | 41490/50000 [27:21:30<4:28:25,  1.89s/it]
Outer training loop:  83%|########2 | 41491/50000 [27:21:32<4:30:39,  1.91s/it]
Outer training loop:  83%|########2 | 41492/50000 [27:21:34<4:31:19,  1.91s/it]
Outer training loop:  83%|########2 | 41493/50000 [27:21:36<4:30:43,  1.91s/it]
Outer training loop:  83%|########2 | 41494/50000 [27:21:38<4:32:27,  1.92s/it]
Outer training loop:  83%|########2 | 41495/50000 [27:21:40<4:35:10,  1.94s/it]
Outer training loop:  83%|########2 | 41496/50000 [27:21:42<4:23:28,  1.86s/it]
Outer training loop:  83%|########2 | 41497/50000 [27:21:43<4:24:37,  1.87s/it]
Outer training loop:  83%|########2 | 41498/50000 [27:21:46<4:34:50,  1.94s/it]
Outer training loop:  83%|########2 | 41499/50000 [27:21:47<4:29:27,  1.90s/it]
Outer training loop:  83%|########2 | 41500/50000 [27:21:49<4:33:45,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:27,  1.80s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:30,  1.86s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.86s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.87s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.89s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.92s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:26,  1.98s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:21,  1.94s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:16,  1.95s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.91s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:03,  1.85s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:00,  1.83s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.91s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:44,  1.82s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:42,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:39,  1.87s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:38,  1.90s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.85s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:38,  1.97s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:32,  1.88s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:25,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:22,  1.83s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:25,  1.98s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.90s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:11,  1.83s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:10,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:07,  1.87s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.84s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:58,  1.81s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:56,  1.84s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.84s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:56,  1.93s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:53,  1.90s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:44,  1.83s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:42,  1.85s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:44,  2.00s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:40,  1.94s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.89s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.81s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:28,  1.80s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.79s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:25,  1.95s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.89s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:14,  1.87s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:12,  1.85s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.85s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.82s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:08,  2.02s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.95s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.89s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.82s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:21,  1.73s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:22,  1.75s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:19,  1.75s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:20,  1.78s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:20,  1.80s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:21,  1.84s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:36,  2.06s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:30,  2.00s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.94s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:22,  1.96s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:14,  1.95s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:07,  1.96s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.92s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.84s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.92s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:47,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:45,  1.85s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:42,  1.83s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:42,  1.90s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.85s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:32,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:35,  1.95s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.91s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:19,  1.82s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:23,  1.95s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.92s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.88s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.91s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:13,  1.98s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:09,  1.92s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:05,  1.87s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.87s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:58,  1.84s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:56,  1.84s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:56,  1.96s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:53,  1.89s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.93s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.92s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.87s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:44,  2.01s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.94s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:38,  1.91s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.90s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.84s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.85s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.86s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:28,  2.02s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.97s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.95s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.90s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.90s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.82s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.84s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.98s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.96s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.93s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.88s/it]

Outer training loop:  83%|########3 | 41501/50000 [27:27:10<230:12:36, 97.51s/it]
Outer training loop:  83%|########3 | 41502/50000 [27:27:12<162:21:09, 68.78s/it]
Outer training loop:  83%|########3 | 41503/50000 [27:27:13<114:53:16, 48.68s/it]
Outer training loop:  83%|########3 | 41504/50000 [27:27:15<81:40:53, 34.61s/it] 
Outer training loop:  83%|########3 | 41505/50000 [27:27:17<58:25:50, 24.76s/it]
Outer training loop:  83%|########3 | 41506/50000 [27:27:19<42:32:03, 18.03s/it]
Outer training loop:  83%|########3 | 41507/50000 [27:27:21<31:05:50, 13.18s/it]
Outer training loop:  83%|########3 | 41508/50000 [27:27:23<23:08:10,  9.81s/it]
Outer training loop:  83%|########3 | 41509/50000 [27:27:25<17:26:38,  7.40s/it]
Outer training loop:  83%|########3 | 41510/50000 [27:27:27<13:31:21,  5.73s/it]
Outer training loop:  83%|########3 | 41511/50000 [27:27:29<10:46:49,  4.57s/it]
Outer training loop:  83%|########3 | 41512/50000 [27:27:30<8:45:25,  3.71s/it] 
Outer training loop:  83%|########3 | 41513/50000 [27:27:32<7:24:19,  3.14s/it]
Outer training loop:  83%|########3 | 41514/50000 [27:27:34<6:37:24,  2.81s/it]
Outer training loop:  83%|########3 | 41515/50000 [27:27:36<5:54:58,  2.51s/it]
Outer training loop:  83%|########3 | 41516/50000 [27:27:38<5:23:47,  2.29s/it]
Outer training loop:  83%|########3 | 41517/50000 [27:27:40<5:07:31,  2.18s/it]
Outer training loop:  83%|########3 | 41518/50000 [27:27:41<4:50:00,  2.05s/it]
Outer training loop:  83%|########3 | 41519/50000 [27:27:43<4:40:59,  1.99s/it]
Outer training loop:  83%|########3 | 41520/50000 [27:27:45<4:42:58,  2.00s/it]
Outer training loop:  83%|########3 | 41521/50000 [27:27:47<4:33:28,  1.94s/it]
Outer training loop:  83%|########3 | 41522/50000 [27:27:49<4:29:12,  1.91s/it]
Outer training loop:  83%|########3 | 41523/50000 [27:27:51<4:21:43,  1.85s/it]
Outer training loop:  83%|########3 | 41524/50000 [27:27:53<4:30:28,  1.91s/it]
Outer training loop:  83%|########3 | 41525/50000 [27:27:55<4:35:23,  1.95s/it]
Outer training loop:  83%|########3 | 41526/50000 [27:27:56<4:25:27,  1.88s/it]
Outer training loop:  83%|########3 | 41527/50000 [27:27:58<4:31:09,  1.92s/it]
Outer training loop:  83%|########3 | 41528/50000 [27:28:00<4:27:03,  1.89s/it]
Outer training loop:  83%|########3 | 41529/50000 [27:28:02<4:26:12,  1.89s/it]
Outer training loop:  83%|########3 | 41530/50000 [27:28:04<4:26:32,  1.89s/it]
Outer training loop:  83%|########3 | 41531/50000 [27:28:06<4:33:26,  1.94s/it]
Outer training loop:  83%|########3 | 41532/50000 [27:28:08<4:31:39,  1.92s/it]
Outer training loop:  83%|########3 | 41533/50000 [27:28:10<4:29:59,  1.91s/it]
Outer training loop:  83%|########3 | 41534/50000 [27:28:12<4:30:50,  1.92s/it]
Outer training loop:  83%|########3 | 41535/50000 [27:28:14<4:32:26,  1.93s/it]
Outer training loop:  83%|########3 | 41536/50000 [27:28:16<4:25:58,  1.89s/it]
Outer training loop:  83%|########3 | 41537/50000 [27:28:17<4:27:02,  1.89s/it]
Outer training loop:  83%|########3 | 41538/50000 [27:28:19<4:30:37,  1.92s/it]
Outer training loop:  83%|########3 | 41539/50000 [27:28:21<4:32:27,  1.93s/it]
Outer training loop:  83%|########3 | 41540/50000 [27:28:23<4:30:42,  1.92s/it]
Outer training loop:  83%|########3 | 41541/50000 [27:28:25<4:34:25,  1.95s/it]
Outer training loop:  83%|########3 | 41542/50000 [27:28:27<4:29:11,  1.91s/it]
Outer training loop:  83%|########3 | 41543/50000 [27:28:29<4:29:00,  1.91s/it]
Outer training loop:  83%|########3 | 41544/50000 [27:28:31<4:32:03,  1.93s/it]
Outer training loop:  83%|########3 | 41545/50000 [27:28:33<4:24:03,  1.87s/it]
Outer training loop:  83%|########3 | 41546/50000 [27:28:35<4:19:12,  1.84s/it]
Outer training loop:  83%|########3 | 41547/50000 [27:28:36<4:19:03,  1.84s/it]
Outer training loop:  83%|########3 | 41548/50000 [27:28:38<4:21:27,  1.86s/it]
Outer training loop:  83%|########3 | 41549/50000 [27:28:40<4:22:04,  1.86s/it]
Outer training loop:  83%|########3 | 41550/50000 [27:28:42<4:19:03,  1.84s/it]
Outer training loop:  83%|########3 | 41551/50000 [27:28:44<4:28:04,  1.90s/it]
Outer training loop:  83%|########3 | 41552/50000 [27:28:46<4:27:48,  1.90s/it]
Outer training loop:  83%|########3 | 41553/50000 [27:28:48<4:20:17,  1.85s/it]
Outer training loop:  83%|########3 | 41554/50000 [27:28:49<4:14:24,  1.81s/it]
Outer training loop:  83%|########3 | 41555/50000 [27:28:51<4:26:58,  1.90s/it]
Outer training loop:  83%|########3 | 41556/50000 [27:28:53<4:23:32,  1.87s/it]
Outer training loop:  83%|########3 | 41557/50000 [27:28:55<4:23:44,  1.87s/it]
Outer training loop:  83%|########3 | 41558/50000 [27:28:57<4:28:41,  1.91s/it]
Outer training loop:  83%|########3 | 41559/50000 [27:28:59<4:25:31,  1.89s/it]
Outer training loop:  83%|########3 | 41560/50000 [27:29:01<4:21:30,  1.86s/it]
Outer training loop:  83%|########3 | 41561/50000 [27:29:02<4:15:16,  1.81s/it]
Outer training loop:  83%|########3 | 41562/50000 [27:29:04<4:21:03,  1.86s/it]
Outer training loop:  83%|########3 | 41563/50000 [27:29:06<4:20:22,  1.85s/it]
Outer training loop:  83%|########3 | 41564/50000 [27:29:08<4:15:52,  1.82s/it]
Outer training loop:  83%|########3 | 41565/50000 [27:29:10<4:25:57,  1.89s/it]
Outer training loop:  83%|########3 | 41566/50000 [27:29:12<4:21:57,  1.86s/it]
Outer training loop:  83%|########3 | 41567/50000 [27:29:14<4:20:28,  1.85s/it]
Outer training loop:  83%|########3 | 41568/50000 [27:29:16<4:28:02,  1.91s/it]
Outer training loop:  83%|########3 | 41569/50000 [27:29:17<4:22:59,  1.87s/it]
Outer training loop:  83%|########3 | 41570/50000 [27:29:19<4:18:39,  1.84s/it]
Outer training loop:  83%|########3 | 41571/50000 [27:29:21<4:28:48,  1.91s/it]
Outer training loop:  83%|########3 | 41572/50000 [27:29:23<4:28:07,  1.91s/it]
Outer training loop:  83%|########3 | 41573/50000 [27:29:25<4:28:11,  1.91s/it]
Outer training loop:  83%|########3 | 41574/50000 [27:29:27<4:26:20,  1.90s/it]
Outer training loop:  83%|########3 | 41575/50000 [27:29:29<4:37:42,  1.98s/it]
Outer training loop:  83%|########3 | 41576/50000 [27:29:31<4:31:12,  1.93s/it]
Outer training loop:  83%|########3 | 41577/50000 [27:29:33<4:22:34,  1.87s/it]
Outer training loop:  83%|########3 | 41578/50000 [27:29:35<4:30:00,  1.92s/it]
Outer training loop:  83%|########3 | 41579/50000 [27:29:37<4:24:42,  1.89s/it]
Outer training loop:  83%|########3 | 41580/50000 [27:29:38<4:16:06,  1.82s/it]
Outer training loop:  83%|########3 | 41581/50000 [27:29:40<4:17:44,  1.84s/it]
Outer training loop:  83%|########3 | 41582/50000 [27:29:42<4:23:41,  1.88s/it]
Outer training loop:  83%|########3 | 41583/50000 [27:29:44<4:20:00,  1.85s/it]
Outer training loop:  83%|########3 | 41584/50000 [27:29:46<4:19:39,  1.85s/it]
Outer training loop:  83%|########3 | 41585/50000 [27:29:48<4:22:38,  1.87s/it]
Outer training loop:  83%|########3 | 41586/50000 [27:29:49<4:19:59,  1.85s/it]
Outer training loop:  83%|########3 | 41587/50000 [27:29:51<4:15:36,  1.82s/it]
Outer training loop:  83%|########3 | 41588/50000 [27:29:53<4:25:37,  1.89s/it]
Outer training loop:  83%|########3 | 41589/50000 [27:29:55<4:30:15,  1.93s/it]
Outer training loop:  83%|########3 | 41590/50000 [27:29:57<4:24:30,  1.89s/it]
Outer training loop:  83%|########3 | 41591/50000 [27:29:59<4:22:49,  1.88s/it]
Outer training loop:  83%|########3 | 41592/50000 [27:30:01<4:27:24,  1.91s/it]
Outer training loop:  83%|########3 | 41593/50000 [27:30:03<4:23:15,  1.88s/it]
Outer training loop:  83%|########3 | 41594/50000 [27:30:05<4:24:47,  1.89s/it]
Outer training loop:  83%|########3 | 41595/50000 [27:30:07<4:32:02,  1.94s/it]
Outer training loop:  83%|########3 | 41596/50000 [27:30:08<4:25:26,  1.90s/it]
Outer training loop:  83%|########3 | 41597/50000 [27:30:10<4:25:22,  1.89s/it]
Outer training loop:  83%|########3 | 41598/50000 [27:30:12<4:24:34,  1.89s/it]
Outer training loop:  83%|########3 | 41599/50000 [27:30:14<4:32:14,  1.94s/it]
Outer training loop:  83%|########3 | 41600/50000 [27:30:16<4:22:34,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.02s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.99s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.90s/it]

Outer training loop:  83%|########3 | 41601/50000 [27:30:23<8:14:23,  3.53s/it]
Outer training loop:  83%|########3 | 41602/50000 [27:30:26<7:16:32,  3.12s/it]
Outer training loop:  83%|########3 | 41603/50000 [27:30:28<6:26:30,  2.76s/it]
Outer training loop:  83%|########3 | 41604/50000 [27:30:29<5:47:41,  2.48s/it]
Outer training loop:  83%|########3 | 41605/50000 [27:30:31<5:20:23,  2.29s/it]
Outer training loop:  83%|########3 | 41606/50000 [27:30:33<5:11:52,  2.23s/it]
Outer training loop:  83%|########3 | 41607/50000 [27:30:35<5:00:05,  2.15s/it]
Outer training loop:  83%|########3 | 41608/50000 [27:30:37<4:43:43,  2.03s/it]
Outer training loop:  83%|########3 | 41609/50000 [27:30:39<4:43:48,  2.03s/it]
Outer training loop:  83%|########3 | 41610/50000 [27:30:41<4:36:52,  1.98s/it]
Outer training loop:  83%|########3 | 41611/50000 [27:30:43<4:33:47,  1.96s/it]
Outer training loop:  83%|########3 | 41612/50000 [27:30:45<4:34:13,  1.96s/it]
Outer training loop:  83%|########3 | 41613/50000 [27:30:47<4:32:37,  1.95s/it]
Outer training loop:  83%|########3 | 41614/50000 [27:30:48<4:24:27,  1.89s/it]
Outer training loop:  83%|########3 | 41615/50000 [27:30:50<4:29:13,  1.93s/it]
Outer training loop:  83%|########3 | 41616/50000 [27:30:52<4:24:16,  1.89s/it]
Outer training loop:  83%|########3 | 41617/50000 [27:30:54<4:21:23,  1.87s/it]
Outer training loop:  83%|########3 | 41618/50000 [27:30:56<4:28:32,  1.92s/it]
Outer training loop:  83%|########3 | 41619/50000 [27:30:58<4:26:09,  1.91s/it]
Outer training loop:  83%|########3 | 41620/50000 [27:31:00<4:24:50,  1.90s/it]
Outer training loop:  83%|########3 | 41621/50000 [27:31:02<4:21:12,  1.87s/it]
Outer training loop:  83%|########3 | 41622/50000 [27:31:04<4:37:55,  1.99s/it]
Outer training loop:  83%|########3 | 41623/50000 [27:31:06<4:34:21,  1.97s/it]
Outer training loop:  83%|########3 | 41624/50000 [27:31:08<4:26:54,  1.91s/it]
Outer training loop:  83%|########3 | 41625/50000 [27:31:10<4:31:00,  1.94s/it]
Outer training loop:  83%|########3 | 41626/50000 [27:31:12<4:29:31,  1.93s/it]
Outer training loop:  83%|########3 | 41627/50000 [27:31:13<4:27:38,  1.92s/it]
Outer training loop:  83%|########3 | 41628/50000 [27:31:16<4:37:51,  1.99s/it]
Outer training loop:  83%|########3 | 41629/50000 [27:31:17<4:28:59,  1.93s/it]
Outer training loop:  83%|########3 | 41630/50000 [27:31:19<4:25:53,  1.91s/it]
Outer training loop:  83%|########3 | 41631/50000 [27:31:21<4:32:29,  1.95s/it]
Outer training loop:  83%|########3 | 41632/50000 [27:31:23<4:27:43,  1.92s/it]
Outer training loop:  83%|########3 | 41633/50000 [27:31:25<4:19:00,  1.86s/it]
Outer training loop:  83%|########3 | 41634/50000 [27:31:27<4:14:51,  1.83s/it]
Outer training loop:  83%|########3 | 41635/50000 [27:31:29<4:23:04,  1.89s/it]
Outer training loop:  83%|########3 | 41636/50000 [27:31:30<4:20:32,  1.87s/it]
Outer training loop:  83%|########3 | 41637/50000 [27:31:32<4:15:24,  1.83s/it]
Outer training loop:  83%|########3 | 41638/50000 [27:31:34<4:22:04,  1.88s/it]
Outer training loop:  83%|########3 | 41639/50000 [27:31:36<4:19:30,  1.86s/it]
Outer training loop:  83%|########3 | 41640/50000 [27:31:38<4:19:11,  1.86s/it]
Outer training loop:  83%|########3 | 41641/50000 [27:31:40<4:23:02,  1.89s/it]
Outer training loop:  83%|########3 | 41642/50000 [27:31:42<4:17:39,  1.85s/it]
Outer training loop:  83%|########3 | 41643/50000 [27:31:43<4:19:07,  1.86s/it]
Outer training loop:  83%|########3 | 41644/50000 [27:31:45<4:19:01,  1.86s/it]
Outer training loop:  83%|########3 | 41645/50000 [27:31:47<4:24:55,  1.90s/it]
Outer training loop:  83%|########3 | 41646/50000 [27:31:49<4:28:08,  1.93s/it]
Outer training loop:  83%|########3 | 41647/50000 [27:31:51<4:18:22,  1.86s/it]
Outer training loop:  83%|########3 | 41648/50000 [27:31:53<4:26:53,  1.92s/it]
Outer training loop:  83%|########3 | 41649/50000 [27:31:55<4:21:49,  1.88s/it]
Outer training loop:  83%|########3 | 41650/50000 [27:31:57<4:15:59,  1.84s/it]
Outer training loop:  83%|########3 | 41651/50000 [27:31:59<4:24:56,  1.90s/it]
Outer training loop:  83%|########3 | 41652/50000 [27:32:01<4:24:13,  1.90s/it]
Outer training loop:  83%|########3 | 41653/50000 [27:32:02<4:17:07,  1.85s/it]
Outer training loop:  83%|########3 | 41654/50000 [27:32:04<4:15:24,  1.84s/it]
Outer training loop:  83%|########3 | 41655/50000 [27:32:06<4:21:50,  1.88s/it]
Outer training loop:  83%|########3 | 41656/50000 [27:32:08<4:17:25,  1.85s/it]
Outer training loop:  83%|########3 | 41657/50000 [27:32:10<4:13:09,  1.82s/it]
Outer training loop:  83%|########3 | 41658/50000 [27:32:12<4:19:24,  1.87s/it]
Outer training loop:  83%|########3 | 41659/50000 [27:32:13<4:13:35,  1.82s/it]
Outer training loop:  83%|########3 | 41660/50000 [27:32:15<4:09:26,  1.79s/it]
Outer training loop:  83%|########3 | 41661/50000 [27:32:17<4:07:14,  1.78s/it]
Outer training loop:  83%|########3 | 41662/50000 [27:32:19<4:17:21,  1.85s/it]
Outer training loop:  83%|########3 | 41663/50000 [27:32:21<4:15:14,  1.84s/it]
Outer training loop:  83%|########3 | 41664/50000 [27:32:22<4:11:27,  1.81s/it]
Outer training loop:  83%|########3 | 41665/50000 [27:32:24<4:20:15,  1.87s/it]
Outer training loop:  83%|########3 | 41666/50000 [27:32:26<4:16:01,  1.84s/it]
Outer training loop:  83%|########3 | 41667/50000 [27:32:28<4:13:05,  1.82s/it]
Outer training loop:  83%|########3 | 41668/50000 [27:32:30<4:17:12,  1.85s/it]
Outer training loop:  83%|########3 | 41669/50000 [27:32:32<4:09:31,  1.80s/it]
Outer training loop:  83%|########3 | 41670/50000 [27:32:33<4:12:29,  1.82s/it]
Outer training loop:  83%|########3 | 41671/50000 [27:32:35<4:12:04,  1.82s/it]
Outer training loop:  83%|########3 | 41672/50000 [27:32:37<4:30:08,  1.95s/it]
Outer training loop:  83%|########3 | 41673/50000 [27:32:39<4:23:24,  1.90s/it]
Outer training loop:  83%|########3 | 41674/50000 [27:32:41<4:20:00,  1.87s/it]
Outer training loop:  83%|########3 | 41675/50000 [27:32:43<4:26:38,  1.92s/it]
Outer training loop:  83%|########3 | 41676/50000 [27:32:45<4:21:00,  1.88s/it]
Outer training loop:  83%|########3 | 41677/50000 [27:32:47<4:13:03,  1.82s/it]
Outer training loop:  83%|########3 | 41678/50000 [27:32:48<4:08:46,  1.79s/it]
Outer training loop:  83%|########3 | 41679/50000 [27:32:50<4:19:17,  1.87s/it]
Outer training loop:  83%|########3 | 41680/50000 [27:32:52<4:16:37,  1.85s/it]
Outer training loop:  83%|########3 | 41681/50000 [27:32:54<4:18:52,  1.87s/it]
Outer training loop:  83%|########3 | 41682/50000 [27:32:56<4:22:54,  1.90s/it]
Outer training loop:  83%|########3 | 41683/50000 [27:32:58<4:28:44,  1.94s/it]
Outer training loop:  83%|########3 | 41684/50000 [27:33:00<4:22:52,  1.90s/it]
Outer training loop:  83%|########3 | 41685/50000 [27:33:02<4:34:24,  1.98s/it]
Outer training loop:  83%|########3 | 41686/50000 [27:33:04<4:28:10,  1.94s/it]
Outer training loop:  83%|########3 | 41687/50000 [27:33:06<4:24:45,  1.91s/it]
Outer training loop:  83%|########3 | 41688/50000 [27:33:08<4:24:44,  1.91s/it]
Outer training loop:  83%|########3 | 41689/50000 [27:33:10<4:28:19,  1.94s/it]
Outer training loop:  83%|########3 | 41690/50000 [27:33:11<4:20:20,  1.88s/it]
Outer training loop:  83%|########3 | 41691/50000 [27:33:13<4:18:56,  1.87s/it]
Outer training loop:  83%|########3 | 41692/50000 [27:33:15<4:28:48,  1.94s/it]
Outer training loop:  83%|########3 | 41693/50000 [27:33:17<4:22:22,  1.90s/it]
Outer training loop:  83%|########3 | 41694/50000 [27:33:19<4:21:51,  1.89s/it]
Outer training loop:  83%|########3 | 41695/50000 [27:33:21<4:24:33,  1.91s/it]
Outer training loop:  83%|########3 | 41696/50000 [27:33:23<4:17:30,  1.86s/it]
Outer training loop:  83%|########3 | 41697/50000 [27:33:24<4:15:10,  1.84s/it]
Outer training loop:  83%|########3 | 41698/50000 [27:33:26<4:12:47,  1.83s/it]
Outer training loop:  83%|########3 | 41699/50000 [27:33:28<4:26:08,  1.92s/it]
Outer training loop:  83%|########3 | 41700/50000 [27:33:30<4:20:53,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.99s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  83%|########3 | 41701/50000 [27:33:38<8:07:55,  3.53s/it]
Outer training loop:  83%|########3 | 41702/50000 [27:33:39<6:56:06,  3.01s/it]
Outer training loop:  83%|########3 | 41703/50000 [27:33:41<6:10:17,  2.68s/it]
Outer training loop:  83%|########3 | 41704/50000 [27:33:43<5:32:02,  2.40s/it]
Outer training loop:  83%|########3 | 41705/50000 [27:33:45<5:08:36,  2.23s/it]
Outer training loop:  83%|########3 | 41706/50000 [27:33:47<4:54:08,  2.13s/it]
Outer training loop:  83%|########3 | 41707/50000 [27:33:49<4:39:14,  2.02s/it]
Outer training loop:  83%|########3 | 41708/50000 [27:33:50<4:31:42,  1.97s/it]
Outer training loop:  83%|########3 | 41709/50000 [27:33:52<4:31:17,  1.96s/it]
Outer training loop:  83%|########3 | 41710/50000 [27:33:54<4:23:29,  1.91s/it]
Outer training loop:  83%|########3 | 41711/50000 [27:33:56<4:19:13,  1.88s/it]
Outer training loop:  83%|########3 | 41712/50000 [27:33:58<4:23:48,  1.91s/it]
Outer training loop:  83%|########3 | 41713/50000 [27:34:00<4:21:03,  1.89s/it]
Outer training loop:  83%|########3 | 41714/50000 [27:34:02<4:18:57,  1.88s/it]
Outer training loop:  83%|########3 | 41715/50000 [27:34:03<4:14:46,  1.85s/it]
Outer training loop:  83%|########3 | 41716/50000 [27:34:05<4:23:27,  1.91s/it]
Outer training loop:  83%|########3 | 41717/50000 [27:34:07<4:18:14,  1.87s/it]
Outer training loop:  83%|########3 | 41718/50000 [27:34:09<4:13:40,  1.84s/it]
Outer training loop:  83%|########3 | 41719/50000 [27:34:11<4:22:38,  1.90s/it]
Outer training loop:  83%|########3 | 41720/50000 [27:34:13<4:26:00,  1.93s/it]
Outer training loop:  83%|########3 | 41721/50000 [27:34:15<4:16:27,  1.86s/it]
Outer training loop:  83%|########3 | 41722/50000 [27:34:16<4:11:17,  1.82s/it]
Outer training loop:  83%|########3 | 41723/50000 [27:34:18<4:20:11,  1.89s/it]
Outer training loop:  83%|########3 | 41724/50000 [27:34:20<4:21:53,  1.90s/it]
Outer training loop:  83%|########3 | 41725/50000 [27:34:22<4:14:25,  1.84s/it]
Outer training loop:  83%|########3 | 41726/50000 [27:34:24<4:24:20,  1.92s/it]
Outer training loop:  83%|########3 | 41727/50000 [27:34:26<4:21:03,  1.89s/it]
Outer training loop:  83%|########3 | 41728/50000 [27:34:28<4:17:22,  1.87s/it]
Outer training loop:  83%|########3 | 41729/50000 [27:34:30<4:14:29,  1.85s/it]
Outer training loop:  83%|########3 | 41730/50000 [27:34:32<4:21:12,  1.90s/it]
Outer training loop:  83%|########3 | 41731/50000 [27:34:33<4:17:53,  1.87s/it]
Outer training loop:  83%|########3 | 41732/50000 [27:34:35<4:15:16,  1.85s/it]
Outer training loop:  83%|########3 | 41733/50000 [27:34:37<4:26:19,  1.93s/it]
Outer training loop:  83%|########3 | 41734/50000 [27:34:39<4:18:00,  1.87s/it]
Outer training loop:  83%|########3 | 41735/50000 [27:34:41<4:18:37,  1.88s/it]
Outer training loop:  83%|########3 | 41736/50000 [27:34:43<4:28:26,  1.95s/it]
Outer training loop:  83%|########3 | 41737/50000 [27:34:45<4:22:50,  1.91s/it]
Outer training loop:  83%|########3 | 41738/50000 [27:34:47<4:16:12,  1.86s/it]
Outer training loop:  83%|########3 | 41739/50000 [27:34:49<4:14:37,  1.85s/it]
Outer training loop:  83%|########3 | 41740/50000 [27:34:51<4:28:33,  1.95s/it]
Outer training loop:  83%|########3 | 41741/50000 [27:34:52<4:20:26,  1.89s/it]
Outer training loop:  83%|########3 | 41742/50000 [27:34:54<4:12:34,  1.84s/it]
Outer training loop:  83%|########3 | 41743/50000 [27:34:56<4:19:18,  1.88s/it]
Outer training loop:  83%|########3 | 41744/50000 [27:34:58<4:19:23,  1.89s/it]
Outer training loop:  83%|########3 | 41745/50000 [27:35:00<4:14:28,  1.85s/it]
Outer training loop:  83%|########3 | 41746/50000 [27:35:02<4:11:03,  1.82s/it]
Outer training loop:  83%|########3 | 41747/50000 [27:35:03<4:13:59,  1.85s/it]
Outer training loop:  83%|########3 | 41748/50000 [27:35:05<4:13:35,  1.84s/it]
Outer training loop:  83%|########3 | 41749/50000 [27:35:07<4:09:36,  1.82s/it]
Outer training loop:  84%|########3 | 41750/50000 [27:35:09<4:13:44,  1.85s/it]
Outer training loop:  84%|########3 | 41751/50000 [27:35:11<4:14:37,  1.85s/it]
Outer training loop:  84%|########3 | 41752/50000 [27:35:13<4:16:01,  1.86s/it]
Outer training loop:  84%|########3 | 41753/50000 [27:35:15<4:24:41,  1.93s/it]
Outer training loop:  84%|########3 | 41754/50000 [27:35:17<4:25:00,  1.93s/it]
Outer training loop:  84%|########3 | 41755/50000 [27:35:19<4:23:33,  1.92s/it]
Outer training loop:  84%|########3 | 41756/50000 [27:35:21<4:23:31,  1.92s/it]
Outer training loop:  84%|########3 | 41757/50000 [27:35:23<4:25:59,  1.94s/it]
Outer training loop:  84%|########3 | 41758/50000 [27:35:24<4:20:59,  1.90s/it]
Outer training loop:  84%|########3 | 41759/50000 [27:35:26<4:20:07,  1.89s/it]
Outer training loop:  84%|########3 | 41760/50000 [27:35:28<4:24:08,  1.92s/it]
Outer training loop:  84%|########3 | 41761/50000 [27:35:30<4:21:29,  1.90s/it]
Outer training loop:  84%|########3 | 41762/50000 [27:35:32<4:16:36,  1.87s/it]
Outer training loop:  84%|########3 | 41763/50000 [27:35:34<4:15:45,  1.86s/it]
Outer training loop:  84%|########3 | 41764/50000 [27:35:36<4:27:07,  1.95s/it]
Outer training loop:  84%|########3 | 41765/50000 [27:35:38<4:19:51,  1.89s/it]
Outer training loop:  84%|########3 | 41766/50000 [27:35:39<4:15:36,  1.86s/it]
Outer training loop:  84%|########3 | 41767/50000 [27:35:41<4:23:50,  1.92s/it]
Outer training loop:  84%|########3 | 41768/50000 [27:35:43<4:23:06,  1.92s/it]
Outer training loop:  84%|########3 | 41769/50000 [27:35:45<4:26:09,  1.94s/it]
Outer training loop:  84%|########3 | 41770/50000 [27:35:47<4:21:53,  1.91s/it]
Outer training loop:  84%|########3 | 41771/50000 [27:35:49<4:26:17,  1.94s/it]
Outer training loop:  84%|########3 | 41772/50000 [27:35:51<4:26:50,  1.95s/it]
Outer training loop:  84%|########3 | 41773/50000 [27:35:53<4:25:42,  1.94s/it]
Outer training loop:  84%|########3 | 41774/50000 [27:35:55<4:34:33,  2.00s/it]
Outer training loop:  84%|########3 | 41775/50000 [27:35:57<4:34:52,  2.01s/it]
Outer training loop:  84%|########3 | 41776/50000 [27:35:59<4:28:30,  1.96s/it]
Outer training loop:  84%|########3 | 41777/50000 [27:36:01<4:28:33,  1.96s/it]
Outer training loop:  84%|########3 | 41778/50000 [27:36:03<4:20:37,  1.90s/it]
Outer training loop:  84%|########3 | 41779/50000 [27:36:05<4:20:45,  1.90s/it]
Outer training loop:  84%|########3 | 41780/50000 [27:36:07<4:27:16,  1.95s/it]
Outer training loop:  84%|########3 | 41781/50000 [27:36:09<4:18:52,  1.89s/it]
Outer training loop:  84%|########3 | 41782/50000 [27:36:10<4:15:27,  1.87s/it]
Outer training loop:  84%|########3 | 41783/50000 [27:36:12<4:15:06,  1.86s/it]
Outer training loop:  84%|########3 | 41784/50000 [27:36:14<4:22:04,  1.91s/it]
Outer training loop:  84%|########3 | 41785/50000 [27:36:16<4:23:04,  1.92s/it]
Outer training loop:  84%|########3 | 41786/50000 [27:36:18<4:19:11,  1.89s/it]
Outer training loop:  84%|########3 | 41787/50000 [27:36:20<4:30:06,  1.97s/it]
Outer training loop:  84%|########3 | 41788/50000 [27:36:22<4:22:02,  1.91s/it]
Outer training loop:  84%|########3 | 41789/50000 [27:36:24<4:14:07,  1.86s/it]
Outer training loop:  84%|########3 | 41790/50000 [27:36:25<4:11:30,  1.84s/it]
Outer training loop:  84%|########3 | 41791/50000 [27:36:28<4:20:05,  1.90s/it]
Outer training loop:  84%|########3 | 41792/50000 [27:36:29<4:21:08,  1.91s/it]
Outer training loop:  84%|########3 | 41793/50000 [27:36:31<4:13:26,  1.85s/it]
Outer training loop:  84%|########3 | 41794/50000 [27:36:33<4:22:09,  1.92s/it]
Outer training loop:  84%|########3 | 41795/50000 [27:36:35<4:20:34,  1.91s/it]
Outer training loop:  84%|########3 | 41796/50000 [27:36:37<4:11:09,  1.84s/it]
Outer training loop:  84%|########3 | 41797/50000 [27:36:39<4:12:44,  1.85s/it]
Outer training loop:  84%|########3 | 41798/50000 [27:36:41<4:16:49,  1.88s/it]
Outer training loop:  84%|########3 | 41799/50000 [27:36:42<4:15:48,  1.87s/it]
Outer training loop:  84%|########3 | 41800/50000 [27:36:44<4:12:52,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  84%|########3 | 41801/50000 [27:36:52<8:08:33,  3.58s/it]
Outer training loop:  84%|########3 | 41802/50000 [27:36:54<6:55:04,  3.04s/it]
Outer training loop:  84%|########3 | 41803/50000 [27:36:55<6:04:42,  2.67s/it]
Outer training loop:  84%|########3 | 41804/50000 [27:36:57<5:26:30,  2.39s/it]
Outer training loop:  84%|########3 | 41805/50000 [27:36:59<5:10:46,  2.28s/it]
Outer training loop:  84%|########3 | 41806/50000 [27:37:01<4:54:14,  2.15s/it]
Outer training loop:  84%|########3 | 41807/50000 [27:37:03<4:38:51,  2.04s/it]
Outer training loop:  84%|########3 | 41808/50000 [27:37:05<4:36:12,  2.02s/it]
Outer training loop:  84%|########3 | 41809/50000 [27:37:07<4:29:25,  1.97s/it]
Outer training loop:  84%|########3 | 41810/50000 [27:37:08<4:21:43,  1.92s/it]
Outer training loop:  84%|########3 | 41811/50000 [27:37:11<4:25:44,  1.95s/it]
Outer training loop:  84%|########3 | 41812/50000 [27:37:12<4:21:49,  1.92s/it]
Outer training loop:  84%|########3 | 41813/50000 [27:37:14<4:18:21,  1.89s/it]
Outer training loop:  84%|########3 | 41814/50000 [27:37:16<4:12:20,  1.85s/it]
Outer training loop:  84%|########3 | 41815/50000 [27:37:18<4:13:17,  1.86s/it]
Outer training loop:  84%|########3 | 41816/50000 [27:37:20<4:21:59,  1.92s/it]
Outer training loop:  84%|########3 | 41817/50000 [27:37:22<4:21:57,  1.92s/it]
Outer training loop:  84%|########3 | 41818/50000 [27:37:24<4:25:57,  1.95s/it]
Outer training loop:  84%|########3 | 41819/50000 [27:37:26<4:22:36,  1.93s/it]
Outer training loop:  84%|########3 | 41820/50000 [27:37:28<4:22:09,  1.92s/it]
Outer training loop:  84%|########3 | 41821/50000 [27:37:29<4:17:21,  1.89s/it]
Outer training loop:  84%|########3 | 41822/50000 [27:37:31<4:23:14,  1.93s/it]
Outer training loop:  84%|########3 | 41823/50000 [27:37:33<4:24:57,  1.94s/it]
Outer training loop:  84%|########3 | 41824/50000 [27:37:35<4:22:06,  1.92s/it]
Outer training loop:  84%|########3 | 41825/50000 [27:37:37<4:28:06,  1.97s/it]
Outer training loop:  84%|########3 | 41826/50000 [27:37:39<4:20:54,  1.92s/it]
Outer training loop:  84%|########3 | 41827/50000 [27:37:41<4:21:06,  1.92s/it]
Outer training loop:  84%|########3 | 41828/50000 [27:37:43<4:18:48,  1.90s/it]
Outer training loop:  84%|########3 | 41829/50000 [27:37:45<4:23:28,  1.93s/it]
Outer training loop:  84%|########3 | 41830/50000 [27:37:47<4:19:18,  1.90s/it]
Outer training loop:  84%|########3 | 41831/50000 [27:37:49<4:14:15,  1.87s/it]
Outer training loop:  84%|########3 | 41832/50000 [27:37:51<4:27:48,  1.97s/it]
Outer training loop:  84%|########3 | 41833/50000 [27:37:53<4:23:32,  1.94s/it]
Outer training loop:  84%|########3 | 41834/50000 [27:37:54<4:17:37,  1.89s/it]
Outer training loop:  84%|########3 | 41835/50000 [27:37:56<4:20:44,  1.92s/it]
Outer training loop:  84%|########3 | 41836/50000 [27:37:58<4:12:39,  1.86s/it]
Outer training loop:  84%|########3 | 41837/50000 [27:38:00<4:06:59,  1.82s/it]
Outer training loop:  84%|########3 | 41838/50000 [27:38:02<4:13:56,  1.87s/it]
Outer training loop:  84%|########3 | 41839/50000 [27:38:04<4:11:45,  1.85s/it]
Outer training loop:  84%|########3 | 41840/50000 [27:38:06<4:13:59,  1.87s/it]
Outer training loop:  84%|########3 | 41841/50000 [27:38:07<4:09:57,  1.84s/it]
Outer training loop:  84%|########3 | 41842/50000 [27:38:09<4:11:46,  1.85s/it]
Outer training loop:  84%|########3 | 41843/50000 [27:38:11<4:07:54,  1.82s/it]
Outer training loop:  84%|########3 | 41844/50000 [27:38:13<4:09:08,  1.83s/it]
Outer training loop:  84%|########3 | 41845/50000 [27:38:15<4:17:40,  1.90s/it]
Outer training loop:  84%|########3 | 41846/50000 [27:38:17<4:13:03,  1.86s/it]
Outer training loop:  84%|########3 | 41847/50000 [27:38:18<4:07:45,  1.82s/it]
Outer training loop:  84%|########3 | 41848/50000 [27:38:20<4:16:16,  1.89s/it]
Outer training loop:  84%|########3 | 41849/50000 [27:38:22<4:19:56,  1.91s/it]
Outer training loop:  84%|########3 | 41850/50000 [27:38:24<4:19:59,  1.91s/it]
Outer training loop:  84%|########3 | 41851/50000 [27:38:26<4:18:16,  1.90s/it]
Outer training loop:  84%|########3 | 41852/50000 [27:38:28<4:23:03,  1.94s/it]
Outer training loop:  84%|########3 | 41853/50000 [27:38:30<4:16:13,  1.89s/it]
Outer training loop:  84%|########3 | 41854/50000 [27:38:32<4:09:45,  1.84s/it]
Outer training loop:  84%|########3 | 41855/50000 [27:38:34<4:16:47,  1.89s/it]
Outer training loop:  84%|########3 | 41856/50000 [27:38:36<4:21:34,  1.93s/it]
Outer training loop:  84%|########3 | 41857/50000 [27:38:37<4:15:21,  1.88s/it]
Outer training loop:  84%|########3 | 41858/50000 [27:38:40<4:23:46,  1.94s/it]
Outer training loop:  84%|########3 | 41859/50000 [27:38:41<4:21:32,  1.93s/it]
Outer training loop:  84%|########3 | 41860/50000 [27:38:43<4:17:43,  1.90s/it]
Outer training loop:  84%|########3 | 41861/50000 [27:38:45<4:23:54,  1.95s/it]
Outer training loop:  84%|########3 | 41862/50000 [27:38:47<4:16:12,  1.89s/it]
Outer training loop:  84%|########3 | 41863/50000 [27:38:49<4:12:12,  1.86s/it]
Outer training loop:  84%|########3 | 41864/50000 [27:38:51<4:09:46,  1.84s/it]
Outer training loop:  84%|########3 | 41865/50000 [27:38:53<4:21:25,  1.93s/it]
Outer training loop:  84%|########3 | 41866/50000 [27:38:55<4:24:23,  1.95s/it]
Outer training loop:  84%|########3 | 41867/50000 [27:38:57<4:17:52,  1.90s/it]
Outer training loop:  84%|########3 | 41868/50000 [27:38:59<4:22:45,  1.94s/it]
Outer training loop:  84%|########3 | 41869/50000 [27:39:00<4:15:40,  1.89s/it]
Outer training loop:  84%|########3 | 41870/50000 [27:39:02<4:14:14,  1.88s/it]
Outer training loop:  84%|########3 | 41871/50000 [27:39:04<4:18:52,  1.91s/it]
Outer training loop:  84%|########3 | 41872/50000 [27:39:06<4:14:01,  1.88s/it]
Outer training loop:  84%|########3 | 41873/50000 [27:39:08<4:07:49,  1.83s/it]
Outer training loop:  84%|########3 | 41874/50000 [27:39:10<4:06:52,  1.82s/it]
Outer training loop:  84%|########3 | 41875/50000 [27:39:12<4:13:58,  1.88s/it]
Outer training loop:  84%|########3 | 41876/50000 [27:39:13<4:12:10,  1.86s/it]
Outer training loop:  84%|########3 | 41877/50000 [27:39:15<4:07:06,  1.83s/it]
Outer training loop:  84%|########3 | 41878/50000 [27:39:17<4:10:46,  1.85s/it]
Outer training loop:  84%|########3 | 41879/50000 [27:39:19<4:11:14,  1.86s/it]
Outer training loop:  84%|########3 | 41880/50000 [27:39:21<4:13:42,  1.87s/it]
Outer training loop:  84%|########3 | 41881/50000 [27:39:23<4:07:06,  1.83s/it]
Outer training loop:  84%|########3 | 41882/50000 [27:39:25<4:16:16,  1.89s/it]
Outer training loop:  84%|########3 | 41883/50000 [27:39:26<4:09:25,  1.84s/it]
Outer training loop:  84%|########3 | 41884/50000 [27:39:28<4:14:01,  1.88s/it]
Outer training loop:  84%|########3 | 41885/50000 [27:39:30<4:17:34,  1.90s/it]
Outer training loop:  84%|########3 | 41886/50000 [27:39:32<4:11:40,  1.86s/it]
Outer training loop:  84%|########3 | 41887/50000 [27:39:34<4:15:38,  1.89s/it]
Outer training loop:  84%|########3 | 41888/50000 [27:39:36<4:21:23,  1.93s/it]
Outer training loop:  84%|########3 | 41889/50000 [27:39:38<4:14:52,  1.89s/it]
Outer training loop:  84%|########3 | 41890/50000 [27:39:40<4:13:58,  1.88s/it]
Outer training loop:  84%|########3 | 41891/50000 [27:39:42<4:15:52,  1.89s/it]
Outer training loop:  84%|########3 | 41892/50000 [27:39:43<4:11:32,  1.86s/it]
Outer training loop:  84%|########3 | 41893/50000 [27:39:45<4:13:51,  1.88s/it]
Outer training loop:  84%|########3 | 41894/50000 [27:39:47<4:03:39,  1.80s/it]
Outer training loop:  84%|########3 | 41895/50000 [27:39:49<4:12:54,  1.87s/it]
Outer training loop:  84%|########3 | 41896/50000 [27:39:51<4:14:51,  1.89s/it]
Outer training loop:  84%|########3 | 41897/50000 [27:39:53<4:12:26,  1.87s/it]
Outer training loop:  84%|########3 | 41898/50000 [27:39:55<4:18:37,  1.92s/it]
Outer training loop:  84%|########3 | 41899/50000 [27:39:57<4:14:41,  1.89s/it]
Outer training loop:  84%|########3 | 41900/50000 [27:39:58<4:14:36,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.87s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  84%|########3 | 41901/50000 [27:40:06<7:51:41,  3.49s/it]
Outer training loop:  84%|########3 | 41902/50000 [27:40:08<6:49:31,  3.03s/it]
Outer training loop:  84%|########3 | 41903/50000 [27:40:09<5:57:39,  2.65s/it]
Outer training loop:  84%|########3 | 41904/50000 [27:40:11<5:27:29,  2.43s/it]
Outer training loop:  84%|########3 | 41905/50000 [27:40:13<4:59:06,  2.22s/it]
Outer training loop:  84%|########3 | 41906/50000 [27:40:15<4:52:04,  2.17s/it]
Outer training loop:  84%|########3 | 41907/50000 [27:40:17<4:38:47,  2.07s/it]
Outer training loop:  84%|########3 | 41908/50000 [27:40:19<4:27:53,  1.99s/it]
Outer training loop:  84%|########3 | 41909/50000 [27:40:21<4:30:16,  2.00s/it]
Outer training loop:  84%|########3 | 41910/50000 [27:40:22<4:18:03,  1.91s/it]
Outer training loop:  84%|########3 | 41911/50000 [27:40:24<4:17:03,  1.91s/it]
Outer training loop:  84%|########3 | 41912/50000 [27:40:26<4:12:31,  1.87s/it]
Outer training loop:  84%|########3 | 41913/50000 [27:40:29<4:34:39,  2.04s/it]
Outer training loop:  84%|########3 | 41914/50000 [27:40:30<4:24:51,  1.97s/it]
Outer training loop:  84%|########3 | 41915/50000 [27:40:32<4:14:16,  1.89s/it]
Outer training loop:  84%|########3 | 41916/50000 [27:40:34<4:19:04,  1.92s/it]
Outer training loop:  84%|########3 | 41917/50000 [27:40:36<4:18:07,  1.92s/it]
Outer training loop:  84%|########3 | 41918/50000 [27:40:38<4:14:54,  1.89s/it]
Outer training loop:  84%|########3 | 41919/50000 [27:40:40<4:27:00,  1.98s/it]
Outer training loop:  84%|########3 | 41920/50000 [27:40:42<4:20:28,  1.93s/it]
Outer training loop:  84%|########3 | 41921/50000 [27:40:44<4:15:59,  1.90s/it]
Outer training loop:  84%|########3 | 41922/50000 [27:40:46<4:16:00,  1.90s/it]
Outer training loop:  84%|########3 | 41923/50000 [27:40:48<4:18:31,  1.92s/it]
Outer training loop:  84%|########3 | 41924/50000 [27:40:49<4:17:26,  1.91s/it]
Outer training loop:  84%|########3 | 41925/50000 [27:40:51<4:09:11,  1.85s/it]
Outer training loop:  84%|########3 | 41926/50000 [27:40:53<4:16:02,  1.90s/it]
Outer training loop:  84%|########3 | 41927/50000 [27:40:55<4:13:38,  1.89s/it]
Outer training loop:  84%|########3 | 41928/50000 [27:40:57<4:10:29,  1.86s/it]
Outer training loop:  84%|########3 | 41929/50000 [27:40:59<4:08:33,  1.85s/it]
Outer training loop:  84%|########3 | 41930/50000 [27:41:01<4:16:38,  1.91s/it]
Outer training loop:  84%|########3 | 41931/50000 [27:41:02<4:10:01,  1.86s/it]
Outer training loop:  84%|########3 | 41932/50000 [27:41:04<4:11:21,  1.87s/it]
Outer training loop:  84%|########3 | 41933/50000 [27:41:06<4:17:01,  1.91s/it]
Outer training loop:  84%|########3 | 41934/50000 [27:41:08<4:20:43,  1.94s/it]
Outer training loop:  84%|########3 | 41935/50000 [27:41:10<4:20:20,  1.94s/it]
Outer training loop:  84%|########3 | 41936/50000 [27:41:12<4:15:44,  1.90s/it]
Outer training loop:  84%|########3 | 41937/50000 [27:41:14<4:24:25,  1.97s/it]
Outer training loop:  84%|########3 | 41938/50000 [27:41:16<4:16:50,  1.91s/it]
Outer training loop:  84%|########3 | 41939/50000 [27:41:18<4:10:07,  1.86s/it]
Outer training loop:  84%|########3 | 41940/50000 [27:41:20<4:18:40,  1.93s/it]
Outer training loop:  84%|########3 | 41941/50000 [27:41:22<4:17:36,  1.92s/it]
Outer training loop:  84%|########3 | 41942/50000 [27:41:23<4:11:44,  1.87s/it]
Outer training loop:  84%|########3 | 41943/50000 [27:41:25<4:10:56,  1.87s/it]
Outer training loop:  84%|########3 | 41944/50000 [27:41:27<4:22:23,  1.95s/it]
Outer training loop:  84%|########3 | 41945/50000 [27:41:29<4:17:50,  1.92s/it]
Outer training loop:  84%|########3 | 41946/50000 [27:41:31<4:15:34,  1.90s/it]
Outer training loop:  84%|########3 | 41947/50000 [27:41:33<4:17:14,  1.92s/it]
Outer training loop:  84%|########3 | 41948/50000 [27:41:35<4:18:18,  1.92s/it]
Outer training loop:  84%|########3 | 41949/50000 [27:41:37<4:08:14,  1.85s/it]
Outer training loop:  84%|########3 | 41950/50000 [27:41:38<4:04:31,  1.82s/it]
Outer training loop:  84%|########3 | 41951/50000 [27:41:41<4:15:54,  1.91s/it]
Outer training loop:  84%|########3 | 41952/50000 [27:41:42<4:15:44,  1.91s/it]
Outer training loop:  84%|########3 | 41953/50000 [27:41:44<4:15:10,  1.90s/it]
Outer training loop:  84%|########3 | 41954/50000 [27:41:46<4:17:45,  1.92s/it]
Outer training loop:  84%|########3 | 41955/50000 [27:41:48<4:16:15,  1.91s/it]
Outer training loop:  84%|########3 | 41956/50000 [27:41:50<4:15:03,  1.90s/it]
Outer training loop:  84%|########3 | 41957/50000 [27:41:52<4:13:24,  1.89s/it]
Outer training loop:  84%|########3 | 41958/50000 [27:41:54<4:17:10,  1.92s/it]
Outer training loop:  84%|########3 | 41959/50000 [27:41:56<4:13:21,  1.89s/it]
Outer training loop:  84%|########3 | 41960/50000 [27:41:58<4:13:37,  1.89s/it]
Outer training loop:  84%|########3 | 41961/50000 [27:42:00<4:17:14,  1.92s/it]
Outer training loop:  84%|########3 | 41962/50000 [27:42:01<4:13:10,  1.89s/it]
Outer training loop:  84%|########3 | 41963/50000 [27:42:04<4:18:18,  1.93s/it]
Outer training loop:  84%|########3 | 41964/50000 [27:42:06<4:25:21,  1.98s/it]
Outer training loop:  84%|########3 | 41965/50000 [27:42:07<4:17:31,  1.92s/it]
Outer training loop:  84%|########3 | 41966/50000 [27:42:09<4:13:17,  1.89s/it]
Outer training loop:  84%|########3 | 41967/50000 [27:42:11<4:16:05,  1.91s/it]
Outer training loop:  84%|########3 | 41968/50000 [27:42:13<4:30:09,  2.02s/it]
Outer training loop:  84%|########3 | 41969/50000 [27:42:15<4:26:53,  1.99s/it]
Outer training loop:  84%|########3 | 41970/50000 [27:42:17<4:25:49,  1.99s/it]
Outer training loop:  84%|########3 | 41971/50000 [27:42:20<4:32:48,  2.04s/it]
Outer training loop:  84%|########3 | 41972/50000 [27:42:21<4:30:16,  2.02s/it]
Outer training loop:  84%|########3 | 41973/50000 [27:42:23<4:22:30,  1.96s/it]
Outer training loop:  84%|########3 | 41974/50000 [27:42:25<4:24:17,  1.98s/it]
Outer training loop:  84%|########3 | 41975/50000 [27:42:27<4:21:58,  1.96s/it]
Outer training loop:  84%|########3 | 41976/50000 [27:42:29<4:23:37,  1.97s/it]
Outer training loop:  84%|########3 | 41977/50000 [27:42:31<4:18:30,  1.93s/it]
Outer training loop:  84%|########3 | 41978/50000 [27:42:33<4:31:17,  2.03s/it]
Outer training loop:  84%|########3 | 41979/50000 [27:42:35<4:28:20,  2.01s/it]
Outer training loop:  84%|########3 | 41980/50000 [27:42:37<4:23:16,  1.97s/it]
Outer training loop:  84%|########3 | 41981/50000 [27:42:39<4:28:09,  2.01s/it]
Outer training loop:  84%|########3 | 41982/50000 [27:42:41<4:20:43,  1.95s/it]
Outer training loop:  84%|########3 | 41983/50000 [27:42:43<4:16:44,  1.92s/it]
Outer training loop:  84%|########3 | 41984/50000 [27:42:45<4:18:09,  1.93s/it]
Outer training loop:  84%|########3 | 41985/50000 [27:42:47<4:11:14,  1.88s/it]
Outer training loop:  84%|########3 | 41986/50000 [27:42:48<4:05:36,  1.84s/it]
Outer training loop:  84%|########3 | 41987/50000 [27:42:50<4:09:47,  1.87s/it]
Outer training loop:  84%|########3 | 41988/50000 [27:42:52<4:17:54,  1.93s/it]
Outer training loop:  84%|########3 | 41989/50000 [27:42:54<4:16:05,  1.92s/it]
Outer training loop:  84%|########3 | 41990/50000 [27:42:56<4:18:05,  1.93s/it]
Outer training loop:  84%|########3 | 41991/50000 [27:42:58<4:22:16,  1.96s/it]
Outer training loop:  84%|########3 | 41992/50000 [27:43:00<4:15:22,  1.91s/it]
Outer training loop:  84%|########3 | 41993/50000 [27:43:02<4:15:51,  1.92s/it]
Outer training loop:  84%|########3 | 41994/50000 [27:43:04<4:16:07,  1.92s/it]
Outer training loop:  84%|########3 | 41995/50000 [27:43:06<4:19:00,  1.94s/it]
Outer training loop:  84%|########3 | 41996/50000 [27:43:08<4:11:26,  1.88s/it]
Outer training loop:  84%|########3 | 41997/50000 [27:43:10<4:11:48,  1.89s/it]
Outer training loop:  84%|########3 | 41998/50000 [27:43:12<4:13:23,  1.90s/it]
Outer training loop:  84%|########3 | 41999/50000 [27:43:13<4:08:39,  1.86s/it]
Outer training loop:  84%|########4 | 42000/50000 [27:43:15<4:07:34,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:50,  2.08s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:42,  2.00s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:36,  1.96s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.94s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.90s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:31,  2.05s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:24,  1.97s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:20,  2.01s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:16,  1.97s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:10,  1.92s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.87s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.94s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.90s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.90s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:58,  1.95s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:55,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:50,  1.98s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.94s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:40,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:40,  1.97s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:37,  1.96s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:33,  1.90s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:27,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:23,  1.90s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:17,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:13,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:11,  1.82s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.83s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:03,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<00:59,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:56,  1.82s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:57,  1.93s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:55,  1.90s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.88s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.91s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.91s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:46,  2.01s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:42,  1.95s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:40,  1.92s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:28,  1.81s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:29,  1.97s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.89s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.89s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.83s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.88s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:12,  2.04s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.98s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.93s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.90s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:19,  1.70s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:18,  1.71s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:19,  1.74s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:20,  1.78s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:32,  1.95s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:25,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:19,  1.86s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:20,  1.90s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.88s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:55,  1.84s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.88s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.88s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:46,  1.83s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:52,  1.98s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:43,  1.88s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:35,  1.84s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:37,  1.92s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:29,  1.86s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:27,  1.85s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:22,  1.88s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:20,  1.87s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:16,  1.86s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:17,  1.99s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:15,  1.98s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:11,  1.93s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:09,  1.92s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:05,  1.87s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.87s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:03,  1.98s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.91s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.89s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.86s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:49,  1.98s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:46,  1.92s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.83s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.82s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:32,  1.94s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.93s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.89s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:23,  1.84s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.86s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.81s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.80s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.98s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.98s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.93s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.87s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.85s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.82s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.87s/it]

Outer training loop:  84%|########4 | 42001/50000 [27:48:36<216:45:17, 97.55s/it]
Outer training loop:  84%|########4 | 42002/50000 [27:48:38<152:53:26, 68.82s/it]
Outer training loop:  84%|########4 | 42003/50000 [27:48:40<108:40:55, 48.93s/it]
Outer training loop:  84%|########4 | 42004/50000 [27:48:42<77:21:07, 34.83s/it] 
Outer training loop:  84%|########4 | 42005/50000 [27:48:44<55:24:40, 24.95s/it]
Outer training loop:  84%|########4 | 42006/50000 [27:48:46<40:10:13, 18.09s/it]
Outer training loop:  84%|########4 | 42007/50000 [27:48:48<29:19:05, 13.20s/it]
Outer training loop:  84%|########4 | 42008/50000 [27:48:50<21:47:16,  9.81s/it]
Outer training loop:  84%|########4 | 42009/50000 [27:48:52<16:30:35,  7.44s/it]
Outer training loop:  84%|########4 | 42010/50000 [27:48:54<12:53:24,  5.81s/it]
Outer training loop:  84%|########4 | 42011/50000 [27:48:56<10:10:36,  4.59s/it]
Outer training loop:  84%|########4 | 42012/50000 [27:48:57<8:15:19,  3.72s/it] 
Outer training loop:  84%|########4 | 42013/50000 [27:48:59<6:59:40,  3.15s/it]
Outer training loop:  84%|########4 | 42014/50000 [27:49:01<6:12:50,  2.80s/it]
Outer training loop:  84%|########4 | 42015/50000 [27:49:03<5:33:21,  2.50s/it]
Outer training loop:  84%|########4 | 42016/50000 [27:49:05<5:11:36,  2.34s/it]
Outer training loop:  84%|########4 | 42017/50000 [27:49:07<4:56:23,  2.23s/it]
Outer training loop:  84%|########4 | 42018/50000 [27:49:09<4:39:58,  2.10s/it]
Outer training loop:  84%|########4 | 42019/50000 [27:49:10<4:28:12,  2.02s/it]
Outer training loop:  84%|########4 | 42020/50000 [27:49:12<4:28:33,  2.02s/it]
Outer training loop:  84%|########4 | 42021/50000 [27:49:14<4:28:14,  2.02s/it]
Outer training loop:  84%|########4 | 42022/50000 [27:49:16<4:20:16,  1.96s/it]
Outer training loop:  84%|########4 | 42023/50000 [27:49:18<4:17:01,  1.93s/it]
Outer training loop:  84%|########4 | 42024/50000 [27:49:20<4:28:14,  2.02s/it]
Outer training loop:  84%|########4 | 42025/50000 [27:49:22<4:22:59,  1.98s/it]
Outer training loop:  84%|########4 | 42026/50000 [27:49:24<4:17:05,  1.93s/it]
Outer training loop:  84%|########4 | 42027/50000 [27:49:26<4:12:07,  1.90s/it]
Outer training loop:  84%|########4 | 42028/50000 [27:49:28<4:22:37,  1.98s/it]
Outer training loop:  84%|########4 | 42029/50000 [27:49:30<4:15:57,  1.93s/it]
Outer training loop:  84%|########4 | 42030/50000 [27:49:32<4:09:45,  1.88s/it]
Outer training loop:  84%|########4 | 42031/50000 [27:49:33<4:08:50,  1.87s/it]
Outer training loop:  84%|########4 | 42032/50000 [27:49:35<4:13:10,  1.91s/it]
Outer training loop:  84%|########4 | 42033/50000 [27:49:37<4:08:50,  1.87s/it]
Outer training loop:  84%|########4 | 42034/50000 [27:49:39<4:03:43,  1.84s/it]
Outer training loop:  84%|########4 | 42035/50000 [27:49:41<4:15:32,  1.92s/it]
Outer training loop:  84%|########4 | 42036/50000 [27:49:43<4:12:50,  1.90s/it]
Outer training loop:  84%|########4 | 42037/50000 [27:49:45<4:09:11,  1.88s/it]
Outer training loop:  84%|########4 | 42038/50000 [27:49:47<4:10:06,  1.88s/it]
Outer training loop:  84%|########4 | 42039/50000 [27:49:48<4:02:54,  1.83s/it]
Outer training loop:  84%|########4 | 42040/50000 [27:49:50<4:05:00,  1.85s/it]
Outer training loop:  84%|########4 | 42041/50000 [27:49:52<4:09:17,  1.88s/it]
Outer training loop:  84%|########4 | 42042/50000 [27:49:54<4:04:12,  1.84s/it]
Outer training loop:  84%|########4 | 42043/50000 [27:49:56<4:02:27,  1.83s/it]
Outer training loop:  84%|########4 | 42044/50000 [27:49:58<4:21:05,  1.97s/it]
Outer training loop:  84%|########4 | 42045/50000 [27:50:00<4:15:24,  1.93s/it]
Outer training loop:  84%|########4 | 42046/50000 [27:50:02<4:06:15,  1.86s/it]
Outer training loop:  84%|########4 | 42047/50000 [27:50:03<4:06:10,  1.86s/it]
Outer training loop:  84%|########4 | 42048/50000 [27:50:06<4:12:57,  1.91s/it]
Outer training loop:  84%|########4 | 42049/50000 [27:50:07<4:09:19,  1.88s/it]
Outer training loop:  84%|########4 | 42050/50000 [27:50:09<4:09:34,  1.88s/it]
Outer training loop:  84%|########4 | 42051/50000 [27:50:11<4:14:01,  1.92s/it]
Outer training loop:  84%|########4 | 42052/50000 [27:50:13<4:10:18,  1.89s/it]
Outer training loop:  84%|########4 | 42053/50000 [27:50:15<4:05:24,  1.85s/it]
Outer training loop:  84%|########4 | 42054/50000 [27:50:17<4:04:23,  1.85s/it]
Outer training loop:  84%|########4 | 42055/50000 [27:50:19<4:08:17,  1.88s/it]
Outer training loop:  84%|########4 | 42056/50000 [27:50:20<4:09:23,  1.88s/it]
Outer training loop:  84%|########4 | 42057/50000 [27:50:22<4:03:01,  1.84s/it]
Outer training loop:  84%|########4 | 42058/50000 [27:50:24<4:16:53,  1.94s/it]
Outer training loop:  84%|########4 | 42059/50000 [27:50:26<4:18:17,  1.95s/it]
Outer training loop:  84%|########4 | 42060/50000 [27:50:28<4:16:34,  1.94s/it]
Outer training loop:  84%|########4 | 42061/50000 [27:50:30<4:11:55,  1.90s/it]
Outer training loop:  84%|########4 | 42062/50000 [27:50:32<4:14:05,  1.92s/it]
Outer training loop:  84%|########4 | 42063/50000 [27:50:34<4:15:05,  1.93s/it]
Outer training loop:  84%|########4 | 42064/50000 [27:50:36<4:11:35,  1.90s/it]
Outer training loop:  84%|########4 | 42065/50000 [27:50:38<4:12:03,  1.91s/it]
Outer training loop:  84%|########4 | 42066/50000 [27:50:39<4:03:22,  1.84s/it]
Outer training loop:  84%|########4 | 42067/50000 [27:50:41<4:01:48,  1.83s/it]
Outer training loop:  84%|########4 | 42068/50000 [27:50:43<4:01:46,  1.83s/it]
Outer training loop:  84%|########4 | 42069/50000 [27:50:45<4:07:57,  1.88s/it]
Outer training loop:  84%|########4 | 42070/50000 [27:50:47<4:03:54,  1.85s/it]
Outer training loop:  84%|########4 | 42071/50000 [27:50:49<3:59:04,  1.81s/it]
Outer training loop:  84%|########4 | 42072/50000 [27:50:50<4:03:07,  1.84s/it]
Outer training loop:  84%|########4 | 42073/50000 [27:50:53<4:11:54,  1.91s/it]
Outer training loop:  84%|########4 | 42074/50000 [27:50:54<4:12:59,  1.92s/it]
Outer training loop:  84%|########4 | 42075/50000 [27:50:56<4:09:07,  1.89s/it]
Outer training loop:  84%|########4 | 42076/50000 [27:50:58<4:13:30,  1.92s/it]
Outer training loop:  84%|########4 | 42077/50000 [27:51:00<4:10:37,  1.90s/it]
Outer training loop:  84%|########4 | 42078/50000 [27:51:02<4:08:01,  1.88s/it]
Outer training loop:  84%|########4 | 42079/50000 [27:51:04<4:13:14,  1.92s/it]
Outer training loop:  84%|########4 | 42080/50000 [27:51:06<4:10:17,  1.90s/it]
Outer training loop:  84%|########4 | 42081/50000 [27:51:08<4:08:05,  1.88s/it]
Outer training loop:  84%|########4 | 42082/50000 [27:51:10<4:06:14,  1.87s/it]
Outer training loop:  84%|########4 | 42083/50000 [27:51:11<4:10:45,  1.90s/it]
Outer training loop:  84%|########4 | 42084/50000 [27:51:13<4:04:17,  1.85s/it]
Outer training loop:  84%|########4 | 42085/50000 [27:51:15<4:04:20,  1.85s/it]
Outer training loop:  84%|########4 | 42086/50000 [27:51:17<4:08:55,  1.89s/it]
Outer training loop:  84%|########4 | 42087/50000 [27:51:19<4:10:54,  1.90s/it]
Outer training loop:  84%|########4 | 42088/50000 [27:51:21<4:09:10,  1.89s/it]
Outer training loop:  84%|########4 | 42089/50000 [27:51:23<4:16:45,  1.95s/it]
Outer training loop:  84%|########4 | 42090/50000 [27:51:25<4:12:53,  1.92s/it]
Outer training loop:  84%|########4 | 42091/50000 [27:51:27<4:09:01,  1.89s/it]
Outer training loop:  84%|########4 | 42092/50000 [27:51:29<4:17:02,  1.95s/it]
Outer training loop:  84%|########4 | 42093/50000 [27:51:30<4:06:23,  1.87s/it]
Outer training loop:  84%|########4 | 42094/50000 [27:51:32<4:15:56,  1.94s/it]
Outer training loop:  84%|########4 | 42095/50000 [27:51:34<4:15:07,  1.94s/it]
Outer training loop:  84%|########4 | 42096/50000 [27:51:36<4:18:00,  1.96s/it]
Outer training loop:  84%|########4 | 42097/50000 [27:51:38<4:12:39,  1.92s/it]
Outer training loop:  84%|########4 | 42098/50000 [27:51:40<4:13:26,  1.92s/it]
Outer training loop:  84%|########4 | 42099/50000 [27:51:42<4:17:45,  1.96s/it]
Outer training loop:  84%|########4 | 42100/50000 [27:51:44<4:10:01,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  84%|########4 | 42101/50000 [27:51:51<7:47:28,  3.55s/it]
Outer training loop:  84%|########4 | 42102/50000 [27:51:53<6:39:28,  3.03s/it]
Outer training loop:  84%|########4 | 42103/50000 [27:51:55<6:01:29,  2.75s/it]
Outer training loop:  84%|########4 | 42104/50000 [27:51:57<5:27:46,  2.49s/it]
Outer training loop:  84%|########4 | 42105/50000 [27:51:59<5:01:52,  2.29s/it]
Outer training loop:  84%|########4 | 42106/50000 [27:52:01<4:41:37,  2.14s/it]
Outer training loop:  84%|########4 | 42107/50000 [27:52:03<4:36:03,  2.10s/it]
Outer training loop:  84%|########4 | 42108/50000 [27:52:05<4:28:28,  2.04s/it]
Outer training loop:  84%|########4 | 42109/50000 [27:52:07<4:21:19,  1.99s/it]
Outer training loop:  84%|########4 | 42110/50000 [27:52:09<4:21:02,  1.99s/it]
Outer training loop:  84%|########4 | 42111/50000 [27:52:10<4:13:46,  1.93s/it]
Outer training loop:  84%|########4 | 42112/50000 [27:52:12<4:06:06,  1.87s/it]
Outer training loop:  84%|########4 | 42113/50000 [27:52:14<4:13:05,  1.93s/it]
Outer training loop:  84%|########4 | 42114/50000 [27:52:16<4:08:47,  1.89s/it]
Outer training loop:  84%|########4 | 42115/50000 [27:52:18<4:07:37,  1.88s/it]
Outer training loop:  84%|########4 | 42116/50000 [27:52:20<4:04:37,  1.86s/it]
Outer training loop:  84%|########4 | 42117/50000 [27:52:22<4:12:12,  1.92s/it]
Outer training loop:  84%|########4 | 42118/50000 [27:52:23<4:04:06,  1.86s/it]
Outer training loop:  84%|########4 | 42119/50000 [27:52:25<4:03:29,  1.85s/it]
Outer training loop:  84%|########4 | 42120/50000 [27:52:27<4:08:29,  1.89s/it]
Outer training loop:  84%|########4 | 42121/50000 [27:52:29<4:00:14,  1.83s/it]
Outer training loop:  84%|########4 | 42122/50000 [27:52:31<4:01:13,  1.84s/it]
Outer training loop:  84%|########4 | 42123/50000 [27:52:33<4:00:33,  1.83s/it]
Outer training loop:  84%|########4 | 42124/50000 [27:52:35<4:08:49,  1.90s/it]
Outer training loop:  84%|########4 | 42125/50000 [27:52:37<4:09:14,  1.90s/it]
Outer training loop:  84%|########4 | 42126/50000 [27:52:38<4:04:14,  1.86s/it]
Outer training loop:  84%|########4 | 42127/50000 [27:52:40<4:11:17,  1.92s/it]
Outer training loop:  84%|########4 | 42128/50000 [27:52:42<4:06:44,  1.88s/it]
Outer training loop:  84%|########4 | 42129/50000 [27:52:44<4:03:12,  1.85s/it]
Outer training loop:  84%|########4 | 42130/50000 [27:52:46<4:10:23,  1.91s/it]
Outer training loop:  84%|########4 | 42131/50000 [27:52:48<4:09:27,  1.90s/it]
Outer training loop:  84%|########4 | 42132/50000 [27:52:50<4:04:06,  1.86s/it]
Outer training loop:  84%|########4 | 42133/50000 [27:52:52<4:05:48,  1.87s/it]
Outer training loop:  84%|########4 | 42134/50000 [27:52:53<4:07:32,  1.89s/it]
Outer training loop:  84%|########4 | 42135/50000 [27:52:55<4:04:48,  1.87s/it]
Outer training loop:  84%|########4 | 42136/50000 [27:52:57<4:03:39,  1.86s/it]
Outer training loop:  84%|########4 | 42137/50000 [27:52:59<4:09:15,  1.90s/it]
Outer training loop:  84%|########4 | 42138/50000 [27:53:01<4:01:55,  1.85s/it]
Outer training loop:  84%|########4 | 42139/50000 [27:53:03<4:02:42,  1.85s/it]
Outer training loop:  84%|########4 | 42140/50000 [27:53:04<3:58:55,  1.82s/it]
Outer training loop:  84%|########4 | 42141/50000 [27:53:07<4:13:52,  1.94s/it]
Outer training loop:  84%|########4 | 42142/50000 [27:53:09<4:10:03,  1.91s/it]
Outer training loop:  84%|########4 | 42143/50000 [27:53:10<4:09:41,  1.91s/it]
Outer training loop:  84%|########4 | 42144/50000 [27:53:12<4:13:43,  1.94s/it]
Outer training loop:  84%|########4 | 42145/50000 [27:53:14<4:11:28,  1.92s/it]
Outer training loop:  84%|########4 | 42146/50000 [27:53:16<4:14:25,  1.94s/it]
Outer training loop:  84%|########4 | 42147/50000 [27:53:18<4:16:04,  1.96s/it]
Outer training loop:  84%|########4 | 42148/50000 [27:53:20<4:07:58,  1.89s/it]
Outer training loop:  84%|########4 | 42149/50000 [27:53:22<4:03:57,  1.86s/it]
Outer training loop:  84%|########4 | 42150/50000 [27:53:24<4:04:29,  1.87s/it]
Outer training loop:  84%|########4 | 42151/50000 [27:53:26<4:10:41,  1.92s/it]
Outer training loop:  84%|########4 | 42152/50000 [27:53:28<4:10:03,  1.91s/it]
Outer training loop:  84%|########4 | 42153/50000 [27:53:29<4:05:09,  1.87s/it]
Outer training loop:  84%|########4 | 42154/50000 [27:53:32<4:14:14,  1.94s/it]
Outer training loop:  84%|########4 | 42155/50000 [27:53:33<4:14:04,  1.94s/it]
Outer training loop:  84%|########4 | 42156/50000 [27:53:35<4:10:30,  1.92s/it]
Outer training loop:  84%|########4 | 42157/50000 [27:53:37<4:05:37,  1.88s/it]
Outer training loop:  84%|########4 | 42158/50000 [27:53:39<4:10:13,  1.91s/it]
Outer training loop:  84%|########4 | 42159/50000 [27:53:41<4:13:40,  1.94s/it]
Outer training loop:  84%|########4 | 42160/50000 [27:53:43<4:10:23,  1.92s/it]
Outer training loop:  84%|########4 | 42161/50000 [27:53:45<4:16:12,  1.96s/it]
Outer training loop:  84%|########4 | 42162/50000 [27:53:47<4:08:37,  1.90s/it]
Outer training loop:  84%|########4 | 42163/50000 [27:53:49<4:07:44,  1.90s/it]
Outer training loop:  84%|########4 | 42164/50000 [27:53:51<4:12:42,  1.93s/it]
Outer training loop:  84%|########4 | 42165/50000 [27:53:52<4:04:10,  1.87s/it]
Outer training loop:  84%|########4 | 42166/50000 [27:53:54<3:58:01,  1.82s/it]
Outer training loop:  84%|########4 | 42167/50000 [27:53:56<3:57:17,  1.82s/it]
Outer training loop:  84%|########4 | 42168/50000 [27:53:58<4:03:51,  1.87s/it]
Outer training loop:  84%|########4 | 42169/50000 [27:54:00<3:57:15,  1.82s/it]
Outer training loop:  84%|########4 | 42170/50000 [27:54:01<3:53:43,  1.79s/it]
Outer training loop:  84%|########4 | 42171/50000 [27:54:03<4:04:07,  1.87s/it]
Outer training loop:  84%|########4 | 42172/50000 [27:54:05<4:01:12,  1.85s/it]
Outer training loop:  84%|########4 | 42173/50000 [27:54:07<3:58:22,  1.83s/it]
Outer training loop:  84%|########4 | 42174/50000 [27:54:09<4:02:17,  1.86s/it]
Outer training loop:  84%|########4 | 42175/50000 [27:54:11<4:01:22,  1.85s/it]
Outer training loop:  84%|########4 | 42176/50000 [27:54:13<4:00:36,  1.85s/it]
Outer training loop:  84%|########4 | 42177/50000 [27:54:14<3:58:22,  1.83s/it]
Outer training loop:  84%|########4 | 42178/50000 [27:54:16<3:59:55,  1.84s/it]
Outer training loop:  84%|########4 | 42179/50000 [27:54:18<3:55:13,  1.80s/it]
Outer training loop:  84%|########4 | 42180/50000 [27:54:20<3:55:48,  1.81s/it]
Outer training loop:  84%|########4 | 42181/50000 [27:54:22<4:02:19,  1.86s/it]
Outer training loop:  84%|########4 | 42182/50000 [27:54:23<3:57:04,  1.82s/it]
Outer training loop:  84%|########4 | 42183/50000 [27:54:26<4:04:49,  1.88s/it]
Outer training loop:  84%|########4 | 42184/50000 [27:54:28<4:09:00,  1.91s/it]
Outer training loop:  84%|########4 | 42185/50000 [27:54:29<4:10:56,  1.93s/it]
Outer training loop:  84%|########4 | 42186/50000 [27:54:31<4:06:25,  1.89s/it]
Outer training loop:  84%|########4 | 42187/50000 [27:54:33<4:01:22,  1.85s/it]
Outer training loop:  84%|########4 | 42188/50000 [27:54:35<3:59:21,  1.84s/it]
Outer training loop:  84%|########4 | 42189/50000 [27:54:37<4:01:39,  1.86s/it]
Outer training loop:  84%|########4 | 42190/50000 [27:54:38<3:57:42,  1.83s/it]
Outer training loop:  84%|########4 | 42191/50000 [27:54:40<4:01:15,  1.85s/it]
Outer training loop:  84%|########4 | 42192/50000 [27:54:42<4:10:15,  1.92s/it]
Outer training loop:  84%|########4 | 42193/50000 [27:54:44<4:02:34,  1.86s/it]
Outer training loop:  84%|########4 | 42194/50000 [27:54:46<4:06:33,  1.90s/it]
Outer training loop:  84%|########4 | 42195/50000 [27:54:48<4:13:34,  1.95s/it]
Outer training loop:  84%|########4 | 42196/50000 [27:54:50<4:15:45,  1.97s/it]
Outer training loop:  84%|########4 | 42197/50000 [27:54:52<4:08:58,  1.91s/it]
Outer training loop:  84%|########4 | 42198/50000 [27:54:54<4:02:21,  1.86s/it]
Outer training loop:  84%|########4 | 42199/50000 [27:54:56<4:12:44,  1.94s/it]
Outer training loop:  84%|########4 | 42200/50000 [27:54:58<4:08:03,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.93s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]

Outer training loop:  84%|########4 | 42201/50000 [27:55:05<7:45:56,  3.58s/it]
Outer training loop:  84%|########4 | 42202/50000 [27:55:07<6:40:58,  3.09s/it]
Outer training loop:  84%|########4 | 42203/50000 [27:55:09<5:58:59,  2.76s/it]
Outer training loop:  84%|########4 | 42204/50000 [27:55:11<5:26:07,  2.51s/it]
Outer training loop:  84%|########4 | 42205/50000 [27:55:13<5:00:12,  2.31s/it]
Outer training loop:  84%|########4 | 42206/50000 [27:55:15<4:52:41,  2.25s/it]
Outer training loop:  84%|########4 | 42207/50000 [27:55:17<4:36:39,  2.13s/it]
Outer training loop:  84%|########4 | 42208/50000 [27:55:19<4:22:06,  2.02s/it]
Outer training loop:  84%|########4 | 42209/50000 [27:55:20<4:12:27,  1.94s/it]
Outer training loop:  84%|########4 | 42210/50000 [27:55:22<4:16:02,  1.97s/it]
Outer training loop:  84%|########4 | 42211/50000 [27:55:24<4:10:28,  1.93s/it]
Outer training loop:  84%|########4 | 42212/50000 [27:55:26<4:05:15,  1.89s/it]
Outer training loop:  84%|########4 | 42213/50000 [27:55:28<4:09:58,  1.93s/it]
Outer training loop:  84%|########4 | 42214/50000 [27:55:30<4:04:39,  1.89s/it]
Outer training loop:  84%|########4 | 42215/50000 [27:55:32<4:01:08,  1.86s/it]
Outer training loop:  84%|########4 | 42216/50000 [27:55:34<4:06:41,  1.90s/it]
Outer training loop:  84%|########4 | 42217/50000 [27:55:35<4:01:37,  1.86s/it]
Outer training loop:  84%|########4 | 42218/50000 [27:55:37<3:56:07,  1.82s/it]
Outer training loop:  84%|########4 | 42219/50000 [27:55:39<3:57:03,  1.83s/it]
Outer training loop:  84%|########4 | 42220/50000 [27:55:41<4:07:19,  1.91s/it]
Outer training loop:  84%|########4 | 42221/50000 [27:55:43<4:04:46,  1.89s/it]
Outer training loop:  84%|########4 | 42222/50000 [27:55:45<4:01:13,  1.86s/it]
Outer training loop:  84%|########4 | 42223/50000 [27:55:47<3:57:06,  1.83s/it]
Outer training loop:  84%|########4 | 42224/50000 [27:55:48<4:01:06,  1.86s/it]
Outer training loop:  84%|########4 | 42225/50000 [27:55:50<4:00:29,  1.86s/it]
Outer training loop:  84%|########4 | 42226/50000 [27:55:52<3:53:14,  1.80s/it]
Outer training loop:  84%|########4 | 42227/50000 [27:55:54<3:55:13,  1.82s/it]
Outer training loop:  84%|########4 | 42228/50000 [27:55:56<3:56:12,  1.82s/it]
Outer training loop:  84%|########4 | 42229/50000 [27:55:58<3:56:03,  1.82s/it]
Outer training loop:  84%|########4 | 42230/50000 [27:56:00<4:04:24,  1.89s/it]
Outer training loop:  84%|########4 | 42231/50000 [27:56:01<3:58:21,  1.84s/it]
Outer training loop:  84%|########4 | 42232/50000 [27:56:03<4:00:43,  1.86s/it]
Outer training loop:  84%|########4 | 42233/50000 [27:56:05<4:00:46,  1.86s/it]
Outer training loop:  84%|########4 | 42234/50000 [27:56:07<4:06:04,  1.90s/it]
Outer training loop:  84%|########4 | 42235/50000 [27:56:09<3:58:22,  1.84s/it]
Outer training loop:  84%|########4 | 42236/50000 [27:56:11<3:58:39,  1.84s/it]
Outer training loop:  84%|########4 | 42237/50000 [27:56:13<4:10:37,  1.94s/it]
Outer training loop:  84%|########4 | 42238/50000 [27:56:15<4:08:37,  1.92s/it]
Outer training loop:  84%|########4 | 42239/50000 [27:56:16<3:59:51,  1.85s/it]
Outer training loop:  84%|########4 | 42240/50000 [27:56:18<3:59:54,  1.85s/it]
Outer training loop:  84%|########4 | 42241/50000 [27:56:20<4:09:29,  1.93s/it]
Outer training loop:  84%|########4 | 42242/50000 [27:56:22<4:02:24,  1.87s/it]
Outer training loop:  84%|########4 | 42243/50000 [27:56:24<4:01:46,  1.87s/it]
Outer training loop:  84%|########4 | 42244/50000 [27:56:26<4:06:16,  1.91s/it]
Outer training loop:  84%|########4 | 42245/50000 [27:56:28<4:06:41,  1.91s/it]
Outer training loop:  84%|########4 | 42246/50000 [27:56:30<3:59:43,  1.86s/it]
Outer training loop:  84%|########4 | 42247/50000 [27:56:31<3:59:50,  1.86s/it]
Outer training loop:  84%|########4 | 42248/50000 [27:56:33<4:07:14,  1.91s/it]
Outer training loop:  84%|########4 | 42249/50000 [27:56:35<4:08:44,  1.93s/it]
Outer training loop:  84%|########4 | 42250/50000 [27:56:37<4:03:21,  1.88s/it]
Outer training loop:  85%|########4 | 42251/50000 [27:56:39<4:07:03,  1.91s/it]
Outer training loop:  85%|########4 | 42252/50000 [27:56:41<4:08:25,  1.92s/it]
Outer training loop:  85%|########4 | 42253/50000 [27:56:43<4:06:05,  1.91s/it]
Outer training loop:  85%|########4 | 42254/50000 [27:56:45<4:02:17,  1.88s/it]
Outer training loop:  85%|########4 | 42255/50000 [27:56:47<4:08:02,  1.92s/it]
Outer training loop:  85%|########4 | 42256/50000 [27:56:49<4:05:10,  1.90s/it]
Outer training loop:  85%|########4 | 42257/50000 [27:56:51<4:06:21,  1.91s/it]
Outer training loop:  85%|########4 | 42258/50000 [27:56:53<4:10:07,  1.94s/it]
Outer training loop:  85%|########4 | 42259/50000 [27:56:55<4:09:10,  1.93s/it]
Outer training loop:  85%|########4 | 42260/50000 [27:56:56<4:05:14,  1.90s/it]
Outer training loop:  85%|########4 | 42261/50000 [27:56:58<4:01:45,  1.87s/it]
Outer training loop:  85%|########4 | 42262/50000 [27:57:00<4:04:15,  1.89s/it]
Outer training loop:  85%|########4 | 42263/50000 [27:57:02<4:01:11,  1.87s/it]
Outer training loop:  85%|########4 | 42264/50000 [27:57:04<3:58:52,  1.85s/it]
Outer training loop:  85%|########4 | 42265/50000 [27:57:06<4:06:38,  1.91s/it]
Outer training loop:  85%|########4 | 42266/50000 [27:57:08<4:01:52,  1.88s/it]
Outer training loop:  85%|########4 | 42267/50000 [27:57:09<3:59:34,  1.86s/it]
Outer training loop:  85%|########4 | 42268/50000 [27:57:11<3:54:02,  1.82s/it]
Outer training loop:  85%|########4 | 42269/50000 [27:57:13<4:02:28,  1.88s/it]
Outer training loop:  85%|########4 | 42270/50000 [27:57:15<3:56:29,  1.84s/it]
Outer training loop:  85%|########4 | 42271/50000 [27:57:17<3:54:37,  1.82s/it]
Outer training loop:  85%|########4 | 42272/50000 [27:57:19<4:01:33,  1.88s/it]
Outer training loop:  85%|########4 | 42273/50000 [27:57:21<4:04:48,  1.90s/it]
Outer training loop:  85%|########4 | 42274/50000 [27:57:22<4:03:41,  1.89s/it]
Outer training loop:  85%|########4 | 42275/50000 [27:57:24<4:01:14,  1.87s/it]
Outer training loop:  85%|########4 | 42276/50000 [27:57:26<4:04:48,  1.90s/it]
Outer training loop:  85%|########4 | 42277/50000 [27:57:28<4:01:37,  1.88s/it]
Outer training loop:  85%|########4 | 42278/50000 [27:57:30<3:55:23,  1.83s/it]
Outer training loop:  85%|########4 | 42279/50000 [27:57:32<3:58:46,  1.86s/it]
Outer training loop:  85%|########4 | 42280/50000 [27:57:34<4:00:17,  1.87s/it]
Outer training loop:  85%|########4 | 42281/50000 [27:57:36<4:02:54,  1.89s/it]
Outer training loop:  85%|########4 | 42282/50000 [27:57:38<4:05:58,  1.91s/it]
Outer training loop:  85%|########4 | 42283/50000 [27:57:39<4:00:18,  1.87s/it]
Outer training loop:  85%|########4 | 42284/50000 [27:57:41<3:54:44,  1.83s/it]
Outer training loop:  85%|########4 | 42285/50000 [27:57:43<3:51:08,  1.80s/it]
Outer training loop:  85%|########4 | 42286/50000 [27:57:45<3:55:47,  1.83s/it]
Outer training loop:  85%|########4 | 42287/50000 [27:57:46<3:53:44,  1.82s/it]
Outer training loop:  85%|########4 | 42288/50000 [27:57:48<3:58:34,  1.86s/it]
Outer training loop:  85%|########4 | 42289/50000 [27:57:50<4:01:39,  1.88s/it]
Outer training loop:  85%|########4 | 42290/50000 [27:57:52<3:59:09,  1.86s/it]
Outer training loop:  85%|########4 | 42291/50000 [27:57:54<3:58:45,  1.86s/it]
Outer training loop:  85%|########4 | 42292/50000 [27:57:56<4:06:09,  1.92s/it]
Outer training loop:  85%|########4 | 42293/50000 [27:57:58<4:02:21,  1.89s/it]
Outer training loop:  85%|########4 | 42294/50000 [27:58:00<3:59:08,  1.86s/it]
Outer training loop:  85%|########4 | 42295/50000 [27:58:01<3:55:48,  1.84s/it]
Outer training loop:  85%|########4 | 42296/50000 [27:58:03<4:03:43,  1.90s/it]
Outer training loop:  85%|########4 | 42297/50000 [27:58:06<4:08:16,  1.93s/it]
Outer training loop:  85%|########4 | 42298/50000 [27:58:07<4:03:47,  1.90s/it]
Outer training loop:  85%|########4 | 42299/50000 [27:58:09<4:12:42,  1.97s/it]
Outer training loop:  85%|########4 | 42300/50000 [27:58:11<4:07:23,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  85%|########4 | 42301/50000 [27:58:19<7:35:55,  3.55s/it]
Outer training loop:  85%|########4 | 42302/50000 [27:58:21<6:34:12,  3.07s/it]
Outer training loop:  85%|########4 | 42303/50000 [27:58:23<5:53:05,  2.75s/it]
Outer training loop:  85%|########4 | 42304/50000 [27:58:24<5:16:49,  2.47s/it]
Outer training loop:  85%|########4 | 42305/50000 [27:58:26<4:56:37,  2.31s/it]
Outer training loop:  85%|########4 | 42306/50000 [27:58:28<4:40:42,  2.19s/it]
Outer training loop:  85%|########4 | 42307/50000 [27:58:30<4:28:26,  2.09s/it]
Outer training loop:  85%|########4 | 42308/50000 [27:58:32<4:17:17,  2.01s/it]
Outer training loop:  85%|########4 | 42309/50000 [27:58:34<4:07:18,  1.93s/it]
Outer training loop:  85%|########4 | 42310/50000 [27:58:36<4:12:15,  1.97s/it]
Outer training loop:  85%|########4 | 42311/50000 [27:58:38<4:10:06,  1.95s/it]
Outer training loop:  85%|########4 | 42312/50000 [27:58:40<4:06:24,  1.92s/it]
Outer training loop:  85%|########4 | 42313/50000 [27:58:41<4:07:25,  1.93s/it]
Outer training loop:  85%|########4 | 42314/50000 [27:58:44<4:12:16,  1.97s/it]
Outer training loop:  85%|########4 | 42315/50000 [27:58:45<4:05:25,  1.92s/it]
Outer training loop:  85%|########4 | 42316/50000 [27:58:47<3:58:51,  1.87s/it]
Outer training loop:  85%|########4 | 42317/50000 [27:58:49<4:00:03,  1.87s/it]
Outer training loop:  85%|########4 | 42318/50000 [27:58:51<3:58:10,  1.86s/it]
Outer training loop:  85%|########4 | 42319/50000 [27:58:53<3:55:54,  1.84s/it]
Outer training loop:  85%|########4 | 42320/50000 [27:58:54<3:55:55,  1.84s/it]
Outer training loop:  85%|########4 | 42321/50000 [27:58:56<3:58:31,  1.86s/it]
Outer training loop:  85%|########4 | 42322/50000 [27:58:58<3:58:57,  1.87s/it]
Outer training loop:  85%|########4 | 42323/50000 [27:59:00<3:55:13,  1.84s/it]
Outer training loop:  85%|########4 | 42324/50000 [27:59:02<3:59:42,  1.87s/it]
Outer training loop:  85%|########4 | 42325/50000 [27:59:04<3:57:56,  1.86s/it]
Outer training loop:  85%|########4 | 42326/50000 [27:59:06<3:58:50,  1.87s/it]
Outer training loop:  85%|########4 | 42327/50000 [27:59:08<4:06:38,  1.93s/it]
Outer training loop:  85%|########4 | 42328/50000 [27:59:10<4:04:32,  1.91s/it]
Outer training loop:  85%|########4 | 42329/50000 [27:59:11<3:58:57,  1.87s/it]
Outer training loop:  85%|########4 | 42330/50000 [27:59:13<3:56:33,  1.85s/it]
Outer training loop:  85%|########4 | 42331/50000 [27:59:15<4:04:00,  1.91s/it]
Outer training loop:  85%|########4 | 42332/50000 [27:59:17<3:59:14,  1.87s/it]
Outer training loop:  85%|########4 | 42333/50000 [27:59:19<3:58:13,  1.86s/it]
Outer training loop:  85%|########4 | 42334/50000 [27:59:21<4:01:32,  1.89s/it]
Outer training loop:  85%|########4 | 42335/50000 [27:59:23<4:06:04,  1.93s/it]
Outer training loop:  85%|########4 | 42336/50000 [27:59:25<4:04:32,  1.91s/it]
Outer training loop:  85%|########4 | 42337/50000 [27:59:26<3:58:16,  1.87s/it]
Outer training loop:  85%|########4 | 42338/50000 [27:59:29<4:06:41,  1.93s/it]
Outer training loop:  85%|########4 | 42339/50000 [27:59:30<3:57:30,  1.86s/it]
Outer training loop:  85%|########4 | 42340/50000 [27:59:32<3:52:26,  1.82s/it]
Outer training loop:  85%|########4 | 42341/50000 [27:59:34<4:02:54,  1.90s/it]
Outer training loop:  85%|########4 | 42342/50000 [27:59:36<4:04:12,  1.91s/it]
Outer training loop:  85%|########4 | 42343/50000 [27:59:38<4:03:24,  1.91s/it]
Outer training loop:  85%|########4 | 42344/50000 [27:59:40<4:01:46,  1.89s/it]
Outer training loop:  85%|########4 | 42345/50000 [27:59:42<4:07:14,  1.94s/it]
Outer training loop:  85%|########4 | 42346/50000 [27:59:44<4:07:33,  1.94s/it]
Outer training loop:  85%|########4 | 42347/50000 [27:59:46<4:03:09,  1.91s/it]
Outer training loop:  85%|########4 | 42348/50000 [27:59:48<4:05:51,  1.93s/it]
Outer training loop:  85%|########4 | 42349/50000 [27:59:49<4:06:56,  1.94s/it]
Outer training loop:  85%|########4 | 42350/50000 [27:59:51<4:03:28,  1.91s/it]
Outer training loop:  85%|########4 | 42351/50000 [27:59:53<3:57:45,  1.87s/it]
Outer training loop:  85%|########4 | 42352/50000 [27:59:55<4:05:19,  1.92s/it]
Outer training loop:  85%|########4 | 42353/50000 [27:59:57<4:02:17,  1.90s/it]
Outer training loop:  85%|########4 | 42354/50000 [27:59:59<4:02:19,  1.90s/it]
Outer training loop:  85%|########4 | 42355/50000 [28:00:01<4:03:55,  1.91s/it]
Outer training loop:  85%|########4 | 42356/50000 [28:00:03<4:00:51,  1.89s/it]
Outer training loop:  85%|########4 | 42357/50000 [28:00:05<3:58:03,  1.87s/it]
Outer training loop:  85%|########4 | 42358/50000 [28:00:06<4:02:27,  1.90s/it]
Outer training loop:  85%|########4 | 42359/50000 [28:00:08<4:00:31,  1.89s/it]
Outer training loop:  85%|########4 | 42360/50000 [28:00:10<3:55:15,  1.85s/it]
Outer training loop:  85%|########4 | 42361/50000 [28:00:12<4:05:11,  1.93s/it]
Outer training loop:  85%|########4 | 42362/50000 [28:00:14<4:06:26,  1.94s/it]
Outer training loop:  85%|########4 | 42363/50000 [28:00:16<4:01:28,  1.90s/it]
Outer training loop:  85%|########4 | 42364/50000 [28:00:18<3:58:02,  1.87s/it]
Outer training loop:  85%|########4 | 42365/50000 [28:00:20<4:08:14,  1.95s/it]
Outer training loop:  85%|########4 | 42366/50000 [28:00:22<4:08:02,  1.95s/it]
Outer training loop:  85%|########4 | 42367/50000 [28:00:24<4:03:03,  1.91s/it]
Outer training loop:  85%|########4 | 42368/50000 [28:00:26<4:06:44,  1.94s/it]
Outer training loop:  85%|########4 | 42369/50000 [28:00:28<4:08:37,  1.95s/it]
Outer training loop:  85%|########4 | 42370/50000 [28:00:29<3:59:55,  1.89s/it]
Outer training loop:  85%|########4 | 42371/50000 [28:00:31<3:55:13,  1.85s/it]
Outer training loop:  85%|########4 | 42372/50000 [28:00:33<3:59:46,  1.89s/it]
Outer training loop:  85%|########4 | 42373/50000 [28:00:35<3:54:11,  1.84s/it]
Outer training loop:  85%|########4 | 42374/50000 [28:00:37<3:54:38,  1.85s/it]
Outer training loop:  85%|########4 | 42375/50000 [28:00:39<3:57:57,  1.87s/it]
Outer training loop:  85%|########4 | 42376/50000 [28:00:41<3:58:05,  1.87s/it]
Outer training loop:  85%|########4 | 42377/50000 [28:00:42<3:54:43,  1.85s/it]
Outer training loop:  85%|########4 | 42378/50000 [28:00:44<4:03:10,  1.91s/it]
Outer training loop:  85%|########4 | 42379/50000 [28:00:46<3:57:31,  1.87s/it]
Outer training loop:  85%|########4 | 42380/50000 [28:00:48<3:56:45,  1.86s/it]
Outer training loop:  85%|########4 | 42381/50000 [28:00:50<3:58:18,  1.88s/it]
Outer training loop:  85%|########4 | 42382/50000 [28:00:52<3:57:58,  1.87s/it]
Outer training loop:  85%|########4 | 42383/50000 [28:00:54<3:55:19,  1.85s/it]
Outer training loop:  85%|########4 | 42384/50000 [28:00:55<3:53:31,  1.84s/it]
Outer training loop:  85%|########4 | 42385/50000 [28:00:58<4:07:07,  1.95s/it]
Outer training loop:  85%|########4 | 42386/50000 [28:00:59<4:02:37,  1.91s/it]
Outer training loop:  85%|########4 | 42387/50000 [28:01:01<4:00:04,  1.89s/it]
Outer training loop:  85%|########4 | 42388/50000 [28:01:03<3:55:48,  1.86s/it]
Outer training loop:  85%|########4 | 42389/50000 [28:01:05<4:05:56,  1.94s/it]
Outer training loop:  85%|########4 | 42390/50000 [28:01:07<3:59:45,  1.89s/it]
Outer training loop:  85%|########4 | 42391/50000 [28:01:09<3:57:34,  1.87s/it]
Outer training loop:  85%|########4 | 42392/50000 [28:01:11<4:03:43,  1.92s/it]
Outer training loop:  85%|########4 | 42393/50000 [28:01:13<4:06:45,  1.95s/it]
Outer training loop:  85%|########4 | 42394/50000 [28:01:15<3:59:01,  1.89s/it]
Outer training loop:  85%|########4 | 42395/50000 [28:01:17<4:05:44,  1.94s/it]
Outer training loop:  85%|########4 | 42396/50000 [28:01:18<3:59:27,  1.89s/it]
Outer training loop:  85%|########4 | 42397/50000 [28:01:20<4:00:59,  1.90s/it]
Outer training loop:  85%|########4 | 42398/50000 [28:01:22<3:54:58,  1.85s/it]
Outer training loop:  85%|########4 | 42399/50000 [28:01:24<3:59:13,  1.89s/it]
Outer training loop:  85%|########4 | 42400/50000 [28:01:26<4:06:05,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.96s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  85%|########4 | 42401/50000 [28:01:33<7:28:14,  3.54s/it]
Outer training loop:  85%|########4 | 42402/50000 [28:01:35<6:30:01,  3.08s/it]
Outer training loop:  85%|########4 | 42403/50000 [28:01:37<5:36:11,  2.66s/it]
Outer training loop:  85%|########4 | 42404/50000 [28:01:39<5:05:06,  2.41s/it]
Outer training loop:  85%|########4 | 42405/50000 [28:01:41<4:50:12,  2.29s/it]
Outer training loop:  85%|########4 | 42406/50000 [28:01:43<4:41:06,  2.22s/it]
Outer training loop:  85%|########4 | 42407/50000 [28:01:45<4:23:25,  2.08s/it]
Outer training loop:  85%|########4 | 42408/50000 [28:01:47<4:16:22,  2.03s/it]
Outer training loop:  85%|########4 | 42409/50000 [28:01:49<4:13:55,  2.01s/it]
Outer training loop:  85%|########4 | 42410/50000 [28:01:50<4:08:23,  1.96s/it]
Outer training loop:  85%|########4 | 42411/50000 [28:01:52<3:58:37,  1.89s/it]
Outer training loop:  85%|########4 | 42412/50000 [28:01:54<3:54:30,  1.85s/it]
Outer training loop:  85%|########4 | 42413/50000 [28:01:56<4:03:08,  1.92s/it]
Outer training loop:  85%|########4 | 42414/50000 [28:01:58<4:02:40,  1.92s/it]
Outer training loop:  85%|########4 | 42415/50000 [28:02:00<3:58:11,  1.88s/it]
Outer training loop:  85%|########4 | 42416/50000 [28:02:02<4:04:18,  1.93s/it]
Outer training loop:  85%|########4 | 42417/50000 [28:02:04<4:02:54,  1.92s/it]
Outer training loop:  85%|########4 | 42418/50000 [28:02:06<4:01:24,  1.91s/it]
Outer training loop:  85%|########4 | 42419/50000 [28:02:07<3:57:41,  1.88s/it]
Outer training loop:  85%|########4 | 42420/50000 [28:02:10<4:07:06,  1.96s/it]
Outer training loop:  85%|########4 | 42421/50000 [28:02:11<4:01:49,  1.91s/it]
Outer training loop:  85%|########4 | 42422/50000 [28:02:13<4:01:46,  1.91s/it]
Outer training loop:  85%|########4 | 42423/50000 [28:02:15<4:04:25,  1.94s/it]
Outer training loop:  85%|########4 | 42424/50000 [28:02:17<4:00:33,  1.91s/it]
Outer training loop:  85%|########4 | 42425/50000 [28:02:19<4:00:02,  1.90s/it]
Outer training loop:  85%|########4 | 42426/50000 [28:02:21<3:54:53,  1.86s/it]
Outer training loop:  85%|########4 | 42427/50000 [28:02:23<3:59:50,  1.90s/it]
Outer training loop:  85%|########4 | 42428/50000 [28:02:25<3:57:43,  1.88s/it]
Outer training loop:  85%|########4 | 42429/50000 [28:02:26<3:55:38,  1.87s/it]
Outer training loop:  85%|########4 | 42430/50000 [28:02:28<3:58:12,  1.89s/it]
Outer training loop:  85%|########4 | 42431/50000 [28:02:30<3:52:56,  1.85s/it]
Outer training loop:  85%|########4 | 42432/50000 [28:02:32<3:52:50,  1.85s/it]
Outer training loop:  85%|########4 | 42433/50000 [28:02:34<3:48:36,  1.81s/it]
Outer training loop:  85%|########4 | 42434/50000 [28:02:36<3:56:24,  1.87s/it]
Outer training loop:  85%|########4 | 42435/50000 [28:02:38<3:55:24,  1.87s/it]
Outer training loop:  85%|########4 | 42436/50000 [28:02:39<3:53:53,  1.86s/it]
Outer training loop:  85%|########4 | 42437/50000 [28:02:41<3:58:49,  1.89s/it]
Outer training loop:  85%|########4 | 42438/50000 [28:02:43<4:00:22,  1.91s/it]
Outer training loop:  85%|########4 | 42439/50000 [28:02:45<3:57:26,  1.88s/it]
Outer training loop:  85%|########4 | 42440/50000 [28:02:47<4:00:24,  1.91s/it]
Outer training loop:  85%|########4 | 42441/50000 [28:02:49<4:00:53,  1.91s/it]
Outer training loop:  85%|########4 | 42442/50000 [28:02:51<3:58:15,  1.89s/it]
Outer training loop:  85%|########4 | 42443/50000 [28:02:53<3:57:59,  1.89s/it]
Outer training loop:  85%|########4 | 42444/50000 [28:02:55<4:04:43,  1.94s/it]
Outer training loop:  85%|########4 | 42445/50000 [28:02:57<3:58:39,  1.90s/it]
Outer training loop:  85%|########4 | 42446/50000 [28:02:58<3:54:39,  1.86s/it]
Outer training loop:  85%|########4 | 42447/50000 [28:03:00<3:57:53,  1.89s/it]
Outer training loop:  85%|########4 | 42448/50000 [28:03:02<3:53:10,  1.85s/it]
Outer training loop:  85%|########4 | 42449/50000 [28:03:04<3:47:02,  1.80s/it]
Outer training loop:  85%|########4 | 42450/50000 [28:03:06<3:52:27,  1.85s/it]
Outer training loop:  85%|########4 | 42451/50000 [28:03:08<3:59:35,  1.90s/it]
Outer training loop:  85%|########4 | 42452/50000 [28:03:10<3:58:12,  1.89s/it]
Outer training loop:  85%|########4 | 42453/50000 [28:03:12<3:57:38,  1.89s/it]
Outer training loop:  85%|########4 | 42454/50000 [28:03:14<4:02:58,  1.93s/it]
Outer training loop:  85%|########4 | 42455/50000 [28:03:15<3:56:58,  1.88s/it]
Outer training loop:  85%|########4 | 42456/50000 [28:03:17<3:53:12,  1.85s/it]
Outer training loop:  85%|########4 | 42457/50000 [28:03:19<3:58:28,  1.90s/it]
Outer training loop:  85%|########4 | 42458/50000 [28:03:21<3:56:05,  1.88s/it]
Outer training loop:  85%|########4 | 42459/50000 [28:03:23<3:52:52,  1.85s/it]
Outer training loop:  85%|########4 | 42460/50000 [28:03:25<3:53:08,  1.86s/it]
Outer training loop:  85%|########4 | 42461/50000 [28:03:27<4:00:38,  1.92s/it]
Outer training loop:  85%|########4 | 42462/50000 [28:03:28<3:54:26,  1.87s/it]
Outer training loop:  85%|########4 | 42463/50000 [28:03:30<3:50:44,  1.84s/it]
Outer training loop:  85%|########4 | 42464/50000 [28:03:32<3:57:24,  1.89s/it]
Outer training loop:  85%|########4 | 42465/50000 [28:03:34<3:53:14,  1.86s/it]
Outer training loop:  85%|########4 | 42466/50000 [28:03:36<3:49:22,  1.83s/it]
Outer training loop:  85%|########4 | 42467/50000 [28:03:38<3:51:30,  1.84s/it]
Outer training loop:  85%|########4 | 42468/50000 [28:03:40<3:56:01,  1.88s/it]
Outer training loop:  85%|########4 | 42469/50000 [28:03:41<3:47:34,  1.81s/it]
Outer training loop:  85%|########4 | 42470/50000 [28:03:43<3:44:20,  1.79s/it]
Outer training loop:  85%|########4 | 42471/50000 [28:03:45<3:53:13,  1.86s/it]
Outer training loop:  85%|########4 | 42472/50000 [28:03:47<3:52:37,  1.85s/it]
Outer training loop:  85%|########4 | 42473/50000 [28:03:49<3:52:37,  1.85s/it]
Outer training loop:  85%|########4 | 42474/50000 [28:03:51<4:00:59,  1.92s/it]
Outer training loop:  85%|########4 | 42475/50000 [28:03:53<3:55:41,  1.88s/it]
Outer training loop:  85%|########4 | 42476/50000 [28:03:54<3:53:17,  1.86s/it]
Outer training loop:  85%|########4 | 42477/50000 [28:03:56<3:52:47,  1.86s/it]
Outer training loop:  85%|########4 | 42478/50000 [28:03:58<3:59:16,  1.91s/it]
Outer training loop:  85%|########4 | 42479/50000 [28:04:00<3:55:25,  1.88s/it]
Outer training loop:  85%|########4 | 42480/50000 [28:04:02<3:52:12,  1.85s/it]
Outer training loop:  85%|########4 | 42481/50000 [28:04:04<3:57:57,  1.90s/it]
Outer training loop:  85%|########4 | 42482/50000 [28:04:06<4:02:52,  1.94s/it]
Outer training loop:  85%|########4 | 42483/50000 [28:04:08<4:02:34,  1.94s/it]
Outer training loop:  85%|########4 | 42484/50000 [28:04:10<4:06:08,  1.96s/it]
Outer training loop:  85%|########4 | 42485/50000 [28:04:12<4:05:23,  1.96s/it]
Outer training loop:  85%|########4 | 42486/50000 [28:04:14<4:00:49,  1.92s/it]
Outer training loop:  85%|########4 | 42487/50000 [28:04:15<3:57:43,  1.90s/it]
Outer training loop:  85%|########4 | 42488/50000 [28:04:17<4:00:58,  1.92s/it]
Outer training loop:  85%|########4 | 42489/50000 [28:04:19<3:57:09,  1.89s/it]
Outer training loop:  85%|########4 | 42490/50000 [28:04:21<3:54:59,  1.88s/it]
Outer training loop:  85%|########4 | 42491/50000 [28:04:23<4:03:03,  1.94s/it]
Outer training loop:  85%|########4 | 42492/50000 [28:04:25<3:59:24,  1.91s/it]
Outer training loop:  85%|########4 | 42493/50000 [28:04:27<3:56:52,  1.89s/it]
Outer training loop:  85%|########4 | 42494/50000 [28:04:29<3:59:19,  1.91s/it]
Outer training loop:  85%|########4 | 42495/50000 [28:04:31<4:00:36,  1.92s/it]
Outer training loop:  85%|########4 | 42496/50000 [28:04:33<4:01:05,  1.93s/it]
Outer training loop:  85%|########4 | 42497/50000 [28:04:35<3:59:52,  1.92s/it]
Outer training loop:  85%|########4 | 42498/50000 [28:04:36<3:57:36,  1.90s/it]
Outer training loop:  85%|########4 | 42499/50000 [28:04:38<3:54:06,  1.87s/it]
Outer training loop:  85%|########5 | 42500/50000 [28:04:40<3:55:16,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.99s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:45,  2.02s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:40,  1.98s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:36,  1.95s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.92s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:33,  1.96s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.92s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.96s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:16,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:19,  1.97s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:14,  1.92s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.93s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:08,  1.95s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:05,  2.02s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:59,  1.96s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:57,  1.96s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:54,  1.94s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:51,  1.92s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:52,  1.97s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:50,  1.98s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:50,  2.00s/it]

Evaluating policy:  35%|###4      | 29/83 [00:56<01:45,  1.95s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [01:00<01:45,  2.04s/it]

Evaluating policy:  39%|###8      | 32/83 [01:02<01:43,  2.02s/it]

Evaluating policy:  40%|###9      | 33/83 [01:04<01:38,  1.97s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:35,  1.94s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:32,  1.92s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:31,  1.94s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:31,  1.99s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:24,  1.92s/it]

Evaluating policy:  48%|####8     | 40/83 [01:17<01:21,  1.88s/it]

Evaluating policy:  49%|####9     | 41/83 [01:19<01:19,  1.90s/it]

Evaluating policy:  51%|#####     | 42/83 [01:21<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:23<01:15,  1.88s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:25<01:17,  1.98s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:27<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:08,  1.86s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:06,  1.86s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:05,  1.88s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:02,  1.85s/it]

Evaluating policy:  60%|######    | 50/83 [01:36<01:00,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<01:01,  1.94s/it]

Evaluating policy:  63%|######2   | 52/83 [01:40<00:59,  1.93s/it]

Evaluating policy:  64%|######3   | 53/83 [01:42<00:57,  1.91s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:54,  1.87s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:49,  1.88s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:47,  1.96s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:44,  1.95s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:41,  1.91s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:06<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:31,  1.98s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:29,  1.95s/it]

Evaluating policy:  83%|########3 | 69/83 [02:12<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:19<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:16,  1.84s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:15,  1.89s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:14,  2.07s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:12,  2.01s/it]

Evaluating policy:  94%|#########3| 78/83 [02:29<00:10,  2.01s/it]

Evaluating policy:  95%|#########5| 79/83 [02:31<00:08,  2.02s/it]

Evaluating policy:  96%|#########6| 80/83 [02:33<00:05,  1.96s/it]

Evaluating policy:  98%|#########7| 81/83 [02:35<00:03,  1.89s/it]

Evaluating policy:  99%|#########8| 82/83 [02:37<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.75s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.75s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:35,  1.94s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.92s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:26,  1.88s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:29,  1.94s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:18,  1.88s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:10,  1.86s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:13,  1.93s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:03,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.87s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.93s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:58,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.86s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:56,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:46,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:43,  1.85s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:48,  1.97s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:42,  1.90s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.86s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:35,  1.83s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:31,  1.83s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:33,  1.91s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.88s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:25,  1.95s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:23,  1.94s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:19,  1.89s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.86s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:13,  1.99s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:10,  1.95s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.90s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<01:00,  2.01s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:55,  1.93s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:53,  1.91s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.92s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:45,  1.96s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.90s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.91s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:34,  1.89s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:30,  2.06s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:28,  2.01s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:25,  1.97s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:23,  1.95s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.91s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.93s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.93s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:12,  2.06s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.98s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.97s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.93s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.90s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.94s/it]

Outer training loop:  85%|########5 | 42501/50000 [28:10:05<205:50:04, 98.81s/it]
Outer training loop:  85%|########5 | 42502/50000 [28:10:07<145:12:30, 69.72s/it]
Outer training loop:  85%|########5 | 42503/50000 [28:10:09<102:47:18, 49.36s/it]
Outer training loop:  85%|########5 | 42504/50000 [28:10:11<73:10:12, 35.14s/it] 
Outer training loop:  85%|########5 | 42505/50000 [28:10:13<52:46:19, 25.35s/it]
Outer training loop:  85%|########5 | 42506/50000 [28:10:15<38:03:32, 18.28s/it]
Outer training loop:  85%|########5 | 42507/50000 [28:10:17<27:47:51, 13.36s/it]
Outer training loop:  85%|########5 | 42508/50000 [28:10:19<20:47:19,  9.99s/it]
Outer training loop:  85%|########5 | 42509/50000 [28:10:21<15:40:42,  7.53s/it]
Outer training loop:  85%|########5 | 42510/50000 [28:10:23<12:08:54,  5.84s/it]
Outer training loop:  85%|########5 | 42511/50000 [28:10:25<9:41:31,  4.66s/it] 
Outer training loop:  85%|########5 | 42512/50000 [28:10:27<8:03:18,  3.87s/it]
Outer training loop:  85%|########5 | 42513/50000 [28:10:29<6:54:23,  3.32s/it]
Outer training loop:  85%|########5 | 42514/50000 [28:10:31<5:54:44,  2.84s/it]
Outer training loop:  85%|########5 | 42515/50000 [28:10:33<5:25:19,  2.61s/it]
Outer training loop:  85%|########5 | 42516/50000 [28:10:35<5:04:05,  2.44s/it]
Outer training loop:  85%|########5 | 42517/50000 [28:10:36<4:41:58,  2.26s/it]
Outer training loop:  85%|########5 | 42518/50000 [28:10:38<4:25:20,  2.13s/it]
Outer training loop:  85%|########5 | 42519/50000 [28:10:40<4:27:59,  2.15s/it]
Outer training loop:  85%|########5 | 42520/50000 [28:10:42<4:21:37,  2.10s/it]
Outer training loop:  85%|########5 | 42521/50000 [28:10:44<4:11:10,  2.01s/it]
Outer training loop:  85%|########5 | 42522/50000 [28:10:46<4:07:15,  1.98s/it]
Outer training loop:  85%|########5 | 42523/50000 [28:10:48<4:03:25,  1.95s/it]
Outer training loop:  85%|########5 | 42524/50000 [28:10:50<4:03:58,  1.96s/it]
Outer training loop:  85%|########5 | 42525/50000 [28:10:52<4:01:00,  1.93s/it]
Outer training loop:  85%|########5 | 42526/50000 [28:10:54<4:09:20,  2.00s/it]
Outer training loop:  85%|########5 | 42527/50000 [28:10:56<4:08:37,  2.00s/it]
Outer training loop:  85%|########5 | 42528/50000 [28:10:58<4:04:00,  1.96s/it]
Outer training loop:  85%|########5 | 42529/50000 [28:11:00<4:03:50,  1.96s/it]
Outer training loop:  85%|########5 | 42530/50000 [28:11:02<3:59:34,  1.92s/it]
Outer training loop:  85%|########5 | 42531/50000 [28:11:04<3:55:52,  1.89s/it]
Outer training loop:  85%|########5 | 42532/50000 [28:11:05<3:51:10,  1.86s/it]
Outer training loop:  85%|########5 | 42533/50000 [28:11:07<4:01:05,  1.94s/it]
Outer training loop:  85%|########5 | 42534/50000 [28:11:09<3:52:16,  1.87s/it]
Outer training loop:  85%|########5 | 42535/50000 [28:11:11<3:52:36,  1.87s/it]
Outer training loop:  85%|########5 | 42536/50000 [28:11:13<4:03:57,  1.96s/it]
Outer training loop:  85%|########5 | 42537/50000 [28:11:15<4:00:05,  1.93s/it]
Outer training loop:  85%|########5 | 42538/50000 [28:11:17<3:53:14,  1.88s/it]
Outer training loop:  85%|########5 | 42539/50000 [28:11:19<3:52:56,  1.87s/it]
Outer training loop:  85%|########5 | 42540/50000 [28:11:21<3:53:21,  1.88s/it]
Outer training loop:  85%|########5 | 42541/50000 [28:11:22<3:54:51,  1.89s/it]
Outer training loop:  85%|########5 | 42542/50000 [28:11:25<4:01:56,  1.95s/it]
Outer training loop:  85%|########5 | 42543/50000 [28:11:26<3:59:20,  1.93s/it]
Outer training loop:  85%|########5 | 42544/50000 [28:11:28<3:51:34,  1.86s/it]
Outer training loop:  85%|########5 | 42545/50000 [28:11:30<3:46:48,  1.83s/it]
Outer training loop:  85%|########5 | 42546/50000 [28:11:32<3:56:12,  1.90s/it]
Outer training loop:  85%|########5 | 42547/50000 [28:11:34<3:53:19,  1.88s/it]
Outer training loop:  85%|########5 | 42548/50000 [28:11:36<3:49:42,  1.85s/it]
Outer training loop:  85%|########5 | 42549/50000 [28:11:38<3:57:36,  1.91s/it]
Outer training loop:  85%|########5 | 42550/50000 [28:11:39<3:50:58,  1.86s/it]
Outer training loop:  85%|########5 | 42551/50000 [28:11:41<3:50:29,  1.86s/it]
Outer training loop:  85%|########5 | 42552/50000 [28:11:43<3:49:03,  1.85s/it]
Outer training loop:  85%|########5 | 42553/50000 [28:11:45<3:58:43,  1.92s/it]
Outer training loop:  85%|########5 | 42554/50000 [28:11:47<3:54:48,  1.89s/it]
Outer training loop:  85%|########5 | 42555/50000 [28:11:49<3:50:40,  1.86s/it]
Outer training loop:  85%|########5 | 42556/50000 [28:11:51<3:53:42,  1.88s/it]
Outer training loop:  85%|########5 | 42557/50000 [28:11:53<3:51:41,  1.87s/it]
Outer training loop:  85%|########5 | 42558/50000 [28:11:54<3:51:37,  1.87s/it]
Outer training loop:  85%|########5 | 42559/50000 [28:11:56<3:54:05,  1.89s/it]
Outer training loop:  85%|########5 | 42560/50000 [28:11:58<4:01:43,  1.95s/it]
Outer training loop:  85%|########5 | 42561/50000 [28:12:00<3:58:51,  1.93s/it]
Outer training loop:  85%|########5 | 42562/50000 [28:12:02<3:51:11,  1.86s/it]
Outer training loop:  85%|########5 | 42563/50000 [28:12:04<4:02:03,  1.95s/it]
Outer training loop:  85%|########5 | 42564/50000 [28:12:06<3:57:06,  1.91s/it]
Outer training loop:  85%|########5 | 42565/50000 [28:12:08<3:59:11,  1.93s/it]
Outer training loop:  85%|########5 | 42566/50000 [28:12:10<3:55:43,  1.90s/it]
Outer training loop:  85%|########5 | 42567/50000 [28:12:12<4:00:14,  1.94s/it]
Outer training loop:  85%|########5 | 42568/50000 [28:12:14<3:58:57,  1.93s/it]
Outer training loop:  85%|########5 | 42569/50000 [28:12:15<3:52:54,  1.88s/it]
Outer training loop:  85%|########5 | 42570/50000 [28:12:17<3:56:19,  1.91s/it]
Outer training loop:  85%|########5 | 42571/50000 [28:12:19<3:51:33,  1.87s/it]
Outer training loop:  85%|########5 | 42572/50000 [28:12:21<3:48:16,  1.84s/it]
Outer training loop:  85%|########5 | 42573/50000 [28:12:23<3:46:23,  1.83s/it]
Outer training loop:  85%|########5 | 42574/50000 [28:12:25<4:01:23,  1.95s/it]
Outer training loop:  85%|########5 | 42575/50000 [28:12:27<3:55:14,  1.90s/it]
Outer training loop:  85%|########5 | 42576/50000 [28:12:29<3:56:09,  1.91s/it]
Outer training loop:  85%|########5 | 42577/50000 [28:12:31<3:58:19,  1.93s/it]
Outer training loop:  85%|########5 | 42578/50000 [28:12:33<3:55:52,  1.91s/it]
Outer training loop:  85%|########5 | 42579/50000 [28:12:34<3:52:34,  1.88s/it]
Outer training loop:  85%|########5 | 42580/50000 [28:12:36<3:52:44,  1.88s/it]
Outer training loop:  85%|########5 | 42581/50000 [28:12:38<3:56:58,  1.92s/it]
Outer training loop:  85%|########5 | 42582/50000 [28:12:40<3:53:39,  1.89s/it]
Outer training loop:  85%|########5 | 42583/50000 [28:12:42<3:54:27,  1.90s/it]
Outer training loop:  85%|########5 | 42584/50000 [28:12:44<3:56:23,  1.91s/it]
Outer training loop:  85%|########5 | 42585/50000 [28:12:46<3:49:48,  1.86s/it]
Outer training loop:  85%|########5 | 42586/50000 [28:12:48<3:50:39,  1.87s/it]
Outer training loop:  85%|########5 | 42587/50000 [28:12:50<4:04:06,  1.98s/it]
Outer training loop:  85%|########5 | 42588/50000 [28:12:52<3:54:25,  1.90s/it]
Outer training loop:  85%|########5 | 42589/50000 [28:12:53<3:52:10,  1.88s/it]
Outer training loop:  85%|########5 | 42590/50000 [28:12:55<3:51:36,  1.88s/it]
Outer training loop:  85%|########5 | 42591/50000 [28:12:57<3:55:52,  1.91s/it]
Outer training loop:  85%|########5 | 42592/50000 [28:12:59<3:53:53,  1.89s/it]
Outer training loop:  85%|########5 | 42593/50000 [28:13:01<3:52:22,  1.88s/it]
Outer training loop:  85%|########5 | 42594/50000 [28:13:03<3:59:13,  1.94s/it]
Outer training loop:  85%|########5 | 42595/50000 [28:13:05<3:56:57,  1.92s/it]
Outer training loop:  85%|########5 | 42596/50000 [28:13:07<3:52:59,  1.89s/it]
Outer training loop:  85%|########5 | 42597/50000 [28:13:09<3:58:20,  1.93s/it]
Outer training loop:  85%|########5 | 42598/50000 [28:13:11<3:59:24,  1.94s/it]
Outer training loop:  85%|########5 | 42599/50000 [28:13:13<4:00:52,  1.95s/it]
Outer training loop:  85%|########5 | 42600/50000 [28:13:15<3:58:34,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  85%|########5 | 42601/50000 [28:13:22<7:33:31,  3.68s/it]
Outer training loop:  85%|########5 | 42602/50000 [28:13:24<6:26:41,  3.14s/it]
Outer training loop:  85%|########5 | 42603/50000 [28:13:26<5:39:39,  2.76s/it]
Outer training loop:  85%|########5 | 42604/50000 [28:13:28<5:04:11,  2.47s/it]
Outer training loop:  85%|########5 | 42605/50000 [28:13:30<4:51:56,  2.37s/it]
Outer training loop:  85%|########5 | 42606/50000 [28:13:32<4:29:17,  2.19s/it]
Outer training loop:  85%|########5 | 42607/50000 [28:13:34<4:14:47,  2.07s/it]
Outer training loop:  85%|########5 | 42608/50000 [28:13:36<4:10:26,  2.03s/it]
Outer training loop:  85%|########5 | 42609/50000 [28:13:38<4:08:51,  2.02s/it]
Outer training loop:  85%|########5 | 42610/50000 [28:13:39<4:07:32,  2.01s/it]
Outer training loop:  85%|########5 | 42611/50000 [28:13:41<4:01:37,  1.96s/it]
Outer training loop:  85%|########5 | 42612/50000 [28:13:43<4:07:59,  2.01s/it]
Outer training loop:  85%|########5 | 42613/50000 [28:13:45<4:05:20,  1.99s/it]
Outer training loop:  85%|########5 | 42614/50000 [28:13:47<3:58:52,  1.94s/it]
Outer training loop:  85%|########5 | 42615/50000 [28:13:49<4:01:43,  1.96s/it]
Outer training loop:  85%|########5 | 42616/50000 [28:13:51<3:57:08,  1.93s/it]
Outer training loop:  85%|########5 | 42617/50000 [28:13:53<3:59:23,  1.95s/it]
Outer training loop:  85%|########5 | 42618/50000 [28:13:55<3:57:20,  1.93s/it]
Outer training loop:  85%|########5 | 42619/50000 [28:13:57<3:59:47,  1.95s/it]
Outer training loop:  85%|########5 | 42620/50000 [28:13:59<3:55:25,  1.91s/it]
Outer training loop:  85%|########5 | 42621/50000 [28:14:01<3:55:25,  1.91s/it]
Outer training loop:  85%|########5 | 42622/50000 [28:14:03<4:00:02,  1.95s/it]
Outer training loop:  85%|########5 | 42623/50000 [28:14:05<4:01:44,  1.97s/it]
Outer training loop:  85%|########5 | 42624/50000 [28:14:07<4:00:47,  1.96s/it]
Outer training loop:  85%|########5 | 42625/50000 [28:14:09<3:58:17,  1.94s/it]
Outer training loop:  85%|########5 | 42626/50000 [28:14:11<4:02:53,  1.98s/it]
Outer training loop:  85%|########5 | 42627/50000 [28:14:13<4:00:44,  1.96s/it]
Outer training loop:  85%|########5 | 42628/50000 [28:14:14<3:53:22,  1.90s/it]
Outer training loop:  85%|########5 | 42629/50000 [28:14:16<3:59:18,  1.95s/it]
Outer training loop:  85%|########5 | 42630/50000 [28:14:18<3:54:13,  1.91s/it]
Outer training loop:  85%|########5 | 42631/50000 [28:14:20<3:55:10,  1.91s/it]
Outer training loop:  85%|########5 | 42632/50000 [28:14:22<4:00:04,  1.96s/it]
Outer training loop:  85%|########5 | 42633/50000 [28:14:24<3:57:46,  1.94s/it]
Outer training loop:  85%|########5 | 42634/50000 [28:14:26<3:57:34,  1.94s/it]
Outer training loop:  85%|########5 | 42635/50000 [28:14:28<3:56:30,  1.93s/it]
Outer training loop:  85%|########5 | 42636/50000 [28:14:30<4:01:34,  1.97s/it]
Outer training loop:  85%|########5 | 42637/50000 [28:14:32<3:52:06,  1.89s/it]
Outer training loop:  85%|########5 | 42638/50000 [28:14:34<3:50:09,  1.88s/it]
Outer training loop:  85%|########5 | 42639/50000 [28:14:36<3:56:36,  1.93s/it]
Outer training loop:  85%|########5 | 42640/50000 [28:14:37<3:54:01,  1.91s/it]
Outer training loop:  85%|########5 | 42641/50000 [28:14:39<3:47:02,  1.85s/it]
Outer training loop:  85%|########5 | 42642/50000 [28:14:41<3:43:53,  1.83s/it]
Outer training loop:  85%|########5 | 42643/50000 [28:14:43<3:50:37,  1.88s/it]
Outer training loop:  85%|########5 | 42644/50000 [28:14:45<3:44:30,  1.83s/it]
Outer training loop:  85%|########5 | 42645/50000 [28:14:46<3:44:18,  1.83s/it]
Outer training loop:  85%|########5 | 42646/50000 [28:14:49<3:53:26,  1.90s/it]
Outer training loop:  85%|########5 | 42647/50000 [28:14:50<3:52:13,  1.89s/it]
Outer training loop:  85%|########5 | 42648/50000 [28:14:52<3:49:33,  1.87s/it]
Outer training loop:  85%|########5 | 42649/50000 [28:14:54<3:51:07,  1.89s/it]
Outer training loop:  85%|########5 | 42650/50000 [28:14:56<3:46:58,  1.85s/it]
Outer training loop:  85%|########5 | 42651/50000 [28:14:58<3:53:11,  1.90s/it]
Outer training loop:  85%|########5 | 42652/50000 [28:15:00<3:49:59,  1.88s/it]
Outer training loop:  85%|########5 | 42653/50000 [28:15:02<3:50:18,  1.88s/it]
Outer training loop:  85%|########5 | 42654/50000 [28:15:03<3:44:59,  1.84s/it]
Outer training loop:  85%|########5 | 42655/50000 [28:15:05<3:47:47,  1.86s/it]
Outer training loop:  85%|########5 | 42656/50000 [28:15:07<3:55:48,  1.93s/it]
Outer training loop:  85%|########5 | 42657/50000 [28:15:09<3:55:35,  1.93s/it]
Outer training loop:  85%|########5 | 42658/50000 [28:15:11<3:54:10,  1.91s/it]
Outer training loop:  85%|########5 | 42659/50000 [28:15:13<3:50:54,  1.89s/it]
Outer training loop:  85%|########5 | 42660/50000 [28:15:15<4:06:20,  2.01s/it]
Outer training loop:  85%|########5 | 42661/50000 [28:15:17<3:58:09,  1.95s/it]
Outer training loop:  85%|########5 | 42662/50000 [28:15:19<3:55:03,  1.92s/it]
Outer training loop:  85%|########5 | 42663/50000 [28:15:21<3:58:40,  1.95s/it]
Outer training loop:  85%|########5 | 42664/50000 [28:15:23<3:52:39,  1.90s/it]
Outer training loop:  85%|########5 | 42665/50000 [28:15:25<3:48:49,  1.87s/it]
Outer training loop:  85%|########5 | 42666/50000 [28:15:27<4:03:04,  1.99s/it]
Outer training loop:  85%|########5 | 42667/50000 [28:15:29<3:54:40,  1.92s/it]
Outer training loop:  85%|########5 | 42668/50000 [28:15:30<3:48:52,  1.87s/it]
Outer training loop:  85%|########5 | 42669/50000 [28:15:32<3:54:03,  1.92s/it]
Outer training loop:  85%|########5 | 42670/50000 [28:15:34<3:51:21,  1.89s/it]
Outer training loop:  85%|########5 | 42671/50000 [28:15:36<3:50:02,  1.88s/it]
Outer training loop:  85%|########5 | 42672/50000 [28:15:38<3:44:42,  1.84s/it]
Outer training loop:  85%|########5 | 42673/50000 [28:15:40<3:53:59,  1.92s/it]
Outer training loop:  85%|########5 | 42674/50000 [28:15:42<3:53:59,  1.92s/it]
Outer training loop:  85%|########5 | 42675/50000 [28:15:44<3:51:38,  1.90s/it]
Outer training loop:  85%|########5 | 42676/50000 [28:15:46<3:54:24,  1.92s/it]
Outer training loop:  85%|########5 | 42677/50000 [28:15:47<3:49:09,  1.88s/it]
Outer training loop:  85%|########5 | 42678/50000 [28:15:49<3:46:36,  1.86s/it]
Outer training loop:  85%|########5 | 42679/50000 [28:15:51<3:52:42,  1.91s/it]
Outer training loop:  85%|########5 | 42680/50000 [28:15:53<3:48:35,  1.87s/it]
Outer training loop:  85%|########5 | 42681/50000 [28:15:55<3:48:28,  1.87s/it]
Outer training loop:  85%|########5 | 42682/50000 [28:15:57<3:54:32,  1.92s/it]
Outer training loop:  85%|########5 | 42683/50000 [28:15:59<3:49:16,  1.88s/it]
Outer training loop:  85%|########5 | 42684/50000 [28:16:01<3:49:56,  1.89s/it]
Outer training loop:  85%|########5 | 42685/50000 [28:16:03<3:48:06,  1.87s/it]
Outer training loop:  85%|########5 | 42686/50000 [28:16:04<3:51:35,  1.90s/it]
Outer training loop:  85%|########5 | 42687/50000 [28:16:06<3:50:47,  1.89s/it]
Outer training loop:  85%|########5 | 42688/50000 [28:16:08<3:48:57,  1.88s/it]
Outer training loop:  85%|########5 | 42689/50000 [28:16:10<3:54:28,  1.92s/it]
Outer training loop:  85%|########5 | 42690/50000 [28:16:12<3:47:53,  1.87s/it]
Outer training loop:  85%|########5 | 42691/50000 [28:16:14<3:49:02,  1.88s/it]
Outer training loop:  85%|########5 | 42692/50000 [28:16:16<3:45:02,  1.85s/it]
Outer training loop:  85%|########5 | 42693/50000 [28:16:18<3:48:21,  1.88s/it]
Outer training loop:  85%|########5 | 42694/50000 [28:16:20<3:49:32,  1.89s/it]
Outer training loop:  85%|########5 | 42695/50000 [28:16:21<3:50:27,  1.89s/it]
Outer training loop:  85%|########5 | 42696/50000 [28:16:23<3:54:05,  1.92s/it]
Outer training loop:  85%|########5 | 42697/50000 [28:16:25<3:53:18,  1.92s/it]
Outer training loop:  85%|########5 | 42698/50000 [28:16:27<3:47:44,  1.87s/it]
Outer training loop:  85%|########5 | 42699/50000 [28:16:29<3:51:26,  1.90s/it]
Outer training loop:  85%|########5 | 42700/50000 [28:16:31<3:56:04,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  85%|########5 | 42701/50000 [28:16:38<7:11:59,  3.55s/it]
Outer training loop:  85%|########5 | 42702/50000 [28:16:40<6:07:56,  3.02s/it]
Outer training loop:  85%|########5 | 42703/50000 [28:16:42<5:26:39,  2.69s/it]
Outer training loop:  85%|########5 | 42704/50000 [28:16:44<4:58:40,  2.46s/it]
Outer training loop:  85%|########5 | 42705/50000 [28:16:46<4:32:07,  2.24s/it]
Outer training loop:  85%|########5 | 42706/50000 [28:16:48<4:15:19,  2.10s/it]
Outer training loop:  85%|########5 | 42707/50000 [28:16:50<4:25:21,  2.18s/it]
Outer training loop:  85%|########5 | 42708/50000 [28:16:52<4:15:19,  2.10s/it]
Outer training loop:  85%|########5 | 42709/50000 [28:16:54<4:05:20,  2.02s/it]
Outer training loop:  85%|########5 | 42710/50000 [28:16:55<3:55:58,  1.94s/it]
Outer training loop:  85%|########5 | 42711/50000 [28:16:58<4:05:29,  2.02s/it]
Outer training loop:  85%|########5 | 42712/50000 [28:16:59<4:00:16,  1.98s/it]
Outer training loop:  85%|########5 | 42713/50000 [28:17:01<3:54:01,  1.93s/it]
Outer training loop:  85%|########5 | 42714/50000 [28:17:03<3:54:59,  1.94s/it]
Outer training loop:  85%|########5 | 42715/50000 [28:17:05<3:51:25,  1.91s/it]
Outer training loop:  85%|########5 | 42716/50000 [28:17:07<3:51:19,  1.91s/it]
Outer training loop:  85%|########5 | 42717/50000 [28:17:09<3:54:50,  1.93s/it]
Outer training loop:  85%|########5 | 42718/50000 [28:17:11<3:54:16,  1.93s/it]
Outer training loop:  85%|########5 | 42719/50000 [28:17:13<3:45:41,  1.86s/it]
Outer training loop:  85%|########5 | 42720/50000 [28:17:14<3:44:29,  1.85s/it]
Outer training loop:  85%|########5 | 42721/50000 [28:17:16<3:51:46,  1.91s/it]
Outer training loop:  85%|########5 | 42722/50000 [28:17:18<3:48:29,  1.88s/it]
Outer training loop:  85%|########5 | 42723/50000 [28:17:20<3:45:54,  1.86s/it]
Outer training loop:  85%|########5 | 42724/50000 [28:17:22<3:48:59,  1.89s/it]
Outer training loop:  85%|########5 | 42725/50000 [28:17:24<3:45:30,  1.86s/it]
Outer training loop:  85%|########5 | 42726/50000 [28:17:26<3:43:51,  1.85s/it]
Outer training loop:  85%|########5 | 42727/50000 [28:17:28<3:48:26,  1.88s/it]
Outer training loop:  85%|########5 | 42728/50000 [28:17:30<3:50:50,  1.90s/it]
Outer training loop:  85%|########5 | 42729/50000 [28:17:31<3:48:40,  1.89s/it]
Outer training loop:  85%|########5 | 42730/50000 [28:17:33<3:46:18,  1.87s/it]
Outer training loop:  85%|########5 | 42731/50000 [28:17:35<3:57:50,  1.96s/it]
Outer training loop:  85%|########5 | 42732/50000 [28:17:37<3:51:34,  1.91s/it]
Outer training loop:  85%|########5 | 42733/50000 [28:17:39<3:48:39,  1.89s/it]
Outer training loop:  85%|########5 | 42734/50000 [28:17:41<3:54:35,  1.94s/it]
Outer training loop:  85%|########5 | 42735/50000 [28:17:43<3:48:36,  1.89s/it]
Outer training loop:  85%|########5 | 42736/50000 [28:17:45<3:46:33,  1.87s/it]
Outer training loop:  85%|########5 | 42737/50000 [28:17:47<3:48:56,  1.89s/it]
Outer training loop:  85%|########5 | 42738/50000 [28:17:48<3:43:21,  1.85s/it]
Outer training loop:  85%|########5 | 42739/50000 [28:17:50<3:46:18,  1.87s/it]
Outer training loop:  85%|########5 | 42740/50000 [28:17:52<3:44:40,  1.86s/it]
Outer training loop:  85%|########5 | 42741/50000 [28:17:54<3:49:59,  1.90s/it]
Outer training loop:  85%|########5 | 42742/50000 [28:17:56<3:46:35,  1.87s/it]
Outer training loop:  85%|########5 | 42743/50000 [28:17:58<3:42:44,  1.84s/it]
Outer training loop:  85%|########5 | 42744/50000 [28:18:00<3:51:49,  1.92s/it]
Outer training loop:  85%|########5 | 42745/50000 [28:18:02<3:44:38,  1.86s/it]
Outer training loop:  85%|########5 | 42746/50000 [28:18:03<3:42:26,  1.84s/it]
Outer training loop:  85%|########5 | 42747/50000 [28:18:05<3:49:03,  1.89s/it]
Outer training loop:  85%|########5 | 42748/50000 [28:18:07<3:51:04,  1.91s/it]
Outer training loop:  85%|########5 | 42749/50000 [28:18:09<3:50:32,  1.91s/it]
Outer training loop:  86%|########5 | 42750/50000 [28:18:11<3:46:58,  1.88s/it]
Outer training loop:  86%|########5 | 42751/50000 [28:18:13<3:53:08,  1.93s/it]
Outer training loop:  86%|########5 | 42752/50000 [28:18:15<3:46:24,  1.87s/it]
Outer training loop:  86%|########5 | 42753/50000 [28:18:17<3:46:20,  1.87s/it]
Outer training loop:  86%|########5 | 42754/50000 [28:18:19<3:51:15,  1.91s/it]
Outer training loop:  86%|########5 | 42755/50000 [28:18:21<3:52:55,  1.93s/it]
Outer training loop:  86%|########5 | 42756/50000 [28:18:22<3:46:25,  1.88s/it]
Outer training loop:  86%|########5 | 42757/50000 [28:18:24<3:52:43,  1.93s/it]
Outer training loop:  86%|########5 | 42758/50000 [28:18:26<3:55:37,  1.95s/it]
Outer training loop:  86%|########5 | 42759/50000 [28:18:28<3:52:06,  1.92s/it]
Outer training loop:  86%|########5 | 42760/50000 [28:18:30<3:49:11,  1.90s/it]
Outer training loop:  86%|########5 | 42761/50000 [28:18:32<3:54:05,  1.94s/it]
Outer training loop:  86%|########5 | 42762/50000 [28:18:34<3:48:22,  1.89s/it]
Outer training loop:  86%|########5 | 42763/50000 [28:18:36<3:47:47,  1.89s/it]
Outer training loop:  86%|########5 | 42764/50000 [28:18:38<3:49:01,  1.90s/it]
Outer training loop:  86%|########5 | 42765/50000 [28:18:40<3:47:52,  1.89s/it]
Outer training loop:  86%|########5 | 42766/50000 [28:18:41<3:43:13,  1.85s/it]
Outer training loop:  86%|########5 | 42767/50000 [28:18:43<3:41:39,  1.84s/it]
Outer training loop:  86%|########5 | 42768/50000 [28:18:45<3:52:52,  1.93s/it]
Outer training loop:  86%|########5 | 42769/50000 [28:18:47<3:46:29,  1.88s/it]
Outer training loop:  86%|########5 | 42770/50000 [28:18:49<3:46:27,  1.88s/it]
Outer training loop:  86%|########5 | 42771/50000 [28:18:51<3:54:03,  1.94s/it]
Outer training loop:  86%|########5 | 42772/50000 [28:18:53<3:49:36,  1.91s/it]
Outer training loop:  86%|########5 | 42773/50000 [28:18:55<3:50:57,  1.92s/it]
Outer training loop:  86%|########5 | 42774/50000 [28:18:57<3:42:56,  1.85s/it]
Outer training loop:  86%|########5 | 42775/50000 [28:18:59<3:52:43,  1.93s/it]
Outer training loop:  86%|########5 | 42776/50000 [28:19:01<3:47:43,  1.89s/it]
Outer training loop:  86%|########5 | 42777/50000 [28:19:02<3:44:11,  1.86s/it]
Outer training loop:  86%|########5 | 42778/50000 [28:19:04<3:56:05,  1.96s/it]
Outer training loop:  86%|########5 | 42779/50000 [28:19:06<3:54:10,  1.95s/it]
Outer training loop:  86%|########5 | 42780/50000 [28:19:08<3:50:07,  1.91s/it]
Outer training loop:  86%|########5 | 42781/50000 [28:19:10<3:47:08,  1.89s/it]
Outer training loop:  86%|########5 | 42782/50000 [28:19:12<3:46:09,  1.88s/it]
Outer training loop:  86%|########5 | 42783/50000 [28:19:14<3:45:05,  1.87s/it]
Outer training loop:  86%|########5 | 42784/50000 [28:19:16<3:44:27,  1.87s/it]
Outer training loop:  86%|########5 | 42785/50000 [28:19:17<3:40:50,  1.84s/it]
Outer training loop:  86%|########5 | 42786/50000 [28:19:19<3:43:31,  1.86s/it]
Outer training loop:  86%|########5 | 42787/50000 [28:19:21<3:43:49,  1.86s/it]
Outer training loop:  86%|########5 | 42788/50000 [28:19:23<3:41:18,  1.84s/it]
Outer training loop:  86%|########5 | 42789/50000 [28:19:25<3:49:15,  1.91s/it]
Outer training loop:  86%|########5 | 42790/50000 [28:19:27<3:44:08,  1.87s/it]
Outer training loop:  86%|########5 | 42791/50000 [28:19:29<3:42:42,  1.85s/it]
Outer training loop:  86%|########5 | 42792/50000 [28:19:30<3:43:02,  1.86s/it]
Outer training loop:  86%|########5 | 42793/50000 [28:19:32<3:44:46,  1.87s/it]
Outer training loop:  86%|########5 | 42794/50000 [28:19:34<3:46:46,  1.89s/it]
Outer training loop:  86%|########5 | 42795/50000 [28:19:36<3:43:41,  1.86s/it]
Outer training loop:  86%|########5 | 42796/50000 [28:19:38<3:46:16,  1.88s/it]
Outer training loop:  86%|########5 | 42797/50000 [28:19:40<3:46:29,  1.89s/it]
Outer training loop:  86%|########5 | 42798/50000 [28:19:42<3:42:09,  1.85s/it]
Outer training loop:  86%|########5 | 42799/50000 [28:19:44<3:45:47,  1.88s/it]
Outer training loop:  86%|########5 | 42800/50000 [28:19:46<3:44:38,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  86%|########5 | 42801/50000 [28:19:53<7:04:45,  3.54s/it]
Outer training loop:  86%|########5 | 42802/50000 [28:19:55<6:05:08,  3.04s/it]
Outer training loop:  86%|########5 | 42803/50000 [28:19:57<5:22:55,  2.69s/it]
Outer training loop:  86%|########5 | 42804/50000 [28:19:59<5:11:54,  2.60s/it]
Outer training loop:  86%|########5 | 42805/50000 [28:20:01<4:44:52,  2.38s/it]
Outer training loop:  86%|########5 | 42806/50000 [28:20:03<4:25:02,  2.21s/it]
Outer training loop:  86%|########5 | 42807/50000 [28:20:05<4:21:30,  2.18s/it]
Outer training loop:  86%|########5 | 42808/50000 [28:20:07<4:19:35,  2.17s/it]
Outer training loop:  86%|########5 | 42809/50000 [28:20:09<4:04:33,  2.04s/it]
Outer training loop:  86%|########5 | 42810/50000 [28:20:11<4:07:12,  2.06s/it]
Outer training loop:  86%|########5 | 42811/50000 [28:20:13<4:00:42,  2.01s/it]
Outer training loop:  86%|########5 | 42812/50000 [28:20:15<3:55:52,  1.97s/it]
Outer training loop:  86%|########5 | 42813/50000 [28:20:16<3:48:26,  1.91s/it]
Outer training loop:  86%|########5 | 42814/50000 [28:20:18<3:48:34,  1.91s/it]
Outer training loop:  86%|########5 | 42815/50000 [28:20:20<3:44:47,  1.88s/it]
Outer training loop:  86%|########5 | 42816/50000 [28:20:22<3:45:36,  1.88s/it]
Outer training loop:  86%|########5 | 42817/50000 [28:20:24<3:51:24,  1.93s/it]
Outer training loop:  86%|########5 | 42818/50000 [28:20:26<3:49:28,  1.92s/it]
Outer training loop:  86%|########5 | 42819/50000 [28:20:28<3:46:53,  1.90s/it]
Outer training loop:  86%|########5 | 42820/50000 [28:20:30<3:51:02,  1.93s/it]
Outer training loop:  86%|########5 | 42821/50000 [28:20:32<3:55:40,  1.97s/it]
Outer training loop:  86%|########5 | 42822/50000 [28:20:34<3:54:40,  1.96s/it]
Outer training loop:  86%|########5 | 42823/50000 [28:20:36<3:47:01,  1.90s/it]
Outer training loop:  86%|########5 | 42824/50000 [28:20:37<3:46:46,  1.90s/it]
Outer training loop:  86%|########5 | 42825/50000 [28:20:39<3:51:41,  1.94s/it]
Outer training loop:  86%|########5 | 42826/50000 [28:20:41<3:50:07,  1.92s/it]
Outer training loop:  86%|########5 | 42827/50000 [28:20:43<3:52:40,  1.95s/it]
Outer training loop:  86%|########5 | 42828/50000 [28:20:45<3:55:00,  1.97s/it]
Outer training loop:  86%|########5 | 42829/50000 [28:20:47<3:50:47,  1.93s/it]
Outer training loop:  86%|########5 | 42830/50000 [28:20:49<3:48:20,  1.91s/it]
Outer training loop:  86%|########5 | 42831/50000 [28:20:51<3:55:50,  1.97s/it]
Outer training loop:  86%|########5 | 42832/50000 [28:20:53<3:48:47,  1.92s/it]
Outer training loop:  86%|########5 | 42833/50000 [28:20:55<3:50:43,  1.93s/it]
Outer training loop:  86%|########5 | 42834/50000 [28:20:57<4:01:23,  2.02s/it]
Outer training loop:  86%|########5 | 42835/50000 [28:20:59<3:56:36,  1.98s/it]
Outer training loop:  86%|########5 | 42836/50000 [28:21:01<3:54:15,  1.96s/it]
Outer training loop:  86%|########5 | 42837/50000 [28:21:03<3:51:35,  1.94s/it]
Outer training loop:  86%|########5 | 42838/50000 [28:21:05<3:57:49,  1.99s/it]
Outer training loop:  86%|########5 | 42839/50000 [28:21:07<3:51:56,  1.94s/it]
Outer training loop:  86%|########5 | 42840/50000 [28:21:09<3:46:18,  1.90s/it]
Outer training loop:  86%|########5 | 42841/50000 [28:21:11<3:57:50,  1.99s/it]
Outer training loop:  86%|########5 | 42842/50000 [28:21:13<3:55:25,  1.97s/it]
Outer training loop:  86%|########5 | 42843/50000 [28:21:15<3:51:58,  1.94s/it]
Outer training loop:  86%|########5 | 42844/50000 [28:21:17<3:50:37,  1.93s/it]
Outer training loop:  86%|########5 | 42845/50000 [28:21:19<3:52:40,  1.95s/it]
Outer training loop:  86%|########5 | 42846/50000 [28:21:21<3:54:49,  1.97s/it]
Outer training loop:  86%|########5 | 42847/50000 [28:21:22<3:51:32,  1.94s/it]
Outer training loop:  86%|########5 | 42848/50000 [28:21:25<4:01:05,  2.02s/it]
Outer training loop:  86%|########5 | 42849/50000 [28:21:27<3:55:02,  1.97s/it]
Outer training loop:  86%|########5 | 42850/50000 [28:21:28<3:50:50,  1.94s/it]
Outer training loop:  86%|########5 | 42851/50000 [28:21:30<3:48:54,  1.92s/it]
Outer training loop:  86%|########5 | 42852/50000 [28:21:32<3:56:16,  1.98s/it]
Outer training loop:  86%|########5 | 42853/50000 [28:21:34<3:52:39,  1.95s/it]
Outer training loop:  86%|########5 | 42854/50000 [28:21:36<3:57:43,  2.00s/it]
Outer training loop:  86%|########5 | 42855/50000 [28:21:39<4:04:22,  2.05s/it]
Outer training loop:  86%|########5 | 42856/50000 [28:21:40<3:55:50,  1.98s/it]
Outer training loop:  86%|########5 | 42857/50000 [28:21:42<3:59:11,  2.01s/it]
Outer training loop:  86%|########5 | 42858/50000 [28:21:44<3:56:58,  1.99s/it]
Outer training loop:  86%|########5 | 42859/50000 [28:21:46<3:59:23,  2.01s/it]
Outer training loop:  86%|########5 | 42860/50000 [28:21:48<3:54:13,  1.97s/it]
Outer training loop:  86%|########5 | 42861/50000 [28:21:50<3:54:33,  1.97s/it]
Outer training loop:  86%|########5 | 42862/50000 [28:21:52<4:02:27,  2.04s/it]
Outer training loop:  86%|########5 | 42863/50000 [28:21:54<4:00:23,  2.02s/it]
Outer training loop:  86%|########5 | 42864/50000 [28:21:56<4:00:12,  2.02s/it]
Outer training loop:  86%|########5 | 42865/50000 [28:21:59<4:05:34,  2.07s/it]
Outer training loop:  86%|########5 | 42866/50000 [28:22:01<4:00:56,  2.03s/it]
Outer training loop:  86%|########5 | 42867/50000 [28:22:02<3:56:37,  1.99s/it]
Outer training loop:  86%|########5 | 42868/50000 [28:22:04<3:54:12,  1.97s/it]
Outer training loop:  86%|########5 | 42869/50000 [28:22:07<4:01:51,  2.03s/it]
Outer training loop:  86%|########5 | 42870/50000 [28:22:09<3:58:54,  2.01s/it]
Outer training loop:  86%|########5 | 42871/50000 [28:22:10<3:56:22,  1.99s/it]
Outer training loop:  86%|########5 | 42872/50000 [28:22:13<3:58:15,  2.01s/it]
Outer training loop:  86%|########5 | 42873/50000 [28:22:14<3:54:18,  1.97s/it]
Outer training loop:  86%|########5 | 42874/50000 [28:22:16<3:52:30,  1.96s/it]
Outer training loop:  86%|########5 | 42875/50000 [28:22:18<3:49:33,  1.93s/it]
Outer training loop:  86%|########5 | 42876/50000 [28:22:20<3:57:57,  2.00s/it]
Outer training loop:  86%|########5 | 42877/50000 [28:22:22<3:47:10,  1.91s/it]
Outer training loop:  86%|########5 | 42878/50000 [28:22:24<3:41:23,  1.87s/it]
Outer training loop:  86%|########5 | 42879/50000 [28:22:26<3:52:43,  1.96s/it]
Outer training loop:  86%|########5 | 42880/50000 [28:22:28<3:48:40,  1.93s/it]
Outer training loop:  86%|########5 | 42881/50000 [28:22:30<3:46:22,  1.91s/it]
Outer training loop:  86%|########5 | 42882/50000 [28:22:32<3:48:31,  1.93s/it]
Outer training loop:  86%|########5 | 42883/50000 [28:22:34<3:45:47,  1.90s/it]
Outer training loop:  86%|########5 | 42884/50000 [28:22:35<3:41:02,  1.86s/it]
Outer training loop:  86%|########5 | 42885/50000 [28:22:37<3:44:01,  1.89s/it]
Outer training loop:  86%|########5 | 42886/50000 [28:22:39<3:51:52,  1.96s/it]
Outer training loop:  86%|########5 | 42887/50000 [28:22:41<3:51:28,  1.95s/it]
Outer training loop:  86%|########5 | 42888/50000 [28:22:43<3:48:48,  1.93s/it]
Outer training loop:  86%|########5 | 42889/50000 [28:22:45<3:50:28,  1.94s/it]
Outer training loop:  86%|########5 | 42890/50000 [28:22:47<3:46:55,  1.92s/it]
Outer training loop:  86%|########5 | 42891/50000 [28:22:49<3:46:10,  1.91s/it]
Outer training loop:  86%|########5 | 42892/50000 [28:22:51<3:48:33,  1.93s/it]
Outer training loop:  86%|########5 | 42893/50000 [28:22:53<3:53:23,  1.97s/it]
Outer training loop:  86%|########5 | 42894/50000 [28:22:55<3:50:34,  1.95s/it]
Outer training loop:  86%|########5 | 42895/50000 [28:22:57<3:44:22,  1.89s/it]
Outer training loop:  86%|########5 | 42896/50000 [28:22:59<3:49:40,  1.94s/it]
Outer training loop:  86%|########5 | 42897/50000 [28:23:00<3:44:01,  1.89s/it]
Outer training loop:  86%|########5 | 42898/50000 [28:23:02<3:41:04,  1.87s/it]
Outer training loop:  86%|########5 | 42899/50000 [28:23:04<3:43:22,  1.89s/it]
Outer training loop:  86%|########5 | 42900/50000 [28:23:06<3:43:58,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.88s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.91s/it]

Outer training loop:  86%|########5 | 42901/50000 [28:23:14<7:09:12,  3.63s/it]
Outer training loop:  86%|########5 | 42902/50000 [28:23:16<6:06:23,  3.10s/it]
Outer training loop:  86%|########5 | 42903/50000 [28:23:18<5:24:18,  2.74s/it]
Outer training loop:  86%|########5 | 42904/50000 [28:23:19<4:47:22,  2.43s/it]
Outer training loop:  86%|########5 | 42905/50000 [28:23:21<4:26:44,  2.26s/it]
Outer training loop:  86%|########5 | 42906/50000 [28:23:23<4:11:25,  2.13s/it]
Outer training loop:  86%|########5 | 42907/50000 [28:23:25<4:08:20,  2.10s/it]
Outer training loop:  86%|########5 | 42908/50000 [28:23:27<4:04:20,  2.07s/it]
Outer training loop:  86%|########5 | 42909/50000 [28:23:29<3:54:29,  1.98s/it]
Outer training loop:  86%|########5 | 42910/50000 [28:23:31<3:59:21,  2.03s/it]
Outer training loop:  86%|########5 | 42911/50000 [28:23:33<3:53:39,  1.98s/it]
Outer training loop:  86%|########5 | 42912/50000 [28:23:35<3:50:54,  1.95s/it]
Outer training loop:  86%|########5 | 42913/50000 [28:23:36<3:44:52,  1.90s/it]
Outer training loop:  86%|########5 | 42914/50000 [28:23:38<3:48:54,  1.94s/it]
Outer training loop:  86%|########5 | 42915/50000 [28:23:40<3:45:14,  1.91s/it]
Outer training loop:  86%|########5 | 42916/50000 [28:23:42<3:40:40,  1.87s/it]
Outer training loop:  86%|########5 | 42917/50000 [28:23:44<3:46:08,  1.92s/it]
Outer training loop:  86%|########5 | 42918/50000 [28:23:46<3:40:52,  1.87s/it]
Outer training loop:  86%|########5 | 42919/50000 [28:23:48<3:38:02,  1.85s/it]
Outer training loop:  86%|########5 | 42920/50000 [28:23:50<3:38:53,  1.86s/it]
Outer training loop:  86%|########5 | 42921/50000 [28:23:52<3:47:28,  1.93s/it]
Outer training loop:  86%|########5 | 42922/50000 [28:23:54<3:48:46,  1.94s/it]
Outer training loop:  86%|########5 | 42923/50000 [28:23:55<3:42:53,  1.89s/it]
Outer training loop:  86%|########5 | 42924/50000 [28:23:57<3:49:38,  1.95s/it]
Outer training loop:  86%|########5 | 42925/50000 [28:23:59<3:47:30,  1.93s/it]
Outer training loop:  86%|########5 | 42926/50000 [28:24:01<3:43:34,  1.90s/it]
Outer training loop:  86%|########5 | 42927/50000 [28:24:03<3:49:01,  1.94s/it]
Outer training loop:  86%|########5 | 42928/50000 [28:24:05<3:49:35,  1.95s/it]
Outer training loop:  86%|########5 | 42929/50000 [28:24:07<3:42:57,  1.89s/it]
Outer training loop:  86%|########5 | 42930/50000 [28:24:09<3:43:07,  1.89s/it]
Outer training loop:  86%|########5 | 42931/50000 [28:24:11<3:55:35,  2.00s/it]
Outer training loop:  86%|########5 | 42932/50000 [28:24:13<3:50:18,  1.96s/it]
Outer training loop:  86%|########5 | 42933/50000 [28:24:15<3:47:12,  1.93s/it]
Outer training loop:  86%|########5 | 42934/50000 [28:24:17<3:48:47,  1.94s/it]
Outer training loop:  86%|########5 | 42935/50000 [28:24:19<3:42:56,  1.89s/it]
Outer training loop:  86%|########5 | 42936/50000 [28:24:20<3:38:39,  1.86s/it]
Outer training loop:  86%|########5 | 42937/50000 [28:24:22<3:39:46,  1.87s/it]
Outer training loop:  86%|########5 | 42938/50000 [28:24:24<3:43:47,  1.90s/it]
Outer training loop:  86%|########5 | 42939/50000 [28:24:26<3:45:09,  1.91s/it]
Outer training loop:  86%|########5 | 42940/50000 [28:24:28<3:46:05,  1.92s/it]
Outer training loop:  86%|########5 | 42941/50000 [28:24:30<3:41:52,  1.89s/it]
Outer training loop:  86%|########5 | 42942/50000 [28:24:32<3:46:09,  1.92s/it]
Outer training loop:  86%|########5 | 42943/50000 [28:24:34<3:42:24,  1.89s/it]
Outer training loop:  86%|########5 | 42944/50000 [28:24:36<3:43:27,  1.90s/it]
Outer training loop:  86%|########5 | 42945/50000 [28:24:38<3:43:43,  1.90s/it]
Outer training loop:  86%|########5 | 42946/50000 [28:24:39<3:43:02,  1.90s/it]
Outer training loop:  86%|########5 | 42947/50000 [28:24:41<3:42:47,  1.90s/it]
Outer training loop:  86%|########5 | 42948/50000 [28:24:43<3:46:34,  1.93s/it]
Outer training loop:  86%|########5 | 42949/50000 [28:24:45<3:47:35,  1.94s/it]
Outer training loop:  86%|########5 | 42950/50000 [28:24:47<3:44:56,  1.91s/it]
Outer training loop:  86%|########5 | 42951/50000 [28:24:49<3:59:00,  2.03s/it]
Outer training loop:  86%|########5 | 42952/50000 [28:24:51<3:51:58,  1.97s/it]
Outer training loop:  86%|########5 | 42953/50000 [28:24:53<3:48:07,  1.94s/it]
Outer training loop:  86%|########5 | 42954/50000 [28:24:55<3:53:36,  1.99s/it]
Outer training loop:  86%|########5 | 42955/50000 [28:24:57<3:46:47,  1.93s/it]
Outer training loop:  86%|########5 | 42956/50000 [28:24:59<3:44:31,  1.91s/it]
Outer training loop:  86%|########5 | 42957/50000 [28:25:01<3:48:50,  1.95s/it]
Outer training loop:  86%|########5 | 42958/50000 [28:25:03<3:45:06,  1.92s/it]
Outer training loop:  86%|########5 | 42959/50000 [28:25:05<3:44:10,  1.91s/it]
Outer training loop:  86%|########5 | 42960/50000 [28:25:07<3:42:57,  1.90s/it]
Outer training loop:  86%|########5 | 42961/50000 [28:25:09<3:48:11,  1.95s/it]
Outer training loop:  86%|########5 | 42962/50000 [28:25:10<3:44:31,  1.91s/it]
Outer training loop:  86%|########5 | 42963/50000 [28:25:12<3:45:24,  1.92s/it]
Outer training loop:  86%|########5 | 42964/50000 [28:25:14<3:48:37,  1.95s/it]
Outer training loop:  86%|########5 | 42965/50000 [28:25:16<3:46:01,  1.93s/it]
Outer training loop:  86%|########5 | 42966/50000 [28:25:18<3:43:50,  1.91s/it]
Outer training loop:  86%|########5 | 42967/50000 [28:25:20<3:42:40,  1.90s/it]
Outer training loop:  86%|########5 | 42968/50000 [28:25:22<3:45:48,  1.93s/it]
Outer training loop:  86%|########5 | 42969/50000 [28:25:24<3:44:26,  1.92s/it]
Outer training loop:  86%|########5 | 42970/50000 [28:25:26<3:45:41,  1.93s/it]
Outer training loop:  86%|########5 | 42971/50000 [28:25:28<3:50:59,  1.97s/it]
Outer training loop:  86%|########5 | 42972/50000 [28:25:30<3:48:33,  1.95s/it]
Outer training loop:  86%|########5 | 42973/50000 [28:25:32<3:41:56,  1.90s/it]
Outer training loop:  86%|########5 | 42974/50000 [28:25:33<3:40:21,  1.88s/it]
Outer training loop:  86%|########5 | 42975/50000 [28:25:36<3:50:09,  1.97s/it]
Outer training loop:  86%|########5 | 42976/50000 [28:25:37<3:42:18,  1.90s/it]
Outer training loop:  86%|########5 | 42977/50000 [28:25:39<3:39:25,  1.87s/it]
Outer training loop:  86%|########5 | 42978/50000 [28:25:41<3:44:00,  1.91s/it]
Outer training loop:  86%|########5 | 42979/50000 [28:25:43<3:40:47,  1.89s/it]
Outer training loop:  86%|########5 | 42980/50000 [28:25:45<3:36:07,  1.85s/it]
Outer training loop:  86%|########5 | 42981/50000 [28:25:47<3:42:39,  1.90s/it]
Outer training loop:  86%|########5 | 42982/50000 [28:25:49<3:41:50,  1.90s/it]
Outer training loop:  86%|########5 | 42983/50000 [28:25:50<3:35:33,  1.84s/it]
Outer training loop:  86%|########5 | 42984/50000 [28:25:52<3:30:48,  1.80s/it]
Outer training loop:  86%|########5 | 42985/50000 [28:25:54<3:43:15,  1.91s/it]
Outer training loop:  86%|########5 | 42986/50000 [28:25:56<3:43:58,  1.92s/it]
Outer training loop:  86%|########5 | 42987/50000 [28:25:58<3:42:20,  1.90s/it]
Outer training loop:  86%|########5 | 42988/50000 [28:26:00<3:49:55,  1.97s/it]
Outer training loop:  86%|########5 | 42989/50000 [28:26:02<3:42:33,  1.90s/it]
Outer training loop:  86%|########5 | 42990/50000 [28:26:04<3:38:38,  1.87s/it]
Outer training loop:  86%|########5 | 42991/50000 [28:26:06<3:40:05,  1.88s/it]
Outer training loop:  86%|########5 | 42992/50000 [28:26:08<3:47:00,  1.94s/it]
Outer training loop:  86%|########5 | 42993/50000 [28:26:10<3:45:18,  1.93s/it]
Outer training loop:  86%|########5 | 42994/50000 [28:26:11<3:40:12,  1.89s/it]
Outer training loop:  86%|########5 | 42995/50000 [28:26:13<3:42:42,  1.91s/it]
Outer training loop:  86%|########5 | 42996/50000 [28:26:15<3:36:16,  1.85s/it]
Outer training loop:  86%|########5 | 42997/50000 [28:26:17<3:38:38,  1.87s/it]
Outer training loop:  86%|########5 | 42998/50000 [28:26:19<3:39:25,  1.88s/it]
Outer training loop:  86%|########5 | 42999/50000 [28:26:21<3:45:52,  1.94s/it]
Outer training loop:  86%|########6 | 43000/50000 [28:26:23<3:42:38,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.05s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.05s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.94s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:02<02:53,  2.12s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:48,  2.09s/it]

Evaluating policy:   4%|3         | 3/83 [00:06<02:45,  2.07s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:36,  1.98s/it]

Evaluating policy:   6%|6         | 5/83 [00:10<02:37,  2.02s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:31,  1.97s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:28,  2.00s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:22,  1.95s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:20,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:18,  1.98s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:09,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:03,  1.85s/it]

Evaluating policy:  20%|##        | 17/83 [00:33<02:07,  1.93s/it]

Evaluating policy:  22%|##1       | 18/83 [00:35<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.93s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:02,  1.95s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<02:00,  1.98s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:57,  1.96s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:51,  1.92s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:48,  1.90s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:51,  1.99s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:50,  2.02s/it]

Evaluating policy:  35%|###4      | 29/83 [00:56<01:45,  1.96s/it]

Evaluating policy:  36%|###6      | 30/83 [00:58<01:43,  1.96s/it]

Evaluating policy:  37%|###7      | 31/83 [01:00<01:41,  1.95s/it]

Evaluating policy:  39%|###8      | 32/83 [01:02<01:42,  2.01s/it]

Evaluating policy:  40%|###9      | 33/83 [01:04<01:38,  1.98s/it]

Evaluating policy:  41%|####      | 34/83 [01:06<01:34,  1.94s/it]

Evaluating policy:  42%|####2     | 35/83 [01:08<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:10<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:26,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:27,  1.94s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:24,  1.92s/it]

Evaluating policy:  48%|####8     | 40/83 [01:17<01:20,  1.88s/it]

Evaluating policy:  49%|####9     | 41/83 [01:19<01:17,  1.85s/it]

Evaluating policy:  51%|#####     | 42/83 [01:21<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:23<01:13,  1.84s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:25<01:15,  1.95s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:27<01:13,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:09,  1.87s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:07,  1.88s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:36<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<01:02,  1.95s/it]

Evaluating policy:  63%|######2   | 52/83 [01:40<01:00,  1.95s/it]

Evaluating policy:  64%|######3   | 53/83 [01:42<00:58,  1.94s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:54,  1.88s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:52,  1.89s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:51,  1.90s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:50,  1.94s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:47,  1.92s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:48,  2.02s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:46,  2.01s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:43,  1.98s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:40,  1.95s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:01<00:38,  1.92s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:03<00:36,  1.91s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:05<00:33,  1.89s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:07<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:09<00:32,  2.01s/it]

Evaluating policy:  82%|########1 | 68/83 [02:11<00:29,  1.95s/it]

Evaluating policy:  83%|########3 | 69/83 [02:13<00:26,  1.89s/it]

Evaluating policy:  84%|########4 | 70/83 [02:14<00:24,  1.89s/it]

Evaluating policy:  86%|########5 | 71/83 [02:16<00:22,  1.88s/it]

Evaluating policy:  87%|########6 | 72/83 [02:18<00:20,  1.88s/it]

Evaluating policy:  88%|########7 | 73/83 [02:20<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:22<00:17,  1.92s/it]

Evaluating policy:  90%|######### | 75/83 [02:24<00:15,  1.94s/it]

Evaluating policy:  92%|#########1| 76/83 [02:27<00:14,  2.11s/it]

Evaluating policy:  93%|#########2| 77/83 [02:28<00:12,  2.03s/it]

Evaluating policy:  94%|#########3| 78/83 [02:30<00:09,  1.96s/it]

Evaluating policy:  95%|#########5| 79/83 [02:32<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:34<00:05,  1.94s/it]

Evaluating policy:  98%|#########7| 81/83 [02:36<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:38<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:39<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:26,  1.79s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.79s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:36,  1.96s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:35,  1.97s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.94s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:33,  1.99s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:30,  1.98s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:22,  1.96s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:16,  1.93s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:15,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:22,  2.07s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:17,  2.03s/it]

Evaluating policy:  19%|#9        | 16/83 [00:31<02:15,  2.02s/it]

Evaluating policy:  20%|##        | 17/83 [00:33<02:09,  1.96s/it]

Evaluating policy:  22%|##1       | 18/83 [00:35<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:37<02:06,  1.98s/it]

Evaluating policy:  24%|##4       | 20/83 [00:39<02:05,  2.00s/it]

Evaluating policy:  25%|##5       | 21/83 [00:41<02:00,  1.94s/it]

Evaluating policy:  27%|##6       | 22/83 [00:43<01:55,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:45<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:47<01:56,  1.97s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:52,  1.93s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:48,  1.90s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:44,  1.86s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:43,  1.89s/it]

Evaluating policy:  35%|###4      | 29/83 [00:56<01:45,  1.96s/it]

Evaluating policy:  36%|###6      | 30/83 [00:58<01:41,  1.92s/it]

Evaluating policy:  37%|###7      | 31/83 [01:00<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:02<01:34,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:33,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:08<01:35,  2.00s/it]

Evaluating policy:  43%|####3     | 36/83 [01:10<01:34,  2.00s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:29,  1.95s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:26,  1.91s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:17<01:22,  1.92s/it]

Evaluating policy:  49%|####9     | 41/83 [01:19<01:24,  2.01s/it]

Evaluating policy:  51%|#####     | 42/83 [01:21<01:20,  1.95s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:23<01:17,  1.93s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:25<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:27<01:13,  1.93s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:29<01:09,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:31<01:11,  1.98s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:33<01:07,  1.92s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:04,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:36<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:40<00:58,  1.87s/it]

Evaluating policy:  64%|######3   | 53/83 [01:42<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:44<00:56,  1.96s/it]

Evaluating policy:  66%|######6   | 55/83 [01:46<00:53,  1.92s/it]

Evaluating policy:  67%|######7   | 56/83 [01:48<00:51,  1.92s/it]

Evaluating policy:  69%|######8   | 57/83 [01:50<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:42,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:59<00:42,  2.02s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:01<00:39,  1.99s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:03<00:38,  2.00s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:05<00:35,  1.95s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:07<00:32,  1.91s/it]

Evaluating policy:  81%|########  | 67/83 [02:09<00:30,  1.91s/it]

Evaluating policy:  82%|########1 | 68/83 [02:11<00:28,  1.89s/it]

Evaluating policy:  83%|########3 | 69/83 [02:13<00:26,  1.89s/it]

Evaluating policy:  84%|########4 | 70/83 [02:15<00:26,  2.05s/it]

Evaluating policy:  86%|########5 | 71/83 [02:17<00:23,  2.00s/it]

Evaluating policy:  87%|########6 | 72/83 [02:19<00:20,  1.91s/it]

Evaluating policy:  88%|########7 | 73/83 [02:20<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:22<00:16,  1.88s/it]

Evaluating policy:  90%|######### | 75/83 [02:24<00:15,  1.90s/it]

Evaluating policy:  92%|#########1| 76/83 [02:26<00:13,  1.88s/it]

Evaluating policy:  93%|#########2| 77/83 [02:28<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:30<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:32<00:08,  2.07s/it]

Evaluating policy:  96%|#########6| 80/83 [02:34<00:06,  2.00s/it]

Evaluating policy:  98%|#########7| 81/83 [02:36<00:03,  1.92s/it]

Evaluating policy:  99%|#########8| 82/83 [02:38<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 83/83 [02:40<00:00,  1.93s/it]

Outer training loop:  86%|########6 | 43001/50000 [28:31:51<194:08:34, 99.86s/it]
Outer training loop:  86%|########6 | 43002/50000 [28:31:53<136:59:05, 70.47s/it]
Outer training loop:  86%|########6 | 43003/50000 [28:31:55<97:00:41, 49.91s/it] 
Outer training loop:  86%|########6 | 43004/50000 [28:31:57<68:55:37, 35.47s/it]
Outer training loop:  86%|########6 | 43005/50000 [28:31:59<49:21:24, 25.40s/it]
Outer training loop:  86%|########6 | 43006/50000 [28:32:00<35:33:31, 18.30s/it]
Outer training loop:  86%|########6 | 43007/50000 [28:32:03<26:18:34, 13.54s/it]
Outer training loop:  86%|########6 | 43008/50000 [28:32:05<19:36:20, 10.09s/it]
Outer training loop:  86%|########6 | 43009/50000 [28:32:07<14:46:30,  7.61s/it]
Outer training loop:  86%|########6 | 43010/50000 [28:32:09<11:22:50,  5.86s/it]
Outer training loop:  86%|########6 | 43011/50000 [28:32:11<9:08:17,  4.71s/it] 
Outer training loop:  86%|########6 | 43012/50000 [28:32:12<7:30:16,  3.87s/it]
Outer training loop:  86%|########6 | 43013/50000 [28:32:14<6:17:39,  3.24s/it]
Outer training loop:  86%|########6 | 43014/50000 [28:32:16<5:34:48,  2.88s/it]
Outer training loop:  86%|########6 | 43015/50000 [28:32:18<4:58:46,  2.57s/it]
Outer training loop:  86%|########6 | 43016/50000 [28:32:20<4:33:55,  2.35s/it]
Outer training loop:  86%|########6 | 43017/50000 [28:32:22<4:22:54,  2.26s/it]
Outer training loop:  86%|########6 | 43018/50000 [28:32:24<4:08:08,  2.13s/it]
Outer training loop:  86%|########6 | 43019/50000 [28:32:26<4:03:10,  2.09s/it]
Outer training loop:  86%|########6 | 43020/50000 [28:32:28<3:58:07,  2.05s/it]
Outer training loop:  86%|########6 | 43021/50000 [28:32:30<3:56:24,  2.03s/it]
Outer training loop:  86%|########6 | 43022/50000 [28:32:32<3:53:35,  2.01s/it]
Outer training loop:  86%|########6 | 43023/50000 [28:32:34<3:50:41,  1.98s/it]
Outer training loop:  86%|########6 | 43024/50000 [28:32:36<3:47:07,  1.95s/it]
Outer training loop:  86%|########6 | 43025/50000 [28:32:38<3:50:21,  1.98s/it]
Outer training loop:  86%|########6 | 43026/50000 [28:32:40<3:47:37,  1.96s/it]
Outer training loop:  86%|########6 | 43027/50000 [28:32:41<3:37:17,  1.87s/it]
Outer training loop:  86%|########6 | 43028/50000 [28:32:43<3:41:42,  1.91s/it]
Outer training loop:  86%|########6 | 43029/50000 [28:32:45<3:36:56,  1.87s/it]
Outer training loop:  86%|########6 | 43030/50000 [28:32:47<3:34:28,  1.85s/it]
Outer training loop:  86%|########6 | 43031/50000 [28:32:49<3:43:03,  1.92s/it]
Outer training loop:  86%|########6 | 43032/50000 [28:32:51<3:47:24,  1.96s/it]
Outer training loop:  86%|########6 | 43033/50000 [28:32:53<3:42:51,  1.92s/it]
Outer training loop:  86%|########6 | 43034/50000 [28:32:55<3:39:48,  1.89s/it]
Outer training loop:  86%|########6 | 43035/50000 [28:32:57<3:43:43,  1.93s/it]
Outer training loop:  86%|########6 | 43036/50000 [28:32:58<3:39:20,  1.89s/it]
Outer training loop:  86%|########6 | 43037/50000 [28:33:00<3:41:28,  1.91s/it]
Outer training loop:  86%|########6 | 43038/50000 [28:33:02<3:45:03,  1.94s/it]
Outer training loop:  86%|########6 | 43039/50000 [28:33:04<3:43:48,  1.93s/it]
Outer training loop:  86%|########6 | 43040/50000 [28:33:06<3:42:49,  1.92s/it]
Outer training loop:  86%|########6 | 43041/50000 [28:33:08<3:42:23,  1.92s/it]
Outer training loop:  86%|########6 | 43042/50000 [28:33:10<3:47:19,  1.96s/it]
Outer training loop:  86%|########6 | 43043/50000 [28:33:12<3:41:58,  1.91s/it]
Outer training loop:  86%|########6 | 43044/50000 [28:33:14<3:39:35,  1.89s/it]
Outer training loop:  86%|########6 | 43045/50000 [28:33:16<3:39:34,  1.89s/it]
Outer training loop:  86%|########6 | 43046/50000 [28:33:17<3:37:34,  1.88s/it]
Outer training loop:  86%|########6 | 43047/50000 [28:33:19<3:38:20,  1.88s/it]
Outer training loop:  86%|########6 | 43048/50000 [28:33:21<3:35:58,  1.86s/it]
Outer training loop:  86%|########6 | 43049/50000 [28:33:23<3:37:42,  1.88s/it]
Outer training loop:  86%|########6 | 43050/50000 [28:33:25<3:40:03,  1.90s/it]
Outer training loop:  86%|########6 | 43051/50000 [28:33:27<3:39:34,  1.90s/it]
Outer training loop:  86%|########6 | 43052/50000 [28:33:29<3:47:03,  1.96s/it]
Outer training loop:  86%|########6 | 43053/50000 [28:33:31<3:42:03,  1.92s/it]
Outer training loop:  86%|########6 | 43054/50000 [28:33:33<3:37:23,  1.88s/it]
Outer training loop:  86%|########6 | 43055/50000 [28:33:35<3:49:21,  1.98s/it]
Outer training loop:  86%|########6 | 43056/50000 [28:33:37<3:49:45,  1.99s/it]
Outer training loop:  86%|########6 | 43057/50000 [28:33:39<3:44:40,  1.94s/it]
Outer training loop:  86%|########6 | 43058/50000 [28:33:41<3:45:57,  1.95s/it]
Outer training loop:  86%|########6 | 43059/50000 [28:33:43<3:49:24,  1.98s/it]
Outer training loop:  86%|########6 | 43060/50000 [28:33:45<3:45:02,  1.95s/it]
Outer training loop:  86%|########6 | 43061/50000 [28:33:47<3:43:32,  1.93s/it]
Outer training loop:  86%|########6 | 43062/50000 [28:33:49<3:46:25,  1.96s/it]
Outer training loop:  86%|########6 | 43063/50000 [28:33:51<3:48:01,  1.97s/it]
Outer training loop:  86%|########6 | 43064/50000 [28:33:52<3:44:26,  1.94s/it]
Outer training loop:  86%|########6 | 43065/50000 [28:33:54<3:38:22,  1.89s/it]
Outer training loop:  86%|########6 | 43066/50000 [28:33:56<3:42:49,  1.93s/it]
Outer training loop:  86%|########6 | 43067/50000 [28:33:58<3:42:55,  1.93s/it]
Outer training loop:  86%|########6 | 43068/50000 [28:34:00<3:40:31,  1.91s/it]
Outer training loop:  86%|########6 | 43069/50000 [28:34:02<3:41:21,  1.92s/it]
Outer training loop:  86%|########6 | 43070/50000 [28:34:04<3:38:48,  1.89s/it]
Outer training loop:  86%|########6 | 43071/50000 [28:34:06<3:37:15,  1.88s/it]
Outer training loop:  86%|########6 | 43072/50000 [28:34:08<3:40:53,  1.91s/it]
Outer training loop:  86%|########6 | 43073/50000 [28:34:09<3:38:53,  1.90s/it]
Outer training loop:  86%|########6 | 43074/50000 [28:34:11<3:37:27,  1.88s/it]
Outer training loop:  86%|########6 | 43075/50000 [28:34:13<3:35:35,  1.87s/it]
Outer training loop:  86%|########6 | 43076/50000 [28:34:15<3:42:27,  1.93s/it]
Outer training loop:  86%|########6 | 43077/50000 [28:34:17<3:41:40,  1.92s/it]
Outer training loop:  86%|########6 | 43078/50000 [28:34:19<3:39:48,  1.91s/it]
Outer training loop:  86%|########6 | 43079/50000 [28:34:21<3:45:40,  1.96s/it]
Outer training loop:  86%|########6 | 43080/50000 [28:34:23<3:43:21,  1.94s/it]
Outer training loop:  86%|########6 | 43081/50000 [28:34:25<3:43:47,  1.94s/it]
Outer training loop:  86%|########6 | 43082/50000 [28:34:27<3:46:20,  1.96s/it]
Outer training loop:  86%|########6 | 43083/50000 [28:34:29<3:50:34,  2.00s/it]
Outer training loop:  86%|########6 | 43084/50000 [28:34:31<3:48:29,  1.98s/it]
Outer training loop:  86%|########6 | 43085/50000 [28:34:33<3:47:08,  1.97s/it]
Outer training loop:  86%|########6 | 43086/50000 [28:34:35<3:43:54,  1.94s/it]
Outer training loop:  86%|########6 | 43087/50000 [28:34:37<3:51:48,  2.01s/it]
Outer training loop:  86%|########6 | 43088/50000 [28:34:39<3:47:40,  1.98s/it]
Outer training loop:  86%|########6 | 43089/50000 [28:34:41<3:43:38,  1.94s/it]
Outer training loop:  86%|########6 | 43090/50000 [28:34:43<3:47:40,  1.98s/it]
Outer training loop:  86%|########6 | 43091/50000 [28:34:44<3:39:07,  1.90s/it]
Outer training loop:  86%|########6 | 43092/50000 [28:34:46<3:36:29,  1.88s/it]
Outer training loop:  86%|########6 | 43093/50000 [28:34:48<3:33:02,  1.85s/it]
Outer training loop:  86%|########6 | 43094/50000 [28:34:50<3:38:16,  1.90s/it]
Outer training loop:  86%|########6 | 43095/50000 [28:34:52<3:38:01,  1.89s/it]
Outer training loop:  86%|########6 | 43096/50000 [28:34:54<3:37:20,  1.89s/it]
Outer training loop:  86%|########6 | 43097/50000 [28:34:56<3:45:55,  1.96s/it]
Outer training loop:  86%|########6 | 43098/50000 [28:34:58<3:48:36,  1.99s/it]
Outer training loop:  86%|########6 | 43099/50000 [28:35:00<3:41:35,  1.93s/it]
Outer training loop:  86%|########6 | 43100/50000 [28:35:02<3:44:07,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.83s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  86%|########6 | 43101/50000 [28:35:09<6:52:45,  3.59s/it]
Outer training loop:  86%|########6 | 43102/50000 [28:35:11<5:51:33,  3.06s/it]
Outer training loop:  86%|########6 | 43103/50000 [28:35:13<5:10:03,  2.70s/it]
Outer training loop:  86%|########6 | 43104/50000 [28:35:15<4:45:22,  2.48s/it]
Outer training loop:  86%|########6 | 43105/50000 [28:35:17<4:28:13,  2.33s/it]
Outer training loop:  86%|########6 | 43106/50000 [28:35:19<4:10:31,  2.18s/it]
Outer training loop:  86%|########6 | 43107/50000 [28:35:21<4:08:00,  2.16s/it]
Outer training loop:  86%|########6 | 43108/50000 [28:35:23<3:57:05,  2.06s/it]
Outer training loop:  86%|########6 | 43109/50000 [28:35:25<3:49:58,  2.00s/it]
Outer training loop:  86%|########6 | 43110/50000 [28:35:26<3:42:06,  1.93s/it]
Outer training loop:  86%|########6 | 43111/50000 [28:35:28<3:46:37,  1.97s/it]
Outer training loop:  86%|########6 | 43112/50000 [28:35:30<3:43:50,  1.95s/it]
Outer training loop:  86%|########6 | 43113/50000 [28:35:32<3:36:24,  1.89s/it]
Outer training loop:  86%|########6 | 43114/50000 [28:35:34<3:42:58,  1.94s/it]
Outer training loop:  86%|########6 | 43115/50000 [28:35:36<3:41:00,  1.93s/it]
Outer training loop:  86%|########6 | 43116/50000 [28:35:38<3:37:10,  1.89s/it]
Outer training loop:  86%|########6 | 43117/50000 [28:35:40<3:36:24,  1.89s/it]
Outer training loop:  86%|########6 | 43118/50000 [28:35:42<3:39:47,  1.92s/it]
Outer training loop:  86%|########6 | 43119/50000 [28:35:43<3:37:28,  1.90s/it]
Outer training loop:  86%|########6 | 43120/50000 [28:35:45<3:34:23,  1.87s/it]
Outer training loop:  86%|########6 | 43121/50000 [28:35:47<3:37:01,  1.89s/it]
Outer training loop:  86%|########6 | 43122/50000 [28:35:49<3:40:36,  1.92s/it]
Outer training loop:  86%|########6 | 43123/50000 [28:35:51<3:42:09,  1.94s/it]
Outer training loop:  86%|########6 | 43124/50000 [28:35:53<3:40:28,  1.92s/it]
Outer training loop:  86%|########6 | 43125/50000 [28:35:55<3:49:42,  2.00s/it]
Outer training loop:  86%|########6 | 43126/50000 [28:35:57<3:47:18,  1.98s/it]
Outer training loop:  86%|########6 | 43127/50000 [28:35:59<3:48:30,  1.99s/it]
Outer training loop:  86%|########6 | 43128/50000 [28:36:01<3:49:52,  2.01s/it]
Outer training loop:  86%|########6 | 43129/50000 [28:36:03<3:48:06,  1.99s/it]
Outer training loop:  86%|########6 | 43130/50000 [28:36:05<3:45:25,  1.97s/it]
Outer training loop:  86%|########6 | 43131/50000 [28:36:07<3:39:05,  1.91s/it]
Outer training loop:  86%|########6 | 43132/50000 [28:36:09<3:45:56,  1.97s/it]
Outer training loop:  86%|########6 | 43133/50000 [28:36:11<3:40:47,  1.93s/it]
Outer training loop:  86%|########6 | 43134/50000 [28:36:13<3:37:01,  1.90s/it]
Outer training loop:  86%|########6 | 43135/50000 [28:36:14<3:33:59,  1.87s/it]
Outer training loop:  86%|########6 | 43136/50000 [28:36:17<3:40:53,  1.93s/it]
Outer training loop:  86%|########6 | 43137/50000 [28:36:19<3:41:07,  1.93s/it]
Outer training loop:  86%|########6 | 43138/50000 [28:36:21<3:44:49,  1.97s/it]
Outer training loop:  86%|########6 | 43139/50000 [28:36:23<3:44:39,  1.96s/it]
Outer training loop:  86%|########6 | 43140/50000 [28:36:24<3:45:10,  1.97s/it]
Outer training loop:  86%|########6 | 43141/50000 [28:36:26<3:41:07,  1.93s/it]
Outer training loop:  86%|########6 | 43142/50000 [28:36:28<3:44:54,  1.97s/it]
Outer training loop:  86%|########6 | 43143/50000 [28:36:30<3:38:34,  1.91s/it]
Outer training loop:  86%|########6 | 43144/50000 [28:36:32<3:40:20,  1.93s/it]
Outer training loop:  86%|########6 | 43145/50000 [28:36:34<3:43:03,  1.95s/it]
Outer training loop:  86%|########6 | 43146/50000 [28:36:36<3:45:08,  1.97s/it]
Outer training loop:  86%|########6 | 43147/50000 [28:36:38<3:44:39,  1.97s/it]
Outer training loop:  86%|########6 | 43148/50000 [28:36:40<3:47:03,  1.99s/it]
Outer training loop:  86%|########6 | 43149/50000 [28:36:42<3:49:33,  2.01s/it]
Outer training loop:  86%|########6 | 43150/50000 [28:36:44<3:44:57,  1.97s/it]
Outer training loop:  86%|########6 | 43151/50000 [28:36:46<3:41:34,  1.94s/it]
Outer training loop:  86%|########6 | 43152/50000 [28:36:48<3:50:34,  2.02s/it]
Outer training loop:  86%|########6 | 43153/50000 [28:36:50<3:51:27,  2.03s/it]
Outer training loop:  86%|########6 | 43154/50000 [28:36:52<3:47:20,  1.99s/it]
Outer training loop:  86%|########6 | 43155/50000 [28:36:54<3:39:00,  1.92s/it]
Outer training loop:  86%|########6 | 43156/50000 [28:36:56<3:42:18,  1.95s/it]
Outer training loop:  86%|########6 | 43157/50000 [28:36:58<3:44:06,  1.96s/it]
Outer training loop:  86%|########6 | 43158/50000 [28:37:00<3:42:25,  1.95s/it]
Outer training loop:  86%|########6 | 43159/50000 [28:37:02<3:50:24,  2.02s/it]
Outer training loop:  86%|########6 | 43160/50000 [28:37:04<3:46:53,  1.99s/it]
Outer training loop:  86%|########6 | 43161/50000 [28:37:06<3:44:46,  1.97s/it]
Outer training loop:  86%|########6 | 43162/50000 [28:37:08<3:40:37,  1.94s/it]
Outer training loop:  86%|########6 | 43163/50000 [28:37:10<3:45:24,  1.98s/it]
Outer training loop:  86%|########6 | 43164/50000 [28:37:12<3:45:57,  1.98s/it]
Outer training loop:  86%|########6 | 43165/50000 [28:37:14<3:46:37,  1.99s/it]
Outer training loop:  86%|########6 | 43166/50000 [28:37:16<3:50:47,  2.03s/it]
Outer training loop:  86%|########6 | 43167/50000 [28:37:18<3:47:48,  2.00s/it]
Outer training loop:  86%|########6 | 43168/50000 [28:37:20<3:42:27,  1.95s/it]
Outer training loop:  86%|########6 | 43169/50000 [28:37:21<3:36:38,  1.90s/it]
Outer training loop:  86%|########6 | 43170/50000 [28:37:23<3:40:50,  1.94s/it]
Outer training loop:  86%|########6 | 43171/50000 [28:37:25<3:41:06,  1.94s/it]
Outer training loop:  86%|########6 | 43172/50000 [28:37:27<3:36:41,  1.90s/it]
Outer training loop:  86%|########6 | 43173/50000 [28:37:29<3:40:37,  1.94s/it]
Outer training loop:  86%|########6 | 43174/50000 [28:37:31<3:37:13,  1.91s/it]
Outer training loop:  86%|########6 | 43175/50000 [28:37:33<3:35:17,  1.89s/it]
Outer training loop:  86%|########6 | 43176/50000 [28:37:35<3:41:29,  1.95s/it]
Outer training loop:  86%|########6 | 43177/50000 [28:37:37<3:39:09,  1.93s/it]
Outer training loop:  86%|########6 | 43178/50000 [28:37:39<3:34:43,  1.89s/it]
Outer training loop:  86%|########6 | 43179/50000 [28:37:41<3:39:21,  1.93s/it]
Outer training loop:  86%|########6 | 43180/50000 [28:37:43<3:51:10,  2.03s/it]
Outer training loop:  86%|########6 | 43181/50000 [28:37:45<3:47:06,  2.00s/it]
Outer training loop:  86%|########6 | 43182/50000 [28:37:47<3:40:17,  1.94s/it]
Outer training loop:  86%|########6 | 43183/50000 [28:37:49<3:41:44,  1.95s/it]
Outer training loop:  86%|########6 | 43184/50000 [28:37:51<3:39:05,  1.93s/it]
Outer training loop:  86%|########6 | 43185/50000 [28:37:52<3:35:01,  1.89s/it]
Outer training loop:  86%|########6 | 43186/50000 [28:37:55<3:44:11,  1.97s/it]
Outer training loop:  86%|########6 | 43187/50000 [28:37:56<3:39:41,  1.93s/it]
Outer training loop:  86%|########6 | 43188/50000 [28:37:58<3:42:57,  1.96s/it]
Outer training loop:  86%|########6 | 43189/50000 [28:38:00<3:37:41,  1.92s/it]
Outer training loop:  86%|########6 | 43190/50000 [28:38:02<3:39:57,  1.94s/it]
Outer training loop:  86%|########6 | 43191/50000 [28:38:04<3:35:22,  1.90s/it]
Outer training loop:  86%|########6 | 43192/50000 [28:38:06<3:31:52,  1.87s/it]
Outer training loop:  86%|########6 | 43193/50000 [28:38:08<3:37:15,  1.92s/it]
Outer training loop:  86%|########6 | 43194/50000 [28:38:10<3:37:35,  1.92s/it]
Outer training loop:  86%|########6 | 43195/50000 [28:38:12<3:39:03,  1.93s/it]
Outer training loop:  86%|########6 | 43196/50000 [28:38:14<3:46:25,  2.00s/it]
Outer training loop:  86%|########6 | 43197/50000 [28:38:16<3:41:57,  1.96s/it]
Outer training loop:  86%|########6 | 43198/50000 [28:38:18<3:38:05,  1.92s/it]
Outer training loop:  86%|########6 | 43199/50000 [28:38:20<3:42:43,  1.96s/it]
Outer training loop:  86%|########6 | 43200/50000 [28:38:22<3:40:50,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  86%|########6 | 43201/50000 [28:38:29<6:47:48,  3.60s/it]
Outer training loop:  86%|########6 | 43202/50000 [28:38:31<5:47:25,  3.07s/it]
Outer training loop:  86%|########6 | 43203/50000 [28:38:33<5:09:39,  2.73s/it]
Outer training loop:  86%|########6 | 43204/50000 [28:38:34<4:34:08,  2.42s/it]
Outer training loop:  86%|########6 | 43205/50000 [28:38:36<4:14:02,  2.24s/it]
Outer training loop:  86%|########6 | 43206/50000 [28:38:38<4:02:40,  2.14s/it]
Outer training loop:  86%|########6 | 43207/50000 [28:38:40<3:58:52,  2.11s/it]
Outer training loop:  86%|########6 | 43208/50000 [28:38:42<3:48:36,  2.02s/it]
Outer training loop:  86%|########6 | 43209/50000 [28:38:44<3:44:56,  1.99s/it]
Outer training loop:  86%|########6 | 43210/50000 [28:38:46<3:42:29,  1.97s/it]
Outer training loop:  86%|########6 | 43211/50000 [28:38:48<3:36:08,  1.91s/it]
Outer training loop:  86%|########6 | 43212/50000 [28:38:50<3:35:05,  1.90s/it]
Outer training loop:  86%|########6 | 43213/50000 [28:38:51<3:30:55,  1.86s/it]
Outer training loop:  86%|########6 | 43214/50000 [28:38:53<3:35:38,  1.91s/it]
Outer training loop:  86%|########6 | 43215/50000 [28:38:55<3:34:02,  1.89s/it]
Outer training loop:  86%|########6 | 43216/50000 [28:38:57<3:31:32,  1.87s/it]
Outer training loop:  86%|########6 | 43217/50000 [28:38:59<3:33:39,  1.89s/it]
Outer training loop:  86%|########6 | 43218/50000 [28:39:01<3:28:20,  1.84s/it]
Outer training loop:  86%|########6 | 43219/50000 [28:39:03<3:28:13,  1.84s/it]
Outer training loop:  86%|########6 | 43220/50000 [28:39:04<3:27:36,  1.84s/it]
Outer training loop:  86%|########6 | 43221/50000 [28:39:06<3:34:12,  1.90s/it]
Outer training loop:  86%|########6 | 43222/50000 [28:39:08<3:33:20,  1.89s/it]
Outer training loop:  86%|########6 | 43223/50000 [28:39:10<3:33:05,  1.89s/it]
Outer training loop:  86%|########6 | 43224/50000 [28:39:12<3:37:43,  1.93s/it]
Outer training loop:  86%|########6 | 43225/50000 [28:39:14<3:39:13,  1.94s/it]
Outer training loop:  86%|########6 | 43226/50000 [28:39:16<3:41:11,  1.96s/it]
Outer training loop:  86%|########6 | 43227/50000 [28:39:18<3:41:14,  1.96s/it]
Outer training loop:  86%|########6 | 43228/50000 [28:39:20<3:40:25,  1.95s/it]
Outer training loop:  86%|########6 | 43229/50000 [28:39:22<3:36:38,  1.92s/it]
Outer training loop:  86%|########6 | 43230/50000 [28:39:24<3:33:31,  1.89s/it]
Outer training loop:  86%|########6 | 43231/50000 [28:39:26<3:42:43,  1.97s/it]
Outer training loop:  86%|########6 | 43232/50000 [28:39:28<3:43:43,  1.98s/it]
Outer training loop:  86%|########6 | 43233/50000 [28:39:30<3:35:43,  1.91s/it]
Outer training loop:  86%|########6 | 43234/50000 [28:39:32<3:36:23,  1.92s/it]
Outer training loop:  86%|########6 | 43235/50000 [28:39:33<3:35:20,  1.91s/it]
Outer training loop:  86%|########6 | 43236/50000 [28:39:35<3:37:38,  1.93s/it]
Outer training loop:  86%|########6 | 43237/50000 [28:39:37<3:36:25,  1.92s/it]
Outer training loop:  86%|########6 | 43238/50000 [28:39:39<3:39:54,  1.95s/it]
Outer training loop:  86%|########6 | 43239/50000 [28:39:41<3:41:04,  1.96s/it]
Outer training loop:  86%|########6 | 43240/50000 [28:39:43<3:40:05,  1.95s/it]
Outer training loop:  86%|########6 | 43241/50000 [28:39:45<3:39:21,  1.95s/it]
Outer training loop:  86%|########6 | 43242/50000 [28:39:47<3:36:12,  1.92s/it]
Outer training loop:  86%|########6 | 43243/50000 [28:39:49<3:33:30,  1.90s/it]
Outer training loop:  86%|########6 | 43244/50000 [28:39:51<3:36:15,  1.92s/it]
Outer training loop:  86%|########6 | 43245/50000 [28:39:53<3:30:54,  1.87s/it]
Outer training loop:  86%|########6 | 43246/50000 [28:39:55<3:30:24,  1.87s/it]
Outer training loop:  86%|########6 | 43247/50000 [28:39:56<3:27:49,  1.85s/it]
Outer training loop:  86%|########6 | 43248/50000 [28:39:58<3:38:53,  1.95s/it]
Outer training loop:  86%|########6 | 43249/50000 [28:40:00<3:34:42,  1.91s/it]
Outer training loop:  86%|########6 | 43250/50000 [28:40:02<3:28:22,  1.85s/it]
Outer training loop:  87%|########6 | 43251/50000 [28:40:04<3:34:09,  1.90s/it]
Outer training loop:  87%|########6 | 43252/50000 [28:40:06<3:30:56,  1.88s/it]
Outer training loop:  87%|########6 | 43253/50000 [28:40:08<3:28:33,  1.85s/it]
Outer training loop:  87%|########6 | 43254/50000 [28:40:09<3:26:47,  1.84s/it]
Outer training loop:  87%|########6 | 43255/50000 [28:40:11<3:32:14,  1.89s/it]
Outer training loop:  87%|########6 | 43256/50000 [28:40:13<3:32:28,  1.89s/it]
Outer training loop:  87%|########6 | 43257/50000 [28:40:15<3:27:22,  1.85s/it]
Outer training loop:  87%|########6 | 43258/50000 [28:40:17<3:27:37,  1.85s/it]
Outer training loop:  87%|########6 | 43259/50000 [28:40:19<3:27:22,  1.85s/it]
Outer training loop:  87%|########6 | 43260/50000 [28:40:21<3:28:20,  1.85s/it]
Outer training loop:  87%|########6 | 43261/50000 [28:40:23<3:30:15,  1.87s/it]
Outer training loop:  87%|########6 | 43262/50000 [28:40:24<3:30:04,  1.87s/it]
Outer training loop:  87%|########6 | 43263/50000 [28:40:26<3:26:59,  1.84s/it]
Outer training loop:  87%|########6 | 43264/50000 [28:40:28<3:27:05,  1.84s/it]
Outer training loop:  87%|########6 | 43265/50000 [28:40:30<3:30:50,  1.88s/it]
Outer training loop:  87%|########6 | 43266/50000 [28:40:32<3:28:07,  1.85s/it]
Outer training loop:  87%|########6 | 43267/50000 [28:40:34<3:25:43,  1.83s/it]
Outer training loop:  87%|########6 | 43268/50000 [28:40:36<3:31:05,  1.88s/it]
Outer training loop:  87%|########6 | 43269/50000 [28:40:37<3:25:37,  1.83s/it]
Outer training loop:  87%|########6 | 43270/50000 [28:40:39<3:23:37,  1.82s/it]
Outer training loop:  87%|########6 | 43271/50000 [28:40:41<3:25:00,  1.83s/it]
Outer training loop:  87%|########6 | 43272/50000 [28:40:43<3:32:00,  1.89s/it]
Outer training loop:  87%|########6 | 43273/50000 [28:40:45<3:30:12,  1.87s/it]
Outer training loop:  87%|########6 | 43274/50000 [28:40:47<3:26:07,  1.84s/it]
Outer training loop:  87%|########6 | 43275/50000 [28:40:49<3:29:00,  1.86s/it]
Outer training loop:  87%|########6 | 43276/50000 [28:40:51<3:38:54,  1.95s/it]
Outer training loop:  87%|########6 | 43277/50000 [28:40:52<3:32:06,  1.89s/it]
Outer training loop:  87%|########6 | 43278/50000 [28:40:54<3:35:28,  1.92s/it]
Outer training loop:  87%|########6 | 43279/50000 [28:40:56<3:29:03,  1.87s/it]
Outer training loop:  87%|########6 | 43280/50000 [28:40:58<3:28:06,  1.86s/it]
Outer training loop:  87%|########6 | 43281/50000 [28:41:00<3:22:40,  1.81s/it]
Outer training loop:  87%|########6 | 43282/50000 [28:41:02<3:27:58,  1.86s/it]
Outer training loop:  87%|########6 | 43283/50000 [28:41:03<3:26:53,  1.85s/it]
Outer training loop:  87%|########6 | 43284/50000 [28:41:05<3:32:10,  1.90s/it]
Outer training loop:  87%|########6 | 43285/50000 [28:41:07<3:34:07,  1.91s/it]
Outer training loop:  87%|########6 | 43286/50000 [28:41:09<3:28:13,  1.86s/it]
Outer training loop:  87%|########6 | 43287/50000 [28:41:11<3:30:09,  1.88s/it]
Outer training loop:  87%|########6 | 43288/50000 [28:41:13<3:28:12,  1.86s/it]
Outer training loop:  87%|########6 | 43289/50000 [28:41:15<3:33:27,  1.91s/it]
Outer training loop:  87%|########6 | 43290/50000 [28:41:17<3:31:45,  1.89s/it]
Outer training loop:  87%|########6 | 43291/50000 [28:41:19<3:30:29,  1.88s/it]
Outer training loop:  87%|########6 | 43292/50000 [28:41:21<3:36:49,  1.94s/it]
Outer training loop:  87%|########6 | 43293/50000 [28:41:23<3:32:22,  1.90s/it]
Outer training loop:  87%|########6 | 43294/50000 [28:41:24<3:30:28,  1.88s/it]
Outer training loop:  87%|########6 | 43295/50000 [28:41:26<3:36:06,  1.93s/it]
Outer training loop:  87%|########6 | 43296/50000 [28:41:28<3:33:35,  1.91s/it]
Outer training loop:  87%|########6 | 43297/50000 [28:41:30<3:30:26,  1.88s/it]
Outer training loop:  87%|########6 | 43298/50000 [28:41:32<3:26:38,  1.85s/it]
Outer training loop:  87%|########6 | 43299/50000 [28:41:34<3:34:29,  1.92s/it]
Outer training loop:  87%|########6 | 43300/50000 [28:41:36<3:33:38,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.00s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]

Outer training loop:  87%|########6 | 43301/50000 [28:41:43<6:38:17,  3.57s/it]
Outer training loop:  87%|########6 | 43302/50000 [28:41:45<5:39:56,  3.05s/it]
Outer training loop:  87%|########6 | 43303/50000 [28:41:47<5:04:19,  2.73s/it]
Outer training loop:  87%|########6 | 43304/50000 [28:41:49<4:40:56,  2.52s/it]
Outer training loop:  87%|########6 | 43305/50000 [28:41:51<4:21:30,  2.34s/it]
Outer training loop:  87%|########6 | 43306/50000 [28:41:53<4:06:14,  2.21s/it]
Outer training loop:  87%|########6 | 43307/50000 [28:41:55<3:52:52,  2.09s/it]
Outer training loop:  87%|########6 | 43308/50000 [28:41:57<3:41:25,  1.99s/it]
Outer training loop:  87%|########6 | 43309/50000 [28:41:59<3:46:03,  2.03s/it]
Outer training loop:  87%|########6 | 43310/50000 [28:42:01<3:42:36,  2.00s/it]
Outer training loop:  87%|########6 | 43311/50000 [28:42:02<3:39:30,  1.97s/it]
Outer training loop:  87%|########6 | 43312/50000 [28:42:04<3:33:57,  1.92s/it]
Outer training loop:  87%|########6 | 43313/50000 [28:42:06<3:37:51,  1.95s/it]
Outer training loop:  87%|########6 | 43314/50000 [28:42:08<3:28:50,  1.87s/it]
Outer training loop:  87%|########6 | 43315/50000 [28:42:10<3:29:01,  1.88s/it]
Outer training loop:  87%|########6 | 43316/50000 [28:42:12<3:33:45,  1.92s/it]
Outer training loop:  87%|########6 | 43317/50000 [28:42:14<3:40:02,  1.98s/it]
Outer training loop:  87%|########6 | 43318/50000 [28:42:16<3:36:01,  1.94s/it]
Outer training loop:  87%|########6 | 43319/50000 [28:42:18<3:39:48,  1.97s/it]
Outer training loop:  87%|########6 | 43320/50000 [28:42:20<3:36:20,  1.94s/it]
Outer training loop:  87%|########6 | 43321/50000 [28:42:22<3:39:49,  1.97s/it]
Outer training loop:  87%|########6 | 43322/50000 [28:42:24<3:35:52,  1.94s/it]
Outer training loop:  87%|########6 | 43323/50000 [28:42:26<3:44:33,  2.02s/it]
Outer training loop:  87%|########6 | 43324/50000 [28:42:28<3:39:41,  1.97s/it]
Outer training loop:  87%|########6 | 43325/50000 [28:42:30<3:35:07,  1.93s/it]
Outer training loop:  87%|########6 | 43326/50000 [28:42:32<3:37:02,  1.95s/it]
Outer training loop:  87%|########6 | 43327/50000 [28:42:33<3:35:17,  1.94s/it]
Outer training loop:  87%|########6 | 43328/50000 [28:42:36<3:37:31,  1.96s/it]
Outer training loop:  87%|########6 | 43329/50000 [28:42:37<3:33:09,  1.92s/it]
Outer training loop:  87%|########6 | 43330/50000 [28:42:39<3:41:30,  1.99s/it]
Outer training loop:  87%|########6 | 43331/50000 [28:42:41<3:37:47,  1.96s/it]
Outer training loop:  87%|########6 | 43332/50000 [28:42:43<3:39:25,  1.97s/it]
Outer training loop:  87%|########6 | 43333/50000 [28:42:46<3:46:24,  2.04s/it]
Outer training loop:  87%|########6 | 43334/50000 [28:42:47<3:40:40,  1.99s/it]
Outer training loop:  87%|########6 | 43335/50000 [28:42:49<3:35:11,  1.94s/it]
Outer training loop:  87%|########6 | 43336/50000 [28:42:51<3:34:53,  1.93s/it]
Outer training loop:  87%|########6 | 43337/50000 [28:42:53<3:40:06,  1.98s/it]
Outer training loop:  87%|########6 | 43338/50000 [28:42:55<3:35:48,  1.94s/it]
Outer training loop:  87%|########6 | 43339/50000 [28:42:57<3:34:36,  1.93s/it]
Outer training loop:  87%|########6 | 43340/50000 [28:42:59<3:38:14,  1.97s/it]
Outer training loop:  87%|########6 | 43341/50000 [28:43:01<3:36:52,  1.95s/it]
Outer training loop:  87%|########6 | 43342/50000 [28:43:03<3:35:36,  1.94s/it]
Outer training loop:  87%|########6 | 43343/50000 [28:43:05<3:37:37,  1.96s/it]
Outer training loop:  87%|########6 | 43344/50000 [28:43:07<3:39:45,  1.98s/it]
Outer training loop:  87%|########6 | 43345/50000 [28:43:09<3:35:01,  1.94s/it]
Outer training loop:  87%|########6 | 43346/50000 [28:43:11<3:34:34,  1.93s/it]
Outer training loop:  87%|########6 | 43347/50000 [28:43:13<3:43:43,  2.02s/it]
Outer training loop:  87%|########6 | 43348/50000 [28:43:15<3:35:51,  1.95s/it]
Outer training loop:  87%|########6 | 43349/50000 [28:43:17<3:34:50,  1.94s/it]
Outer training loop:  87%|########6 | 43350/50000 [28:43:19<3:40:43,  1.99s/it]
Outer training loop:  87%|########6 | 43351/50000 [28:43:21<3:37:39,  1.96s/it]
Outer training loop:  87%|########6 | 43352/50000 [28:43:23<3:34:27,  1.94s/it]
Outer training loop:  87%|########6 | 43353/50000 [28:43:25<3:41:10,  2.00s/it]
Outer training loop:  87%|########6 | 43354/50000 [28:43:27<3:36:18,  1.95s/it]
Outer training loop:  87%|########6 | 43355/50000 [28:43:29<3:37:24,  1.96s/it]
Outer training loop:  87%|########6 | 43356/50000 [28:43:31<3:39:10,  1.98s/it]
Outer training loop:  87%|########6 | 43357/50000 [28:43:32<3:33:49,  1.93s/it]
Outer training loop:  87%|########6 | 43358/50000 [28:43:34<3:31:41,  1.91s/it]
Outer training loop:  87%|########6 | 43359/50000 [28:43:36<3:31:29,  1.91s/it]
Outer training loop:  87%|########6 | 43360/50000 [28:43:38<3:34:04,  1.93s/it]
Outer training loop:  87%|########6 | 43361/50000 [28:43:40<3:30:41,  1.90s/it]
Outer training loop:  87%|########6 | 43362/50000 [28:43:42<3:27:59,  1.88s/it]
Outer training loop:  87%|########6 | 43363/50000 [28:43:44<3:31:03,  1.91s/it]
Outer training loop:  87%|########6 | 43364/50000 [28:43:46<3:30:24,  1.90s/it]
Outer training loop:  87%|########6 | 43365/50000 [28:43:47<3:28:21,  1.88s/it]
Outer training loop:  87%|########6 | 43366/50000 [28:43:49<3:32:31,  1.92s/it]
Outer training loop:  87%|########6 | 43367/50000 [28:43:51<3:28:32,  1.89s/it]
Outer training loop:  87%|########6 | 43368/50000 [28:43:53<3:27:42,  1.88s/it]
Outer training loop:  87%|########6 | 43369/50000 [28:43:55<3:27:53,  1.88s/it]
Outer training loop:  87%|########6 | 43370/50000 [28:43:57<3:30:50,  1.91s/it]
Outer training loop:  87%|########6 | 43371/50000 [28:43:59<3:32:35,  1.92s/it]
Outer training loop:  87%|########6 | 43372/50000 [28:44:01<3:27:16,  1.88s/it]
Outer training loop:  87%|########6 | 43373/50000 [28:44:03<3:35:21,  1.95s/it]
Outer training loop:  87%|########6 | 43374/50000 [28:44:05<3:34:19,  1.94s/it]
Outer training loop:  87%|########6 | 43375/50000 [28:44:07<3:34:45,  1.94s/it]
Outer training loop:  87%|########6 | 43376/50000 [28:44:09<3:35:17,  1.95s/it]
Outer training loop:  87%|########6 | 43377/50000 [28:44:11<3:35:55,  1.96s/it]
Outer training loop:  87%|########6 | 43378/50000 [28:44:13<3:33:57,  1.94s/it]
Outer training loop:  87%|########6 | 43379/50000 [28:44:15<3:40:33,  2.00s/it]
Outer training loop:  87%|########6 | 43380/50000 [28:44:16<3:33:23,  1.93s/it]
Outer training loop:  87%|########6 | 43381/50000 [28:44:18<3:31:12,  1.91s/it]
Outer training loop:  87%|########6 | 43382/50000 [28:44:20<3:35:21,  1.95s/it]
Outer training loop:  87%|########6 | 43383/50000 [28:44:22<3:30:43,  1.91s/it]
Outer training loop:  87%|########6 | 43384/50000 [28:44:24<3:28:32,  1.89s/it]
Outer training loop:  87%|########6 | 43385/50000 [28:44:26<3:27:13,  1.88s/it]
Outer training loop:  87%|########6 | 43386/50000 [28:44:28<3:35:56,  1.96s/it]
Outer training loop:  87%|########6 | 43387/50000 [28:44:30<3:30:31,  1.91s/it]
Outer training loop:  87%|########6 | 43388/50000 [28:44:32<3:30:04,  1.91s/it]
Outer training loop:  87%|########6 | 43389/50000 [28:44:33<3:25:18,  1.86s/it]
Outer training loop:  87%|########6 | 43390/50000 [28:44:35<3:29:51,  1.90s/it]
Outer training loop:  87%|########6 | 43391/50000 [28:44:37<3:31:12,  1.92s/it]
Outer training loop:  87%|########6 | 43392/50000 [28:44:39<3:31:47,  1.92s/it]
Outer training loop:  87%|########6 | 43393/50000 [28:44:41<3:35:29,  1.96s/it]
Outer training loop:  87%|########6 | 43394/50000 [28:44:43<3:34:01,  1.94s/it]
Outer training loop:  87%|########6 | 43395/50000 [28:44:45<3:29:28,  1.90s/it]
Outer training loop:  87%|########6 | 43396/50000 [28:44:47<3:35:50,  1.96s/it]
Outer training loop:  87%|########6 | 43397/50000 [28:44:49<3:33:15,  1.94s/it]
Outer training loop:  87%|########6 | 43398/50000 [28:44:51<3:30:16,  1.91s/it]
Outer training loop:  87%|########6 | 43399/50000 [28:44:53<3:26:58,  1.88s/it]
Outer training loop:  87%|########6 | 43400/50000 [28:44:55<3:30:03,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  87%|########6 | 43401/50000 [28:45:02<6:34:47,  3.59s/it]
Outer training loop:  87%|########6 | 43402/50000 [28:45:04<5:35:17,  3.05s/it]
Outer training loop:  87%|########6 | 43403/50000 [28:45:06<4:55:49,  2.69s/it]
Outer training loop:  87%|########6 | 43404/50000 [28:45:08<4:35:10,  2.50s/it]
Outer training loop:  87%|########6 | 43405/50000 [28:45:10<4:16:40,  2.34s/it]
Outer training loop:  87%|########6 | 43406/50000 [28:45:12<4:01:12,  2.19s/it]
Outer training loop:  87%|########6 | 43407/50000 [28:45:14<3:58:20,  2.17s/it]
Outer training loop:  87%|########6 | 43408/50000 [28:45:16<3:50:15,  2.10s/it]
Outer training loop:  87%|########6 | 43409/50000 [28:45:18<3:44:23,  2.04s/it]
Outer training loop:  87%|########6 | 43410/50000 [28:45:20<3:39:22,  2.00s/it]
Outer training loop:  87%|########6 | 43411/50000 [28:45:22<3:41:05,  2.01s/it]
Outer training loop:  87%|########6 | 43412/50000 [28:45:24<3:35:27,  1.96s/it]
Outer training loop:  87%|########6 | 43413/50000 [28:45:25<3:33:59,  1.95s/it]
Outer training loop:  87%|########6 | 43414/50000 [28:45:27<3:35:17,  1.96s/it]
Outer training loop:  87%|########6 | 43415/50000 [28:45:29<3:27:37,  1.89s/it]
Outer training loop:  87%|########6 | 43416/50000 [28:45:31<3:26:07,  1.88s/it]
Outer training loop:  87%|########6 | 43417/50000 [28:45:33<3:24:01,  1.86s/it]
Outer training loop:  87%|########6 | 43418/50000 [28:45:35<3:32:49,  1.94s/it]
Outer training loop:  87%|########6 | 43419/50000 [28:45:37<3:31:14,  1.93s/it]
Outer training loop:  87%|########6 | 43420/50000 [28:45:39<3:33:20,  1.95s/it]
Outer training loop:  87%|########6 | 43421/50000 [28:45:41<3:41:06,  2.02s/it]
Outer training loop:  87%|########6 | 43422/50000 [28:45:43<3:35:41,  1.97s/it]
Outer training loop:  87%|########6 | 43423/50000 [28:45:45<3:30:36,  1.92s/it]
Outer training loop:  87%|########6 | 43424/50000 [28:45:47<3:29:04,  1.91s/it]
Outer training loop:  87%|########6 | 43425/50000 [28:45:49<3:30:41,  1.92s/it]
Outer training loop:  87%|########6 | 43426/50000 [28:45:51<3:37:14,  1.98s/it]
Outer training loop:  87%|########6 | 43427/50000 [28:45:53<3:34:34,  1.96s/it]
Outer training loop:  87%|########6 | 43428/50000 [28:45:55<3:38:40,  2.00s/it]
Outer training loop:  87%|########6 | 43429/50000 [28:45:57<3:37:22,  1.98s/it]
Outer training loop:  87%|########6 | 43430/50000 [28:45:59<3:36:14,  1.97s/it]
Outer training loop:  87%|########6 | 43431/50000 [28:46:01<3:37:26,  1.99s/it]
Outer training loop:  87%|########6 | 43432/50000 [28:46:02<3:30:30,  1.92s/it]
Outer training loop:  87%|########6 | 43433/50000 [28:46:04<3:32:31,  1.94s/it]
Outer training loop:  87%|########6 | 43434/50000 [28:46:06<3:34:39,  1.96s/it]
Outer training loop:  87%|########6 | 43435/50000 [28:46:08<3:28:14,  1.90s/it]
Outer training loop:  87%|########6 | 43436/50000 [28:46:10<3:24:24,  1.87s/it]
Outer training loop:  87%|########6 | 43437/50000 [28:46:12<3:28:15,  1.90s/it]
Outer training loop:  87%|########6 | 43438/50000 [28:46:14<3:26:36,  1.89s/it]
Outer training loop:  87%|########6 | 43439/50000 [28:46:15<3:20:17,  1.83s/it]
Outer training loop:  87%|########6 | 43440/50000 [28:46:17<3:21:11,  1.84s/it]
Outer training loop:  87%|########6 | 43441/50000 [28:46:19<3:30:22,  1.92s/it]
Outer training loop:  87%|########6 | 43442/50000 [28:46:21<3:29:06,  1.91s/it]
Outer training loop:  87%|########6 | 43443/50000 [28:46:23<3:24:58,  1.88s/it]
Outer training loop:  87%|########6 | 43444/50000 [28:46:25<3:28:08,  1.90s/it]
Outer training loop:  87%|########6 | 43445/50000 [28:46:27<3:25:27,  1.88s/it]
Outer training loop:  87%|########6 | 43446/50000 [28:46:29<3:22:55,  1.86s/it]
Outer training loop:  87%|########6 | 43447/50000 [28:46:31<3:27:17,  1.90s/it]
Outer training loop:  87%|########6 | 43448/50000 [28:46:32<3:22:42,  1.86s/it]
Outer training loop:  87%|########6 | 43449/50000 [28:46:34<3:21:27,  1.85s/it]
Outer training loop:  87%|########6 | 43450/50000 [28:46:36<3:22:28,  1.85s/it]
Outer training loop:  87%|########6 | 43451/50000 [28:46:38<3:27:51,  1.90s/it]
Outer training loop:  87%|########6 | 43452/50000 [28:46:40<3:23:58,  1.87s/it]
Outer training loop:  87%|########6 | 43453/50000 [28:46:42<3:20:16,  1.84s/it]
Outer training loop:  87%|########6 | 43454/50000 [28:46:44<3:25:04,  1.88s/it]
Outer training loop:  87%|########6 | 43455/50000 [28:46:45<3:21:03,  1.84s/it]
Outer training loop:  87%|########6 | 43456/50000 [28:46:47<3:15:06,  1.79s/it]
Outer training loop:  87%|########6 | 43457/50000 [28:46:49<3:20:09,  1.84s/it]
Outer training loop:  87%|########6 | 43458/50000 [28:46:51<3:19:24,  1.83s/it]
Outer training loop:  87%|########6 | 43459/50000 [28:46:53<3:20:28,  1.84s/it]
Outer training loop:  87%|########6 | 43460/50000 [28:46:55<3:19:17,  1.83s/it]
Outer training loop:  87%|########6 | 43461/50000 [28:46:57<3:29:21,  1.92s/it]
Outer training loop:  87%|########6 | 43462/50000 [28:46:59<3:29:27,  1.92s/it]
Outer training loop:  87%|########6 | 43463/50000 [28:47:00<3:24:47,  1.88s/it]
Outer training loop:  87%|########6 | 43464/50000 [28:47:02<3:27:51,  1.91s/it]
Outer training loop:  87%|########6 | 43465/50000 [28:47:04<3:24:20,  1.88s/it]
Outer training loop:  87%|########6 | 43466/50000 [28:47:06<3:21:59,  1.85s/it]
Outer training loop:  87%|########6 | 43467/50000 [28:47:08<3:24:09,  1.88s/it]
Outer training loop:  87%|########6 | 43468/50000 [28:47:10<3:20:57,  1.85s/it]
Outer training loop:  87%|########6 | 43469/50000 [28:47:11<3:20:58,  1.85s/it]
Outer training loop:  87%|########6 | 43470/50000 [28:47:13<3:24:09,  1.88s/it]
Outer training loop:  87%|########6 | 43471/50000 [28:47:15<3:27:49,  1.91s/it]
Outer training loop:  87%|########6 | 43472/50000 [28:47:17<3:27:52,  1.91s/it]
Outer training loop:  87%|########6 | 43473/50000 [28:47:19<3:34:24,  1.97s/it]
Outer training loop:  87%|########6 | 43474/50000 [28:47:21<3:30:42,  1.94s/it]
Outer training loop:  87%|########6 | 43475/50000 [28:47:23<3:26:30,  1.90s/it]
Outer training loop:  87%|########6 | 43476/50000 [28:47:25<3:26:50,  1.90s/it]
Outer training loop:  87%|########6 | 43477/50000 [28:47:27<3:28:47,  1.92s/it]
Outer training loop:  87%|########6 | 43478/50000 [28:47:29<3:23:47,  1.87s/it]
Outer training loop:  87%|########6 | 43479/50000 [28:47:31<3:20:16,  1.84s/it]
Outer training loop:  87%|########6 | 43480/50000 [28:47:32<3:15:25,  1.80s/it]
Outer training loop:  87%|########6 | 43481/50000 [28:47:34<3:22:48,  1.87s/it]
Outer training loop:  87%|########6 | 43482/50000 [28:47:36<3:22:25,  1.86s/it]
Outer training loop:  87%|########6 | 43483/50000 [28:47:38<3:21:04,  1.85s/it]
Outer training loop:  87%|########6 | 43484/50000 [28:47:40<3:30:04,  1.93s/it]
Outer training loop:  87%|########6 | 43485/50000 [28:47:42<3:23:55,  1.88s/it]
Outer training loop:  87%|########6 | 43486/50000 [28:47:44<3:22:52,  1.87s/it]
Outer training loop:  87%|########6 | 43487/50000 [28:47:46<3:24:33,  1.88s/it]
Outer training loop:  87%|########6 | 43488/50000 [28:47:47<3:20:36,  1.85s/it]
Outer training loop:  87%|########6 | 43489/50000 [28:47:49<3:21:35,  1.86s/it]
Outer training loop:  87%|########6 | 43490/50000 [28:47:51<3:22:59,  1.87s/it]
Outer training loop:  87%|########6 | 43491/50000 [28:47:53<3:29:16,  1.93s/it]
Outer training loop:  87%|########6 | 43492/50000 [28:47:55<3:24:34,  1.89s/it]
Outer training loop:  87%|########6 | 43493/50000 [28:47:57<3:24:16,  1.88s/it]
Outer training loop:  87%|########6 | 43494/50000 [28:47:59<3:20:57,  1.85s/it]
Outer training loop:  87%|########6 | 43495/50000 [28:48:00<3:21:37,  1.86s/it]
Outer training loop:  87%|########6 | 43496/50000 [28:48:02<3:19:34,  1.84s/it]
Outer training loop:  87%|########6 | 43497/50000 [28:48:04<3:21:15,  1.86s/it]
Outer training loop:  87%|########6 | 43498/50000 [28:48:06<3:24:09,  1.88s/it]
Outer training loop:  87%|########6 | 43499/50000 [28:48:08<3:20:25,  1.85s/it]
Outer training loop:  87%|########7 | 43500/50000 [28:48:10<3:21:12,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.98s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:40,  1.95s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:34,  1.91s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:32,  1.91s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.90s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.95s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.93s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:25,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:20,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.95s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:15,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.90s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.92s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.90s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:02,  1.94s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:59,  1.97s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:56,  1.95s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:51,  1.92s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:51,  1.99s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:47,  1.96s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:44,  1.93s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:38,  1.89s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:39,  1.96s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:36,  1.93s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:32,  1.88s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:27,  1.83s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.88s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:29,  1.95s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:25,  1.89s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:22,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:17,  1.81s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:15,  1.80s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:14,  1.82s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:12,  1.80s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.91s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:07,  1.83s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:05,  1.82s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:02,  1.78s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:00,  1.78s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.91s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:52,  1.82s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:50,  1.79s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:48,  1.79s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:47,  1.82s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:46,  1.95s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:44,  1.93s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.81s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:31,  1.97s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:29,  1.95s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.86s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.86s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:16,  2.01s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.96s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.94s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.92s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.91s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.80s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:24,  1.76s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:21,  1.75s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:33,  1.92s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.87s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.86s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.90s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:17,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:20,  1.95s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.93s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:11,  1.88s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:12,  1.95s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.94s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.94s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:05,  1.96s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:50,  1.84s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.88s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:45,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:41,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:39,  1.81s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:42,  1.90s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:35,  1.84s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:34,  1.97s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:32,  1.97s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:28,  1.92s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:25,  1.90s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.86s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:22,  1.95s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:20,  1.96s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:16,  1.91s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:13,  1.93s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.90s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:12,  2.02s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:07,  1.94s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:05,  1.91s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:01,  1.87s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.85s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.84s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:55,  1.93s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.89s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.90s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.83s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:41,  1.99s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:38,  1.93s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.91s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.85s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.84s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.97s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.89s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.86s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.84s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.83s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.80s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.94s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.90s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.93s/it]

Outer training loop:  87%|########7 | 43501/50000 [28:53:31<176:12:25, 97.61s/it]
Outer training loop:  87%|########7 | 43502/50000 [28:53:33<124:22:33, 68.91s/it]
Outer training loop:  87%|########7 | 43503/50000 [28:53:35<88:03:13, 48.79s/it] 
Outer training loop:  87%|########7 | 43504/50000 [28:53:36<62:36:29, 34.70s/it]
Outer training loop:  87%|########7 | 43505/50000 [28:53:38<44:49:38, 24.85s/it]
Outer training loop:  87%|########7 | 43506/50000 [28:53:40<32:21:43, 17.94s/it]
Outer training loop:  87%|########7 | 43507/50000 [28:53:42<23:40:13, 13.12s/it]
Outer training loop:  87%|########7 | 43508/50000 [28:53:45<17:57:32,  9.96s/it]
Outer training loop:  87%|########7 | 43509/50000 [28:53:46<13:29:35,  7.48s/it]
Outer training loop:  87%|########7 | 43510/50000 [28:53:48<10:24:21,  5.77s/it]
Outer training loop:  87%|########7 | 43511/50000 [28:53:50<8:24:47,  4.67s/it] 
Outer training loop:  87%|########7 | 43512/50000 [28:53:52<6:50:20,  3.79s/it]
Outer training loop:  87%|########7 | 43513/50000 [28:53:54<5:47:56,  3.22s/it]
Outer training loop:  87%|########7 | 43514/50000 [28:53:56<5:05:22,  2.82s/it]
Outer training loop:  87%|########7 | 43515/50000 [28:53:58<4:37:56,  2.57s/it]
Outer training loop:  87%|########7 | 43516/50000 [28:53:59<4:12:29,  2.34s/it]
Outer training loop:  87%|########7 | 43517/50000 [28:54:01<3:51:39,  2.14s/it]
Outer training loop:  87%|########7 | 43518/50000 [28:54:03<3:39:49,  2.03s/it]
Outer training loop:  87%|########7 | 43519/50000 [28:54:05<3:39:47,  2.03s/it]
Outer training loop:  87%|########7 | 43520/50000 [28:54:07<3:34:05,  1.98s/it]
Outer training loop:  87%|########7 | 43521/50000 [28:54:09<3:28:49,  1.93s/it]
Outer training loop:  87%|########7 | 43522/50000 [28:54:11<3:31:09,  1.96s/it]
Outer training loop:  87%|########7 | 43523/50000 [28:54:12<3:27:12,  1.92s/it]
Outer training loop:  87%|########7 | 43524/50000 [28:54:14<3:23:10,  1.88s/it]
Outer training loop:  87%|########7 | 43525/50000 [28:54:16<3:15:45,  1.81s/it]
Outer training loop:  87%|########7 | 43526/50000 [28:54:18<3:19:18,  1.85s/it]
Outer training loop:  87%|########7 | 43527/50000 [28:54:20<3:17:58,  1.84s/it]
Outer training loop:  87%|########7 | 43528/50000 [28:54:22<3:19:02,  1.85s/it]
Outer training loop:  87%|########7 | 43529/50000 [28:54:23<3:22:22,  1.88s/it]
Outer training loop:  87%|########7 | 43530/50000 [28:54:25<3:21:42,  1.87s/it]
Outer training loop:  87%|########7 | 43531/50000 [28:54:27<3:17:55,  1.84s/it]
Outer training loop:  87%|########7 | 43532/50000 [28:54:29<3:14:07,  1.80s/it]
Outer training loop:  87%|########7 | 43533/50000 [28:54:31<3:24:45,  1.90s/it]
Outer training loop:  87%|########7 | 43534/50000 [28:54:33<3:19:47,  1.85s/it]
Outer training loop:  87%|########7 | 43535/50000 [28:54:35<3:24:57,  1.90s/it]
Outer training loop:  87%|########7 | 43536/50000 [28:54:36<3:20:20,  1.86s/it]
Outer training loop:  87%|########7 | 43537/50000 [28:54:38<3:15:26,  1.81s/it]
Outer training loop:  87%|########7 | 43538/50000 [28:54:40<3:13:42,  1.80s/it]
Outer training loop:  87%|########7 | 43539/50000 [28:54:42<3:22:12,  1.88s/it]
Outer training loop:  87%|########7 | 43540/50000 [28:54:44<3:24:37,  1.90s/it]
Outer training loop:  87%|########7 | 43541/50000 [28:54:46<3:20:52,  1.87s/it]
Outer training loop:  87%|########7 | 43542/50000 [28:54:48<3:26:38,  1.92s/it]
Outer training loop:  87%|########7 | 43543/50000 [28:54:50<3:22:07,  1.88s/it]
Outer training loop:  87%|########7 | 43544/50000 [28:54:51<3:21:58,  1.88s/it]
Outer training loop:  87%|########7 | 43545/50000 [28:54:53<3:20:00,  1.86s/it]
Outer training loop:  87%|########7 | 43546/50000 [28:54:55<3:23:45,  1.89s/it]
Outer training loop:  87%|########7 | 43547/50000 [28:54:57<3:21:04,  1.87s/it]
Outer training loop:  87%|########7 | 43548/50000 [28:54:59<3:20:46,  1.87s/it]
Outer training loop:  87%|########7 | 43549/50000 [28:55:01<3:24:58,  1.91s/it]
Outer training loop:  87%|########7 | 43550/50000 [28:55:03<3:21:49,  1.88s/it]
Outer training loop:  87%|########7 | 43551/50000 [28:55:05<3:28:25,  1.94s/it]
Outer training loop:  87%|########7 | 43552/50000 [28:55:07<3:29:12,  1.95s/it]
Outer training loop:  87%|########7 | 43553/50000 [28:55:09<3:26:16,  1.92s/it]
Outer training loop:  87%|########7 | 43554/50000 [28:55:10<3:23:06,  1.89s/it]
Outer training loop:  87%|########7 | 43555/50000 [28:55:12<3:21:27,  1.88s/it]
Outer training loop:  87%|########7 | 43556/50000 [28:55:14<3:26:16,  1.92s/it]
Outer training loop:  87%|########7 | 43557/50000 [28:55:16<3:23:03,  1.89s/it]
Outer training loop:  87%|########7 | 43558/50000 [28:55:18<3:17:15,  1.84s/it]
Outer training loop:  87%|########7 | 43559/50000 [28:55:20<3:24:25,  1.90s/it]
Outer training loop:  87%|########7 | 43560/50000 [28:55:22<3:22:29,  1.89s/it]
Outer training loop:  87%|########7 | 43561/50000 [28:55:24<3:22:13,  1.88s/it]
Outer training loop:  87%|########7 | 43562/50000 [28:55:26<3:29:54,  1.96s/it]
Outer training loop:  87%|########7 | 43563/50000 [28:55:28<3:30:47,  1.96s/it]
Outer training loop:  87%|########7 | 43564/50000 [28:55:30<3:25:35,  1.92s/it]
Outer training loop:  87%|########7 | 43565/50000 [28:55:31<3:23:22,  1.90s/it]
Outer training loop:  87%|########7 | 43566/50000 [28:55:33<3:26:16,  1.92s/it]
Outer training loop:  87%|########7 | 43567/50000 [28:55:35<3:24:53,  1.91s/it]
Outer training loop:  87%|########7 | 43568/50000 [28:55:37<3:19:54,  1.86s/it]
Outer training loop:  87%|########7 | 43569/50000 [28:55:39<3:22:22,  1.89s/it]
Outer training loop:  87%|########7 | 43570/50000 [28:55:41<3:16:30,  1.83s/it]
Outer training loop:  87%|########7 | 43571/50000 [28:55:43<3:17:51,  1.85s/it]
Outer training loop:  87%|########7 | 43572/50000 [28:55:44<3:20:07,  1.87s/it]
Outer training loop:  87%|########7 | 43573/50000 [28:55:46<3:19:53,  1.87s/it]
Outer training loop:  87%|########7 | 43574/50000 [28:55:48<3:17:49,  1.85s/it]
Outer training loop:  87%|########7 | 43575/50000 [28:55:50<3:21:40,  1.88s/it]
Outer training loop:  87%|########7 | 43576/50000 [28:55:52<3:24:45,  1.91s/it]
Outer training loop:  87%|########7 | 43577/50000 [28:55:54<3:22:49,  1.89s/it]
Outer training loop:  87%|########7 | 43578/50000 [28:55:56<3:21:21,  1.88s/it]
Outer training loop:  87%|########7 | 43579/50000 [28:55:58<3:24:10,  1.91s/it]
Outer training loop:  87%|########7 | 43580/50000 [28:56:00<3:23:33,  1.90s/it]
Outer training loop:  87%|########7 | 43581/50000 [28:56:01<3:19:59,  1.87s/it]
Outer training loop:  87%|########7 | 43582/50000 [28:56:03<3:24:17,  1.91s/it]
Outer training loop:  87%|########7 | 43583/50000 [28:56:05<3:24:32,  1.91s/it]
Outer training loop:  87%|########7 | 43584/50000 [28:56:07<3:23:33,  1.90s/it]
Outer training loop:  87%|########7 | 43585/50000 [28:56:09<3:21:48,  1.89s/it]
Outer training loop:  87%|########7 | 43586/50000 [28:56:11<3:25:27,  1.92s/it]
Outer training loop:  87%|########7 | 43587/50000 [28:56:13<3:29:39,  1.96s/it]
Outer training loop:  87%|########7 | 43588/50000 [28:56:15<3:23:43,  1.91s/it]
Outer training loop:  87%|########7 | 43589/50000 [28:56:17<3:28:34,  1.95s/it]
Outer training loop:  87%|########7 | 43590/50000 [28:56:19<3:30:13,  1.97s/it]
Outer training loop:  87%|########7 | 43591/50000 [28:56:21<3:25:13,  1.92s/it]
Outer training loop:  87%|########7 | 43592/50000 [28:56:23<3:28:17,  1.95s/it]
Outer training loop:  87%|########7 | 43593/50000 [28:56:25<3:28:05,  1.95s/it]
Outer training loop:  87%|########7 | 43594/50000 [28:56:27<3:22:36,  1.90s/it]
Outer training loop:  87%|########7 | 43595/50000 [28:56:29<3:26:15,  1.93s/it]
Outer training loop:  87%|########7 | 43596/50000 [28:56:30<3:26:22,  1.93s/it]
Outer training loop:  87%|########7 | 43597/50000 [28:56:32<3:24:27,  1.92s/it]
Outer training loop:  87%|########7 | 43598/50000 [28:56:34<3:21:00,  1.88s/it]
Outer training loop:  87%|########7 | 43599/50000 [28:56:36<3:28:30,  1.95s/it]
Outer training loop:  87%|########7 | 43600/50000 [28:56:38<3:21:58,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.07s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.03s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.94s/it]

Outer training loop:  87%|########7 | 43601/50000 [28:56:46<6:31:13,  3.67s/it]
Outer training loop:  87%|########7 | 43602/50000 [28:56:48<5:35:13,  3.14s/it]
Outer training loop:  87%|########7 | 43603/50000 [28:56:50<5:03:34,  2.85s/it]
Outer training loop:  87%|########7 | 43604/50000 [28:56:52<4:28:48,  2.52s/it]
Outer training loop:  87%|########7 | 43605/50000 [28:56:53<4:04:59,  2.30s/it]
Outer training loop:  87%|########7 | 43606/50000 [28:56:55<3:50:18,  2.16s/it]
Outer training loop:  87%|########7 | 43607/50000 [28:56:57<3:39:55,  2.06s/it]
Outer training loop:  87%|########7 | 43608/50000 [28:56:59<3:30:19,  1.97s/it]
Outer training loop:  87%|########7 | 43609/50000 [28:57:01<3:28:07,  1.95s/it]
Outer training loop:  87%|########7 | 43610/50000 [28:57:03<3:30:41,  1.98s/it]
Outer training loop:  87%|########7 | 43611/50000 [28:57:05<3:29:58,  1.97s/it]
Outer training loop:  87%|########7 | 43612/50000 [28:57:07<3:26:39,  1.94s/it]
Outer training loop:  87%|########7 | 43613/50000 [28:57:09<3:27:29,  1.95s/it]
Outer training loop:  87%|########7 | 43614/50000 [28:57:10<3:24:50,  1.92s/it]
Outer training loop:  87%|########7 | 43615/50000 [28:57:12<3:16:38,  1.85s/it]
Outer training loop:  87%|########7 | 43616/50000 [28:57:14<3:17:55,  1.86s/it]
Outer training loop:  87%|########7 | 43617/50000 [28:57:16<3:18:54,  1.87s/it]
Outer training loop:  87%|########7 | 43618/50000 [28:57:18<3:12:37,  1.81s/it]
Outer training loop:  87%|########7 | 43619/50000 [28:57:20<3:15:40,  1.84s/it]
Outer training loop:  87%|########7 | 43620/50000 [28:57:22<3:23:50,  1.92s/it]
Outer training loop:  87%|########7 | 43621/50000 [28:57:24<3:23:54,  1.92s/it]
Outer training loop:  87%|########7 | 43622/50000 [28:57:25<3:23:29,  1.91s/it]
Outer training loop:  87%|########7 | 43623/50000 [28:57:27<3:21:59,  1.90s/it]
Outer training loop:  87%|########7 | 43624/50000 [28:57:29<3:23:35,  1.92s/it]
Outer training loop:  87%|########7 | 43625/50000 [28:57:31<3:18:46,  1.87s/it]
Outer training loop:  87%|########7 | 43626/50000 [28:57:33<3:16:22,  1.85s/it]
Outer training loop:  87%|########7 | 43627/50000 [28:57:35<3:25:44,  1.94s/it]
Outer training loop:  87%|########7 | 43628/50000 [28:57:37<3:23:17,  1.91s/it]
Outer training loop:  87%|########7 | 43629/50000 [28:57:39<3:22:18,  1.91s/it]
Outer training loop:  87%|########7 | 43630/50000 [28:57:41<3:26:28,  1.94s/it]
Outer training loop:  87%|########7 | 43631/50000 [28:57:43<3:23:46,  1.92s/it]
Outer training loop:  87%|########7 | 43632/50000 [28:57:44<3:20:00,  1.88s/it]
Outer training loop:  87%|########7 | 43633/50000 [28:57:46<3:18:05,  1.87s/it]
Outer training loop:  87%|########7 | 43634/50000 [28:57:48<3:24:44,  1.93s/it]
Outer training loop:  87%|########7 | 43635/50000 [28:57:50<3:22:15,  1.91s/it]
Outer training loop:  87%|########7 | 43636/50000 [28:57:52<3:18:12,  1.87s/it]
Outer training loop:  87%|########7 | 43637/50000 [28:57:54<3:25:42,  1.94s/it]
Outer training loop:  87%|########7 | 43638/50000 [28:57:56<3:21:00,  1.90s/it]
Outer training loop:  87%|########7 | 43639/50000 [28:57:58<3:19:39,  1.88s/it]
Outer training loop:  87%|########7 | 43640/50000 [28:58:00<3:23:21,  1.92s/it]
Outer training loop:  87%|########7 | 43641/50000 [28:58:01<3:18:16,  1.87s/it]
Outer training loop:  87%|########7 | 43642/50000 [28:58:03<3:13:19,  1.82s/it]
Outer training loop:  87%|########7 | 43643/50000 [28:58:05<3:17:11,  1.86s/it]
Outer training loop:  87%|########7 | 43644/50000 [28:58:07<3:20:01,  1.89s/it]
Outer training loop:  87%|########7 | 43645/50000 [28:58:09<3:16:54,  1.86s/it]
Outer training loop:  87%|########7 | 43646/50000 [28:58:11<3:18:04,  1.87s/it]
Outer training loop:  87%|########7 | 43647/50000 [28:58:13<3:27:54,  1.96s/it]
Outer training loop:  87%|########7 | 43648/50000 [28:58:15<3:29:01,  1.97s/it]
Outer training loop:  87%|########7 | 43649/50000 [28:58:17<3:23:00,  1.92s/it]
Outer training loop:  87%|########7 | 43650/50000 [28:58:18<3:18:22,  1.87s/it]
Outer training loop:  87%|########7 | 43651/50000 [28:58:21<3:26:47,  1.95s/it]
Outer training loop:  87%|########7 | 43652/50000 [28:58:23<3:25:05,  1.94s/it]
Outer training loop:  87%|########7 | 43653/50000 [28:58:24<3:22:58,  1.92s/it]
Outer training loop:  87%|########7 | 43654/50000 [28:58:27<3:30:58,  1.99s/it]
Outer training loop:  87%|########7 | 43655/50000 [28:58:28<3:26:02,  1.95s/it]
Outer training loop:  87%|########7 | 43656/50000 [28:58:30<3:23:48,  1.93s/it]
Outer training loop:  87%|########7 | 43657/50000 [28:58:32<3:27:40,  1.96s/it]
Outer training loop:  87%|########7 | 43658/50000 [28:58:34<3:22:24,  1.91s/it]
Outer training loop:  87%|########7 | 43659/50000 [28:58:36<3:23:26,  1.93s/it]
Outer training loop:  87%|########7 | 43660/50000 [28:58:38<3:23:58,  1.93s/it]
Outer training loop:  87%|########7 | 43661/50000 [28:58:40<3:20:37,  1.90s/it]
Outer training loop:  87%|########7 | 43662/50000 [28:58:42<3:18:36,  1.88s/it]
Outer training loop:  87%|########7 | 43663/50000 [28:58:44<3:21:28,  1.91s/it]
Outer training loop:  87%|########7 | 43664/50000 [28:58:46<3:24:07,  1.93s/it]
Outer training loop:  87%|########7 | 43665/50000 [28:58:47<3:19:26,  1.89s/it]
Outer training loop:  87%|########7 | 43666/50000 [28:58:49<3:15:43,  1.85s/it]
Outer training loop:  87%|########7 | 43667/50000 [28:58:51<3:19:08,  1.89s/it]
Outer training loop:  87%|########7 | 43668/50000 [28:58:53<3:17:55,  1.88s/it]
Outer training loop:  87%|########7 | 43669/50000 [28:58:55<3:12:50,  1.83s/it]
Outer training loop:  87%|########7 | 43670/50000 [28:58:57<3:12:06,  1.82s/it]
Outer training loop:  87%|########7 | 43671/50000 [28:58:59<3:18:37,  1.88s/it]
Outer training loop:  87%|########7 | 43672/50000 [28:59:00<3:14:43,  1.85s/it]
Outer training loop:  87%|########7 | 43673/50000 [28:59:02<3:13:20,  1.83s/it]
Outer training loop:  87%|########7 | 43674/50000 [28:59:04<3:17:08,  1.87s/it]
Outer training loop:  87%|########7 | 43675/50000 [28:59:06<3:17:56,  1.88s/it]
Outer training loop:  87%|########7 | 43676/50000 [28:59:08<3:14:58,  1.85s/it]
Outer training loop:  87%|########7 | 43677/50000 [28:59:10<3:15:23,  1.85s/it]
Outer training loop:  87%|########7 | 43678/50000 [28:59:12<3:22:32,  1.92s/it]
Outer training loop:  87%|########7 | 43679/50000 [28:59:14<3:20:27,  1.90s/it]
Outer training loop:  87%|########7 | 43680/50000 [28:59:15<3:20:01,  1.90s/it]
Outer training loop:  87%|########7 | 43681/50000 [28:59:17<3:21:39,  1.91s/it]
Outer training loop:  87%|########7 | 43682/50000 [28:59:19<3:17:25,  1.87s/it]
Outer training loop:  87%|########7 | 43683/50000 [28:59:21<3:11:33,  1.82s/it]
Outer training loop:  87%|########7 | 43684/50000 [28:59:23<3:15:27,  1.86s/it]
Outer training loop:  87%|########7 | 43685/50000 [28:59:25<3:12:39,  1.83s/it]
Outer training loop:  87%|########7 | 43686/50000 [28:59:26<3:10:00,  1.81s/it]
Outer training loop:  87%|########7 | 43687/50000 [28:59:28<3:10:25,  1.81s/it]
Outer training loop:  87%|########7 | 43688/50000 [28:59:30<3:17:02,  1.87s/it]
Outer training loop:  87%|########7 | 43689/50000 [28:59:32<3:14:21,  1.85s/it]
Outer training loop:  87%|########7 | 43690/50000 [28:59:34<3:17:18,  1.88s/it]
Outer training loop:  87%|########7 | 43691/50000 [28:59:36<3:22:43,  1.93s/it]
Outer training loop:  87%|########7 | 43692/50000 [28:59:38<3:23:21,  1.93s/it]
Outer training loop:  87%|########7 | 43693/50000 [28:59:40<3:18:03,  1.88s/it]
Outer training loop:  87%|########7 | 43694/50000 [28:59:42<3:17:15,  1.88s/it]
Outer training loop:  87%|########7 | 43695/50000 [28:59:44<3:24:06,  1.94s/it]
Outer training loop:  87%|########7 | 43696/50000 [28:59:45<3:18:46,  1.89s/it]
Outer training loop:  87%|########7 | 43697/50000 [28:59:47<3:16:32,  1.87s/it]
Outer training loop:  87%|########7 | 43698/50000 [28:59:49<3:25:31,  1.96s/it]
Outer training loop:  87%|########7 | 43699/50000 [28:59:51<3:21:10,  1.92s/it]
Outer training loop:  87%|########7 | 43700/50000 [28:59:53<3:17:31,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  87%|########7 | 43701/50000 [29:00:01<6:15:48,  3.58s/it]
Outer training loop:  87%|########7 | 43702/50000 [29:00:02<5:23:35,  3.08s/it]
Outer training loop:  87%|########7 | 43703/50000 [29:00:04<4:44:48,  2.71s/it]
Outer training loop:  87%|########7 | 43704/50000 [29:00:06<4:16:16,  2.44s/it]
Outer training loop:  87%|########7 | 43705/50000 [29:00:08<4:00:31,  2.29s/it]
Outer training loop:  87%|########7 | 43706/50000 [29:00:10<3:46:14,  2.16s/it]
Outer training loop:  87%|########7 | 43707/50000 [29:00:12<3:32:45,  2.03s/it]
Outer training loop:  87%|########7 | 43708/50000 [29:00:13<3:23:43,  1.94s/it]
Outer training loop:  87%|########7 | 43709/50000 [29:00:15<3:24:19,  1.95s/it]
Outer training loop:  87%|########7 | 43710/50000 [29:00:17<3:17:13,  1.88s/it]
Outer training loop:  87%|########7 | 43711/50000 [29:00:19<3:14:31,  1.86s/it]
Outer training loop:  87%|########7 | 43712/50000 [29:00:21<3:15:38,  1.87s/it]
Outer training loop:  87%|########7 | 43713/50000 [29:00:23<3:19:12,  1.90s/it]
Outer training loop:  87%|########7 | 43714/50000 [29:00:25<3:16:34,  1.88s/it]
Outer training loop:  87%|########7 | 43715/50000 [29:00:26<3:15:52,  1.87s/it]
Outer training loop:  87%|########7 | 43716/50000 [29:00:28<3:16:56,  1.88s/it]
Outer training loop:  87%|########7 | 43717/50000 [29:00:30<3:11:43,  1.83s/it]
Outer training loop:  87%|########7 | 43718/50000 [29:00:32<3:10:27,  1.82s/it]
Outer training loop:  87%|########7 | 43719/50000 [29:00:34<3:10:54,  1.82s/it]
Outer training loop:  87%|########7 | 43720/50000 [29:00:36<3:16:52,  1.88s/it]
Outer training loop:  87%|########7 | 43721/50000 [29:00:38<3:17:13,  1.88s/it]
Outer training loop:  87%|########7 | 43722/50000 [29:00:40<3:20:57,  1.92s/it]
Outer training loop:  87%|########7 | 43723/50000 [29:00:42<3:22:22,  1.93s/it]
Outer training loop:  87%|########7 | 43724/50000 [29:00:43<3:20:12,  1.91s/it]
Outer training loop:  87%|########7 | 43725/50000 [29:00:45<3:13:45,  1.85s/it]
Outer training loop:  87%|########7 | 43726/50000 [29:00:47<3:16:50,  1.88s/it]
Outer training loop:  87%|########7 | 43727/50000 [29:00:49<3:15:51,  1.87s/it]
Outer training loop:  87%|########7 | 43728/50000 [29:00:51<3:12:56,  1.85s/it]
Outer training loop:  87%|########7 | 43729/50000 [29:00:53<3:16:16,  1.88s/it]
Outer training loop:  87%|########7 | 43730/50000 [29:00:55<3:24:16,  1.95s/it]
Outer training loop:  87%|########7 | 43731/50000 [29:00:57<3:23:55,  1.95s/it]
Outer training loop:  87%|########7 | 43732/50000 [29:00:59<3:20:23,  1.92s/it]
Outer training loop:  87%|########7 | 43733/50000 [29:01:01<3:22:05,  1.93s/it]
Outer training loop:  87%|########7 | 43734/50000 [29:01:02<3:18:40,  1.90s/it]
Outer training loop:  87%|########7 | 43735/50000 [29:01:04<3:19:03,  1.91s/it]
Outer training loop:  87%|########7 | 43736/50000 [29:01:06<3:21:45,  1.93s/it]
Outer training loop:  87%|########7 | 43737/50000 [29:01:08<3:21:32,  1.93s/it]
Outer training loop:  87%|########7 | 43738/50000 [29:01:10<3:20:40,  1.92s/it]
Outer training loop:  87%|########7 | 43739/50000 [29:01:12<3:19:19,  1.91s/it]
Outer training loop:  87%|########7 | 43740/50000 [29:01:14<3:28:27,  2.00s/it]
Outer training loop:  87%|########7 | 43741/50000 [29:01:16<3:21:36,  1.93s/it]
Outer training loop:  87%|########7 | 43742/50000 [29:01:18<3:15:56,  1.88s/it]
Outer training loop:  87%|########7 | 43743/50000 [29:01:20<3:20:10,  1.92s/it]
Outer training loop:  87%|########7 | 43744/50000 [29:01:22<3:15:44,  1.88s/it]
Outer training loop:  87%|########7 | 43745/50000 [29:01:24<3:20:31,  1.92s/it]
Outer training loop:  87%|########7 | 43746/50000 [29:01:26<3:21:04,  1.93s/it]
Outer training loop:  87%|########7 | 43747/50000 [29:01:28<3:23:12,  1.95s/it]
Outer training loop:  87%|########7 | 43748/50000 [29:01:29<3:17:40,  1.90s/it]
Outer training loop:  87%|########7 | 43749/50000 [29:01:31<3:17:40,  1.90s/it]
Outer training loop:  88%|########7 | 43750/50000 [29:01:33<3:25:36,  1.97s/it]
Outer training loop:  88%|########7 | 43751/50000 [29:01:35<3:27:33,  1.99s/it]
Outer training loop:  88%|########7 | 43752/50000 [29:01:37<3:20:35,  1.93s/it]
Outer training loop:  88%|########7 | 43753/50000 [29:01:39<3:20:14,  1.92s/it]
Outer training loop:  88%|########7 | 43754/50000 [29:01:41<3:24:44,  1.97s/it]
Outer training loop:  88%|########7 | 43755/50000 [29:01:43<3:24:07,  1.96s/it]
Outer training loop:  88%|########7 | 43756/50000 [29:01:45<3:21:56,  1.94s/it]
Outer training loop:  88%|########7 | 43757/50000 [29:01:47<3:27:22,  1.99s/it]
Outer training loop:  88%|########7 | 43758/50000 [29:01:49<3:25:35,  1.98s/it]
Outer training loop:  88%|########7 | 43759/50000 [29:01:51<3:23:15,  1.95s/it]
Outer training loop:  88%|########7 | 43760/50000 [29:01:53<3:15:29,  1.88s/it]
Outer training loop:  88%|########7 | 43761/50000 [29:01:55<3:24:14,  1.96s/it]
Outer training loop:  88%|########7 | 43762/50000 [29:01:57<3:19:38,  1.92s/it]
Outer training loop:  88%|########7 | 43763/50000 [29:01:59<3:19:03,  1.91s/it]
Outer training loop:  88%|########7 | 43764/50000 [29:02:01<3:21:14,  1.94s/it]
Outer training loop:  88%|########7 | 43765/50000 [29:02:02<3:18:19,  1.91s/it]
Outer training loop:  88%|########7 | 43766/50000 [29:02:04<3:17:34,  1.90s/it]
Outer training loop:  88%|########7 | 43767/50000 [29:02:06<3:18:27,  1.91s/it]
Outer training loop:  88%|########7 | 43768/50000 [29:02:08<3:25:23,  1.98s/it]
Outer training loop:  88%|########7 | 43769/50000 [29:02:10<3:19:54,  1.93s/it]
Outer training loop:  88%|########7 | 43770/50000 [29:02:12<3:20:21,  1.93s/it]
Outer training loop:  88%|########7 | 43771/50000 [29:02:14<3:25:53,  1.98s/it]
Outer training loop:  88%|########7 | 43772/50000 [29:02:16<3:19:24,  1.92s/it]
Outer training loop:  88%|########7 | 43773/50000 [29:02:18<3:16:43,  1.90s/it]
Outer training loop:  88%|########7 | 43774/50000 [29:02:20<3:16:26,  1.89s/it]
Outer training loop:  88%|########7 | 43775/50000 [29:02:22<3:18:49,  1.92s/it]
Outer training loop:  88%|########7 | 43776/50000 [29:02:23<3:16:26,  1.89s/it]
Outer training loop:  88%|########7 | 43777/50000 [29:02:25<3:17:25,  1.90s/it]
Outer training loop:  88%|########7 | 43778/50000 [29:02:28<3:24:06,  1.97s/it]
Outer training loop:  88%|########7 | 43779/50000 [29:02:29<3:22:02,  1.95s/it]
Outer training loop:  88%|########7 | 43780/50000 [29:02:31<3:21:26,  1.94s/it]
Outer training loop:  88%|########7 | 43781/50000 [29:02:33<3:17:41,  1.91s/it]
Outer training loop:  88%|########7 | 43782/50000 [29:02:36<3:32:37,  2.05s/it]
Outer training loop:  88%|########7 | 43783/50000 [29:02:37<3:25:47,  1.99s/it]
Outer training loop:  88%|########7 | 43784/50000 [29:02:39<3:16:35,  1.90s/it]
Outer training loop:  88%|########7 | 43785/50000 [29:02:41<3:29:06,  2.02s/it]
Outer training loop:  88%|########7 | 43786/50000 [29:02:43<3:23:55,  1.97s/it]
Outer training loop:  88%|########7 | 43787/50000 [29:02:45<3:14:44,  1.88s/it]
Outer training loop:  88%|########7 | 43788/50000 [29:02:47<3:14:40,  1.88s/it]
Outer training loop:  88%|########7 | 43789/50000 [29:02:49<3:18:06,  1.91s/it]
Outer training loop:  88%|########7 | 43790/50000 [29:02:51<3:15:07,  1.89s/it]
Outer training loop:  88%|########7 | 43791/50000 [29:02:52<3:14:31,  1.88s/it]
Outer training loop:  88%|########7 | 43792/50000 [29:02:55<3:21:24,  1.95s/it]
Outer training loop:  88%|########7 | 43793/50000 [29:02:56<3:17:02,  1.90s/it]
Outer training loop:  88%|########7 | 43794/50000 [29:02:58<3:17:41,  1.91s/it]
Outer training loop:  88%|########7 | 43795/50000 [29:03:00<3:16:27,  1.90s/it]
Outer training loop:  88%|########7 | 43796/50000 [29:03:02<3:20:17,  1.94s/it]
Outer training loop:  88%|########7 | 43797/50000 [29:03:04<3:19:53,  1.93s/it]
Outer training loop:  88%|########7 | 43798/50000 [29:03:06<3:19:11,  1.93s/it]
Outer training loop:  88%|########7 | 43799/50000 [29:03:08<3:22:59,  1.96s/it]
Outer training loop:  88%|########7 | 43800/50000 [29:03:10<3:20:53,  1.94s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  88%|########7 | 43801/50000 [29:03:18<6:18:32,  3.66s/it]
Outer training loop:  88%|########7 | 43802/50000 [29:03:20<5:23:44,  3.13s/it]
Outer training loop:  88%|########7 | 43803/50000 [29:03:21<4:45:38,  2.77s/it]
Outer training loop:  88%|########7 | 43804/50000 [29:03:23<4:20:16,  2.52s/it]
Outer training loop:  88%|########7 | 43805/50000 [29:03:25<3:59:26,  2.32s/it]
Outer training loop:  88%|########7 | 43806/50000 [29:03:27<3:44:45,  2.18s/it]
Outer training loop:  88%|########7 | 43807/50000 [29:03:29<3:35:55,  2.09s/it]
Outer training loop:  88%|########7 | 43808/50000 [29:03:31<3:33:48,  2.07s/it]
Outer training loop:  88%|########7 | 43809/50000 [29:03:33<3:23:56,  1.98s/it]
Outer training loop:  88%|########7 | 43810/50000 [29:03:35<3:23:25,  1.97s/it]
Outer training loop:  88%|########7 | 43811/50000 [29:03:37<3:24:12,  1.98s/it]
Outer training loop:  88%|########7 | 43812/50000 [29:03:39<3:19:10,  1.93s/it]
Outer training loop:  88%|########7 | 43813/50000 [29:03:40<3:17:02,  1.91s/it]
Outer training loop:  88%|########7 | 43814/50000 [29:03:42<3:20:19,  1.94s/it]
Outer training loop:  88%|########7 | 43815/50000 [29:03:44<3:15:44,  1.90s/it]
Outer training loop:  88%|########7 | 43816/50000 [29:03:46<3:11:51,  1.86s/it]
Outer training loop:  88%|########7 | 43817/50000 [29:03:48<3:15:13,  1.89s/it]
Outer training loop:  88%|########7 | 43818/50000 [29:03:50<3:15:48,  1.90s/it]
Outer training loop:  88%|########7 | 43819/50000 [29:03:52<3:13:06,  1.87s/it]
Outer training loop:  88%|########7 | 43820/50000 [29:03:54<3:15:06,  1.89s/it]
Outer training loop:  88%|########7 | 43821/50000 [29:03:56<3:20:44,  1.95s/it]
Outer training loop:  88%|########7 | 43822/50000 [29:03:58<3:19:13,  1.93s/it]
Outer training loop:  88%|########7 | 43823/50000 [29:03:59<3:13:14,  1.88s/it]
Outer training loop:  88%|########7 | 43824/50000 [29:04:01<3:14:52,  1.89s/it]
Outer training loop:  88%|########7 | 43825/50000 [29:04:03<3:11:45,  1.86s/it]
Outer training loop:  88%|########7 | 43826/50000 [29:04:05<3:11:25,  1.86s/it]
Outer training loop:  88%|########7 | 43827/50000 [29:04:07<3:06:54,  1.82s/it]
Outer training loop:  88%|########7 | 43828/50000 [29:04:09<3:15:31,  1.90s/it]
Outer training loop:  88%|########7 | 43829/50000 [29:04:11<3:12:36,  1.87s/it]
Outer training loop:  88%|########7 | 43830/50000 [29:04:13<3:16:49,  1.91s/it]
Outer training loop:  88%|########7 | 43831/50000 [29:04:14<3:16:53,  1.92s/it]
Outer training loop:  88%|########7 | 43832/50000 [29:04:16<3:09:57,  1.85s/it]
Outer training loop:  88%|########7 | 43833/50000 [29:04:18<3:10:57,  1.86s/it]
Outer training loop:  88%|########7 | 43834/50000 [29:04:20<3:10:22,  1.85s/it]
Outer training loop:  88%|########7 | 43835/50000 [29:04:22<3:09:53,  1.85s/it]
Outer training loop:  88%|########7 | 43836/50000 [29:04:24<3:09:42,  1.85s/it]
Outer training loop:  88%|########7 | 43837/50000 [29:04:25<3:10:30,  1.85s/it]
Outer training loop:  88%|########7 | 43838/50000 [29:04:27<3:13:19,  1.88s/it]
Outer training loop:  88%|########7 | 43839/50000 [29:04:29<3:13:24,  1.88s/it]
Outer training loop:  88%|########7 | 43840/50000 [29:04:31<3:11:17,  1.86s/it]
Outer training loop:  88%|########7 | 43841/50000 [29:04:33<3:10:05,  1.85s/it]
Outer training loop:  88%|########7 | 43842/50000 [29:04:35<3:26:06,  2.01s/it]
Outer training loop:  88%|########7 | 43843/50000 [29:04:37<3:23:39,  1.98s/it]
Outer training loop:  88%|########7 | 43844/50000 [29:04:39<3:17:05,  1.92s/it]
Outer training loop:  88%|########7 | 43845/50000 [29:04:41<3:16:07,  1.91s/it]
Outer training loop:  88%|########7 | 43846/50000 [29:04:43<3:20:59,  1.96s/it]
Outer training loop:  88%|########7 | 43847/50000 [29:04:45<3:19:07,  1.94s/it]
Outer training loop:  88%|########7 | 43848/50000 [29:04:47<3:14:00,  1.89s/it]
Outer training loop:  88%|########7 | 43849/50000 [29:04:49<3:17:22,  1.93s/it]
Outer training loop:  88%|########7 | 43850/50000 [29:04:51<3:15:52,  1.91s/it]
Outer training loop:  88%|########7 | 43851/50000 [29:04:52<3:14:30,  1.90s/it]
Outer training loop:  88%|########7 | 43852/50000 [29:04:54<3:18:19,  1.94s/it]
Outer training loop:  88%|########7 | 43853/50000 [29:04:57<3:25:38,  2.01s/it]
Outer training loop:  88%|########7 | 43854/50000 [29:04:59<3:22:38,  1.98s/it]
Outer training loop:  88%|########7 | 43855/50000 [29:05:00<3:18:00,  1.93s/it]
Outer training loop:  88%|########7 | 43856/50000 [29:05:02<3:20:27,  1.96s/it]
Outer training loop:  88%|########7 | 43857/50000 [29:05:04<3:13:57,  1.89s/it]
Outer training loop:  88%|########7 | 43858/50000 [29:05:06<3:12:27,  1.88s/it]
Outer training loop:  88%|########7 | 43859/50000 [29:05:08<3:19:27,  1.95s/it]
Outer training loop:  88%|########7 | 43860/50000 [29:05:10<3:19:11,  1.95s/it]
Outer training loop:  88%|########7 | 43861/50000 [29:05:12<3:16:37,  1.92s/it]
Outer training loop:  88%|########7 | 43862/50000 [29:05:14<3:23:38,  1.99s/it]
Outer training loop:  88%|########7 | 43863/50000 [29:05:16<3:21:24,  1.97s/it]
Outer training loop:  88%|########7 | 43864/50000 [29:05:18<3:16:38,  1.92s/it]
Outer training loop:  88%|########7 | 43865/50000 [29:05:20<3:14:22,  1.90s/it]
Outer training loop:  88%|########7 | 43866/50000 [29:05:21<3:13:44,  1.90s/it]
Outer training loop:  88%|########7 | 43867/50000 [29:05:23<3:13:34,  1.89s/it]
Outer training loop:  88%|########7 | 43868/50000 [29:05:25<3:15:36,  1.91s/it]
Outer training loop:  88%|########7 | 43869/50000 [29:05:27<3:20:35,  1.96s/it]
Outer training loop:  88%|########7 | 43870/50000 [29:05:29<3:15:24,  1.91s/it]
Outer training loop:  88%|########7 | 43871/50000 [29:05:31<3:11:48,  1.88s/it]
Outer training loop:  88%|########7 | 43872/50000 [29:05:33<3:12:59,  1.89s/it]
Outer training loop:  88%|########7 | 43873/50000 [29:05:35<3:20:51,  1.97s/it]
Outer training loop:  88%|########7 | 43874/50000 [29:05:37<3:14:46,  1.91s/it]
Outer training loop:  88%|########7 | 43875/50000 [29:05:39<3:09:33,  1.86s/it]
Outer training loop:  88%|########7 | 43876/50000 [29:05:41<3:17:32,  1.94s/it]
Outer training loop:  88%|########7 | 43877/50000 [29:05:43<3:14:09,  1.90s/it]
Outer training loop:  88%|########7 | 43878/50000 [29:05:44<3:12:47,  1.89s/it]
Outer training loop:  88%|########7 | 43879/50000 [29:05:46<3:10:35,  1.87s/it]
Outer training loop:  88%|########7 | 43880/50000 [29:05:48<3:15:20,  1.92s/it]
Outer training loop:  88%|########7 | 43881/50000 [29:05:50<3:15:59,  1.92s/it]
Outer training loop:  88%|########7 | 43882/50000 [29:05:52<3:13:41,  1.90s/it]
Outer training loop:  88%|########7 | 43883/50000 [29:05:54<3:17:22,  1.94s/it]
Outer training loop:  88%|########7 | 43884/50000 [29:05:56<3:14:08,  1.90s/it]
Outer training loop:  88%|########7 | 43885/50000 [29:05:58<3:14:32,  1.91s/it]
Outer training loop:  88%|########7 | 43886/50000 [29:06:00<3:11:12,  1.88s/it]
Outer training loop:  88%|########7 | 43887/50000 [29:06:02<3:15:44,  1.92s/it]
Outer training loop:  88%|########7 | 43888/50000 [29:06:03<3:13:47,  1.90s/it]
Outer training loop:  88%|########7 | 43889/50000 [29:06:05<3:16:27,  1.93s/it]
Outer training loop:  88%|########7 | 43890/50000 [29:06:07<3:12:08,  1.89s/it]
Outer training loop:  88%|########7 | 43891/50000 [29:06:09<3:19:39,  1.96s/it]
Outer training loop:  88%|########7 | 43892/50000 [29:06:11<3:23:22,  2.00s/it]
Outer training loop:  88%|########7 | 43893/50000 [29:06:13<3:22:23,  1.99s/it]
Outer training loop:  88%|########7 | 43894/50000 [29:06:15<3:22:32,  1.99s/it]
Outer training loop:  88%|########7 | 43895/50000 [29:06:17<3:18:09,  1.95s/it]
Outer training loop:  88%|########7 | 43896/50000 [29:06:19<3:16:18,  1.93s/it]
Outer training loop:  88%|########7 | 43897/50000 [29:06:21<3:15:17,  1.92s/it]
Outer training loop:  88%|########7 | 43898/50000 [29:06:23<3:14:14,  1.91s/it]
Outer training loop:  88%|########7 | 43899/50000 [29:06:25<3:16:57,  1.94s/it]
Outer training loop:  88%|########7 | 43900/50000 [29:06:27<3:14:40,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  88%|########7 | 43901/50000 [29:06:34<6:08:52,  3.63s/it]
Outer training loop:  88%|########7 | 43902/50000 [29:06:36<5:13:13,  3.08s/it]
Outer training loop:  88%|########7 | 43903/50000 [29:06:38<4:36:11,  2.72s/it]
Outer training loop:  88%|########7 | 43904/50000 [29:06:40<4:11:02,  2.47s/it]
Outer training loop:  88%|########7 | 43905/50000 [29:06:42<3:57:55,  2.34s/it]
Outer training loop:  88%|########7 | 43906/50000 [29:06:44<3:41:45,  2.18s/it]
Outer training loop:  88%|########7 | 43907/50000 [29:06:46<3:28:56,  2.06s/it]
Outer training loop:  88%|########7 | 43908/50000 [29:06:48<3:25:44,  2.03s/it]
Outer training loop:  88%|########7 | 43909/50000 [29:06:49<3:20:02,  1.97s/it]
Outer training loop:  88%|########7 | 43910/50000 [29:06:51<3:15:52,  1.93s/it]
Outer training loop:  88%|########7 | 43911/50000 [29:06:53<3:09:16,  1.87s/it]
Outer training loop:  88%|########7 | 43912/50000 [29:06:55<3:16:33,  1.94s/it]
Outer training loop:  88%|########7 | 43913/50000 [29:06:57<3:12:09,  1.89s/it]
Outer training loop:  88%|########7 | 43914/50000 [29:06:59<3:09:13,  1.87s/it]
Outer training loop:  88%|########7 | 43915/50000 [29:07:01<3:14:15,  1.92s/it]
Outer training loop:  88%|########7 | 43916/50000 [29:07:03<3:16:55,  1.94s/it]
Outer training loop:  88%|########7 | 43917/50000 [29:07:05<3:15:53,  1.93s/it]
Outer training loop:  88%|########7 | 43918/50000 [29:07:07<3:15:47,  1.93s/it]
Outer training loop:  88%|########7 | 43919/50000 [29:07:08<3:16:20,  1.94s/it]
Outer training loop:  88%|########7 | 43920/50000 [29:07:10<3:15:47,  1.93s/it]
Outer training loop:  88%|########7 | 43921/50000 [29:07:12<3:13:02,  1.91s/it]
Outer training loop:  88%|########7 | 43922/50000 [29:07:14<3:17:47,  1.95s/it]
Outer training loop:  88%|########7 | 43923/50000 [29:07:16<3:12:21,  1.90s/it]
Outer training loop:  88%|########7 | 43924/50000 [29:07:18<3:09:49,  1.87s/it]
Outer training loop:  88%|########7 | 43925/50000 [29:07:20<3:08:37,  1.86s/it]
Outer training loop:  88%|########7 | 43926/50000 [29:07:22<3:12:58,  1.91s/it]
Outer training loop:  88%|########7 | 43927/50000 [29:07:24<3:09:58,  1.88s/it]
Outer training loop:  88%|########7 | 43928/50000 [29:07:26<3:13:11,  1.91s/it]
Outer training loop:  88%|########7 | 43929/50000 [29:07:27<3:07:35,  1.85s/it]
Outer training loop:  88%|########7 | 43930/50000 [29:07:29<3:09:32,  1.87s/it]
Outer training loop:  88%|########7 | 43931/50000 [29:07:31<3:08:18,  1.86s/it]
Outer training loop:  88%|########7 | 43932/50000 [29:07:33<3:06:37,  1.85s/it]
Outer training loop:  88%|########7 | 43933/50000 [29:07:35<3:13:22,  1.91s/it]
Outer training loop:  88%|########7 | 43934/50000 [29:07:37<3:07:57,  1.86s/it]
Outer training loop:  88%|########7 | 43935/50000 [29:07:38<3:06:46,  1.85s/it]
Outer training loop:  88%|########7 | 43936/50000 [29:07:40<3:12:34,  1.91s/it]
Outer training loop:  88%|########7 | 43937/50000 [29:07:42<3:09:07,  1.87s/it]
Outer training loop:  88%|########7 | 43938/50000 [29:07:44<3:08:50,  1.87s/it]
Outer training loop:  88%|########7 | 43939/50000 [29:07:46<3:13:50,  1.92s/it]
Outer training loop:  88%|########7 | 43940/50000 [29:07:48<3:15:49,  1.94s/it]
Outer training loop:  88%|########7 | 43941/50000 [29:07:50<3:15:18,  1.93s/it]
Outer training loop:  88%|########7 | 43942/50000 [29:07:52<3:08:03,  1.86s/it]
Outer training loop:  88%|########7 | 43943/50000 [29:07:54<3:13:59,  1.92s/it]
Outer training loop:  88%|########7 | 43944/50000 [29:07:56<3:14:59,  1.93s/it]
Outer training loop:  88%|########7 | 43945/50000 [29:07:58<3:13:11,  1.91s/it]
Outer training loop:  88%|########7 | 43946/50000 [29:07:59<3:10:40,  1.89s/it]
Outer training loop:  88%|########7 | 43947/50000 [29:08:02<3:16:46,  1.95s/it]
Outer training loop:  88%|########7 | 43948/50000 [29:08:04<3:15:43,  1.94s/it]
Outer training loop:  88%|########7 | 43949/50000 [29:08:05<3:15:14,  1.94s/it]
Outer training loop:  88%|########7 | 43950/50000 [29:08:08<3:21:07,  1.99s/it]
Outer training loop:  88%|########7 | 43951/50000 [29:08:09<3:18:09,  1.97s/it]
Outer training loop:  88%|########7 | 43952/50000 [29:08:11<3:14:43,  1.93s/it]
Outer training loop:  88%|########7 | 43953/50000 [29:08:13<3:17:16,  1.96s/it]
Outer training loop:  88%|########7 | 43954/50000 [29:08:15<3:11:30,  1.90s/it]
Outer training loop:  88%|########7 | 43955/50000 [29:08:17<3:09:44,  1.88s/it]
Outer training loop:  88%|########7 | 43956/50000 [29:08:19<3:17:58,  1.97s/it]
Outer training loop:  88%|########7 | 43957/50000 [29:08:21<3:14:11,  1.93s/it]
Outer training loop:  88%|########7 | 43958/50000 [29:08:23<3:13:31,  1.92s/it]
Outer training loop:  88%|########7 | 43959/50000 [29:08:25<3:09:36,  1.88s/it]
Outer training loop:  88%|########7 | 43960/50000 [29:08:27<3:10:15,  1.89s/it]
Outer training loop:  88%|########7 | 43961/50000 [29:08:28<3:08:07,  1.87s/it]
Outer training loop:  88%|########7 | 43962/50000 [29:08:30<3:07:26,  1.86s/it]
Outer training loop:  88%|########7 | 43963/50000 [29:08:32<3:11:11,  1.90s/it]
Outer training loop:  88%|########7 | 43964/50000 [29:08:34<3:07:17,  1.86s/it]
Outer training loop:  88%|########7 | 43965/50000 [29:08:36<3:07:37,  1.87s/it]
Outer training loop:  88%|########7 | 43966/50000 [29:08:38<3:03:06,  1.82s/it]
Outer training loop:  88%|########7 | 43967/50000 [29:08:40<3:09:17,  1.88s/it]
Outer training loop:  88%|########7 | 43968/50000 [29:08:41<3:09:45,  1.89s/it]
Outer training loop:  88%|########7 | 43969/50000 [29:08:43<3:08:10,  1.87s/it]
Outer training loop:  88%|########7 | 43970/50000 [29:08:45<3:12:23,  1.91s/it]
Outer training loop:  88%|########7 | 43971/50000 [29:08:47<3:09:49,  1.89s/it]
Outer training loop:  88%|########7 | 43972/50000 [29:08:49<3:10:44,  1.90s/it]
Outer training loop:  88%|########7 | 43973/50000 [29:08:51<3:14:54,  1.94s/it]
Outer training loop:  88%|########7 | 43974/50000 [29:08:53<3:11:28,  1.91s/it]
Outer training loop:  88%|########7 | 43975/50000 [29:08:55<3:06:42,  1.86s/it]
Outer training loop:  88%|########7 | 43976/50000 [29:08:57<3:13:37,  1.93s/it]
Outer training loop:  88%|########7 | 43977/50000 [29:08:59<3:11:26,  1.91s/it]
Outer training loop:  88%|########7 | 43978/50000 [29:09:00<3:08:14,  1.88s/it]
Outer training loop:  88%|########7 | 43979/50000 [29:09:02<3:11:15,  1.91s/it]
Outer training loop:  88%|########7 | 43980/50000 [29:09:04<3:08:10,  1.88s/it]
Outer training loop:  88%|########7 | 43981/50000 [29:09:06<3:07:23,  1.87s/it]
Outer training loop:  88%|########7 | 43982/50000 [29:09:08<3:03:49,  1.83s/it]
Outer training loop:  88%|########7 | 43983/50000 [29:09:10<3:09:15,  1.89s/it]
Outer training loop:  88%|########7 | 43984/50000 [29:09:12<3:08:53,  1.88s/it]
Outer training loop:  88%|########7 | 43985/50000 [29:09:14<3:07:51,  1.87s/it]
Outer training loop:  88%|########7 | 43986/50000 [29:09:15<3:06:27,  1.86s/it]
Outer training loop:  88%|########7 | 43987/50000 [29:09:17<3:12:14,  1.92s/it]
Outer training loop:  88%|########7 | 43988/50000 [29:09:19<3:10:31,  1.90s/it]
Outer training loop:  88%|########7 | 43989/50000 [29:09:21<3:11:05,  1.91s/it]
Outer training loop:  88%|########7 | 43990/50000 [29:09:23<3:14:46,  1.94s/it]
Outer training loop:  88%|########7 | 43991/50000 [29:09:25<3:13:34,  1.93s/it]
Outer training loop:  88%|########7 | 43992/50000 [29:09:27<3:09:25,  1.89s/it]
Outer training loop:  88%|########7 | 43993/50000 [29:09:29<3:12:11,  1.92s/it]
Outer training loop:  88%|########7 | 43994/50000 [29:09:31<3:08:55,  1.89s/it]
Outer training loop:  88%|########7 | 43995/50000 [29:09:33<3:09:49,  1.90s/it]
Outer training loop:  88%|########7 | 43996/50000 [29:09:35<3:12:48,  1.93s/it]
Outer training loop:  88%|########7 | 43997/50000 [29:09:37<3:12:47,  1.93s/it]
Outer training loop:  88%|########7 | 43998/50000 [29:09:38<3:08:57,  1.89s/it]
Outer training loop:  88%|########7 | 43999/50000 [29:09:40<3:13:50,  1.94s/it]
Outer training loop:  88%|########8 | 44000/50000 [29:09:42<3:13:18,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.77s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.78s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.86s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:24,  1.83s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:23,  1.84s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:20,  1.83s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:21,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:18,  1.88s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:20,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:08,  1.86s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.94s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.88s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:59,  1.84s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:03,  1.96s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.86s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.85s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:46,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:42,  1.87s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:38,  1.82s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:41,  1.92s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:39,  1.91s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.90s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:33,  1.98s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:22,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:17,  1.85s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:19,  1.95s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:16,  1.92s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.89s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:09,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:06,  1.80s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:04,  1.79s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:05,  1.93s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.87s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:50,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:53,  1.97s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.96s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:48,  1.93s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.90s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:37,  1.97s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:34,  1.91s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.82s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:21,  1.81s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:21,  1.97s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:19,  1.91s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:17,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.85s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.83s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.86s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:02,  2.04s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.91s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:24,  1.76s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.82s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.80s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.82s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:22,  1.83s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:20,  1.82s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:20,  1.85s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:19,  1.85s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:17,  1.85s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:27,  2.02s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:22,  1.98s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.93s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:16,  1.97s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:12,  1.94s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.93s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.92s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.90s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:51,  1.83s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:56,  1.94s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:45,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:43,  1.84s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:47,  1.96s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.92s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:35,  1.86s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:35,  1.96s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.90s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:31,  1.96s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:28,  1.91s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:24,  1.87s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:24,  1.98s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:21,  1.94s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.91s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.90s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:13,  1.99s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:10,  1.95s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:07,  1.92s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.91s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:00,  1.85s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.92s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<01:00,  2.03s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:55,  1.93s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:53,  1.93s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:52,  1.96s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.90s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.91s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:45,  1.88s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:44,  1.93s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:44,  2.03s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:41,  1.99s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:38,  1.93s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:36,  1.90s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:33,  1.88s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:32,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:30,  1.88s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:27,  1.86s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:26,  2.03s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:23,  1.96s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.90s/it]

Evaluating policy:  88%|########7 | 73/83 [02:19<00:18,  1.89s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:14,  1.86s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.87s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.86s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.82s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.97s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.94s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.87s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.87s/it]

Outer training loop:  88%|########8 | 44001/50000 [29:15:04<163:13:28, 97.95s/it]
Outer training loop:  88%|########8 | 44002/50000 [29:15:06<115:12:03, 69.14s/it]
Outer training loop:  88%|########8 | 44003/50000 [29:15:08<81:33:34, 48.96s/it] 
Outer training loop:  88%|########8 | 44004/50000 [29:15:10<58:00:49, 34.83s/it]
Outer training loop:  88%|########8 | 44005/50000 [29:15:12<41:29:58, 24.92s/it]
Outer training loop:  88%|########8 | 44006/50000 [29:15:14<29:56:02, 17.98s/it]
Outer training loop:  88%|########8 | 44007/50000 [29:15:15<21:51:22, 13.13s/it]
Outer training loop:  88%|########8 | 44008/50000 [29:15:18<16:29:16,  9.91s/it]
Outer training loop:  88%|########8 | 44009/50000 [29:15:20<12:28:16,  7.49s/it]
Outer training loop:  88%|########8 | 44010/50000 [29:15:21<9:35:27,  5.76s/it] 
Outer training loop:  88%|########8 | 44011/50000 [29:15:23<7:42:13,  4.63s/it]
Outer training loop:  88%|########8 | 44012/50000 [29:15:25<6:19:52,  3.81s/it]
Outer training loop:  88%|########8 | 44013/50000 [29:15:27<5:20:14,  3.21s/it]
Outer training loop:  88%|########8 | 44014/50000 [29:15:29<4:46:27,  2.87s/it]
Outer training loop:  88%|########8 | 44015/50000 [29:15:31<4:14:14,  2.55s/it]
Outer training loop:  88%|########8 | 44016/50000 [29:15:33<3:52:02,  2.33s/it]
Outer training loop:  88%|########8 | 44017/50000 [29:15:35<3:40:05,  2.21s/it]
Outer training loop:  88%|########8 | 44018/50000 [29:15:37<3:28:27,  2.09s/it]
Outer training loop:  88%|########8 | 44019/50000 [29:15:38<3:18:47,  1.99s/it]
Outer training loop:  88%|########8 | 44020/50000 [29:15:40<3:17:41,  1.98s/it]
Outer training loop:  88%|########8 | 44021/50000 [29:15:42<3:18:05,  1.99s/it]
Outer training loop:  88%|########8 | 44022/50000 [29:15:44<3:14:53,  1.96s/it]
Outer training loop:  88%|########8 | 44023/50000 [29:15:46<3:08:52,  1.90s/it]
Outer training loop:  88%|########8 | 44024/50000 [29:15:48<3:10:01,  1.91s/it]
Outer training loop:  88%|########8 | 44025/50000 [29:15:50<3:07:43,  1.89s/it]
Outer training loop:  88%|########8 | 44026/50000 [29:15:51<3:03:34,  1.84s/it]
Outer training loop:  88%|########8 | 44027/50000 [29:15:53<3:05:03,  1.86s/it]
Outer training loop:  88%|########8 | 44028/50000 [29:15:55<3:10:58,  1.92s/it]
Outer training loop:  88%|########8 | 44029/50000 [29:15:57<3:07:52,  1.89s/it]
Outer training loop:  88%|########8 | 44030/50000 [29:15:59<3:08:24,  1.89s/it]
Outer training loop:  88%|########8 | 44031/50000 [29:16:01<3:10:46,  1.92s/it]
Outer training loop:  88%|########8 | 44032/50000 [29:16:03<3:09:17,  1.90s/it]
Outer training loop:  88%|########8 | 44033/50000 [29:16:05<3:07:03,  1.88s/it]
Outer training loop:  88%|########8 | 44034/50000 [29:16:07<3:05:56,  1.87s/it]
Outer training loop:  88%|########8 | 44035/50000 [29:16:09<3:12:20,  1.93s/it]
Outer training loop:  88%|########8 | 44036/50000 [29:16:10<3:07:15,  1.88s/it]
Outer training loop:  88%|########8 | 44037/50000 [29:16:12<3:06:25,  1.88s/it]
Outer training loop:  88%|########8 | 44038/50000 [29:16:14<3:08:51,  1.90s/it]
Outer training loop:  88%|########8 | 44039/50000 [29:16:16<3:06:17,  1.88s/it]
Outer training loop:  88%|########8 | 44040/50000 [29:16:18<3:05:47,  1.87s/it]
Outer training loop:  88%|########8 | 44041/50000 [29:16:20<3:06:20,  1.88s/it]
Outer training loop:  88%|########8 | 44042/50000 [29:16:22<3:10:31,  1.92s/it]
Outer training loop:  88%|########8 | 44043/50000 [29:16:24<3:10:45,  1.92s/it]
Outer training loop:  88%|########8 | 44044/50000 [29:16:26<3:11:20,  1.93s/it]
Outer training loop:  88%|########8 | 44045/50000 [29:16:28<3:14:04,  1.96s/it]
Outer training loop:  88%|########8 | 44046/50000 [29:16:29<3:07:18,  1.89s/it]
Outer training loop:  88%|########8 | 44047/50000 [29:16:31<3:02:53,  1.84s/it]
Outer training loop:  88%|########8 | 44048/50000 [29:16:33<3:06:23,  1.88s/it]
Outer training loop:  88%|########8 | 44049/50000 [29:16:35<3:07:59,  1.90s/it]
Outer training loop:  88%|########8 | 44050/50000 [29:16:37<3:07:31,  1.89s/it]
Outer training loop:  88%|########8 | 44051/50000 [29:16:39<3:14:34,  1.96s/it]
Outer training loop:  88%|########8 | 44052/50000 [29:16:41<3:10:59,  1.93s/it]
Outer training loop:  88%|########8 | 44053/50000 [29:16:43<3:06:39,  1.88s/it]
Outer training loop:  88%|########8 | 44054/50000 [29:16:45<3:07:15,  1.89s/it]
Outer training loop:  88%|########8 | 44055/50000 [29:16:47<3:11:42,  1.93s/it]
Outer training loop:  88%|########8 | 44056/50000 [29:16:48<3:06:52,  1.89s/it]
Outer training loop:  88%|########8 | 44057/50000 [29:16:50<3:07:48,  1.90s/it]
Outer training loop:  88%|########8 | 44058/50000 [29:16:52<3:05:56,  1.88s/it]
Outer training loop:  88%|########8 | 44059/50000 [29:16:54<3:00:47,  1.83s/it]
Outer training loop:  88%|########8 | 44060/50000 [29:16:56<3:01:59,  1.84s/it]
Outer training loop:  88%|########8 | 44061/50000 [29:16:58<3:01:07,  1.83s/it]
Outer training loop:  88%|########8 | 44062/50000 [29:17:00<3:08:39,  1.91s/it]
Outer training loop:  88%|########8 | 44063/50000 [29:17:01<3:03:42,  1.86s/it]
Outer training loop:  88%|########8 | 44064/50000 [29:17:03<3:02:56,  1.85s/it]
Outer training loop:  88%|########8 | 44065/50000 [29:17:05<3:08:15,  1.90s/it]
Outer training loop:  88%|########8 | 44066/50000 [29:17:07<3:03:27,  1.85s/it]
Outer training loop:  88%|########8 | 44067/50000 [29:17:09<3:01:02,  1.83s/it]
Outer training loop:  88%|########8 | 44068/50000 [29:17:11<3:01:57,  1.84s/it]
Outer training loop:  88%|########8 | 44069/50000 [29:17:13<3:10:58,  1.93s/it]
Outer training loop:  88%|########8 | 44070/50000 [29:17:15<3:12:56,  1.95s/it]
Outer training loop:  88%|########8 | 44071/50000 [29:17:17<3:09:14,  1.92s/it]
Outer training loop:  88%|########8 | 44072/50000 [29:17:19<3:10:43,  1.93s/it]
Outer training loop:  88%|########8 | 44073/50000 [29:17:20<3:07:51,  1.90s/it]
Outer training loop:  88%|########8 | 44074/50000 [29:17:22<3:07:07,  1.89s/it]
Outer training loop:  88%|########8 | 44075/50000 [29:17:24<3:09:20,  1.92s/it]
Outer training loop:  88%|########8 | 44076/50000 [29:17:26<3:11:43,  1.94s/it]
Outer training loop:  88%|########8 | 44077/50000 [29:17:28<3:08:03,  1.90s/it]
Outer training loop:  88%|########8 | 44078/50000 [29:17:30<3:02:36,  1.85s/it]
Outer training loop:  88%|########8 | 44079/50000 [29:17:32<3:08:57,  1.91s/it]
Outer training loop:  88%|########8 | 44080/50000 [29:17:34<3:04:39,  1.87s/it]
Outer training loop:  88%|########8 | 44081/50000 [29:17:35<3:02:50,  1.85s/it]
Outer training loop:  88%|########8 | 44082/50000 [29:17:37<3:05:52,  1.88s/it]
Outer training loop:  88%|########8 | 44083/50000 [29:17:39<3:04:21,  1.87s/it]
Outer training loop:  88%|########8 | 44084/50000 [29:17:41<3:01:27,  1.84s/it]
Outer training loop:  88%|########8 | 44085/50000 [29:17:43<3:02:36,  1.85s/it]
Outer training loop:  88%|########8 | 44086/50000 [29:17:45<3:06:13,  1.89s/it]
Outer training loop:  88%|########8 | 44087/50000 [29:17:47<3:04:30,  1.87s/it]
Outer training loop:  88%|########8 | 44088/50000 [29:17:49<3:05:23,  1.88s/it]
Outer training loop:  88%|########8 | 44089/50000 [29:17:51<3:13:08,  1.96s/it]
Outer training loop:  88%|########8 | 44090/50000 [29:17:53<3:09:19,  1.92s/it]
Outer training loop:  88%|########8 | 44091/50000 [29:17:54<3:05:48,  1.89s/it]
Outer training loop:  88%|########8 | 44092/50000 [29:17:56<3:10:34,  1.94s/it]
Outer training loop:  88%|########8 | 44093/50000 [29:17:58<3:08:32,  1.92s/it]
Outer training loop:  88%|########8 | 44094/50000 [29:18:00<3:05:15,  1.88s/it]
Outer training loop:  88%|########8 | 44095/50000 [29:18:02<3:11:43,  1.95s/it]
Outer training loop:  88%|########8 | 44096/50000 [29:18:04<3:09:18,  1.92s/it]
Outer training loop:  88%|########8 | 44097/50000 [29:18:06<3:06:11,  1.89s/it]
Outer training loop:  88%|########8 | 44098/50000 [29:18:08<3:13:59,  1.97s/it]
Outer training loop:  88%|########8 | 44099/50000 [29:18:10<3:10:55,  1.94s/it]
Outer training loop:  88%|########8 | 44100/50000 [29:18:12<3:09:41,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.91s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  88%|########8 | 44101/50000 [29:18:19<5:50:44,  3.57s/it]
Outer training loop:  88%|########8 | 44102/50000 [29:18:21<5:03:38,  3.09s/it]
Outer training loop:  88%|########8 | 44103/50000 [29:18:23<4:25:45,  2.70s/it]
Outer training loop:  88%|########8 | 44104/50000 [29:18:25<4:00:14,  2.44s/it]
Outer training loop:  88%|########8 | 44105/50000 [29:18:27<3:40:24,  2.24s/it]
Outer training loop:  88%|########8 | 44106/50000 [29:18:29<3:34:13,  2.18s/it]
Outer training loop:  88%|########8 | 44107/50000 [29:18:30<3:21:13,  2.05s/it]
Outer training loop:  88%|########8 | 44108/50000 [29:18:32<3:14:19,  1.98s/it]
Outer training loop:  88%|########8 | 44109/50000 [29:18:34<3:10:42,  1.94s/it]
Outer training loop:  88%|########8 | 44110/50000 [29:18:36<3:12:08,  1.96s/it]
Outer training loop:  88%|########8 | 44111/50000 [29:18:38<3:04:23,  1.88s/it]
Outer training loop:  88%|########8 | 44112/50000 [29:18:40<3:01:40,  1.85s/it]
Outer training loop:  88%|########8 | 44113/50000 [29:18:42<3:08:08,  1.92s/it]
Outer training loop:  88%|########8 | 44114/50000 [29:18:44<3:08:06,  1.92s/it]
Outer training loop:  88%|########8 | 44115/50000 [29:18:45<3:01:06,  1.85s/it]
Outer training loop:  88%|########8 | 44116/50000 [29:18:47<3:05:43,  1.89s/it]
Outer training loop:  88%|########8 | 44117/50000 [29:18:49<3:10:56,  1.95s/it]
Outer training loop:  88%|########8 | 44118/50000 [29:18:51<3:08:49,  1.93s/it]
Outer training loop:  88%|########8 | 44119/50000 [29:18:53<3:08:13,  1.92s/it]
Outer training loop:  88%|########8 | 44120/50000 [29:18:55<3:07:26,  1.91s/it]
Outer training loop:  88%|########8 | 44121/50000 [29:18:57<3:02:35,  1.86s/it]
Outer training loop:  88%|########8 | 44122/50000 [29:18:59<3:03:07,  1.87s/it]
Outer training loop:  88%|########8 | 44123/50000 [29:19:01<3:07:35,  1.92s/it]
Outer training loop:  88%|########8 | 44124/50000 [29:19:02<3:05:14,  1.89s/it]
Outer training loop:  88%|########8 | 44125/50000 [29:19:04<3:02:59,  1.87s/it]
Outer training loop:  88%|########8 | 44126/50000 [29:19:06<3:11:04,  1.95s/it]
Outer training loop:  88%|########8 | 44127/50000 [29:19:08<3:07:08,  1.91s/it]
Outer training loop:  88%|########8 | 44128/50000 [29:19:10<3:07:26,  1.92s/it]
Outer training loop:  88%|########8 | 44129/50000 [29:19:12<3:00:03,  1.84s/it]
Outer training loop:  88%|########8 | 44130/50000 [29:19:14<3:10:05,  1.94s/it]
Outer training loop:  88%|########8 | 44131/50000 [29:19:16<3:07:43,  1.92s/it]
Outer training loop:  88%|########8 | 44132/50000 [29:19:18<3:04:07,  1.88s/it]
Outer training loop:  88%|########8 | 44133/50000 [29:19:20<3:09:23,  1.94s/it]
Outer training loop:  88%|########8 | 44134/50000 [29:19:22<3:08:40,  1.93s/it]
Outer training loop:  88%|########8 | 44135/50000 [29:19:24<3:06:50,  1.91s/it]
Outer training loop:  88%|########8 | 44136/50000 [29:19:26<3:15:27,  2.00s/it]
Outer training loop:  88%|########8 | 44137/50000 [29:19:28<3:10:31,  1.95s/it]
Outer training loop:  88%|########8 | 44138/50000 [29:19:29<3:08:13,  1.93s/it]
Outer training loop:  88%|########8 | 44139/50000 [29:19:31<3:05:52,  1.90s/it]
Outer training loop:  88%|########8 | 44140/50000 [29:19:33<3:10:43,  1.95s/it]
Outer training loop:  88%|########8 | 44141/50000 [29:19:35<3:10:35,  1.95s/it]
Outer training loop:  88%|########8 | 44142/50000 [29:19:37<3:06:39,  1.91s/it]
Outer training loop:  88%|########8 | 44143/50000 [29:19:39<3:08:46,  1.93s/it]
Outer training loop:  88%|########8 | 44144/50000 [29:19:41<3:02:50,  1.87s/it]
Outer training loop:  88%|########8 | 44145/50000 [29:19:43<2:59:50,  1.84s/it]
Outer training loop:  88%|########8 | 44146/50000 [29:19:44<2:57:11,  1.82s/it]
Outer training loop:  88%|########8 | 44147/50000 [29:19:46<3:05:40,  1.90s/it]
Outer training loop:  88%|########8 | 44148/50000 [29:19:48<3:02:41,  1.87s/it]
Outer training loop:  88%|########8 | 44149/50000 [29:19:50<3:03:24,  1.88s/it]
Outer training loop:  88%|########8 | 44150/50000 [29:19:52<3:05:28,  1.90s/it]
Outer training loop:  88%|########8 | 44151/50000 [29:19:54<3:01:53,  1.87s/it]
Outer training loop:  88%|########8 | 44152/50000 [29:19:56<3:02:27,  1.87s/it]
Outer training loop:  88%|########8 | 44153/50000 [29:19:58<3:04:56,  1.90s/it]
Outer training loop:  88%|########8 | 44154/50000 [29:20:00<3:05:24,  1.90s/it]
Outer training loop:  88%|########8 | 44155/50000 [29:20:02<3:06:24,  1.91s/it]
Outer training loop:  88%|########8 | 44156/50000 [29:20:03<3:02:05,  1.87s/it]
Outer training loop:  88%|########8 | 44157/50000 [29:20:05<3:05:36,  1.91s/it]
Outer training loop:  88%|########8 | 44158/50000 [29:20:07<3:05:42,  1.91s/it]
Outer training loop:  88%|########8 | 44159/50000 [29:20:09<3:07:28,  1.93s/it]
Outer training loop:  88%|########8 | 44160/50000 [29:20:11<3:03:07,  1.88s/it]
Outer training loop:  88%|########8 | 44161/50000 [29:20:13<3:00:57,  1.86s/it]
Outer training loop:  88%|########8 | 44162/50000 [29:20:15<2:58:21,  1.83s/it]
Outer training loop:  88%|########8 | 44163/50000 [29:20:17<3:04:25,  1.90s/it]
Outer training loop:  88%|########8 | 44164/50000 [29:20:18<3:00:32,  1.86s/it]
Outer training loop:  88%|########8 | 44165/50000 [29:20:20<3:00:15,  1.85s/it]
Outer training loop:  88%|########8 | 44166/50000 [29:20:22<3:05:14,  1.91s/it]
Outer training loop:  88%|########8 | 44167/50000 [29:20:24<3:09:38,  1.95s/it]
Outer training loop:  88%|########8 | 44168/50000 [29:20:26<3:07:29,  1.93s/it]
Outer training loop:  88%|########8 | 44169/50000 [29:20:28<3:08:38,  1.94s/it]
Outer training loop:  88%|########8 | 44170/50000 [29:20:30<3:13:24,  1.99s/it]
Outer training loop:  88%|########8 | 44171/50000 [29:20:32<3:10:12,  1.96s/it]
Outer training loop:  88%|########8 | 44172/50000 [29:20:34<3:05:30,  1.91s/it]
Outer training loop:  88%|########8 | 44173/50000 [29:20:36<3:08:17,  1.94s/it]
Outer training loop:  88%|########8 | 44174/50000 [29:20:38<3:03:24,  1.89s/it]
Outer training loop:  88%|########8 | 44175/50000 [29:20:40<3:00:47,  1.86s/it]
Outer training loop:  88%|########8 | 44176/50000 [29:20:42<3:06:21,  1.92s/it]
Outer training loop:  88%|########8 | 44177/50000 [29:20:43<3:02:44,  1.88s/it]
Outer training loop:  88%|########8 | 44178/50000 [29:20:45<2:58:39,  1.84s/it]
Outer training loop:  88%|########8 | 44179/50000 [29:20:47<3:00:17,  1.86s/it]
Outer training loop:  88%|########8 | 44180/50000 [29:20:49<3:05:37,  1.91s/it]
Outer training loop:  88%|########8 | 44181/50000 [29:20:51<3:06:11,  1.92s/it]
Outer training loop:  88%|########8 | 44182/50000 [29:20:53<3:05:25,  1.91s/it]
Outer training loop:  88%|########8 | 44183/50000 [29:20:55<3:05:03,  1.91s/it]
Outer training loop:  88%|########8 | 44184/50000 [29:20:57<2:59:37,  1.85s/it]
Outer training loop:  88%|########8 | 44185/50000 [29:20:58<3:02:01,  1.88s/it]
Outer training loop:  88%|########8 | 44186/50000 [29:21:01<3:06:31,  1.92s/it]
Outer training loop:  88%|########8 | 44187/50000 [29:21:02<2:59:57,  1.86s/it]
Outer training loop:  88%|########8 | 44188/50000 [29:21:04<2:58:34,  1.84s/it]
Outer training loop:  88%|########8 | 44189/50000 [29:21:06<2:57:07,  1.83s/it]
Outer training loop:  88%|########8 | 44190/50000 [29:21:08<2:58:23,  1.84s/it]
Outer training loop:  88%|########8 | 44191/50000 [29:21:10<2:58:59,  1.85s/it]
Outer training loop:  88%|########8 | 44192/50000 [29:21:11<2:55:23,  1.81s/it]
Outer training loop:  88%|########8 | 44193/50000 [29:21:13<2:58:32,  1.84s/it]
Outer training loop:  88%|########8 | 44194/50000 [29:21:15<2:57:12,  1.83s/it]
Outer training loop:  88%|########8 | 44195/50000 [29:21:17<3:00:14,  1.86s/it]
Outer training loop:  88%|########8 | 44196/50000 [29:21:19<3:06:11,  1.92s/it]
Outer training loop:  88%|########8 | 44197/50000 [29:21:21<3:04:42,  1.91s/it]
Outer training loop:  88%|########8 | 44198/50000 [29:21:23<3:03:53,  1.90s/it]
Outer training loop:  88%|########8 | 44199/50000 [29:21:25<3:05:56,  1.92s/it]
Outer training loop:  88%|########8 | 44200/50000 [29:21:27<3:03:32,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.93s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  88%|########8 | 44201/50000 [29:21:34<5:42:30,  3.54s/it]
Outer training loop:  88%|########8 | 44202/50000 [29:21:36<4:52:22,  3.03s/it]
Outer training loop:  88%|########8 | 44203/50000 [29:21:38<4:20:31,  2.70s/it]
Outer training loop:  88%|########8 | 44204/50000 [29:21:39<3:53:14,  2.41s/it]
Outer training loop:  88%|########8 | 44205/50000 [29:21:41<3:36:02,  2.24s/it]
Outer training loop:  88%|########8 | 44206/50000 [29:21:43<3:27:50,  2.15s/it]
Outer training loop:  88%|########8 | 44207/50000 [29:21:45<3:15:49,  2.03s/it]
Outer training loop:  88%|########8 | 44208/50000 [29:21:47<3:06:24,  1.93s/it]
Outer training loop:  88%|########8 | 44209/50000 [29:21:49<3:03:59,  1.91s/it]
Outer training loop:  88%|########8 | 44210/50000 [29:21:51<3:08:18,  1.95s/it]
Outer training loop:  88%|########8 | 44211/50000 [29:21:52<3:04:25,  1.91s/it]
Outer training loop:  88%|########8 | 44212/50000 [29:21:54<2:59:54,  1.86s/it]
Outer training loop:  88%|########8 | 44213/50000 [29:21:56<3:02:54,  1.90s/it]
Outer training loop:  88%|########8 | 44214/50000 [29:21:58<3:04:22,  1.91s/it]
Outer training loop:  88%|########8 | 44215/50000 [29:22:00<3:00:18,  1.87s/it]
Outer training loop:  88%|########8 | 44216/50000 [29:22:02<2:58:17,  1.85s/it]
Outer training loop:  88%|########8 | 44217/50000 [29:22:04<3:01:45,  1.89s/it]
Outer training loop:  88%|########8 | 44218/50000 [29:22:06<3:03:24,  1.90s/it]
Outer training loop:  88%|########8 | 44219/50000 [29:22:07<3:02:53,  1.90s/it]
Outer training loop:  88%|########8 | 44220/50000 [29:22:09<3:07:13,  1.94s/it]
Outer training loop:  88%|########8 | 44221/50000 [29:22:11<3:05:12,  1.92s/it]
Outer training loop:  88%|########8 | 44222/50000 [29:22:13<3:05:08,  1.92s/it]
Outer training loop:  88%|########8 | 44223/50000 [29:22:15<3:09:11,  1.97s/it]
Outer training loop:  88%|########8 | 44224/50000 [29:22:17<3:04:54,  1.92s/it]
Outer training loop:  88%|########8 | 44225/50000 [29:22:19<3:03:38,  1.91s/it]
Outer training loop:  88%|########8 | 44226/50000 [29:22:21<3:01:52,  1.89s/it]
Outer training loop:  88%|########8 | 44227/50000 [29:22:23<3:05:04,  1.92s/it]
Outer training loop:  88%|########8 | 44228/50000 [29:22:25<2:59:34,  1.87s/it]
Outer training loop:  88%|########8 | 44229/50000 [29:22:26<2:58:11,  1.85s/it]
Outer training loop:  88%|########8 | 44230/50000 [29:22:28<3:02:07,  1.89s/it]
Outer training loop:  88%|########8 | 44231/50000 [29:22:30<3:00:10,  1.87s/it]
Outer training loop:  88%|########8 | 44232/50000 [29:22:32<2:58:20,  1.86s/it]
Outer training loop:  88%|########8 | 44233/50000 [29:22:34<2:57:02,  1.84s/it]
Outer training loop:  88%|########8 | 44234/50000 [29:22:36<3:02:40,  1.90s/it]
Outer training loop:  88%|########8 | 44235/50000 [29:22:38<2:59:25,  1.87s/it]
Outer training loop:  88%|########8 | 44236/50000 [29:22:39<2:56:32,  1.84s/it]
Outer training loop:  88%|########8 | 44237/50000 [29:22:41<3:00:53,  1.88s/it]
Outer training loop:  88%|########8 | 44238/50000 [29:22:43<2:57:42,  1.85s/it]
Outer training loop:  88%|########8 | 44239/50000 [29:22:45<2:55:07,  1.82s/it]
Outer training loop:  88%|########8 | 44240/50000 [29:22:47<2:53:45,  1.81s/it]
Outer training loop:  88%|########8 | 44241/50000 [29:22:49<3:00:42,  1.88s/it]
Outer training loop:  88%|########8 | 44242/50000 [29:22:51<2:57:53,  1.85s/it]
Outer training loop:  88%|########8 | 44243/50000 [29:22:52<2:55:14,  1.83s/it]
Outer training loop:  88%|########8 | 44244/50000 [29:22:54<2:52:21,  1.80s/it]
Outer training loop:  88%|########8 | 44245/50000 [29:22:56<2:55:32,  1.83s/it]
Outer training loop:  88%|########8 | 44246/50000 [29:22:58<2:53:54,  1.81s/it]
Outer training loop:  88%|########8 | 44247/50000 [29:23:00<2:53:44,  1.81s/it]
Outer training loop:  88%|########8 | 44248/50000 [29:23:02<3:01:38,  1.89s/it]
Outer training loop:  88%|########8 | 44249/50000 [29:23:03<2:56:13,  1.84s/it]
Outer training loop:  88%|########8 | 44250/50000 [29:23:05<2:56:40,  1.84s/it]
Outer training loop:  89%|########8 | 44251/50000 [29:23:07<2:51:15,  1.79s/it]
Outer training loop:  89%|########8 | 44252/50000 [29:23:09<2:58:19,  1.86s/it]
Outer training loop:  89%|########8 | 44253/50000 [29:23:11<2:56:58,  1.85s/it]
Outer training loop:  89%|########8 | 44254/50000 [29:23:13<2:56:34,  1.84s/it]
Outer training loop:  89%|########8 | 44255/50000 [29:23:15<3:02:02,  1.90s/it]
Outer training loop:  89%|########8 | 44256/50000 [29:23:17<3:01:49,  1.90s/it]
Outer training loop:  89%|########8 | 44257/50000 [29:23:18<2:56:26,  1.84s/it]
Outer training loop:  89%|########8 | 44258/50000 [29:23:20<2:56:37,  1.85s/it]
Outer training loop:  89%|########8 | 44259/50000 [29:23:22<2:58:28,  1.87s/it]
Outer training loop:  89%|########8 | 44260/50000 [29:23:24<2:59:26,  1.88s/it]
Outer training loop:  89%|########8 | 44261/50000 [29:23:26<3:00:54,  1.89s/it]
Outer training loop:  89%|########8 | 44262/50000 [29:23:28<3:05:43,  1.94s/it]
Outer training loop:  89%|########8 | 44263/50000 [29:23:30<2:58:50,  1.87s/it]
Outer training loop:  89%|########8 | 44264/50000 [29:23:32<3:02:26,  1.91s/it]
Outer training loop:  89%|########8 | 44265/50000 [29:23:34<3:05:52,  1.94s/it]
Outer training loop:  89%|########8 | 44266/50000 [29:23:36<3:07:12,  1.96s/it]
Outer training loop:  89%|########8 | 44267/50000 [29:23:38<3:05:14,  1.94s/it]
Outer training loop:  89%|########8 | 44268/50000 [29:23:39<3:06:19,  1.95s/it]
Outer training loop:  89%|########8 | 44269/50000 [29:23:41<3:06:30,  1.95s/it]
Outer training loop:  89%|########8 | 44270/50000 [29:23:43<3:05:09,  1.94s/it]
Outer training loop:  89%|########8 | 44271/50000 [29:23:45<3:06:10,  1.95s/it]
Outer training loop:  89%|########8 | 44272/50000 [29:23:47<2:57:51,  1.86s/it]
Outer training loop:  89%|########8 | 44273/50000 [29:23:49<2:57:38,  1.86s/it]
Outer training loop:  89%|########8 | 44274/50000 [29:23:51<3:03:22,  1.92s/it]
Outer training loop:  89%|########8 | 44275/50000 [29:23:53<3:00:05,  1.89s/it]
Outer training loop:  89%|########8 | 44276/50000 [29:23:55<3:01:15,  1.90s/it]
Outer training loop:  89%|########8 | 44277/50000 [29:23:57<3:01:26,  1.90s/it]
Outer training loop:  89%|########8 | 44278/50000 [29:23:59<3:08:20,  1.97s/it]
Outer training loop:  89%|########8 | 44279/50000 [29:24:00<3:02:07,  1.91s/it]
Outer training loop:  89%|########8 | 44280/50000 [29:24:02<2:57:53,  1.87s/it]
Outer training loop:  89%|########8 | 44281/50000 [29:24:04<3:03:58,  1.93s/it]
Outer training loop:  89%|########8 | 44282/50000 [29:24:06<3:00:29,  1.89s/it]
Outer training loop:  89%|########8 | 44283/50000 [29:24:08<2:59:06,  1.88s/it]
Outer training loop:  89%|########8 | 44284/50000 [29:24:10<2:56:51,  1.86s/it]
Outer training loop:  89%|########8 | 44285/50000 [29:24:12<3:02:58,  1.92s/it]
Outer training loop:  89%|########8 | 44286/50000 [29:24:14<3:04:01,  1.93s/it]
Outer training loop:  89%|########8 | 44287/50000 [29:24:16<2:59:59,  1.89s/it]
Outer training loop:  89%|########8 | 44288/50000 [29:24:18<3:02:29,  1.92s/it]
Outer training loop:  89%|########8 | 44289/50000 [29:24:19<2:57:47,  1.87s/it]
Outer training loop:  89%|########8 | 44290/50000 [29:24:21<3:00:42,  1.90s/it]
Outer training loop:  89%|########8 | 44291/50000 [29:24:23<2:58:12,  1.87s/it]
Outer training loop:  89%|########8 | 44292/50000 [29:24:25<3:03:00,  1.92s/it]
Outer training loop:  89%|########8 | 44293/50000 [29:24:27<3:01:46,  1.91s/it]
Outer training loop:  89%|########8 | 44294/50000 [29:24:29<3:01:24,  1.91s/it]
Outer training loop:  89%|########8 | 44295/50000 [29:24:31<3:03:09,  1.93s/it]
Outer training loop:  89%|########8 | 44296/50000 [29:24:33<2:59:11,  1.88s/it]
Outer training loop:  89%|########8 | 44297/50000 [29:24:34<2:56:36,  1.86s/it]
Outer training loop:  89%|########8 | 44298/50000 [29:24:37<3:06:54,  1.97s/it]
Outer training loop:  89%|########8 | 44299/50000 [29:24:38<2:59:13,  1.89s/it]
Outer training loop:  89%|########8 | 44300/50000 [29:24:40<3:02:02,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  89%|########8 | 44301/50000 [29:24:48<5:42:58,  3.61s/it]
Outer training loop:  89%|########8 | 44302/50000 [29:24:50<5:04:12,  3.20s/it]
Outer training loop:  89%|########8 | 44303/50000 [29:24:52<4:27:21,  2.82s/it]
Outer training loop:  89%|########8 | 44304/50000 [29:24:54<4:01:28,  2.54s/it]
Outer training loop:  89%|########8 | 44305/50000 [29:24:56<3:39:58,  2.32s/it]
Outer training loop:  89%|########8 | 44306/50000 [29:24:58<3:32:17,  2.24s/it]
Outer training loop:  89%|########8 | 44307/50000 [29:25:00<3:19:56,  2.11s/it]
Outer training loop:  89%|########8 | 44308/50000 [29:25:01<3:09:49,  2.00s/it]
Outer training loop:  89%|########8 | 44309/50000 [29:25:04<3:13:02,  2.04s/it]
Outer training loop:  89%|########8 | 44310/50000 [29:25:05<3:08:13,  1.98s/it]
Outer training loop:  89%|########8 | 44311/50000 [29:25:07<3:09:30,  2.00s/it]
Outer training loop:  89%|########8 | 44312/50000 [29:25:09<3:08:02,  1.98s/it]
Outer training loop:  89%|########8 | 44313/50000 [29:25:11<3:09:14,  2.00s/it]
Outer training loop:  89%|########8 | 44314/50000 [29:25:13<3:03:54,  1.94s/it]
Outer training loop:  89%|########8 | 44315/50000 [29:25:15<3:08:01,  1.98s/it]
Outer training loop:  89%|########8 | 44316/50000 [29:25:17<3:07:15,  1.98s/it]
Outer training loop:  89%|########8 | 44317/50000 [29:25:19<3:07:15,  1.98s/it]
Outer training loop:  89%|########8 | 44318/50000 [29:25:21<3:06:54,  1.97s/it]
Outer training loop:  89%|########8 | 44319/50000 [29:25:23<3:10:24,  2.01s/it]
Outer training loop:  89%|########8 | 44320/50000 [29:25:25<3:05:08,  1.96s/it]
Outer training loop:  89%|########8 | 44321/50000 [29:25:27<3:00:37,  1.91s/it]
Outer training loop:  89%|########8 | 44322/50000 [29:25:29<2:57:44,  1.88s/it]
Outer training loop:  89%|########8 | 44323/50000 [29:25:31<2:58:44,  1.89s/it]
Outer training loop:  89%|########8 | 44324/50000 [29:25:32<2:57:19,  1.87s/it]
Outer training loop:  89%|########8 | 44325/50000 [29:25:34<2:54:51,  1.85s/it]
Outer training loop:  89%|########8 | 44326/50000 [29:25:36<2:55:34,  1.86s/it]
Outer training loop:  89%|########8 | 44327/50000 [29:25:38<2:59:53,  1.90s/it]
Outer training loop:  89%|########8 | 44328/50000 [29:25:40<2:58:40,  1.89s/it]
Outer training loop:  89%|########8 | 44329/50000 [29:25:42<2:57:02,  1.87s/it]
Outer training loop:  89%|########8 | 44330/50000 [29:25:44<3:02:30,  1.93s/it]
Outer training loop:  89%|########8 | 44331/50000 [29:25:46<2:56:37,  1.87s/it]
Outer training loop:  89%|########8 | 44332/50000 [29:25:47<2:55:15,  1.86s/it]
Outer training loop:  89%|########8 | 44333/50000 [29:25:49<2:55:37,  1.86s/it]
Outer training loop:  89%|########8 | 44334/50000 [29:25:51<2:59:24,  1.90s/it]
Outer training loop:  89%|########8 | 44335/50000 [29:25:53<2:57:09,  1.88s/it]
Outer training loop:  89%|########8 | 44336/50000 [29:25:55<2:57:23,  1.88s/it]
Outer training loop:  89%|########8 | 44337/50000 [29:25:57<3:00:51,  1.92s/it]
Outer training loop:  89%|########8 | 44338/50000 [29:25:59<3:00:27,  1.91s/it]
Outer training loop:  89%|########8 | 44339/50000 [29:26:01<2:56:19,  1.87s/it]
Outer training loop:  89%|########8 | 44340/50000 [29:26:03<2:59:19,  1.90s/it]
Outer training loop:  89%|########8 | 44341/50000 [29:26:05<3:02:54,  1.94s/it]
Outer training loop:  89%|########8 | 44342/50000 [29:26:07<3:02:17,  1.93s/it]
Outer training loop:  89%|########8 | 44343/50000 [29:26:09<3:02:48,  1.94s/it]
Outer training loop:  89%|########8 | 44344/50000 [29:26:10<2:57:59,  1.89s/it]
Outer training loop:  89%|########8 | 44345/50000 [29:26:12<2:54:08,  1.85s/it]
Outer training loop:  89%|########8 | 44346/50000 [29:26:14<2:58:28,  1.89s/it]
Outer training loop:  89%|########8 | 44347/50000 [29:26:16<2:57:23,  1.88s/it]
Outer training loop:  89%|########8 | 44348/50000 [29:26:18<2:56:25,  1.87s/it]
Outer training loop:  89%|########8 | 44349/50000 [29:26:20<2:54:31,  1.85s/it]
Outer training loop:  89%|########8 | 44350/50000 [29:26:22<2:59:28,  1.91s/it]
Outer training loop:  89%|########8 | 44351/50000 [29:26:23<2:57:09,  1.88s/it]
Outer training loop:  89%|########8 | 44352/50000 [29:26:25<2:57:39,  1.89s/it]
Outer training loop:  89%|########8 | 44353/50000 [29:26:28<3:05:23,  1.97s/it]
Outer training loop:  89%|########8 | 44354/50000 [29:26:29<3:01:43,  1.93s/it]
Outer training loop:  89%|########8 | 44355/50000 [29:26:31<2:58:36,  1.90s/it]
Outer training loop:  89%|########8 | 44356/50000 [29:26:33<2:56:23,  1.88s/it]
Outer training loop:  89%|########8 | 44357/50000 [29:26:35<3:00:36,  1.92s/it]
Outer training loop:  89%|########8 | 44358/50000 [29:26:37<2:55:48,  1.87s/it]
Outer training loop:  89%|########8 | 44359/50000 [29:26:39<2:57:47,  1.89s/it]
Outer training loop:  89%|########8 | 44360/50000 [29:26:41<2:59:35,  1.91s/it]
Outer training loop:  89%|########8 | 44361/50000 [29:26:43<3:03:57,  1.96s/it]
Outer training loop:  89%|########8 | 44362/50000 [29:26:45<3:01:39,  1.93s/it]
Outer training loop:  89%|########8 | 44363/50000 [29:26:47<3:04:53,  1.97s/it]
Outer training loop:  89%|########8 | 44364/50000 [29:26:49<3:02:42,  1.95s/it]
Outer training loop:  89%|########8 | 44365/50000 [29:26:50<2:59:37,  1.91s/it]
Outer training loop:  89%|########8 | 44366/50000 [29:26:52<2:53:41,  1.85s/it]
Outer training loop:  89%|########8 | 44367/50000 [29:26:54<3:01:33,  1.93s/it]
Outer training loop:  89%|########8 | 44368/50000 [29:26:56<2:58:13,  1.90s/it]
Outer training loop:  89%|########8 | 44369/50000 [29:26:58<2:59:54,  1.92s/it]
Outer training loop:  89%|########8 | 44370/50000 [29:27:00<3:01:10,  1.93s/it]
Outer training loop:  89%|########8 | 44371/50000 [29:27:02<3:00:34,  1.92s/it]
Outer training loop:  89%|########8 | 44372/50000 [29:27:04<2:58:24,  1.90s/it]
Outer training loop:  89%|########8 | 44373/50000 [29:27:06<3:01:47,  1.94s/it]
Outer training loop:  89%|########8 | 44374/50000 [29:27:08<3:06:35,  1.99s/it]
Outer training loop:  89%|########8 | 44375/50000 [29:27:10<3:01:22,  1.93s/it]
Outer training loop:  89%|########8 | 44376/50000 [29:27:12<2:59:18,  1.91s/it]
Outer training loop:  89%|########8 | 44377/50000 [29:27:14<3:02:45,  1.95s/it]
Outer training loop:  89%|########8 | 44378/50000 [29:27:15<3:00:23,  1.93s/it]
Outer training loop:  89%|########8 | 44379/50000 [29:27:17<2:58:08,  1.90s/it]
Outer training loop:  89%|########8 | 44380/50000 [29:27:19<2:59:54,  1.92s/it]
Outer training loop:  89%|########8 | 44381/50000 [29:27:21<2:55:32,  1.87s/it]
Outer training loop:  89%|########8 | 44382/50000 [29:27:23<2:53:00,  1.85s/it]
Outer training loop:  89%|########8 | 44383/50000 [29:27:25<2:55:58,  1.88s/it]
Outer training loop:  89%|########8 | 44384/50000 [29:27:27<2:55:54,  1.88s/it]
Outer training loop:  89%|########8 | 44385/50000 [29:27:29<2:55:01,  1.87s/it]
Outer training loop:  89%|########8 | 44386/50000 [29:27:31<2:58:41,  1.91s/it]
Outer training loop:  89%|########8 | 44387/50000 [29:27:32<2:54:22,  1.86s/it]
Outer training loop:  89%|########8 | 44388/50000 [29:27:34<2:54:30,  1.87s/it]
Outer training loop:  89%|########8 | 44389/50000 [29:27:36<2:52:28,  1.84s/it]
Outer training loop:  89%|########8 | 44390/50000 [29:27:38<2:58:57,  1.91s/it]
Outer training loop:  89%|########8 | 44391/50000 [29:27:40<2:53:18,  1.85s/it]
Outer training loop:  89%|########8 | 44392/50000 [29:27:42<2:56:16,  1.89s/it]
Outer training loop:  89%|########8 | 44393/50000 [29:27:44<3:01:39,  1.94s/it]
Outer training loop:  89%|########8 | 44394/50000 [29:27:46<2:58:38,  1.91s/it]
Outer training loop:  89%|########8 | 44395/50000 [29:27:47<2:56:34,  1.89s/it]
Outer training loop:  89%|########8 | 44396/50000 [29:27:49<3:00:22,  1.93s/it]
Outer training loop:  89%|########8 | 44397/50000 [29:27:51<2:59:07,  1.92s/it]
Outer training loop:  89%|########8 | 44398/50000 [29:27:53<2:57:44,  1.90s/it]
Outer training loop:  89%|########8 | 44399/50000 [29:27:55<2:54:40,  1.87s/it]
Outer training loop:  89%|########8 | 44400/50000 [29:27:57<2:56:37,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  89%|########8 | 44401/50000 [29:28:04<5:27:34,  3.51s/it]
Outer training loop:  89%|########8 | 44402/50000 [29:28:06<4:43:03,  3.03s/it]
Outer training loop:  89%|########8 | 44403/50000 [29:28:08<4:07:57,  2.66s/it]
Outer training loop:  89%|########8 | 44404/50000 [29:28:10<3:49:45,  2.46s/it]
Outer training loop:  89%|########8 | 44405/50000 [29:28:12<3:33:24,  2.29s/it]
Outer training loop:  89%|########8 | 44406/50000 [29:28:14<3:18:32,  2.13s/it]
Outer training loop:  89%|########8 | 44407/50000 [29:28:16<3:16:39,  2.11s/it]
Outer training loop:  89%|########8 | 44408/50000 [29:28:18<3:11:46,  2.06s/it]
Outer training loop:  89%|########8 | 44409/50000 [29:28:19<3:05:00,  1.99s/it]
Outer training loop:  89%|########8 | 44410/50000 [29:28:21<3:01:37,  1.95s/it]
Outer training loop:  89%|########8 | 44411/50000 [29:28:23<3:04:02,  1.98s/it]
Outer training loop:  89%|########8 | 44412/50000 [29:28:25<3:02:47,  1.96s/it]
Outer training loop:  89%|########8 | 44413/50000 [29:28:27<2:58:49,  1.92s/it]
Outer training loop:  89%|########8 | 44414/50000 [29:28:29<3:02:16,  1.96s/it]
Outer training loop:  89%|########8 | 44415/50000 [29:28:31<3:01:39,  1.95s/it]
Outer training loop:  89%|########8 | 44416/50000 [29:28:33<2:55:43,  1.89s/it]
Outer training loop:  89%|########8 | 44417/50000 [29:28:35<2:53:22,  1.86s/it]
Outer training loop:  89%|########8 | 44418/50000 [29:28:37<2:56:24,  1.90s/it]
Outer training loop:  89%|########8 | 44419/50000 [29:28:38<2:55:46,  1.89s/it]
Outer training loop:  89%|########8 | 44420/50000 [29:28:40<2:56:48,  1.90s/it]
Outer training loop:  89%|########8 | 44421/50000 [29:28:43<3:05:12,  1.99s/it]
Outer training loop:  89%|########8 | 44422/50000 [29:28:44<3:00:55,  1.95s/it]
Outer training loop:  89%|########8 | 44423/50000 [29:28:46<2:57:32,  1.91s/it]
Outer training loop:  89%|########8 | 44424/50000 [29:28:48<2:59:55,  1.94s/it]
Outer training loop:  89%|########8 | 44425/50000 [29:28:50<3:00:51,  1.95s/it]
Outer training loop:  89%|########8 | 44426/50000 [29:28:52<2:57:06,  1.91s/it]
Outer training loop:  89%|########8 | 44427/50000 [29:28:54<2:54:07,  1.87s/it]
Outer training loop:  89%|########8 | 44428/50000 [29:28:56<2:58:01,  1.92s/it]
Outer training loop:  89%|########8 | 44429/50000 [29:28:58<2:59:33,  1.93s/it]
Outer training loop:  89%|########8 | 44430/50000 [29:29:00<2:55:35,  1.89s/it]
Outer training loop:  89%|########8 | 44431/50000 [29:29:02<2:56:23,  1.90s/it]
Outer training loop:  89%|########8 | 44432/50000 [29:29:03<2:54:25,  1.88s/it]
Outer training loop:  89%|########8 | 44433/50000 [29:29:05<2:54:24,  1.88s/it]
Outer training loop:  89%|########8 | 44434/50000 [29:29:07<2:55:05,  1.89s/it]
Outer training loop:  89%|########8 | 44435/50000 [29:29:09<2:56:11,  1.90s/it]
Outer training loop:  89%|########8 | 44436/50000 [29:29:11<2:55:32,  1.89s/it]
Outer training loop:  89%|########8 | 44437/50000 [29:29:13<2:58:00,  1.92s/it]
Outer training loop:  89%|########8 | 44438/50000 [29:29:15<2:56:36,  1.91s/it]
Outer training loop:  89%|########8 | 44439/50000 [29:29:17<2:53:45,  1.87s/it]
Outer training loop:  89%|########8 | 44440/50000 [29:29:18<2:53:08,  1.87s/it]
Outer training loop:  89%|########8 | 44441/50000 [29:29:21<3:00:41,  1.95s/it]
Outer training loop:  89%|########8 | 44442/50000 [29:29:22<2:58:18,  1.92s/it]
Outer training loop:  89%|########8 | 44443/50000 [29:29:24<2:54:38,  1.89s/it]
Outer training loop:  89%|########8 | 44444/50000 [29:29:26<2:55:44,  1.90s/it]
Outer training loop:  89%|########8 | 44445/50000 [29:29:28<2:58:59,  1.93s/it]
Outer training loop:  89%|########8 | 44446/50000 [29:29:30<2:54:43,  1.89s/it]
Outer training loop:  89%|########8 | 44447/50000 [29:29:32<2:54:36,  1.89s/it]
Outer training loop:  89%|########8 | 44448/50000 [29:29:34<2:56:16,  1.91s/it]
Outer training loop:  89%|########8 | 44449/50000 [29:29:36<2:56:19,  1.91s/it]
Outer training loop:  89%|########8 | 44450/50000 [29:29:38<2:53:57,  1.88s/it]
Outer training loop:  89%|########8 | 44451/50000 [29:29:40<2:57:21,  1.92s/it]
Outer training loop:  89%|########8 | 44452/50000 [29:29:41<2:53:53,  1.88s/it]
Outer training loop:  89%|########8 | 44453/50000 [29:29:43<2:54:39,  1.89s/it]
Outer training loop:  89%|########8 | 44454/50000 [29:29:45<2:51:18,  1.85s/it]
Outer training loop:  89%|########8 | 44455/50000 [29:29:47<2:52:40,  1.87s/it]
Outer training loop:  89%|########8 | 44456/50000 [29:29:49<2:50:21,  1.84s/it]
Outer training loop:  89%|########8 | 44457/50000 [29:29:50<2:48:39,  1.83s/it]
Outer training loop:  89%|########8 | 44458/50000 [29:29:53<2:58:59,  1.94s/it]
Outer training loop:  89%|########8 | 44459/50000 [29:29:54<2:55:14,  1.90s/it]
Outer training loop:  89%|########8 | 44460/50000 [29:29:56<2:51:57,  1.86s/it]
Outer training loop:  89%|########8 | 44461/50000 [29:29:58<3:00:21,  1.95s/it]
Outer training loop:  89%|########8 | 44462/50000 [29:30:00<2:57:31,  1.92s/it]
Outer training loop:  89%|########8 | 44463/50000 [29:30:02<2:51:38,  1.86s/it]
Outer training loop:  89%|########8 | 44464/50000 [29:30:04<2:51:38,  1.86s/it]
Outer training loop:  89%|########8 | 44465/50000 [29:30:06<2:57:02,  1.92s/it]
Outer training loop:  89%|########8 | 44466/50000 [29:30:08<2:50:49,  1.85s/it]
Outer training loop:  89%|########8 | 44467/50000 [29:30:09<2:51:11,  1.86s/it]
Outer training loop:  89%|########8 | 44468/50000 [29:30:12<2:55:32,  1.90s/it]
Outer training loop:  89%|########8 | 44469/50000 [29:30:13<2:54:37,  1.89s/it]
Outer training loop:  89%|########8 | 44470/50000 [29:30:15<2:50:56,  1.85s/it]
Outer training loop:  89%|########8 | 44471/50000 [29:30:17<2:52:56,  1.88s/it]
Outer training loop:  89%|########8 | 44472/50000 [29:30:19<2:55:42,  1.91s/it]
Outer training loop:  89%|########8 | 44473/50000 [29:30:21<2:54:23,  1.89s/it]
Outer training loop:  89%|########8 | 44474/50000 [29:30:23<2:52:47,  1.88s/it]
Outer training loop:  89%|########8 | 44475/50000 [29:30:25<2:56:52,  1.92s/it]
Outer training loop:  89%|########8 | 44476/50000 [29:30:27<2:52:44,  1.88s/it]
Outer training loop:  89%|########8 | 44477/50000 [29:30:28<2:54:18,  1.89s/it]
Outer training loop:  89%|########8 | 44478/50000 [29:30:30<2:52:34,  1.88s/it]
Outer training loop:  89%|########8 | 44479/50000 [29:30:32<2:56:19,  1.92s/it]
Outer training loop:  89%|########8 | 44480/50000 [29:30:34<2:54:48,  1.90s/it]
Outer training loop:  89%|########8 | 44481/50000 [29:30:36<2:54:15,  1.89s/it]
Outer training loop:  89%|########8 | 44482/50000 [29:30:38<2:54:37,  1.90s/it]
Outer training loop:  89%|########8 | 44483/50000 [29:30:40<2:51:32,  1.87s/it]
Outer training loop:  89%|########8 | 44484/50000 [29:30:42<2:50:54,  1.86s/it]
Outer training loop:  89%|########8 | 44485/50000 [29:30:43<2:50:58,  1.86s/it]
Outer training loop:  89%|########8 | 44486/50000 [29:30:45<2:53:17,  1.89s/it]
Outer training loop:  89%|########8 | 44487/50000 [29:30:47<2:53:30,  1.89s/it]
Outer training loop:  89%|########8 | 44488/50000 [29:30:49<2:50:00,  1.85s/it]
Outer training loop:  89%|########8 | 44489/50000 [29:30:51<2:55:09,  1.91s/it]
Outer training loop:  89%|########8 | 44490/50000 [29:30:53<2:52:45,  1.88s/it]
Outer training loop:  89%|########8 | 44491/50000 [29:30:55<2:50:07,  1.85s/it]
Outer training loop:  89%|########8 | 44492/50000 [29:30:57<2:49:22,  1.85s/it]
Outer training loop:  89%|########8 | 44493/50000 [29:30:59<2:56:17,  1.92s/it]
Outer training loop:  89%|########8 | 44494/50000 [29:31:00<2:51:50,  1.87s/it]
Outer training loop:  89%|########8 | 44495/50000 [29:31:02<2:47:46,  1.83s/it]
Outer training loop:  89%|########8 | 44496/50000 [29:31:04<2:57:19,  1.93s/it]
Outer training loop:  89%|########8 | 44497/50000 [29:31:06<2:51:53,  1.87s/it]
Outer training loop:  89%|########8 | 44498/50000 [29:31:08<2:52:59,  1.89s/it]
Outer training loop:  89%|########8 | 44499/50000 [29:31:10<2:49:02,  1.84s/it]
Outer training loop:  89%|########9 | 44500/50000 [29:31:12<2:56:10,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.67s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.78s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:31,  1.85s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:33,  1.89s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:30,  1.88s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:32,  1.93s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.94s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:30,  1.95s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.94s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.92s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:22,  1.96s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:14,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:14,  1.95s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<01:58,  1.82s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:04,  1.94s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:02,  1.94s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:56,  1.88s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:55,  1.95s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.92s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.89s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:44,  1.94s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:40,  1.92s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:34,  1.89s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.92s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.92s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:22,  1.84s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:19,  1.84s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:21,  1.94s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:19,  1.94s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:12,  1.85s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:05,  1.78s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:03,  1.77s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:07,  1.94s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:04,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.84s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:54,  1.82s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:53,  1.99s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:50,  1.94s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.91s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.88s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.89s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.87s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.98s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.93s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.88s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.86s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.88s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.83s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:23,  1.84s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.95s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.91s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.86s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.88s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.89s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.90s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.90s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.95s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:06,  2.08s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.98s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.88s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:27,  1.80s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.78s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:21,  1.77s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.82s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:21,  1.81s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:21,  1.83s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:18,  1.82s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:29,  2.00s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:27,  1.99s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:20,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:19,  1.97s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:13,  1.99s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:09,  1.96s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.90s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:04,  2.01s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:59,  1.96s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.93s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:51,  1.96s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:41,  1.95s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:38,  1.93s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.89s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:32,  1.88s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.86s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:31,  2.00s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:23,  1.89s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:21,  1.89s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:15,  1.84s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:19,  1.98s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:17,  1.98s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:15,  1.98s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:12,  1.95s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:09,  1.94s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:07,  1.92s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.90s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:05,  1.99s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:03,  1.97s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<01:00,  1.95s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:57,  1.91s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:54,  1.87s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:51,  1.96s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:48,  1.94s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:46,  1.92s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:43,  1.90s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:38,  1.84s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.82s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:32,  1.79s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:33,  1.94s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:30,  1.90s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.89s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.87s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:17,  1.99s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:15,  1.97s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.90s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.87s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.89s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.88s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.84s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.86s/it]

Outer training loop:  89%|########9 | 44501/50000 [29:36:34<149:55:30, 98.15s/it]
Outer training loop:  89%|########9 | 44502/50000 [29:36:37<106:03:16, 69.44s/it]
Outer training loop:  89%|########9 | 44503/50000 [29:36:39<75:04:36, 49.17s/it] 
Outer training loop:  89%|########9 | 44504/50000 [29:36:41<53:25:04, 34.99s/it]
Outer training loop:  89%|########9 | 44505/50000 [29:36:43<38:12:34, 25.03s/it]
Outer training loop:  89%|########9 | 44506/50000 [29:36:44<27:38:40, 18.11s/it]
Outer training loop:  89%|########9 | 44507/50000 [29:36:46<20:12:06, 13.24s/it]
Outer training loop:  89%|########9 | 44508/50000 [29:36:48<14:56:28,  9.79s/it]
Outer training loop:  89%|########9 | 44509/50000 [29:36:50<11:25:05,  7.49s/it]
Outer training loop:  89%|########9 | 44510/50000 [29:36:52<8:49:01,  5.78s/it] 
Outer training loop:  89%|########9 | 44511/50000 [29:36:54<6:59:07,  4.58s/it]
Outer training loop:  89%|########9 | 44512/50000 [29:36:56<5:48:35,  3.81s/it]
Outer training loop:  89%|########9 | 44513/50000 [29:36:58<4:55:15,  3.23s/it]
Outer training loop:  89%|########9 | 44514/50000 [29:37:00<4:18:42,  2.83s/it]
Outer training loop:  89%|########9 | 44515/50000 [29:37:02<3:54:39,  2.57s/it]
Outer training loop:  89%|########9 | 44516/50000 [29:37:04<3:42:09,  2.43s/it]
Outer training loop:  89%|########9 | 44517/50000 [29:37:06<3:27:59,  2.28s/it]
Outer training loop:  89%|########9 | 44518/50000 [29:37:07<3:17:18,  2.16s/it]
Outer training loop:  89%|########9 | 44519/50000 [29:37:09<3:14:23,  2.13s/it]
Outer training loop:  89%|########9 | 44520/50000 [29:37:11<3:07:12,  2.05s/it]
Outer training loop:  89%|########9 | 44521/50000 [29:37:13<3:01:20,  1.99s/it]
Outer training loop:  89%|########9 | 44522/50000 [29:37:15<3:03:53,  2.01s/it]
Outer training loop:  89%|########9 | 44523/50000 [29:37:17<3:00:17,  1.98s/it]
Outer training loop:  89%|########9 | 44524/50000 [29:37:19<2:56:30,  1.93s/it]
Outer training loop:  89%|########9 | 44525/50000 [29:37:21<2:53:39,  1.90s/it]
Outer training loop:  89%|########9 | 44526/50000 [29:37:23<2:57:45,  1.95s/it]
Outer training loop:  89%|########9 | 44527/50000 [29:37:25<2:56:54,  1.94s/it]
Outer training loop:  89%|########9 | 44528/50000 [29:37:27<2:54:41,  1.92s/it]
Outer training loop:  89%|########9 | 44529/50000 [29:37:29<2:58:10,  1.95s/it]
Outer training loop:  89%|########9 | 44530/50000 [29:37:30<2:53:09,  1.90s/it]
Outer training loop:  89%|########9 | 44531/50000 [29:37:32<2:49:37,  1.86s/it]
Outer training loop:  89%|########9 | 44532/50000 [29:37:34<2:51:27,  1.88s/it]
Outer training loop:  89%|########9 | 44533/50000 [29:37:36<2:52:03,  1.89s/it]
Outer training loop:  89%|########9 | 44534/50000 [29:37:38<2:49:52,  1.86s/it]
Outer training loop:  89%|########9 | 44535/50000 [29:37:40<2:51:36,  1.88s/it]
Outer training loop:  89%|########9 | 44536/50000 [29:37:42<2:54:55,  1.92s/it]
Outer training loop:  89%|########9 | 44537/50000 [29:37:44<2:53:49,  1.91s/it]
Outer training loop:  89%|########9 | 44538/50000 [29:37:45<2:49:09,  1.86s/it]
Outer training loop:  89%|########9 | 44539/50000 [29:37:48<2:56:21,  1.94s/it]
Outer training loop:  89%|########9 | 44540/50000 [29:37:50<2:58:37,  1.96s/it]
Outer training loop:  89%|########9 | 44541/50000 [29:37:51<2:55:36,  1.93s/it]
Outer training loop:  89%|########9 | 44542/50000 [29:37:53<2:53:09,  1.90s/it]
Outer training loop:  89%|########9 | 44543/50000 [29:37:55<2:57:17,  1.95s/it]
Outer training loop:  89%|########9 | 44544/50000 [29:37:57<2:55:45,  1.93s/it]
Outer training loop:  89%|########9 | 44545/50000 [29:37:59<2:54:08,  1.92s/it]
Outer training loop:  89%|########9 | 44546/50000 [29:38:01<2:56:44,  1.94s/it]
Outer training loop:  89%|########9 | 44547/50000 [29:38:03<2:54:32,  1.92s/it]
Outer training loop:  89%|########9 | 44548/50000 [29:38:05<2:53:46,  1.91s/it]
Outer training loop:  89%|########9 | 44549/50000 [29:38:07<2:48:46,  1.86s/it]
Outer training loop:  89%|########9 | 44550/50000 [29:38:09<2:51:19,  1.89s/it]
Outer training loop:  89%|########9 | 44551/50000 [29:38:10<2:51:44,  1.89s/it]
Outer training loop:  89%|########9 | 44552/50000 [29:38:12<2:49:42,  1.87s/it]
Outer training loop:  89%|########9 | 44553/50000 [29:38:14<2:52:00,  1.89s/it]
Outer training loop:  89%|########9 | 44554/50000 [29:38:16<2:49:26,  1.87s/it]
Outer training loop:  89%|########9 | 44555/50000 [29:38:18<2:48:46,  1.86s/it]
Outer training loop:  89%|########9 | 44556/50000 [29:38:20<2:51:38,  1.89s/it]
Outer training loop:  89%|########9 | 44557/50000 [29:38:22<2:52:05,  1.90s/it]
Outer training loop:  89%|########9 | 44558/50000 [29:38:24<2:52:00,  1.90s/it]
Outer training loop:  89%|########9 | 44559/50000 [29:38:26<2:52:52,  1.91s/it]
Outer training loop:  89%|########9 | 44560/50000 [29:38:28<2:55:19,  1.93s/it]
Outer training loop:  89%|########9 | 44561/50000 [29:38:29<2:51:36,  1.89s/it]
Outer training loop:  89%|########9 | 44562/50000 [29:38:31<2:47:38,  1.85s/it]
Outer training loop:  89%|########9 | 44563/50000 [29:38:33<2:51:24,  1.89s/it]
Outer training loop:  89%|########9 | 44564/50000 [29:38:35<2:55:37,  1.94s/it]
Outer training loop:  89%|########9 | 44565/50000 [29:38:37<2:51:18,  1.89s/it]
Outer training loop:  89%|########9 | 44566/50000 [29:38:39<2:48:32,  1.86s/it]
Outer training loop:  89%|########9 | 44567/50000 [29:38:41<2:53:03,  1.91s/it]
Outer training loop:  89%|########9 | 44568/50000 [29:38:43<2:53:38,  1.92s/it]
Outer training loop:  89%|########9 | 44569/50000 [29:38:45<2:52:08,  1.90s/it]
Outer training loop:  89%|########9 | 44570/50000 [29:38:47<2:56:06,  1.95s/it]
Outer training loop:  89%|########9 | 44571/50000 [29:38:48<2:51:05,  1.89s/it]
Outer training loop:  89%|########9 | 44572/50000 [29:38:50<2:51:10,  1.89s/it]
Outer training loop:  89%|########9 | 44573/50000 [29:38:52<2:55:51,  1.94s/it]
Outer training loop:  89%|########9 | 44574/50000 [29:38:54<2:49:09,  1.87s/it]
Outer training loop:  89%|########9 | 44575/50000 [29:38:56<2:50:18,  1.88s/it]
Outer training loop:  89%|########9 | 44576/50000 [29:38:58<2:49:48,  1.88s/it]
Outer training loop:  89%|########9 | 44577/50000 [29:39:00<2:56:36,  1.95s/it]
Outer training loop:  89%|########9 | 44578/50000 [29:39:02<2:53:37,  1.92s/it]
Outer training loop:  89%|########9 | 44579/50000 [29:39:04<2:51:18,  1.90s/it]
Outer training loop:  89%|########9 | 44580/50000 [29:39:06<2:57:05,  1.96s/it]
Outer training loop:  89%|########9 | 44581/50000 [29:39:08<2:52:14,  1.91s/it]
Outer training loop:  89%|########9 | 44582/50000 [29:39:09<2:48:31,  1.87s/it]
Outer training loop:  89%|########9 | 44583/50000 [29:39:11<2:45:10,  1.83s/it]
Outer training loop:  89%|########9 | 44584/50000 [29:39:13<2:53:31,  1.92s/it]
Outer training loop:  89%|########9 | 44585/50000 [29:39:15<2:49:46,  1.88s/it]
Outer training loop:  89%|########9 | 44586/50000 [29:39:17<2:46:58,  1.85s/it]
Outer training loop:  89%|########9 | 44587/50000 [29:39:19<2:46:18,  1.84s/it]
Outer training loop:  89%|########9 | 44588/50000 [29:39:21<2:52:26,  1.91s/it]
Outer training loop:  89%|########9 | 44589/50000 [29:39:23<2:56:31,  1.96s/it]
Outer training loop:  89%|########9 | 44590/50000 [29:39:25<2:53:56,  1.93s/it]
Outer training loop:  89%|########9 | 44591/50000 [29:39:27<2:58:49,  1.98s/it]
Outer training loop:  89%|########9 | 44592/50000 [29:39:29<2:54:34,  1.94s/it]
Outer training loop:  89%|########9 | 44593/50000 [29:39:30<2:51:25,  1.90s/it]
Outer training loop:  89%|########9 | 44594/50000 [29:39:32<2:57:04,  1.97s/it]
Outer training loop:  89%|########9 | 44595/50000 [29:39:34<2:56:17,  1.96s/it]
Outer training loop:  89%|########9 | 44596/50000 [29:39:36<2:53:13,  1.92s/it]
Outer training loop:  89%|########9 | 44597/50000 [29:39:38<2:49:57,  1.89s/it]
Outer training loop:  89%|########9 | 44598/50000 [29:39:40<2:55:03,  1.94s/it]
Outer training loop:  89%|########9 | 44599/50000 [29:39:42<2:52:32,  1.92s/it]
Outer training loop:  89%|########9 | 44600/50000 [29:39:44<2:52:53,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  89%|########9 | 44601/50000 [29:39:51<5:22:45,  3.59s/it]
Outer training loop:  89%|########9 | 44602/50000 [29:39:53<4:32:32,  3.03s/it]
Outer training loop:  89%|########9 | 44603/50000 [29:39:55<3:59:34,  2.66s/it]
Outer training loop:  89%|########9 | 44604/50000 [29:39:57<3:35:53,  2.40s/it]
Outer training loop:  89%|########9 | 44605/50000 [29:39:59<3:26:19,  2.29s/it]
Outer training loop:  89%|########9 | 44606/50000 [29:40:01<3:14:07,  2.16s/it]
Outer training loop:  89%|########9 | 44607/50000 [29:40:03<3:08:30,  2.10s/it]
Outer training loop:  89%|########9 | 44608/50000 [29:40:05<3:09:07,  2.10s/it]
Outer training loop:  89%|########9 | 44609/50000 [29:40:06<3:01:53,  2.02s/it]
Outer training loop:  89%|########9 | 44610/50000 [29:40:09<3:01:35,  2.02s/it]
Outer training loop:  89%|########9 | 44611/50000 [29:40:11<3:02:30,  2.03s/it]
Outer training loop:  89%|########9 | 44612/50000 [29:40:12<2:57:00,  1.97s/it]
Outer training loop:  89%|########9 | 44613/50000 [29:40:14<2:54:33,  1.94s/it]
Outer training loop:  89%|########9 | 44614/50000 [29:40:16<2:51:04,  1.91s/it]
Outer training loop:  89%|########9 | 44615/50000 [29:40:18<2:55:52,  1.96s/it]
Outer training loop:  89%|########9 | 44616/50000 [29:40:20<2:55:17,  1.95s/it]
Outer training loop:  89%|########9 | 44617/50000 [29:40:22<2:49:42,  1.89s/it]
Outer training loop:  89%|########9 | 44618/50000 [29:40:24<2:52:52,  1.93s/it]
Outer training loop:  89%|########9 | 44619/50000 [29:40:26<2:52:20,  1.92s/it]
Outer training loop:  89%|########9 | 44620/50000 [29:40:28<2:48:55,  1.88s/it]
Outer training loop:  89%|########9 | 44621/50000 [29:40:29<2:46:13,  1.85s/it]
Outer training loop:  89%|########9 | 44622/50000 [29:40:31<2:49:28,  1.89s/it]
Outer training loop:  89%|########9 | 44623/50000 [29:40:33<2:48:28,  1.88s/it]
Outer training loop:  89%|########9 | 44624/50000 [29:40:35<2:51:07,  1.91s/it]
Outer training loop:  89%|########9 | 44625/50000 [29:40:37<2:52:03,  1.92s/it]
Outer training loop:  89%|########9 | 44626/50000 [29:40:39<2:47:05,  1.87s/it]
Outer training loop:  89%|########9 | 44627/50000 [29:40:41<2:45:04,  1.84s/it]
Outer training loop:  89%|########9 | 44628/50000 [29:40:43<2:47:12,  1.87s/it]
Outer training loop:  89%|########9 | 44629/50000 [29:40:45<2:50:14,  1.90s/it]
Outer training loop:  89%|########9 | 44630/50000 [29:40:46<2:45:55,  1.85s/it]
Outer training loop:  89%|########9 | 44631/50000 [29:40:48<2:45:23,  1.85s/it]
Outer training loop:  89%|########9 | 44632/50000 [29:40:50<2:48:01,  1.88s/it]
Outer training loop:  89%|########9 | 44633/50000 [29:40:52<2:50:40,  1.91s/it]
Outer training loop:  89%|########9 | 44634/50000 [29:40:54<2:49:24,  1.89s/it]
Outer training loop:  89%|########9 | 44635/50000 [29:40:56<2:46:24,  1.86s/it]
Outer training loop:  89%|########9 | 44636/50000 [29:40:58<2:57:22,  1.98s/it]
Outer training loop:  89%|########9 | 44637/50000 [29:41:00<2:52:25,  1.93s/it]
Outer training loop:  89%|########9 | 44638/50000 [29:41:02<2:50:44,  1.91s/it]
Outer training loop:  89%|########9 | 44639/50000 [29:41:04<2:58:13,  1.99s/it]
Outer training loop:  89%|########9 | 44640/50000 [29:41:06<2:57:09,  1.98s/it]
Outer training loop:  89%|########9 | 44641/50000 [29:41:08<2:53:49,  1.95s/it]
Outer training loop:  89%|########9 | 44642/50000 [29:41:09<2:49:05,  1.89s/it]
Outer training loop:  89%|########9 | 44643/50000 [29:41:11<2:51:55,  1.93s/it]
Outer training loop:  89%|########9 | 44644/50000 [29:41:13<2:53:24,  1.94s/it]
Outer training loop:  89%|########9 | 44645/50000 [29:41:15<2:50:23,  1.91s/it]
Outer training loop:  89%|########9 | 44646/50000 [29:41:17<2:52:27,  1.93s/it]
Outer training loop:  89%|########9 | 44647/50000 [29:41:19<2:53:21,  1.94s/it]
Outer training loop:  89%|########9 | 44648/50000 [29:41:21<2:54:18,  1.95s/it]
Outer training loop:  89%|########9 | 44649/50000 [29:41:23<2:52:11,  1.93s/it]
Outer training loop:  89%|########9 | 44650/50000 [29:41:25<2:51:05,  1.92s/it]
Outer training loop:  89%|########9 | 44651/50000 [29:41:27<2:52:24,  1.93s/it]
Outer training loop:  89%|########9 | 44652/50000 [29:41:29<2:57:01,  1.99s/it]
Outer training loop:  89%|########9 | 44653/50000 [29:41:31<2:54:27,  1.96s/it]
Outer training loop:  89%|########9 | 44654/50000 [29:41:33<2:51:09,  1.92s/it]
Outer training loop:  89%|########9 | 44655/50000 [29:41:35<2:47:34,  1.88s/it]
Outer training loop:  89%|########9 | 44656/50000 [29:41:37<2:55:53,  1.97s/it]
Outer training loop:  89%|########9 | 44657/50000 [29:41:39<2:51:17,  1.92s/it]
Outer training loop:  89%|########9 | 44658/50000 [29:41:40<2:52:38,  1.94s/it]
Outer training loop:  89%|########9 | 44659/50000 [29:41:43<2:57:05,  1.99s/it]
Outer training loop:  89%|########9 | 44660/50000 [29:41:44<2:49:13,  1.90s/it]
Outer training loop:  89%|########9 | 44661/50000 [29:41:46<2:45:08,  1.86s/it]
Outer training loop:  89%|########9 | 44662/50000 [29:41:48<2:46:36,  1.87s/it]
Outer training loop:  89%|########9 | 44663/50000 [29:41:50<2:51:05,  1.92s/it]
Outer training loop:  89%|########9 | 44664/50000 [29:41:52<2:45:46,  1.86s/it]
Outer training loop:  89%|########9 | 44665/50000 [29:41:54<2:44:05,  1.85s/it]
Outer training loop:  89%|########9 | 44666/50000 [29:41:56<2:49:07,  1.90s/it]
Outer training loop:  89%|########9 | 44667/50000 [29:41:57<2:48:30,  1.90s/it]
Outer training loop:  89%|########9 | 44668/50000 [29:41:59<2:46:13,  1.87s/it]
Outer training loop:  89%|########9 | 44669/50000 [29:42:01<2:43:10,  1.84s/it]
Outer training loop:  89%|########9 | 44670/50000 [29:42:03<2:49:03,  1.90s/it]
Outer training loop:  89%|########9 | 44671/50000 [29:42:05<2:48:26,  1.90s/it]
Outer training loop:  89%|########9 | 44672/50000 [29:42:07<2:45:43,  1.87s/it]
Outer training loop:  89%|########9 | 44673/50000 [29:42:09<2:47:51,  1.89s/it]
Outer training loop:  89%|########9 | 44674/50000 [29:42:11<2:46:35,  1.88s/it]
Outer training loop:  89%|########9 | 44675/50000 [29:42:12<2:43:24,  1.84s/it]
Outer training loop:  89%|########9 | 44676/50000 [29:42:14<2:48:38,  1.90s/it]
Outer training loop:  89%|########9 | 44677/50000 [29:42:16<2:46:29,  1.88s/it]
Outer training loop:  89%|########9 | 44678/50000 [29:42:18<2:46:49,  1.88s/it]
Outer training loop:  89%|########9 | 44679/50000 [29:42:20<2:46:28,  1.88s/it]
Outer training loop:  89%|########9 | 44680/50000 [29:42:22<2:51:47,  1.94s/it]
Outer training loop:  89%|########9 | 44681/50000 [29:42:24<2:48:24,  1.90s/it]
Outer training loop:  89%|########9 | 44682/50000 [29:42:26<2:48:31,  1.90s/it]
Outer training loop:  89%|########9 | 44683/50000 [29:42:28<2:50:02,  1.92s/it]
Outer training loop:  89%|########9 | 44684/50000 [29:42:29<2:45:42,  1.87s/it]
Outer training loop:  89%|########9 | 44685/50000 [29:42:31<2:40:19,  1.81s/it]
Outer training loop:  89%|########9 | 44686/50000 [29:42:33<2:43:28,  1.85s/it]
Outer training loop:  89%|########9 | 44687/50000 [29:42:35<2:48:44,  1.91s/it]
Outer training loop:  89%|########9 | 44688/50000 [29:42:37<2:47:04,  1.89s/it]
Outer training loop:  89%|########9 | 44689/50000 [29:42:39<2:44:39,  1.86s/it]
Outer training loop:  89%|########9 | 44690/50000 [29:42:41<2:51:18,  1.94s/it]
Outer training loop:  89%|########9 | 44691/50000 [29:42:43<2:51:47,  1.94s/it]
Outer training loop:  89%|########9 | 44692/50000 [29:42:45<2:47:58,  1.90s/it]
Outer training loop:  89%|########9 | 44693/50000 [29:42:47<2:51:52,  1.94s/it]
Outer training loop:  89%|########9 | 44694/50000 [29:42:48<2:48:59,  1.91s/it]
Outer training loop:  89%|########9 | 44695/50000 [29:42:50<2:47:08,  1.89s/it]
Outer training loop:  89%|########9 | 44696/50000 [29:42:52<2:48:22,  1.90s/it]
Outer training loop:  89%|########9 | 44697/50000 [29:42:54<2:51:26,  1.94s/it]
Outer training loop:  89%|########9 | 44698/50000 [29:42:56<2:47:31,  1.90s/it]
Outer training loop:  89%|########9 | 44699/50000 [29:42:58<2:47:33,  1.90s/it]
Outer training loop:  89%|########9 | 44700/50000 [29:43:00<2:50:43,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.93s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  89%|########9 | 44701/50000 [29:43:07<5:14:31,  3.56s/it]
Outer training loop:  89%|########9 | 44702/50000 [29:43:09<4:32:46,  3.09s/it]
Outer training loop:  89%|########9 | 44703/50000 [29:43:11<4:00:33,  2.72s/it]
Outer training loop:  89%|########9 | 44704/50000 [29:43:13<3:45:12,  2.55s/it]
Outer training loop:  89%|########9 | 44705/50000 [29:43:15<3:25:33,  2.33s/it]
Outer training loop:  89%|########9 | 44706/50000 [29:43:17<3:12:28,  2.18s/it]
Outer training loop:  89%|########9 | 44707/50000 [29:43:19<3:06:36,  2.12s/it]
Outer training loop:  89%|########9 | 44708/50000 [29:43:21<3:02:52,  2.07s/it]
Outer training loop:  89%|########9 | 44709/50000 [29:43:23<2:56:24,  2.00s/it]
Outer training loop:  89%|########9 | 44710/50000 [29:43:24<2:49:13,  1.92s/it]
Outer training loop:  89%|########9 | 44711/50000 [29:43:27<2:53:13,  1.97s/it]
Outer training loop:  89%|########9 | 44712/50000 [29:43:28<2:48:48,  1.92s/it]
Outer training loop:  89%|########9 | 44713/50000 [29:43:30<2:45:07,  1.87s/it]
Outer training loop:  89%|########9 | 44714/50000 [29:43:32<2:47:17,  1.90s/it]
Outer training loop:  89%|########9 | 44715/50000 [29:43:34<2:47:19,  1.90s/it]
Outer training loop:  89%|########9 | 44716/50000 [29:43:36<2:45:20,  1.88s/it]
Outer training loop:  89%|########9 | 44717/50000 [29:43:38<2:41:46,  1.84s/it]
Outer training loop:  89%|########9 | 44718/50000 [29:43:40<2:48:39,  1.92s/it]
Outer training loop:  89%|########9 | 44719/50000 [29:43:42<2:47:24,  1.90s/it]
Outer training loop:  89%|########9 | 44720/50000 [29:43:44<2:49:48,  1.93s/it]
Outer training loop:  89%|########9 | 44721/50000 [29:43:46<2:55:25,  1.99s/it]
Outer training loop:  89%|########9 | 44722/50000 [29:43:47<2:50:18,  1.94s/it]
Outer training loop:  89%|########9 | 44723/50000 [29:43:49<2:47:21,  1.90s/it]
Outer training loop:  89%|########9 | 44724/50000 [29:43:51<2:47:06,  1.90s/it]
Outer training loop:  89%|########9 | 44725/50000 [29:43:53<2:51:04,  1.95s/it]
Outer training loop:  89%|########9 | 44726/50000 [29:43:55<2:48:22,  1.92s/it]
Outer training loop:  89%|########9 | 44727/50000 [29:43:57<2:47:34,  1.91s/it]
Outer training loop:  89%|########9 | 44728/50000 [29:43:59<2:49:59,  1.93s/it]
Outer training loop:  89%|########9 | 44729/50000 [29:44:01<2:45:50,  1.89s/it]
Outer training loop:  89%|########9 | 44730/50000 [29:44:03<2:44:42,  1.88s/it]
Outer training loop:  89%|########9 | 44731/50000 [29:44:05<2:52:16,  1.96s/it]
Outer training loop:  89%|########9 | 44732/50000 [29:44:07<2:51:15,  1.95s/it]
Outer training loop:  89%|########9 | 44733/50000 [29:44:09<2:55:54,  2.00s/it]
Outer training loop:  89%|########9 | 44734/50000 [29:44:11<2:48:24,  1.92s/it]
Outer training loop:  89%|########9 | 44735/50000 [29:44:13<2:55:16,  2.00s/it]
Outer training loop:  89%|########9 | 44736/50000 [29:44:15<2:52:39,  1.97s/it]
Outer training loop:  89%|########9 | 44737/50000 [29:44:17<2:51:01,  1.95s/it]
Outer training loop:  89%|########9 | 44738/50000 [29:44:19<2:55:23,  2.00s/it]
Outer training loop:  89%|########9 | 44739/50000 [29:44:21<2:53:26,  1.98s/it]
Outer training loop:  89%|########9 | 44740/50000 [29:44:22<2:50:49,  1.95s/it]
Outer training loop:  89%|########9 | 44741/50000 [29:44:25<2:54:27,  1.99s/it]
Outer training loop:  89%|########9 | 44742/50000 [29:44:26<2:52:08,  1.96s/it]
Outer training loop:  89%|########9 | 44743/50000 [29:44:28<2:47:40,  1.91s/it]
Outer training loop:  89%|########9 | 44744/50000 [29:44:30<2:46:19,  1.90s/it]
Outer training loop:  89%|########9 | 44745/50000 [29:44:32<2:46:27,  1.90s/it]
Outer training loop:  89%|########9 | 44746/50000 [29:44:34<2:46:11,  1.90s/it]
Outer training loop:  89%|########9 | 44747/50000 [29:44:36<2:45:46,  1.89s/it]
Outer training loop:  89%|########9 | 44748/50000 [29:44:38<2:48:56,  1.93s/it]
Outer training loop:  89%|########9 | 44749/50000 [29:44:40<2:45:52,  1.90s/it]
Outer training loop:  90%|########9 | 44750/50000 [29:44:41<2:42:28,  1.86s/it]
Outer training loop:  90%|########9 | 44751/50000 [29:44:43<2:42:43,  1.86s/it]
Outer training loop:  90%|########9 | 44752/50000 [29:44:45<2:46:20,  1.90s/it]
Outer training loop:  90%|########9 | 44753/50000 [29:44:47<2:47:34,  1.92s/it]
Outer training loop:  90%|########9 | 44754/50000 [29:44:49<2:46:12,  1.90s/it]
Outer training loop:  90%|########9 | 44755/50000 [29:44:51<2:51:15,  1.96s/it]
Outer training loop:  90%|########9 | 44756/50000 [29:44:53<2:48:13,  1.92s/it]
Outer training loop:  90%|########9 | 44757/50000 [29:44:55<2:44:19,  1.88s/it]
Outer training loop:  90%|########9 | 44758/50000 [29:44:57<2:40:27,  1.84s/it]
Outer training loop:  90%|########9 | 44759/50000 [29:44:59<2:47:08,  1.91s/it]
Outer training loop:  90%|########9 | 44760/50000 [29:45:00<2:41:52,  1.85s/it]
Outer training loop:  90%|########9 | 44761/50000 [29:45:02<2:39:37,  1.83s/it]
Outer training loop:  90%|########9 | 44762/50000 [29:45:04<2:44:47,  1.89s/it]
Outer training loop:  90%|########9 | 44763/50000 [29:45:06<2:42:41,  1.86s/it]
Outer training loop:  90%|########9 | 44764/50000 [29:45:08<2:40:00,  1.83s/it]
Outer training loop:  90%|########9 | 44765/50000 [29:45:10<2:40:42,  1.84s/it]
Outer training loop:  90%|########9 | 44766/50000 [29:45:11<2:42:43,  1.87s/it]
Outer training loop:  90%|########9 | 44767/50000 [29:45:13<2:44:20,  1.88s/it]
Outer training loop:  90%|########9 | 44768/50000 [29:45:15<2:41:53,  1.86s/it]
Outer training loop:  90%|########9 | 44769/50000 [29:45:17<2:38:45,  1.82s/it]
Outer training loop:  90%|########9 | 44770/50000 [29:45:19<2:43:40,  1.88s/it]
Outer training loop:  90%|########9 | 44771/50000 [29:45:21<2:43:21,  1.87s/it]
Outer training loop:  90%|########9 | 44772/50000 [29:45:23<2:41:46,  1.86s/it]
Outer training loop:  90%|########9 | 44773/50000 [29:45:25<2:49:00,  1.94s/it]
Outer training loop:  90%|########9 | 44774/50000 [29:45:27<2:45:16,  1.90s/it]
Outer training loop:  90%|########9 | 44775/50000 [29:45:28<2:43:29,  1.88s/it]
Outer training loop:  90%|########9 | 44776/50000 [29:45:30<2:48:00,  1.93s/it]
Outer training loop:  90%|########9 | 44777/50000 [29:45:32<2:43:40,  1.88s/it]
Outer training loop:  90%|########9 | 44778/50000 [29:45:34<2:43:03,  1.87s/it]
Outer training loop:  90%|########9 | 44779/50000 [29:45:36<2:41:36,  1.86s/it]
Outer training loop:  90%|########9 | 44780/50000 [29:45:38<2:45:36,  1.90s/it]
Outer training loop:  90%|########9 | 44781/50000 [29:45:40<2:43:26,  1.88s/it]
Outer training loop:  90%|########9 | 44782/50000 [29:45:42<2:41:28,  1.86s/it]
Outer training loop:  90%|########9 | 44783/50000 [29:45:44<2:50:44,  1.96s/it]
Outer training loop:  90%|########9 | 44784/50000 [29:45:46<2:46:51,  1.92s/it]
Outer training loop:  90%|########9 | 44785/50000 [29:45:47<2:46:29,  1.92s/it]
Outer training loop:  90%|########9 | 44786/50000 [29:45:49<2:49:45,  1.95s/it]
Outer training loop:  90%|########9 | 44787/50000 [29:45:51<2:46:21,  1.91s/it]
Outer training loop:  90%|########9 | 44788/50000 [29:45:53<2:42:38,  1.87s/it]
Outer training loop:  90%|########9 | 44789/50000 [29:45:55<2:45:56,  1.91s/it]
Outer training loop:  90%|########9 | 44790/50000 [29:45:57<2:49:47,  1.96s/it]
Outer training loop:  90%|########9 | 44791/50000 [29:45:59<2:48:48,  1.94s/it]
Outer training loop:  90%|########9 | 44792/50000 [29:46:01<2:46:15,  1.92s/it]
Outer training loop:  90%|########9 | 44793/50000 [29:46:03<2:47:50,  1.93s/it]
Outer training loop:  90%|########9 | 44794/50000 [29:46:05<2:45:33,  1.91s/it]
Outer training loop:  90%|########9 | 44795/50000 [29:46:07<2:44:53,  1.90s/it]
Outer training loop:  90%|########9 | 44796/50000 [29:46:08<2:44:09,  1.89s/it]
Outer training loop:  90%|########9 | 44797/50000 [29:46:10<2:46:48,  1.92s/it]
Outer training loop:  90%|########9 | 44798/50000 [29:46:12<2:43:33,  1.89s/it]
Outer training loop:  90%|########9 | 44799/50000 [29:46:14<2:42:59,  1.88s/it]
Outer training loop:  90%|########9 | 44800/50000 [29:46:16<2:46:53,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.83s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  90%|########9 | 44801/50000 [29:46:23<5:05:23,  3.52s/it]
Outer training loop:  90%|########9 | 44802/50000 [29:46:25<4:23:43,  3.04s/it]
Outer training loop:  90%|########9 | 44803/50000 [29:46:27<3:50:45,  2.66s/it]
Outer training loop:  90%|########9 | 44804/50000 [29:46:29<3:35:01,  2.48s/it]
Outer training loop:  90%|########9 | 44805/50000 [29:46:31<3:17:40,  2.28s/it]
Outer training loop:  90%|########9 | 44806/50000 [29:46:33<3:05:46,  2.15s/it]
Outer training loop:  90%|########9 | 44807/50000 [29:46:35<2:57:45,  2.05s/it]
Outer training loop:  90%|########9 | 44808/50000 [29:46:37<2:54:54,  2.02s/it]
Outer training loop:  90%|########9 | 44809/50000 [29:46:38<2:49:19,  1.96s/it]
Outer training loop:  90%|########9 | 44810/50000 [29:46:40<2:46:28,  1.92s/it]
Outer training loop:  90%|########9 | 44811/50000 [29:46:42<2:46:17,  1.92s/it]
Outer training loop:  90%|########9 | 44812/50000 [29:46:44<2:43:36,  1.89s/it]
Outer training loop:  90%|########9 | 44813/50000 [29:46:46<2:41:44,  1.87s/it]
Outer training loop:  90%|########9 | 44814/50000 [29:46:48<2:45:43,  1.92s/it]
Outer training loop:  90%|########9 | 44815/50000 [29:46:50<2:47:12,  1.93s/it]
Outer training loop:  90%|########9 | 44816/50000 [29:46:52<2:43:42,  1.89s/it]
Outer training loop:  90%|########9 | 44817/50000 [29:46:53<2:41:41,  1.87s/it]
Outer training loop:  90%|########9 | 44818/50000 [29:46:56<2:48:19,  1.95s/it]
Outer training loop:  90%|########9 | 44819/50000 [29:46:57<2:43:24,  1.89s/it]
Outer training loop:  90%|########9 | 44820/50000 [29:46:59<2:44:08,  1.90s/it]
Outer training loop:  90%|########9 | 44821/50000 [29:47:01<2:46:48,  1.93s/it]
Outer training loop:  90%|########9 | 44822/50000 [29:47:03<2:43:02,  1.89s/it]
Outer training loop:  90%|########9 | 44823/50000 [29:47:05<2:43:05,  1.89s/it]
Outer training loop:  90%|########9 | 44824/50000 [29:47:07<2:44:58,  1.91s/it]
Outer training loop:  90%|########9 | 44825/50000 [29:47:09<2:46:22,  1.93s/it]
Outer training loop:  90%|########9 | 44826/50000 [29:47:11<2:43:39,  1.90s/it]
Outer training loop:  90%|########9 | 44827/50000 [29:47:13<2:41:03,  1.87s/it]
Outer training loop:  90%|########9 | 44828/50000 [29:47:15<2:46:15,  1.93s/it]
Outer training loop:  90%|########9 | 44829/50000 [29:47:16<2:42:32,  1.89s/it]
Outer training loop:  90%|########9 | 44830/50000 [29:47:19<2:50:36,  1.98s/it]
Outer training loop:  90%|########9 | 44831/50000 [29:47:21<2:53:49,  2.02s/it]
Outer training loop:  90%|########9 | 44832/50000 [29:47:22<2:48:42,  1.96s/it]
Outer training loop:  90%|########9 | 44833/50000 [29:47:24<2:42:50,  1.89s/it]
Outer training loop:  90%|########9 | 44834/50000 [29:47:27<2:53:10,  2.01s/it]
Outer training loop:  90%|########9 | 44835/50000 [29:47:29<2:53:35,  2.02s/it]
Outer training loop:  90%|########9 | 44836/50000 [29:47:31<2:51:52,  2.00s/it]
Outer training loop:  90%|########9 | 44837/50000 [29:47:32<2:47:34,  1.95s/it]
Outer training loop:  90%|########9 | 44838/50000 [29:47:35<2:53:13,  2.01s/it]
Outer training loop:  90%|########9 | 44839/50000 [29:47:36<2:49:25,  1.97s/it]
Outer training loop:  90%|########9 | 44840/50000 [29:47:38<2:46:42,  1.94s/it]
Outer training loop:  90%|########9 | 44841/50000 [29:47:40<2:50:06,  1.98s/it]
Outer training loop:  90%|########9 | 44842/50000 [29:47:42<2:46:03,  1.93s/it]
Outer training loop:  90%|########9 | 44843/50000 [29:47:44<2:45:17,  1.92s/it]
Outer training loop:  90%|########9 | 44844/50000 [29:47:46<2:42:35,  1.89s/it]
Outer training loop:  90%|########9 | 44845/50000 [29:47:48<2:41:50,  1.88s/it]
Outer training loop:  90%|########9 | 44846/50000 [29:47:50<2:44:14,  1.91s/it]
Outer training loop:  90%|########9 | 44847/50000 [29:47:51<2:40:03,  1.86s/it]
Outer training loop:  90%|########9 | 44848/50000 [29:47:53<2:43:24,  1.90s/it]
Outer training loop:  90%|########9 | 44849/50000 [29:47:55<2:40:25,  1.87s/it]
Outer training loop:  90%|########9 | 44850/50000 [29:47:57<2:39:45,  1.86s/it]
Outer training loop:  90%|########9 | 44851/50000 [29:47:59<2:41:31,  1.88s/it]
Outer training loop:  90%|########9 | 44852/50000 [29:48:01<2:45:32,  1.93s/it]
Outer training loop:  90%|########9 | 44853/50000 [29:48:03<2:42:55,  1.90s/it]
Outer training loop:  90%|########9 | 44854/50000 [29:48:05<2:40:48,  1.88s/it]
Outer training loop:  90%|########9 | 44855/50000 [29:48:07<2:43:39,  1.91s/it]
Outer training loop:  90%|########9 | 44856/50000 [29:48:09<2:41:46,  1.89s/it]
Outer training loop:  90%|########9 | 44857/50000 [29:48:10<2:39:01,  1.86s/it]
Outer training loop:  90%|########9 | 44858/50000 [29:48:12<2:39:37,  1.86s/it]
Outer training loop:  90%|########9 | 44859/50000 [29:48:14<2:45:14,  1.93s/it]
Outer training loop:  90%|########9 | 44860/50000 [29:48:16<2:41:47,  1.89s/it]
Outer training loop:  90%|########9 | 44861/50000 [29:48:18<2:39:49,  1.87s/it]
Outer training loop:  90%|########9 | 44862/50000 [29:48:20<2:43:02,  1.90s/it]
Outer training loop:  90%|########9 | 44863/50000 [29:48:22<2:40:57,  1.88s/it]
Outer training loop:  90%|########9 | 44864/50000 [29:48:24<2:42:43,  1.90s/it]
Outer training loop:  90%|########9 | 44865/50000 [29:48:25<2:41:27,  1.89s/it]
Outer training loop:  90%|########9 | 44866/50000 [29:48:28<2:47:36,  1.96s/it]
Outer training loop:  90%|########9 | 44867/50000 [29:48:29<2:44:48,  1.93s/it]
Outer training loop:  90%|########9 | 44868/50000 [29:48:31<2:44:08,  1.92s/it]
Outer training loop:  90%|########9 | 44869/50000 [29:48:33<2:46:19,  1.94s/it]
Outer training loop:  90%|########9 | 44870/50000 [29:48:35<2:46:42,  1.95s/it]
Outer training loop:  90%|########9 | 44871/50000 [29:48:37<2:40:13,  1.87s/it]
Outer training loop:  90%|########9 | 44872/50000 [29:48:39<2:36:57,  1.84s/it]
Outer training loop:  90%|########9 | 44873/50000 [29:48:41<2:39:35,  1.87s/it]
Outer training loop:  90%|########9 | 44874/50000 [29:48:43<2:37:58,  1.85s/it]
Outer training loop:  90%|########9 | 44875/50000 [29:48:44<2:34:23,  1.81s/it]
Outer training loop:  90%|########9 | 44876/50000 [29:48:46<2:38:46,  1.86s/it]
Outer training loop:  90%|########9 | 44877/50000 [29:48:48<2:35:15,  1.82s/it]
Outer training loop:  90%|########9 | 44878/50000 [29:48:50<2:37:19,  1.84s/it]
Outer training loop:  90%|########9 | 44879/50000 [29:48:52<2:36:47,  1.84s/it]
Outer training loop:  90%|########9 | 44880/50000 [29:48:54<2:46:20,  1.95s/it]
Outer training loop:  90%|########9 | 44881/50000 [29:48:56<2:44:49,  1.93s/it]
Outer training loop:  90%|########9 | 44882/50000 [29:48:58<2:43:12,  1.91s/it]
Outer training loop:  90%|########9 | 44883/50000 [29:49:00<2:42:16,  1.90s/it]
Outer training loop:  90%|########9 | 44884/50000 [29:49:01<2:42:33,  1.91s/it]
Outer training loop:  90%|########9 | 44885/50000 [29:49:03<2:40:44,  1.89s/it]
Outer training loop:  90%|########9 | 44886/50000 [29:49:05<2:37:14,  1.84s/it]
Outer training loop:  90%|########9 | 44887/50000 [29:49:07<2:43:25,  1.92s/it]
Outer training loop:  90%|########9 | 44888/50000 [29:49:09<2:42:32,  1.91s/it]
Outer training loop:  90%|########9 | 44889/50000 [29:49:11<2:39:58,  1.88s/it]
Outer training loop:  90%|########9 | 44890/50000 [29:49:13<2:37:37,  1.85s/it]
Outer training loop:  90%|########9 | 44891/50000 [29:49:15<2:41:59,  1.90s/it]
Outer training loop:  90%|########9 | 44892/50000 [29:49:16<2:40:53,  1.89s/it]
Outer training loop:  90%|########9 | 44893/50000 [29:49:18<2:37:37,  1.85s/it]
Outer training loop:  90%|########9 | 44894/50000 [29:49:20<2:44:47,  1.94s/it]
Outer training loop:  90%|########9 | 44895/50000 [29:49:22<2:41:12,  1.89s/it]
Outer training loop:  90%|########9 | 44896/50000 [29:49:24<2:37:31,  1.85s/it]
Outer training loop:  90%|########9 | 44897/50000 [29:49:26<2:41:41,  1.90s/it]
Outer training loop:  90%|########9 | 44898/50000 [29:49:28<2:39:01,  1.87s/it]
Outer training loop:  90%|########9 | 44899/50000 [29:49:30<2:36:41,  1.84s/it]
Outer training loop:  90%|########9 | 44900/50000 [29:49:31<2:36:07,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.86s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]

Outer training loop:  90%|########9 | 44901/50000 [29:49:39<5:01:43,  3.55s/it]
Outer training loop:  90%|########9 | 44902/50000 [29:49:41<4:19:05,  3.05s/it]
Outer training loop:  90%|########9 | 44903/50000 [29:49:43<3:48:23,  2.69s/it]
Outer training loop:  90%|########9 | 44904/50000 [29:49:44<3:26:30,  2.43s/it]
Outer training loop:  90%|########9 | 44905/50000 [29:49:46<3:15:20,  2.30s/it]
Outer training loop:  90%|########9 | 44906/50000 [29:49:48<3:02:24,  2.15s/it]
Outer training loop:  90%|########9 | 44907/50000 [29:49:50<2:57:56,  2.10s/it]
Outer training loop:  90%|########9 | 44908/50000 [29:49:52<2:57:56,  2.10s/it]
Outer training loop:  90%|########9 | 44909/50000 [29:49:54<2:51:22,  2.02s/it]
Outer training loop:  90%|########9 | 44910/50000 [29:49:56<2:48:33,  1.99s/it]
Outer training loop:  90%|########9 | 44911/50000 [29:49:58<2:45:01,  1.95s/it]
Outer training loop:  90%|########9 | 44912/50000 [29:50:00<2:45:00,  1.95s/it]
Outer training loop:  90%|########9 | 44913/50000 [29:50:02<2:41:34,  1.91s/it]
Outer training loop:  90%|########9 | 44914/50000 [29:50:03<2:38:04,  1.86s/it]
Outer training loop:  90%|########9 | 44915/50000 [29:50:05<2:41:51,  1.91s/it]
Outer training loop:  90%|########9 | 44916/50000 [29:50:07<2:37:19,  1.86s/it]
Outer training loop:  90%|########9 | 44917/50000 [29:50:09<2:40:09,  1.89s/it]
Outer training loop:  90%|########9 | 44918/50000 [29:50:11<2:44:19,  1.94s/it]
Outer training loop:  90%|########9 | 44919/50000 [29:50:13<2:43:45,  1.93s/it]
Outer training loop:  90%|########9 | 44920/50000 [29:50:15<2:43:04,  1.93s/it]
Outer training loop:  90%|########9 | 44921/50000 [29:50:17<2:42:31,  1.92s/it]
Outer training loop:  90%|########9 | 44922/50000 [29:50:19<2:37:50,  1.87s/it]
Outer training loop:  90%|########9 | 44923/50000 [29:50:20<2:34:50,  1.83s/it]
Outer training loop:  90%|########9 | 44924/50000 [29:50:22<2:34:18,  1.82s/it]
Outer training loop:  90%|########9 | 44925/50000 [29:50:24<2:35:44,  1.84s/it]
Outer training loop:  90%|########9 | 44926/50000 [29:50:26<2:39:52,  1.89s/it]
Outer training loop:  90%|########9 | 44927/50000 [29:50:28<2:46:12,  1.97s/it]
Outer training loop:  90%|########9 | 44928/50000 [29:50:30<2:40:25,  1.90s/it]
Outer training loop:  90%|########9 | 44929/50000 [29:50:32<2:41:34,  1.91s/it]
Outer training loop:  90%|########9 | 44930/50000 [29:50:34<2:41:44,  1.91s/it]
Outer training loop:  90%|########9 | 44931/50000 [29:50:36<2:40:19,  1.90s/it]
Outer training loop:  90%|########9 | 44932/50000 [29:50:38<2:41:32,  1.91s/it]
Outer training loop:  90%|########9 | 44933/50000 [29:50:40<2:39:43,  1.89s/it]
Outer training loop:  90%|########9 | 44934/50000 [29:50:41<2:34:56,  1.84s/it]
Outer training loop:  90%|########9 | 44935/50000 [29:50:43<2:35:58,  1.85s/it]
Outer training loop:  90%|########9 | 44936/50000 [29:50:45<2:39:47,  1.89s/it]
Outer training loop:  90%|########9 | 44937/50000 [29:50:47<2:37:43,  1.87s/it]
Outer training loop:  90%|########9 | 44938/50000 [29:50:49<2:37:47,  1.87s/it]
Outer training loop:  90%|########9 | 44939/50000 [29:50:51<2:40:56,  1.91s/it]
Outer training loop:  90%|########9 | 44940/50000 [29:50:53<2:39:32,  1.89s/it]
Outer training loop:  90%|########9 | 44941/50000 [29:50:54<2:38:51,  1.88s/it]
Outer training loop:  90%|########9 | 44942/50000 [29:50:56<2:41:46,  1.92s/it]
Outer training loop:  90%|########9 | 44943/50000 [29:50:58<2:36:24,  1.86s/it]
Outer training loop:  90%|########9 | 44944/50000 [29:51:00<2:35:04,  1.84s/it]
Outer training loop:  90%|########9 | 44945/50000 [29:51:02<2:36:42,  1.86s/it]
Outer training loop:  90%|########9 | 44946/50000 [29:51:04<2:39:03,  1.89s/it]
Outer training loop:  90%|########9 | 44947/50000 [29:51:06<2:38:58,  1.89s/it]
Outer training loop:  90%|########9 | 44948/50000 [29:51:08<2:35:36,  1.85s/it]
Outer training loop:  90%|########9 | 44949/50000 [29:51:10<2:41:01,  1.91s/it]
Outer training loop:  90%|########9 | 44950/50000 [29:51:11<2:37:55,  1.88s/it]
Outer training loop:  90%|########9 | 44951/50000 [29:51:13<2:37:19,  1.87s/it]
Outer training loop:  90%|########9 | 44952/50000 [29:51:15<2:33:19,  1.82s/it]
Outer training loop:  90%|########9 | 44953/50000 [29:51:17<2:37:36,  1.87s/it]
Outer training loop:  90%|########9 | 44954/50000 [29:51:19<2:37:54,  1.88s/it]
Outer training loop:  90%|########9 | 44955/50000 [29:51:21<2:37:07,  1.87s/it]
Outer training loop:  90%|########9 | 44956/50000 [29:51:23<2:42:24,  1.93s/it]
Outer training loop:  90%|########9 | 44957/50000 [29:51:25<2:40:02,  1.90s/it]
Outer training loop:  90%|########9 | 44958/50000 [29:51:26<2:39:07,  1.89s/it]
Outer training loop:  90%|########9 | 44959/50000 [29:51:28<2:43:08,  1.94s/it]
Outer training loop:  90%|########9 | 44960/50000 [29:51:30<2:38:25,  1.89s/it]
Outer training loop:  90%|########9 | 44961/50000 [29:51:32<2:34:16,  1.84s/it]
Outer training loop:  90%|########9 | 44962/50000 [29:51:34<2:34:02,  1.83s/it]
Outer training loop:  90%|########9 | 44963/50000 [29:51:36<2:36:56,  1.87s/it]
Outer training loop:  90%|########9 | 44964/50000 [29:51:38<2:34:34,  1.84s/it]
Outer training loop:  90%|########9 | 44965/50000 [29:51:39<2:34:04,  1.84s/it]
Outer training loop:  90%|########9 | 44966/50000 [29:51:41<2:38:10,  1.89s/it]
Outer training loop:  90%|########9 | 44967/50000 [29:51:43<2:38:29,  1.89s/it]
Outer training loop:  90%|########9 | 44968/50000 [29:51:45<2:35:45,  1.86s/it]
Outer training loop:  90%|########9 | 44969/50000 [29:51:47<2:33:02,  1.83s/it]
Outer training loop:  90%|########9 | 44970/50000 [29:51:49<2:38:25,  1.89s/it]
Outer training loop:  90%|########9 | 44971/50000 [29:51:51<2:39:48,  1.91s/it]
Outer training loop:  90%|########9 | 44972/50000 [29:51:53<2:36:31,  1.87s/it]
Outer training loop:  90%|########9 | 44973/50000 [29:51:55<2:40:44,  1.92s/it]
Outer training loop:  90%|########9 | 44974/50000 [29:51:56<2:37:17,  1.88s/it]
Outer training loop:  90%|########9 | 44975/50000 [29:51:58<2:37:47,  1.88s/it]
Outer training loop:  90%|########9 | 44976/50000 [29:52:00<2:39:32,  1.91s/it]
Outer training loop:  90%|########9 | 44977/50000 [29:52:02<2:39:36,  1.91s/it]
Outer training loop:  90%|########9 | 44978/50000 [29:52:04<2:36:02,  1.86s/it]
Outer training loop:  90%|########9 | 44979/50000 [29:52:06<2:34:16,  1.84s/it]
Outer training loop:  90%|########9 | 44980/50000 [29:52:08<2:36:40,  1.87s/it]
Outer training loop:  90%|########9 | 44981/50000 [29:52:09<2:34:29,  1.85s/it]
Outer training loop:  90%|########9 | 44982/50000 [29:52:11<2:35:18,  1.86s/it]
Outer training loop:  90%|########9 | 44983/50000 [29:52:13<2:42:07,  1.94s/it]
Outer training loop:  90%|########9 | 44984/50000 [29:52:15<2:41:04,  1.93s/it]
Outer training loop:  90%|########9 | 44985/50000 [29:52:17<2:36:13,  1.87s/it]
Outer training loop:  90%|########9 | 44986/50000 [29:52:19<2:38:48,  1.90s/it]
Outer training loop:  90%|########9 | 44987/50000 [29:52:21<2:38:32,  1.90s/it]
Outer training loop:  90%|########9 | 44988/50000 [29:52:23<2:36:15,  1.87s/it]
Outer training loop:  90%|########9 | 44989/50000 [29:52:24<2:33:27,  1.84s/it]
Outer training loop:  90%|########9 | 44990/50000 [29:52:27<2:40:14,  1.92s/it]
Outer training loop:  90%|########9 | 44991/50000 [29:52:28<2:38:23,  1.90s/it]
Outer training loop:  90%|########9 | 44992/50000 [29:52:30<2:34:27,  1.85s/it]
Outer training loop:  90%|########9 | 44993/50000 [29:52:32<2:33:13,  1.84s/it]
Outer training loop:  90%|########9 | 44994/50000 [29:52:34<2:34:47,  1.86s/it]
Outer training loop:  90%|########9 | 44995/50000 [29:52:36<2:34:41,  1.85s/it]
Outer training loop:  90%|########9 | 44996/50000 [29:52:38<2:34:09,  1.85s/it]
Outer training loop:  90%|########9 | 44997/50000 [29:52:40<2:40:00,  1.92s/it]
Outer training loop:  90%|########9 | 44998/50000 [29:52:41<2:36:33,  1.88s/it]
Outer training loop:  90%|########9 | 44999/50000 [29:52:43<2:35:14,  1.86s/it]
Outer training loop:  90%|######### | 45000/50000 [29:52:45<2:34:28,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:28,  1.81s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.83s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.86s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:24,  1.85s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.87s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:25,  1.94s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.95s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:18,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:21,  1.99s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:17,  1.96s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:15,  1.96s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:10,  1.92s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.93s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.90s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:00,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.90s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:59,  1.93s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:56,  1.91s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:58,  1.97s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:51,  1.92s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:52,  1.98s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:49,  1.96s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.94s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:42,  1.89s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:38,  1.90s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:40,  1.97s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:36,  1.92s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:33,  1.91s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.88s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:27,  1.93s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:25,  1.94s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:20,  1.86s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:18,  1.87s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:16,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:13,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:15,  1.94s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.90s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:03,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:58,  1.82s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<01:00,  1.95s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.90s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.88s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:46,  1.88s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:46,  1.96s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:44,  1.92s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:36,  1.84s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:31,  1.96s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.91s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.85s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:21,  1.81s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:19,  1.80s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.87s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.89s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.89s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:14,  2.05s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  2.00s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.97s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.98s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.96s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.93s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.86s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:24,  1.76s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:36,  1.93s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:33,  1.92s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:30,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.86s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.93s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.90s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:23,  1.96s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.94s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:15,  1.91s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:17,  2.00s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:13,  1.96s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:03,  1.87s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:06,  1.95s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:56,  1.85s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:55,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:56,  1.95s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:52,  1.91s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.89s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:42,  1.84s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.94s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:37,  1.87s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:31,  1.82s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:35,  1.94s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.90s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:20,  1.83s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:26,  2.02s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:22,  1.96s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:20,  1.97s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:17,  1.93s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:14,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:10,  1.97s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.90s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:56,  1.95s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:54,  1.93s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:52,  1.96s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.91s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.90s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:44,  1.86s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:42,  1.85s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:40,  1.95s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:38,  1.91s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:35,  1.88s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.87s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.85s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.86s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:25,  1.97s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.92s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.81s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.82s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:12,  1.81s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:10,  1.79s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:08,  1.77s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.94s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.84s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.85s/it]

Outer training loop:  90%|######### | 45001/50000 [29:58:08<136:09:37, 98.06s/it]
Outer training loop:  90%|######### | 45002/50000 [29:58:09<96:00:36, 69.16s/it] 
Outer training loop:  90%|######### | 45003/50000 [29:58:11<67:57:08, 48.96s/it]
Outer training loop:  90%|######### | 45004/50000 [29:58:13<48:19:30, 34.82s/it]
Outer training loop:  90%|######### | 45005/50000 [29:58:15<34:34:20, 24.92s/it]
Outer training loop:  90%|######### | 45006/50000 [29:58:17<24:57:33, 17.99s/it]
Outer training loop:  90%|######### | 45007/50000 [29:58:18<18:11:49, 13.12s/it]
Outer training loop:  90%|######### | 45008/50000 [29:58:21<13:47:03,  9.94s/it]
Outer training loop:  90%|######### | 45009/50000 [29:58:23<10:24:41,  7.51s/it]
Outer training loop:  90%|######### | 45010/50000 [29:58:25<8:02:06,  5.80s/it] 
Outer training loop:  90%|######### | 45011/50000 [29:58:27<6:28:30,  4.67s/it]
Outer training loop:  90%|######### | 45012/50000 [29:58:29<5:20:43,  3.86s/it]
Outer training loop:  90%|######### | 45013/50000 [29:58:31<4:32:40,  3.28s/it]
Outer training loop:  90%|######### | 45014/50000 [29:58:32<3:52:13,  2.79s/it]
Outer training loop:  90%|######### | 45015/50000 [29:58:34<3:31:41,  2.55s/it]
Outer training loop:  90%|######### | 45016/50000 [29:58:36<3:11:58,  2.31s/it]
Outer training loop:  90%|######### | 45017/50000 [29:58:38<2:58:00,  2.14s/it]
Outer training loop:  90%|######### | 45018/50000 [29:58:40<2:53:42,  2.09s/it]
Outer training loop:  90%|######### | 45019/50000 [29:58:41<2:45:41,  2.00s/it]
Outer training loop:  90%|######### | 45020/50000 [29:58:43<2:44:29,  1.98s/it]
Outer training loop:  90%|######### | 45021/50000 [29:58:45<2:42:14,  1.96s/it]
Outer training loop:  90%|######### | 45022/50000 [29:58:47<2:44:03,  1.98s/it]
Outer training loop:  90%|######### | 45023/50000 [29:58:49<2:40:12,  1.93s/it]
Outer training loop:  90%|######### | 45024/50000 [29:58:51<2:38:07,  1.91s/it]
Outer training loop:  90%|######### | 45025/50000 [29:58:53<2:38:31,  1.91s/it]
Outer training loop:  90%|######### | 45026/50000 [29:58:55<2:33:01,  1.85s/it]
Outer training loop:  90%|######### | 45027/50000 [29:58:56<2:29:00,  1.80s/it]
Outer training loop:  90%|######### | 45028/50000 [29:58:58<2:30:07,  1.81s/it]
Outer training loop:  90%|######### | 45029/50000 [29:59:00<2:38:00,  1.91s/it]
Outer training loop:  90%|######### | 45030/50000 [29:59:02<2:34:03,  1.86s/it]
Outer training loop:  90%|######### | 45031/50000 [29:59:04<2:33:24,  1.85s/it]
Outer training loop:  90%|######### | 45032/50000 [29:59:06<2:40:54,  1.94s/it]
Outer training loop:  90%|######### | 45033/50000 [29:59:08<2:36:06,  1.89s/it]
Outer training loop:  90%|######### | 45034/50000 [29:59:10<2:37:27,  1.90s/it]
Outer training loop:  90%|######### | 45035/50000 [29:59:12<2:37:37,  1.90s/it]
Outer training loop:  90%|######### | 45036/50000 [29:59:13<2:36:52,  1.90s/it]
Outer training loop:  90%|######### | 45037/50000 [29:59:15<2:32:56,  1.85s/it]
Outer training loop:  90%|######### | 45038/50000 [29:59:17<2:30:46,  1.82s/it]
Outer training loop:  90%|######### | 45039/50000 [29:59:19<2:34:59,  1.87s/it]
Outer training loop:  90%|######### | 45040/50000 [29:59:21<2:30:39,  1.82s/it]
Outer training loop:  90%|######### | 45041/50000 [29:59:22<2:30:56,  1.83s/it]
Outer training loop:  90%|######### | 45042/50000 [29:59:25<2:36:01,  1.89s/it]
Outer training loop:  90%|######### | 45043/50000 [29:59:27<2:38:47,  1.92s/it]
Outer training loop:  90%|######### | 45044/50000 [29:59:28<2:40:17,  1.94s/it]
Outer training loop:  90%|######### | 45045/50000 [29:59:31<2:43:15,  1.98s/it]
Outer training loop:  90%|######### | 45046/50000 [29:59:32<2:38:45,  1.92s/it]
Outer training loop:  90%|######### | 45047/50000 [29:59:34<2:36:14,  1.89s/it]
Outer training loop:  90%|######### | 45048/50000 [29:59:36<2:42:08,  1.96s/it]
Outer training loop:  90%|######### | 45049/50000 [29:59:38<2:39:28,  1.93s/it]
Outer training loop:  90%|######### | 45050/50000 [29:59:40<2:38:03,  1.92s/it]
Outer training loop:  90%|######### | 45051/50000 [29:59:42<2:39:56,  1.94s/it]
Outer training loop:  90%|######### | 45052/50000 [29:59:44<2:38:18,  1.92s/it]
Outer training loop:  90%|######### | 45053/50000 [29:59:46<2:35:07,  1.88s/it]
Outer training loop:  90%|######### | 45054/50000 [29:59:47<2:33:07,  1.86s/it]
Outer training loop:  90%|######### | 45055/50000 [29:59:49<2:34:53,  1.88s/it]
Outer training loop:  90%|######### | 45056/50000 [29:59:51<2:34:28,  1.87s/it]
Outer training loop:  90%|######### | 45057/50000 [29:59:53<2:32:39,  1.85s/it]
Outer training loop:  90%|######### | 45058/50000 [29:59:55<2:39:28,  1.94s/it]
Outer training loop:  90%|######### | 45059/50000 [29:59:57<2:39:41,  1.94s/it]
Outer training loop:  90%|######### | 45060/50000 [29:59:59<2:36:55,  1.91s/it]
Outer training loop:  90%|######### | 45061/50000 [30:00:01<2:34:40,  1.88s/it]
Outer training loop:  90%|######### | 45062/50000 [30:00:03<2:37:26,  1.91s/it]
Outer training loop:  90%|######### | 45063/50000 [30:00:05<2:34:05,  1.87s/it]
Outer training loop:  90%|######### | 45064/50000 [30:00:06<2:34:27,  1.88s/it]
Outer training loop:  90%|######### | 45065/50000 [30:00:09<2:38:05,  1.92s/it]
Outer training loop:  90%|######### | 45066/50000 [30:00:10<2:33:45,  1.87s/it]
Outer training loop:  90%|######### | 45067/50000 [30:00:12<2:32:25,  1.85s/it]
Outer training loop:  90%|######### | 45068/50000 [30:00:14<2:37:54,  1.92s/it]
Outer training loop:  90%|######### | 45069/50000 [30:00:16<2:36:26,  1.90s/it]
Outer training loop:  90%|######### | 45070/50000 [30:00:18<2:35:46,  1.90s/it]
Outer training loop:  90%|######### | 45071/50000 [30:00:20<2:34:58,  1.89s/it]
Outer training loop:  90%|######### | 45072/50000 [30:00:22<2:37:05,  1.91s/it]
Outer training loop:  90%|######### | 45073/50000 [30:00:24<2:34:58,  1.89s/it]
Outer training loop:  90%|######### | 45074/50000 [30:00:25<2:35:19,  1.89s/it]
Outer training loop:  90%|######### | 45075/50000 [30:00:27<2:36:09,  1.90s/it]
Outer training loop:  90%|######### | 45076/50000 [30:00:29<2:34:01,  1.88s/it]
Outer training loop:  90%|######### | 45077/50000 [30:00:31<2:30:56,  1.84s/it]
Outer training loop:  90%|######### | 45078/50000 [30:00:33<2:30:38,  1.84s/it]
Outer training loop:  90%|######### | 45079/50000 [30:00:35<2:37:48,  1.92s/it]
Outer training loop:  90%|######### | 45080/50000 [30:00:37<2:35:30,  1.90s/it]
Outer training loop:  90%|######### | 45081/50000 [30:00:38<2:31:52,  1.85s/it]
Outer training loop:  90%|######### | 45082/50000 [30:00:41<2:36:14,  1.91s/it]
Outer training loop:  90%|######### | 45083/50000 [30:00:42<2:35:20,  1.90s/it]
Outer training loop:  90%|######### | 45084/50000 [30:00:44<2:31:21,  1.85s/it]
Outer training loop:  90%|######### | 45085/50000 [30:00:46<2:34:09,  1.88s/it]
Outer training loop:  90%|######### | 45086/50000 [30:00:48<2:29:35,  1.83s/it]
Outer training loop:  90%|######### | 45087/50000 [30:00:50<2:30:30,  1.84s/it]
Outer training loop:  90%|######### | 45088/50000 [30:00:52<2:34:32,  1.89s/it]
Outer training loop:  90%|######### | 45089/50000 [30:00:53<2:33:17,  1.87s/it]
Outer training loop:  90%|######### | 45090/50000 [30:00:55<2:32:44,  1.87s/it]
Outer training loop:  90%|######### | 45091/50000 [30:00:57<2:30:59,  1.85s/it]
Outer training loop:  90%|######### | 45092/50000 [30:00:59<2:34:52,  1.89s/it]
Outer training loop:  90%|######### | 45093/50000 [30:01:01<2:30:46,  1.84s/it]
Outer training loop:  90%|######### | 45094/50000 [30:01:03<2:30:37,  1.84s/it]
Outer training loop:  90%|######### | 45095/50000 [30:01:05<2:39:46,  1.95s/it]
Outer training loop:  90%|######### | 45096/50000 [30:01:07<2:38:46,  1.94s/it]
Outer training loop:  90%|######### | 45097/50000 [30:01:09<2:34:21,  1.89s/it]
Outer training loop:  90%|######### | 45098/50000 [30:01:11<2:39:58,  1.96s/it]
Outer training loop:  90%|######### | 45099/50000 [30:01:13<2:39:22,  1.95s/it]
Outer training loop:  90%|######### | 45100/50000 [30:01:14<2:35:49,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.90s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  90%|######### | 45101/50000 [30:01:22<4:51:27,  3.57s/it]
Outer training loop:  90%|######### | 45102/50000 [30:01:24<4:14:26,  3.12s/it]
Outer training loop:  90%|######### | 45103/50000 [30:01:26<3:42:35,  2.73s/it]
Outer training loop:  90%|######### | 45104/50000 [30:01:28<3:20:14,  2.45s/it]
Outer training loop:  90%|######### | 45105/50000 [30:01:30<3:07:24,  2.30s/it]
Outer training loop:  90%|######### | 45106/50000 [30:01:31<2:56:50,  2.17s/it]
Outer training loop:  90%|######### | 45107/50000 [30:01:33<2:48:29,  2.07s/it]
Outer training loop:  90%|######### | 45108/50000 [30:01:35<2:43:16,  2.00s/it]
Outer training loop:  90%|######### | 45109/50000 [30:01:37<2:40:12,  1.97s/it]
Outer training loop:  90%|######### | 45110/50000 [30:01:39<2:43:38,  2.01s/it]
Outer training loop:  90%|######### | 45111/50000 [30:01:41<2:39:37,  1.96s/it]
Outer training loop:  90%|######### | 45112/50000 [30:01:43<2:38:03,  1.94s/it]
Outer training loop:  90%|######### | 45113/50000 [30:01:45<2:37:49,  1.94s/it]
Outer training loop:  90%|######### | 45114/50000 [30:01:47<2:34:49,  1.90s/it]
Outer training loop:  90%|######### | 45115/50000 [30:01:48<2:34:55,  1.90s/it]
Outer training loop:  90%|######### | 45116/50000 [30:01:50<2:32:59,  1.88s/it]
Outer training loop:  90%|######### | 45117/50000 [30:01:52<2:35:50,  1.91s/it]
Outer training loop:  90%|######### | 45118/50000 [30:01:54<2:32:20,  1.87s/it]
Outer training loop:  90%|######### | 45119/50000 [30:01:56<2:33:15,  1.88s/it]
Outer training loop:  90%|######### | 45120/50000 [30:01:58<2:39:11,  1.96s/it]
Outer training loop:  90%|######### | 45121/50000 [30:02:00<2:36:30,  1.92s/it]
Outer training loop:  90%|######### | 45122/50000 [30:02:02<2:34:07,  1.90s/it]
Outer training loop:  90%|######### | 45123/50000 [30:02:04<2:36:13,  1.92s/it]
Outer training loop:  90%|######### | 45124/50000 [30:02:06<2:36:34,  1.93s/it]
Outer training loop:  90%|######### | 45125/50000 [30:02:08<2:34:15,  1.90s/it]
Outer training loop:  90%|######### | 45126/50000 [30:02:10<2:35:52,  1.92s/it]
Outer training loop:  90%|######### | 45127/50000 [30:02:11<2:34:52,  1.91s/it]
Outer training loop:  90%|######### | 45128/50000 [30:02:13<2:32:00,  1.87s/it]
Outer training loop:  90%|######### | 45129/50000 [30:02:15<2:31:04,  1.86s/it]
Outer training loop:  90%|######### | 45130/50000 [30:02:17<2:34:29,  1.90s/it]
Outer training loop:  90%|######### | 45131/50000 [30:02:19<2:34:40,  1.91s/it]
Outer training loop:  90%|######### | 45132/50000 [30:02:21<2:35:34,  1.92s/it]
Outer training loop:  90%|######### | 45133/50000 [30:02:23<2:29:32,  1.84s/it]
Outer training loop:  90%|######### | 45134/50000 [30:02:25<2:36:11,  1.93s/it]
Outer training loop:  90%|######### | 45135/50000 [30:02:27<2:34:54,  1.91s/it]
Outer training loop:  90%|######### | 45136/50000 [30:02:28<2:32:26,  1.88s/it]
Outer training loop:  90%|######### | 45137/50000 [30:02:30<2:36:38,  1.93s/it]
Outer training loop:  90%|######### | 45138/50000 [30:02:32<2:35:49,  1.92s/it]
Outer training loop:  90%|######### | 45139/50000 [30:02:34<2:32:41,  1.88s/it]
Outer training loop:  90%|######### | 45140/50000 [30:02:36<2:30:35,  1.86s/it]
Outer training loop:  90%|######### | 45141/50000 [30:02:38<2:33:24,  1.89s/it]
Outer training loop:  90%|######### | 45142/50000 [30:02:40<2:30:57,  1.86s/it]
Outer training loop:  90%|######### | 45143/50000 [30:02:41<2:28:54,  1.84s/it]
Outer training loop:  90%|######### | 45144/50000 [30:02:44<2:34:16,  1.91s/it]
Outer training loop:  90%|######### | 45145/50000 [30:02:45<2:31:51,  1.88s/it]
Outer training loop:  90%|######### | 45146/50000 [30:02:47<2:28:39,  1.84s/it]
Outer training loop:  90%|######### | 45147/50000 [30:02:49<2:28:14,  1.83s/it]
Outer training loop:  90%|######### | 45148/50000 [30:02:51<2:30:10,  1.86s/it]
Outer training loop:  90%|######### | 45149/50000 [30:02:53<2:30:00,  1.86s/it]
Outer training loop:  90%|######### | 45150/50000 [30:02:54<2:29:08,  1.84s/it]
Outer training loop:  90%|######### | 45151/50000 [30:02:56<2:31:16,  1.87s/it]
Outer training loop:  90%|######### | 45152/50000 [30:02:58<2:29:08,  1.85s/it]
Outer training loop:  90%|######### | 45153/50000 [30:03:00<2:27:08,  1.82s/it]
Outer training loop:  90%|######### | 45154/50000 [30:03:02<2:28:21,  1.84s/it]
Outer training loop:  90%|######### | 45155/50000 [30:03:04<2:39:23,  1.97s/it]
Outer training loop:  90%|######### | 45156/50000 [30:03:06<2:35:52,  1.93s/it]
Outer training loop:  90%|######### | 45157/50000 [30:03:08<2:32:08,  1.88s/it]
Outer training loop:  90%|######### | 45158/50000 [30:03:10<2:38:46,  1.97s/it]
Outer training loop:  90%|######### | 45159/50000 [30:03:12<2:35:05,  1.92s/it]
Outer training loop:  90%|######### | 45160/50000 [30:03:13<2:31:42,  1.88s/it]
Outer training loop:  90%|######### | 45161/50000 [30:03:16<2:35:29,  1.93s/it]
Outer training loop:  90%|######### | 45162/50000 [30:03:17<2:31:13,  1.88s/it]
Outer training loop:  90%|######### | 45163/50000 [30:03:19<2:32:20,  1.89s/it]
Outer training loop:  90%|######### | 45164/50000 [30:03:21<2:33:00,  1.90s/it]
Outer training loop:  90%|######### | 45165/50000 [30:03:23<2:37:43,  1.96s/it]
Outer training loop:  90%|######### | 45166/50000 [30:03:25<2:36:25,  1.94s/it]
Outer training loop:  90%|######### | 45167/50000 [30:03:27<2:34:17,  1.92s/it]
Outer training loop:  90%|######### | 45168/50000 [30:03:29<2:36:05,  1.94s/it]
Outer training loop:  90%|######### | 45169/50000 [30:03:31<2:34:56,  1.92s/it]
Outer training loop:  90%|######### | 45170/50000 [30:03:33<2:33:58,  1.91s/it]
Outer training loop:  90%|######### | 45171/50000 [30:03:35<2:35:18,  1.93s/it]
Outer training loop:  90%|######### | 45172/50000 [30:03:37<2:33:17,  1.91s/it]
Outer training loop:  90%|######### | 45173/50000 [30:03:38<2:29:49,  1.86s/it]
Outer training loop:  90%|######### | 45174/50000 [30:03:40<2:35:42,  1.94s/it]
Outer training loop:  90%|######### | 45175/50000 [30:03:42<2:32:50,  1.90s/it]
Outer training loop:  90%|######### | 45176/50000 [30:03:44<2:30:16,  1.87s/it]
Outer training loop:  90%|######### | 45177/50000 [30:03:46<2:29:22,  1.86s/it]
Outer training loop:  90%|######### | 45178/50000 [30:03:48<2:33:12,  1.91s/it]
Outer training loop:  90%|######### | 45179/50000 [30:03:50<2:35:15,  1.93s/it]
Outer training loop:  90%|######### | 45180/50000 [30:03:52<2:32:39,  1.90s/it]
Outer training loop:  90%|######### | 45181/50000 [30:03:54<2:35:38,  1.94s/it]
Outer training loop:  90%|######### | 45182/50000 [30:03:56<2:34:46,  1.93s/it]
Outer training loop:  90%|######### | 45183/50000 [30:03:57<2:32:02,  1.89s/it]
Outer training loop:  90%|######### | 45184/50000 [30:03:59<2:33:53,  1.92s/it]
Outer training loop:  90%|######### | 45185/50000 [30:04:01<2:32:37,  1.90s/it]
Outer training loop:  90%|######### | 45186/50000 [30:04:03<2:29:38,  1.87s/it]
Outer training loop:  90%|######### | 45187/50000 [30:04:05<2:32:02,  1.90s/it]
Outer training loop:  90%|######### | 45188/50000 [30:04:07<2:38:05,  1.97s/it]
Outer training loop:  90%|######### | 45189/50000 [30:04:09<2:34:58,  1.93s/it]
Outer training loop:  90%|######### | 45190/50000 [30:04:11<2:32:18,  1.90s/it]
Outer training loop:  90%|######### | 45191/50000 [30:04:13<2:37:55,  1.97s/it]
Outer training loop:  90%|######### | 45192/50000 [30:04:15<2:31:59,  1.90s/it]
Outer training loop:  90%|######### | 45193/50000 [30:04:17<2:30:13,  1.88s/it]
Outer training loop:  90%|######### | 45194/50000 [30:04:18<2:27:43,  1.84s/it]
Outer training loop:  90%|######### | 45195/50000 [30:04:20<2:31:11,  1.89s/it]
Outer training loop:  90%|######### | 45196/50000 [30:04:22<2:29:13,  1.86s/it]
Outer training loop:  90%|######### | 45197/50000 [30:04:24<2:27:46,  1.85s/it]
Outer training loop:  90%|######### | 45198/50000 [30:04:26<2:27:47,  1.85s/it]
Outer training loop:  90%|######### | 45199/50000 [30:04:28<2:29:43,  1.87s/it]
Outer training loop:  90%|######### | 45200/50000 [30:04:30<2:28:27,  1.86s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  90%|######### | 45201/50000 [30:04:37<4:38:25,  3.48s/it]
Outer training loop:  90%|######### | 45202/50000 [30:04:39<4:04:11,  3.05s/it]
Outer training loop:  90%|######### | 45203/50000 [30:04:41<3:35:57,  2.70s/it]
Outer training loop:  90%|######### | 45204/50000 [30:04:43<3:14:26,  2.43s/it]
Outer training loop:  90%|######### | 45205/50000 [30:04:44<2:59:44,  2.25s/it]
Outer training loop:  90%|######### | 45206/50000 [30:04:46<2:52:33,  2.16s/it]
Outer training loop:  90%|######### | 45207/50000 [30:04:48<2:44:23,  2.06s/it]
Outer training loop:  90%|######### | 45208/50000 [30:04:50<2:43:17,  2.04s/it]
Outer training loop:  90%|######### | 45209/50000 [30:04:52<2:36:38,  1.96s/it]
Outer training loop:  90%|######### | 45210/50000 [30:04:54<2:38:39,  1.99s/it]
Outer training loop:  90%|######### | 45211/50000 [30:04:56<2:34:48,  1.94s/it]
Outer training loop:  90%|######### | 45212/50000 [30:04:58<2:34:09,  1.93s/it]
Outer training loop:  90%|######### | 45213/50000 [30:05:00<2:39:40,  2.00s/it]
Outer training loop:  90%|######### | 45214/50000 [30:05:02<2:39:26,  2.00s/it]
Outer training loop:  90%|######### | 45215/50000 [30:05:04<2:36:49,  1.97s/it]
Outer training loop:  90%|######### | 45216/50000 [30:05:06<2:38:03,  1.98s/it]
Outer training loop:  90%|######### | 45217/50000 [30:05:08<2:35:57,  1.96s/it]
Outer training loop:  90%|######### | 45218/50000 [30:05:10<2:35:09,  1.95s/it]
Outer training loop:  90%|######### | 45219/50000 [30:05:11<2:33:09,  1.92s/it]
Outer training loop:  90%|######### | 45220/50000 [30:05:14<2:36:45,  1.97s/it]
Outer training loop:  90%|######### | 45221/50000 [30:05:15<2:31:53,  1.91s/it]
Outer training loop:  90%|######### | 45222/50000 [30:05:17<2:32:59,  1.92s/it]
Outer training loop:  90%|######### | 45223/50000 [30:05:19<2:35:41,  1.96s/it]
Outer training loop:  90%|######### | 45224/50000 [30:05:21<2:31:53,  1.91s/it]
Outer training loop:  90%|######### | 45225/50000 [30:05:23<2:33:11,  1.92s/it]
Outer training loop:  90%|######### | 45226/50000 [30:05:25<2:35:09,  1.95s/it]
Outer training loop:  90%|######### | 45227/50000 [30:05:27<2:39:43,  2.01s/it]
Outer training loop:  90%|######### | 45228/50000 [30:05:29<2:37:41,  1.98s/it]
Outer training loop:  90%|######### | 45229/50000 [30:05:31<2:34:16,  1.94s/it]
Outer training loop:  90%|######### | 45230/50000 [30:05:33<2:34:58,  1.95s/it]
Outer training loop:  90%|######### | 45231/50000 [30:05:35<2:35:26,  1.96s/it]
Outer training loop:  90%|######### | 45232/50000 [30:05:37<2:29:42,  1.88s/it]
Outer training loop:  90%|######### | 45233/50000 [30:05:38<2:27:27,  1.86s/it]
Outer training loop:  90%|######### | 45234/50000 [30:05:40<2:29:20,  1.88s/it]
Outer training loop:  90%|######### | 45235/50000 [30:05:42<2:29:24,  1.88s/it]
Outer training loop:  90%|######### | 45236/50000 [30:05:44<2:31:48,  1.91s/it]
Outer training loop:  90%|######### | 45237/50000 [30:05:46<2:33:52,  1.94s/it]
Outer training loop:  90%|######### | 45238/50000 [30:05:48<2:27:50,  1.86s/it]
Outer training loop:  90%|######### | 45239/50000 [30:05:50<2:28:02,  1.87s/it]
Outer training loop:  90%|######### | 45240/50000 [30:05:52<2:30:35,  1.90s/it]
Outer training loop:  90%|######### | 45241/50000 [30:05:53<2:27:04,  1.85s/it]
Outer training loop:  90%|######### | 45242/50000 [30:05:55<2:23:04,  1.80s/it]
Outer training loop:  90%|######### | 45243/50000 [30:05:57<2:23:29,  1.81s/it]
Outer training loop:  90%|######### | 45244/50000 [30:05:59<2:30:34,  1.90s/it]
Outer training loop:  90%|######### | 45245/50000 [30:06:01<2:28:08,  1.87s/it]
Outer training loop:  90%|######### | 45246/50000 [30:06:03<2:24:46,  1.83s/it]
Outer training loop:  90%|######### | 45247/50000 [30:06:05<2:29:32,  1.89s/it]
Outer training loop:  90%|######### | 45248/50000 [30:06:07<2:29:17,  1.89s/it]
Outer training loop:  90%|######### | 45249/50000 [30:06:08<2:26:47,  1.85s/it]
Outer training loop:  90%|######### | 45250/50000 [30:06:10<2:28:31,  1.88s/it]
Outer training loop:  91%|######### | 45251/50000 [30:06:12<2:34:13,  1.95s/it]
Outer training loop:  91%|######### | 45252/50000 [30:06:14<2:34:54,  1.96s/it]
Outer training loop:  91%|######### | 45253/50000 [30:06:16<2:30:01,  1.90s/it]
Outer training loop:  91%|######### | 45254/50000 [30:06:18<2:33:41,  1.94s/it]
Outer training loop:  91%|######### | 45255/50000 [30:06:20<2:33:28,  1.94s/it]
Outer training loop:  91%|######### | 45256/50000 [30:06:22<2:32:53,  1.93s/it]
Outer training loop:  91%|######### | 45257/50000 [30:06:24<2:37:20,  1.99s/it]
Outer training loop:  91%|######### | 45258/50000 [30:06:26<2:34:03,  1.95s/it]
Outer training loop:  91%|######### | 45259/50000 [30:06:28<2:32:54,  1.94s/it]
Outer training loop:  91%|######### | 45260/50000 [30:06:30<2:30:09,  1.90s/it]
Outer training loop:  91%|######### | 45261/50000 [30:06:32<2:34:20,  1.95s/it]
Outer training loop:  91%|######### | 45262/50000 [30:06:34<2:34:35,  1.96s/it]
Outer training loop:  91%|######### | 45263/50000 [30:06:36<2:31:59,  1.93s/it]
Outer training loop:  91%|######### | 45264/50000 [30:06:38<2:34:42,  1.96s/it]
Outer training loop:  91%|######### | 45265/50000 [30:06:40<2:33:15,  1.94s/it]
Outer training loop:  91%|######### | 45266/50000 [30:06:41<2:32:22,  1.93s/it]
Outer training loop:  91%|######### | 45267/50000 [30:06:43<2:30:38,  1.91s/it]
Outer training loop:  91%|######### | 45268/50000 [30:06:45<2:31:05,  1.92s/it]
Outer training loop:  91%|######### | 45269/50000 [30:06:47<2:28:33,  1.88s/it]
Outer training loop:  91%|######### | 45270/50000 [30:06:49<2:24:16,  1.83s/it]
Outer training loop:  91%|######### | 45271/50000 [30:06:51<2:30:09,  1.91s/it]
Outer training loop:  91%|######### | 45272/50000 [30:06:53<2:27:58,  1.88s/it]
Outer training loop:  91%|######### | 45273/50000 [30:06:55<2:28:02,  1.88s/it]
Outer training loop:  91%|######### | 45274/50000 [30:06:56<2:29:06,  1.89s/it]
Outer training loop:  91%|######### | 45275/50000 [30:06:59<2:33:43,  1.95s/it]
Outer training loop:  91%|######### | 45276/50000 [30:07:00<2:30:33,  1.91s/it]
Outer training loop:  91%|######### | 45277/50000 [30:07:02<2:27:11,  1.87s/it]
Outer training loop:  91%|######### | 45278/50000 [30:07:04<2:32:48,  1.94s/it]
Outer training loop:  91%|######### | 45279/50000 [30:07:06<2:33:58,  1.96s/it]
Outer training loop:  91%|######### | 45280/50000 [30:07:08<2:29:33,  1.90s/it]
Outer training loop:  91%|######### | 45281/50000 [30:07:10<2:29:36,  1.90s/it]
Outer training loop:  91%|######### | 45282/50000 [30:07:12<2:31:52,  1.93s/it]
Outer training loop:  91%|######### | 45283/50000 [30:07:14<2:29:59,  1.91s/it]
Outer training loop:  91%|######### | 45284/50000 [30:07:16<2:27:36,  1.88s/it]
Outer training loop:  91%|######### | 45285/50000 [30:07:18<2:30:32,  1.92s/it]
Outer training loop:  91%|######### | 45286/50000 [30:07:19<2:27:08,  1.87s/it]
Outer training loop:  91%|######### | 45287/50000 [30:07:21<2:29:53,  1.91s/it]
Outer training loop:  91%|######### | 45288/50000 [30:07:23<2:28:24,  1.89s/it]
Outer training loop:  91%|######### | 45289/50000 [30:07:25<2:35:06,  1.98s/it]
Outer training loop:  91%|######### | 45290/50000 [30:07:27<2:32:11,  1.94s/it]
Outer training loop:  91%|######### | 45291/50000 [30:07:29<2:30:31,  1.92s/it]
Outer training loop:  91%|######### | 45292/50000 [30:07:31<2:30:56,  1.92s/it]
Outer training loop:  91%|######### | 45293/50000 [30:07:33<2:26:30,  1.87s/it]
Outer training loop:  91%|######### | 45294/50000 [30:07:35<2:24:09,  1.84s/it]
Outer training loop:  91%|######### | 45295/50000 [30:07:36<2:26:04,  1.86s/it]
Outer training loop:  91%|######### | 45296/50000 [30:07:38<2:26:02,  1.86s/it]
Outer training loop:  91%|######### | 45297/50000 [30:07:40<2:25:44,  1.86s/it]
Outer training loop:  91%|######### | 45298/50000 [30:07:42<2:23:30,  1.83s/it]
Outer training loop:  91%|######### | 45299/50000 [30:07:44<2:28:05,  1.89s/it]
Outer training loop:  91%|######### | 45300/50000 [30:07:46<2:24:37,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.19s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:02,  2.12s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.96s/it]

Outer training loop:  91%|######### | 45301/50000 [30:07:53<4:37:12,  3.54s/it]
Outer training loop:  91%|######### | 45302/50000 [30:07:55<3:58:54,  3.05s/it]
Outer training loop:  91%|######### | 45303/50000 [30:07:57<3:34:07,  2.74s/it]
Outer training loop:  91%|######### | 45304/50000 [30:07:59<3:13:21,  2.47s/it]
Outer training loop:  91%|######### | 45305/50000 [30:08:01<2:56:11,  2.25s/it]
Outer training loop:  91%|######### | 45306/50000 [30:08:03<2:54:28,  2.23s/it]
Outer training loop:  91%|######### | 45307/50000 [30:08:05<2:46:37,  2.13s/it]
Outer training loop:  91%|######### | 45308/50000 [30:08:07<2:40:00,  2.05s/it]
Outer training loop:  91%|######### | 45309/50000 [30:08:09<2:38:23,  2.03s/it]
Outer training loop:  91%|######### | 45310/50000 [30:08:10<2:32:46,  1.95s/it]
Outer training loop:  91%|######### | 45311/50000 [30:08:12<2:26:29,  1.87s/it]
Outer training loop:  91%|######### | 45312/50000 [30:08:14<2:26:30,  1.88s/it]
Outer training loop:  91%|######### | 45313/50000 [30:08:16<2:29:26,  1.91s/it]
Outer training loop:  91%|######### | 45314/50000 [30:08:18<2:26:35,  1.88s/it]
Outer training loop:  91%|######### | 45315/50000 [30:08:19<2:23:12,  1.83s/it]
Outer training loop:  91%|######### | 45316/50000 [30:08:22<2:28:46,  1.91s/it]
Outer training loop:  91%|######### | 45317/50000 [30:08:23<2:27:52,  1.89s/it]
Outer training loop:  91%|######### | 45318/50000 [30:08:25<2:27:49,  1.89s/it]
Outer training loop:  91%|######### | 45319/50000 [30:08:27<2:24:13,  1.85s/it]
Outer training loop:  91%|######### | 45320/50000 [30:08:29<2:28:43,  1.91s/it]
Outer training loop:  91%|######### | 45321/50000 [30:08:31<2:26:23,  1.88s/it]
Outer training loop:  91%|######### | 45322/50000 [30:08:33<2:24:59,  1.86s/it]
Outer training loop:  91%|######### | 45323/50000 [30:08:35<2:26:04,  1.87s/it]
Outer training loop:  91%|######### | 45324/50000 [30:08:37<2:32:42,  1.96s/it]
Outer training loop:  91%|######### | 45325/50000 [30:08:39<2:28:39,  1.91s/it]
Outer training loop:  91%|######### | 45326/50000 [30:08:41<2:29:46,  1.92s/it]
Outer training loop:  91%|######### | 45327/50000 [30:08:43<2:34:07,  1.98s/it]
Outer training loop:  91%|######### | 45328/50000 [30:08:44<2:28:39,  1.91s/it]
Outer training loop:  91%|######### | 45329/50000 [30:08:46<2:25:02,  1.86s/it]
Outer training loop:  91%|######### | 45330/50000 [30:08:48<2:28:02,  1.90s/it]
Outer training loop:  91%|######### | 45331/50000 [30:08:50<2:27:03,  1.89s/it]
Outer training loop:  91%|######### | 45332/50000 [30:08:52<2:25:01,  1.86s/it]
Outer training loop:  91%|######### | 45333/50000 [30:08:53<2:20:27,  1.81s/it]
Outer training loop:  91%|######### | 45334/50000 [30:08:56<2:26:27,  1.88s/it]
Outer training loop:  91%|######### | 45335/50000 [30:08:57<2:23:59,  1.85s/it]
Outer training loop:  91%|######### | 45336/50000 [30:08:59<2:24:19,  1.86s/it]
Outer training loop:  91%|######### | 45337/50000 [30:09:01<2:29:27,  1.92s/it]
Outer training loop:  91%|######### | 45338/50000 [30:09:03<2:25:43,  1.88s/it]
Outer training loop:  91%|######### | 45339/50000 [30:09:05<2:27:24,  1.90s/it]
Outer training loop:  91%|######### | 45340/50000 [30:09:07<2:30:06,  1.93s/it]
Outer training loop:  91%|######### | 45341/50000 [30:09:09<2:25:46,  1.88s/it]
Outer training loop:  91%|######### | 45342/50000 [30:09:10<2:21:59,  1.83s/it]
Outer training loop:  91%|######### | 45343/50000 [30:09:13<2:28:11,  1.91s/it]
Outer training loop:  91%|######### | 45344/50000 [30:09:14<2:25:07,  1.87s/it]
Outer training loop:  91%|######### | 45345/50000 [30:09:16<2:23:35,  1.85s/it]
Outer training loop:  91%|######### | 45346/50000 [30:09:18<2:23:09,  1.85s/it]
Outer training loop:  91%|######### | 45347/50000 [30:09:20<2:28:30,  1.91s/it]
Outer training loop:  91%|######### | 45348/50000 [30:09:22<2:23:47,  1.85s/it]
Outer training loop:  91%|######### | 45349/50000 [30:09:24<2:29:11,  1.92s/it]
Outer training loop:  91%|######### | 45350/50000 [30:09:26<2:33:53,  1.99s/it]
Outer training loop:  91%|######### | 45351/50000 [30:09:28<2:28:47,  1.92s/it]
Outer training loop:  91%|######### | 45352/50000 [30:09:30<2:25:46,  1.88s/it]
Outer training loop:  91%|######### | 45353/50000 [30:09:32<2:30:19,  1.94s/it]
Outer training loop:  91%|######### | 45354/50000 [30:09:33<2:26:18,  1.89s/it]
Outer training loop:  91%|######### | 45355/50000 [30:09:35<2:26:41,  1.89s/it]
Outer training loop:  91%|######### | 45356/50000 [30:09:37<2:30:27,  1.94s/it]
Outer training loop:  91%|######### | 45357/50000 [30:09:39<2:26:24,  1.89s/it]
Outer training loop:  91%|######### | 45358/50000 [30:09:41<2:24:28,  1.87s/it]
Outer training loop:  91%|######### | 45359/50000 [30:09:43<2:24:28,  1.87s/it]
Outer training loop:  91%|######### | 45360/50000 [30:09:45<2:28:10,  1.92s/it]
Outer training loop:  91%|######### | 45361/50000 [30:09:47<2:25:27,  1.88s/it]
Outer training loop:  91%|######### | 45362/50000 [30:09:48<2:24:09,  1.86s/it]
Outer training loop:  91%|######### | 45363/50000 [30:09:50<2:26:28,  1.90s/it]
Outer training loop:  91%|######### | 45364/50000 [30:09:52<2:28:58,  1.93s/it]
Outer training loop:  91%|######### | 45365/50000 [30:09:54<2:24:19,  1.87s/it]
Outer training loop:  91%|######### | 45366/50000 [30:09:56<2:24:11,  1.87s/it]
Outer training loop:  91%|######### | 45367/50000 [30:09:58<2:29:47,  1.94s/it]
Outer training loop:  91%|######### | 45368/50000 [30:10:00<2:29:24,  1.94s/it]
Outer training loop:  91%|######### | 45369/50000 [30:10:02<2:26:34,  1.90s/it]
Outer training loop:  91%|######### | 45370/50000 [30:10:04<2:28:14,  1.92s/it]
Outer training loop:  91%|######### | 45371/50000 [30:10:06<2:30:51,  1.96s/it]
Outer training loop:  91%|######### | 45372/50000 [30:10:08<2:25:50,  1.89s/it]
Outer training loop:  91%|######### | 45373/50000 [30:10:10<2:29:11,  1.93s/it]
Outer training loop:  91%|######### | 45374/50000 [30:10:11<2:24:17,  1.87s/it]
Outer training loop:  91%|######### | 45375/50000 [30:10:13<2:23:37,  1.86s/it]
Outer training loop:  91%|######### | 45376/50000 [30:10:15<2:25:09,  1.88s/it]
Outer training loop:  91%|######### | 45377/50000 [30:10:17<2:23:46,  1.87s/it]
Outer training loop:  91%|######### | 45378/50000 [30:10:19<2:24:29,  1.88s/it]
Outer training loop:  91%|######### | 45379/50000 [30:10:21<2:24:53,  1.88s/it]
Outer training loop:  91%|######### | 45380/50000 [30:10:23<2:28:09,  1.92s/it]
Outer training loop:  91%|######### | 45381/50000 [30:10:25<2:28:06,  1.92s/it]
Outer training loop:  91%|######### | 45382/50000 [30:10:27<2:27:24,  1.92s/it]
Outer training loop:  91%|######### | 45383/50000 [30:10:29<2:29:10,  1.94s/it]
Outer training loop:  91%|######### | 45384/50000 [30:10:30<2:27:39,  1.92s/it]
Outer training loop:  91%|######### | 45385/50000 [30:10:32<2:25:19,  1.89s/it]
Outer training loop:  91%|######### | 45386/50000 [30:10:34<2:24:13,  1.88s/it]
Outer training loop:  91%|######### | 45387/50000 [30:10:36<2:28:03,  1.93s/it]
Outer training loop:  91%|######### | 45388/50000 [30:10:38<2:21:50,  1.85s/it]
Outer training loop:  91%|######### | 45389/50000 [30:10:40<2:20:34,  1.83s/it]
Outer training loop:  91%|######### | 45390/50000 [30:10:42<2:23:11,  1.86s/it]
Outer training loop:  91%|######### | 45391/50000 [30:10:43<2:23:21,  1.87s/it]
Outer training loop:  91%|######### | 45392/50000 [30:10:45<2:23:35,  1.87s/it]
Outer training loop:  91%|######### | 45393/50000 [30:10:47<2:27:34,  1.92s/it]
Outer training loop:  91%|######### | 45394/50000 [30:10:49<2:25:29,  1.90s/it]
Outer training loop:  91%|######### | 45395/50000 [30:10:51<2:24:50,  1.89s/it]
Outer training loop:  91%|######### | 45396/50000 [30:10:53<2:21:50,  1.85s/it]
Outer training loop:  91%|######### | 45397/50000 [30:10:55<2:25:15,  1.89s/it]
Outer training loop:  91%|######### | 45398/50000 [30:10:57<2:24:24,  1.88s/it]
Outer training loop:  91%|######### | 45399/50000 [30:10:59<2:28:41,  1.94s/it]
Outer training loop:  91%|######### | 45400/50000 [30:11:01<2:31:04,  1.97s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  91%|######### | 45401/50000 [30:11:08<4:35:18,  3.59s/it]
Outer training loop:  91%|######### | 45402/50000 [30:11:10<3:53:58,  3.05s/it]
Outer training loop:  91%|######### | 45403/50000 [30:11:12<3:24:03,  2.66s/it]
Outer training loop:  91%|######### | 45404/50000 [30:11:14<3:14:06,  2.53s/it]
Outer training loop:  91%|######### | 45405/50000 [30:11:16<2:56:25,  2.30s/it]
Outer training loop:  91%|######### | 45406/50000 [30:11:18<2:45:06,  2.16s/it]
Outer training loop:  91%|######### | 45407/50000 [30:11:19<2:36:24,  2.04s/it]
Outer training loop:  91%|######### | 45408/50000 [30:11:21<2:33:33,  2.01s/it]
Outer training loop:  91%|######### | 45409/50000 [30:11:23<2:31:12,  1.98s/it]
Outer training loop:  91%|######### | 45410/50000 [30:11:25<2:27:21,  1.93s/it]
Outer training loop:  91%|######### | 45411/50000 [30:11:27<2:30:42,  1.97s/it]
Outer training loop:  91%|######### | 45412/50000 [30:11:29<2:31:42,  1.98s/it]
Outer training loop:  91%|######### | 45413/50000 [30:11:31<2:27:47,  1.93s/it]
Outer training loop:  91%|######### | 45414/50000 [30:11:33<2:29:51,  1.96s/it]
Outer training loop:  91%|######### | 45415/50000 [30:11:35<2:25:01,  1.90s/it]
Outer training loop:  91%|######### | 45416/50000 [30:11:36<2:23:03,  1.87s/it]
Outer training loop:  91%|######### | 45417/50000 [30:11:38<2:22:09,  1.86s/it]
Outer training loop:  91%|######### | 45418/50000 [30:11:40<2:26:40,  1.92s/it]
Outer training loop:  91%|######### | 45419/50000 [30:11:42<2:23:39,  1.88s/it]
Outer training loop:  91%|######### | 45420/50000 [30:11:44<2:24:44,  1.90s/it]
Outer training loop:  91%|######### | 45421/50000 [30:11:46<2:26:42,  1.92s/it]
Outer training loop:  91%|######### | 45422/50000 [30:11:48<2:24:16,  1.89s/it]
Outer training loop:  91%|######### | 45423/50000 [30:11:50<2:24:03,  1.89s/it]
Outer training loop:  91%|######### | 45424/50000 [30:11:52<2:23:48,  1.89s/it]
Outer training loop:  91%|######### | 45425/50000 [30:11:54<2:29:16,  1.96s/it]
Outer training loop:  91%|######### | 45426/50000 [30:11:56<2:25:48,  1.91s/it]
Outer training loop:  91%|######### | 45427/50000 [30:11:57<2:20:41,  1.85s/it]
Outer training loop:  91%|######### | 45428/50000 [30:11:59<2:23:43,  1.89s/it]
Outer training loop:  91%|######### | 45429/50000 [30:12:01<2:23:33,  1.88s/it]
Outer training loop:  91%|######### | 45430/50000 [30:12:03<2:21:02,  1.85s/it]
Outer training loop:  91%|######### | 45431/50000 [30:12:05<2:19:19,  1.83s/it]
Outer training loop:  91%|######### | 45432/50000 [30:12:07<2:22:19,  1.87s/it]
Outer training loop:  91%|######### | 45433/50000 [30:12:08<2:19:28,  1.83s/it]
Outer training loop:  91%|######### | 45434/50000 [30:12:10<2:17:04,  1.80s/it]
Outer training loop:  91%|######### | 45435/50000 [30:12:12<2:22:54,  1.88s/it]
Outer training loop:  91%|######### | 45436/50000 [30:12:14<2:20:48,  1.85s/it]
Outer training loop:  91%|######### | 45437/50000 [30:12:16<2:19:14,  1.83s/it]
Outer training loop:  91%|######### | 45438/50000 [30:12:18<2:19:13,  1.83s/it]
Outer training loop:  91%|######### | 45439/50000 [30:12:20<2:24:40,  1.90s/it]
Outer training loop:  91%|######### | 45440/50000 [30:12:22<2:24:35,  1.90s/it]
Outer training loop:  91%|######### | 45441/50000 [30:12:23<2:22:34,  1.88s/it]
Outer training loop:  91%|######### | 45442/50000 [30:12:25<2:26:40,  1.93s/it]
Outer training loop:  91%|######### | 45443/50000 [30:12:27<2:25:57,  1.92s/it]
Outer training loop:  91%|######### | 45444/50000 [30:12:29<2:25:59,  1.92s/it]
Outer training loop:  91%|######### | 45445/50000 [30:12:31<2:25:08,  1.91s/it]
Outer training loop:  91%|######### | 45446/50000 [30:12:33<2:31:08,  1.99s/it]
Outer training loop:  91%|######### | 45447/50000 [30:12:35<2:29:05,  1.96s/it]
Outer training loop:  91%|######### | 45448/50000 [30:12:37<2:24:19,  1.90s/it]
Outer training loop:  91%|######### | 45449/50000 [30:12:39<2:23:21,  1.89s/it]
Outer training loop:  91%|######### | 45450/50000 [30:12:41<2:22:18,  1.88s/it]
Outer training loop:  91%|######### | 45451/50000 [30:12:43<2:23:27,  1.89s/it]
Outer training loop:  91%|######### | 45452/50000 [30:12:44<2:21:06,  1.86s/it]
Outer training loop:  91%|######### | 45453/50000 [30:12:46<2:21:34,  1.87s/it]
Outer training loop:  91%|######### | 45454/50000 [30:12:48<2:18:37,  1.83s/it]
Outer training loop:  91%|######### | 45455/50000 [30:12:50<2:20:23,  1.85s/it]
Outer training loop:  91%|######### | 45456/50000 [30:12:52<2:25:58,  1.93s/it]
Outer training loop:  91%|######### | 45457/50000 [30:12:54<2:24:29,  1.91s/it]
Outer training loop:  91%|######### | 45458/50000 [30:12:56<2:22:09,  1.88s/it]
Outer training loop:  91%|######### | 45459/50000 [30:12:57<2:19:48,  1.85s/it]
Outer training loop:  91%|######### | 45460/50000 [30:12:59<2:21:17,  1.87s/it]
Outer training loop:  91%|######### | 45461/50000 [30:13:01<2:19:47,  1.85s/it]
Outer training loop:  91%|######### | 45462/50000 [30:13:03<2:18:21,  1.83s/it]
Outer training loop:  91%|######### | 45463/50000 [30:13:05<2:22:41,  1.89s/it]
Outer training loop:  91%|######### | 45464/50000 [30:13:07<2:21:49,  1.88s/it]
Outer training loop:  91%|######### | 45465/50000 [30:13:09<2:19:05,  1.84s/it]
Outer training loop:  91%|######### | 45466/50000 [30:13:11<2:21:04,  1.87s/it]
Outer training loop:  91%|######### | 45467/50000 [30:13:12<2:23:06,  1.89s/it]
Outer training loop:  91%|######### | 45468/50000 [30:13:14<2:22:09,  1.88s/it]
Outer training loop:  91%|######### | 45469/50000 [30:13:16<2:19:22,  1.85s/it]
Outer training loop:  91%|######### | 45470/50000 [30:13:18<2:21:50,  1.88s/it]
Outer training loop:  91%|######### | 45471/50000 [30:13:20<2:23:21,  1.90s/it]
Outer training loop:  91%|######### | 45472/50000 [30:13:22<2:21:14,  1.87s/it]
Outer training loop:  91%|######### | 45473/50000 [30:13:24<2:20:42,  1.86s/it]
Outer training loop:  91%|######### | 45474/50000 [30:13:26<2:23:09,  1.90s/it]
Outer training loop:  91%|######### | 45475/50000 [30:13:27<2:20:50,  1.87s/it]
Outer training loop:  91%|######### | 45476/50000 [30:13:29<2:18:56,  1.84s/it]
Outer training loop:  91%|######### | 45477/50000 [30:13:31<2:20:28,  1.86s/it]
Outer training loop:  91%|######### | 45478/50000 [30:13:33<2:17:48,  1.83s/it]
Outer training loop:  91%|######### | 45479/50000 [30:13:35<2:16:45,  1.81s/it]
Outer training loop:  91%|######### | 45480/50000 [30:13:36<2:17:13,  1.82s/it]
Outer training loop:  91%|######### | 45481/50000 [30:13:38<2:21:15,  1.88s/it]
Outer training loop:  91%|######### | 45482/50000 [30:13:40<2:21:20,  1.88s/it]
Outer training loop:  91%|######### | 45483/50000 [30:13:42<2:18:50,  1.84s/it]
Outer training loop:  91%|######### | 45484/50000 [30:13:44<2:22:27,  1.89s/it]
Outer training loop:  91%|######### | 45485/50000 [30:13:46<2:21:23,  1.88s/it]
Outer training loop:  91%|######### | 45486/50000 [30:13:48<2:19:59,  1.86s/it]
Outer training loop:  91%|######### | 45487/50000 [30:13:50<2:23:53,  1.91s/it]
Outer training loop:  91%|######### | 45488/50000 [30:13:52<2:21:08,  1.88s/it]
Outer training loop:  91%|######### | 45489/50000 [30:13:53<2:19:46,  1.86s/it]
Outer training loop:  91%|######### | 45490/50000 [30:13:55<2:17:12,  1.83s/it]
Outer training loop:  91%|######### | 45491/50000 [30:13:57<2:20:44,  1.87s/it]
Outer training loop:  91%|######### | 45492/50000 [30:13:59<2:19:20,  1.85s/it]
Outer training loop:  91%|######### | 45493/50000 [30:14:01<2:17:54,  1.84s/it]
Outer training loop:  91%|######### | 45494/50000 [30:14:03<2:21:21,  1.88s/it]
Outer training loop:  91%|######### | 45495/50000 [30:14:05<2:20:53,  1.88s/it]
Outer training loop:  91%|######### | 45496/50000 [30:14:07<2:26:32,  1.95s/it]
Outer training loop:  91%|######### | 45497/50000 [30:14:09<2:22:41,  1.90s/it]
Outer training loop:  91%|######### | 45498/50000 [30:14:11<2:24:42,  1.93s/it]
Outer training loop:  91%|######### | 45499/50000 [30:14:12<2:18:39,  1.85s/it]
Outer training loop:  91%|#########1| 45500/50000 [30:14:14<2:22:20,  1.90s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.76s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:26,  1.79s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.82s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:26,  1.83s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.86s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.87s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.87s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.88s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:18,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:18,  1.96s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:01,  1.87s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:02,  1.95s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.86s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.88s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.88s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:51,  1.95s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.94s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.83s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:41,  1.95s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.89s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:25,  1.82s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:26,  1.89s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:22,  1.83s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:17,  1.81s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:15,  1.80s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.87s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:15,  1.88s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:15,  1.94s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:07,  1.84s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:05,  1.83s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:31<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:58,  1.78s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<01:01,  1.91s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:42<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:44<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:46<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:49,  1.99s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:46,  1.94s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:44,  1.92s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:01<00:33,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:33,  1.99s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:30,  1.94s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:28,  1.90s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:25,  1.84s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:23,  1.83s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:21,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:17,  1.80s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:16,  2.02s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.96s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.91s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.89s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.92s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.82s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:11,  1.60s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:26,  1.81s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.78s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:25,  1.84s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:22,  1.83s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.86s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:18,  1.85s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.95s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:18,  1.90s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:16,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:14,  1.90s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:16,  1.95s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:08,  1.94s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.90s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:58,  1.86s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.86s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:53,  1.84s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.87s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:47,  1.85s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:45,  1.85s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:46,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.90s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.83s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:35,  1.83s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:38,  1.93s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.90s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:27,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:26,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:28,  1.96s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.90s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:21,  1.90s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:16,  1.87s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:17,  1.99s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:10,  1.90s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:07,  1.88s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:05,  1.86s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.85s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:02,  1.96s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.93s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.89s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:50,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:47,  1.83s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:48,  2.02s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:45,  1.98s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.85s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.85s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.84s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:29,  1.99s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:27,  1.99s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:25,  1.93s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.83s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.84s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.98s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.93s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.91s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.90s/it]

Outer training loop:  91%|#########1| 45501/50000 [30:19:35<121:44:55, 97.42s/it]
Outer training loop:  91%|#########1| 45502/50000 [30:19:37<85:59:27, 68.82s/it] 
Outer training loop:  91%|#########1| 45503/50000 [30:19:38<60:52:56, 48.74s/it]
Outer training loop:  91%|#########1| 45504/50000 [30:19:40<43:17:42, 34.67s/it]
Outer training loop:  91%|#########1| 45505/50000 [30:19:43<31:12:22, 24.99s/it]
Outer training loop:  91%|#########1| 45506/50000 [30:19:45<22:31:18, 18.04s/it]
Outer training loop:  91%|#########1| 45507/50000 [30:19:46<16:27:40, 13.19s/it]
Outer training loop:  91%|#########1| 45508/50000 [30:19:48<12:14:43,  9.81s/it]
Outer training loop:  91%|#########1| 45509/50000 [30:19:50<9:18:10,  7.46s/it] 
Outer training loop:  91%|#########1| 45510/50000 [30:19:52<7:10:46,  5.76s/it]
Outer training loop:  91%|#########1| 45511/50000 [30:19:54<5:48:06,  4.65s/it]
Outer training loop:  91%|#########1| 45512/50000 [30:19:56<4:48:23,  3.86s/it]
Outer training loop:  91%|#########1| 45513/50000 [30:19:58<4:04:36,  3.27s/it]
Outer training loop:  91%|#########1| 45514/50000 [30:20:00<3:32:33,  2.84s/it]
Outer training loop:  91%|#########1| 45515/50000 [30:20:02<3:11:42,  2.56s/it]
Outer training loop:  91%|#########1| 45516/50000 [30:20:04<2:56:48,  2.37s/it]
Outer training loop:  91%|#########1| 45517/50000 [30:20:06<2:46:25,  2.23s/it]
Outer training loop:  91%|#########1| 45518/50000 [30:20:08<2:41:30,  2.16s/it]
Outer training loop:  91%|#########1| 45519/50000 [30:20:10<2:34:01,  2.06s/it]
Outer training loop:  91%|#########1| 45520/50000 [30:20:11<2:27:03,  1.97s/it]
Outer training loop:  91%|#########1| 45521/50000 [30:20:13<2:26:24,  1.96s/it]
Outer training loop:  91%|#########1| 45522/50000 [30:20:15<2:25:26,  1.95s/it]
Outer training loop:  91%|#########1| 45523/50000 [30:20:17<2:22:53,  1.91s/it]
Outer training loop:  91%|#########1| 45524/50000 [30:20:19<2:20:13,  1.88s/it]
Outer training loop:  91%|#########1| 45525/50000 [30:20:21<2:23:54,  1.93s/it]
Outer training loop:  91%|#########1| 45526/50000 [30:20:23<2:22:07,  1.91s/it]
Outer training loop:  91%|#########1| 45527/50000 [30:20:25<2:24:59,  1.94s/it]
Outer training loop:  91%|#########1| 45528/50000 [30:20:27<2:27:50,  1.98s/it]
Outer training loop:  91%|#########1| 45529/50000 [30:20:29<2:27:24,  1.98s/it]
Outer training loop:  91%|#########1| 45530/50000 [30:20:30<2:21:55,  1.90s/it]
Outer training loop:  91%|#########1| 45531/50000 [30:20:32<2:22:57,  1.92s/it]
Outer training loop:  91%|#########1| 45532/50000 [30:20:34<2:20:40,  1.89s/it]
Outer training loop:  91%|#########1| 45533/50000 [30:20:36<2:20:51,  1.89s/it]
Outer training loop:  91%|#########1| 45534/50000 [30:20:38<2:22:33,  1.92s/it]
Outer training loop:  91%|#########1| 45535/50000 [30:20:40<2:22:04,  1.91s/it]
Outer training loop:  91%|#########1| 45536/50000 [30:20:42<2:20:15,  1.89s/it]
Outer training loop:  91%|#########1| 45537/50000 [30:20:44<2:22:22,  1.91s/it]
Outer training loop:  91%|#########1| 45538/50000 [30:20:46<2:20:20,  1.89s/it]
Outer training loop:  91%|#########1| 45539/50000 [30:20:47<2:18:40,  1.87s/it]
Outer training loop:  91%|#########1| 45540/50000 [30:20:49<2:20:06,  1.88s/it]
Outer training loop:  91%|#########1| 45541/50000 [30:20:52<2:27:59,  1.99s/it]
Outer training loop:  91%|#########1| 45542/50000 [30:20:53<2:25:26,  1.96s/it]
Outer training loop:  91%|#########1| 45543/50000 [30:20:55<2:21:24,  1.90s/it]
Outer training loop:  91%|#########1| 45544/50000 [30:20:57<2:25:05,  1.95s/it]
Outer training loop:  91%|#########1| 45545/50000 [30:20:59<2:24:28,  1.95s/it]
Outer training loop:  91%|#########1| 45546/50000 [30:21:01<2:17:39,  1.85s/it]
Outer training loop:  91%|#########1| 45547/50000 [30:21:03<2:15:40,  1.83s/it]
Outer training loop:  91%|#########1| 45548/50000 [30:21:05<2:17:19,  1.85s/it]
Outer training loop:  91%|#########1| 45549/50000 [30:21:06<2:16:31,  1.84s/it]
Outer training loop:  91%|#########1| 45550/50000 [30:21:08<2:20:01,  1.89s/it]
Outer training loop:  91%|#########1| 45551/50000 [30:21:10<2:20:18,  1.89s/it]
Outer training loop:  91%|#########1| 45552/50000 [30:21:12<2:16:58,  1.85s/it]
Outer training loop:  91%|#########1| 45553/50000 [30:21:14<2:18:22,  1.87s/it]
Outer training loop:  91%|#########1| 45554/50000 [30:21:16<2:22:03,  1.92s/it]
Outer training loop:  91%|#########1| 45555/50000 [30:21:18<2:20:16,  1.89s/it]
Outer training loop:  91%|#########1| 45556/50000 [30:21:20<2:19:46,  1.89s/it]
Outer training loop:  91%|#########1| 45557/50000 [30:21:22<2:25:38,  1.97s/it]
Outer training loop:  91%|#########1| 45558/50000 [30:21:24<2:21:03,  1.91s/it]
Outer training loop:  91%|#########1| 45559/50000 [30:21:26<2:21:49,  1.92s/it]
Outer training loop:  91%|#########1| 45560/50000 [30:21:27<2:18:53,  1.88s/it]
Outer training loop:  91%|#########1| 45561/50000 [30:21:29<2:22:51,  1.93s/it]
Outer training loop:  91%|#########1| 45562/50000 [30:21:31<2:23:10,  1.94s/it]
Outer training loop:  91%|#########1| 45563/50000 [30:21:33<2:18:54,  1.88s/it]
Outer training loop:  91%|#########1| 45564/50000 [30:21:35<2:23:16,  1.94s/it]
Outer training loop:  91%|#########1| 45565/50000 [30:21:37<2:22:14,  1.92s/it]
Outer training loop:  91%|#########1| 45566/50000 [30:21:39<2:18:45,  1.88s/it]
Outer training loop:  91%|#########1| 45567/50000 [30:21:41<2:21:43,  1.92s/it]
Outer training loop:  91%|#########1| 45568/50000 [30:21:43<2:21:43,  1.92s/it]
Outer training loop:  91%|#########1| 45569/50000 [30:21:45<2:20:31,  1.90s/it]
Outer training loop:  91%|#########1| 45570/50000 [30:21:46<2:17:44,  1.87s/it]
Outer training loop:  91%|#########1| 45571/50000 [30:21:48<2:19:44,  1.89s/it]
Outer training loop:  91%|#########1| 45572/50000 [30:21:50<2:19:22,  1.89s/it]
Outer training loop:  91%|#########1| 45573/50000 [30:21:52<2:18:06,  1.87s/it]
Outer training loop:  91%|#########1| 45574/50000 [30:21:54<2:22:13,  1.93s/it]
Outer training loop:  91%|#########1| 45575/50000 [30:21:56<2:18:29,  1.88s/it]
Outer training loop:  91%|#########1| 45576/50000 [30:21:58<2:19:06,  1.89s/it]
Outer training loop:  91%|#########1| 45577/50000 [30:22:00<2:26:41,  1.99s/it]
Outer training loop:  91%|#########1| 45578/50000 [30:22:02<2:20:33,  1.91s/it]
Outer training loop:  91%|#########1| 45579/50000 [30:22:03<2:16:55,  1.86s/it]
Outer training loop:  91%|#########1| 45580/50000 [30:22:06<2:20:53,  1.91s/it]
Outer training loop:  91%|#########1| 45581/50000 [30:22:07<2:19:01,  1.89s/it]
Outer training loop:  91%|#########1| 45582/50000 [30:22:09<2:17:41,  1.87s/it]
Outer training loop:  91%|#########1| 45583/50000 [30:22:11<2:16:41,  1.86s/it]
Outer training loop:  91%|#########1| 45584/50000 [30:22:13<2:19:10,  1.89s/it]
Outer training loop:  91%|#########1| 45585/50000 [30:22:15<2:17:51,  1.87s/it]
Outer training loop:  91%|#########1| 45586/50000 [30:22:17<2:16:37,  1.86s/it]
Outer training loop:  91%|#########1| 45587/50000 [30:22:19<2:18:46,  1.89s/it]
Outer training loop:  91%|#########1| 45588/50000 [30:22:21<2:19:25,  1.90s/it]
Outer training loop:  91%|#########1| 45589/50000 [30:22:22<2:17:58,  1.88s/it]
Outer training loop:  91%|#########1| 45590/50000 [30:22:24<2:17:28,  1.87s/it]
Outer training loop:  91%|#########1| 45591/50000 [30:22:26<2:24:45,  1.97s/it]
Outer training loop:  91%|#########1| 45592/50000 [30:22:28<2:23:37,  1.95s/it]
Outer training loop:  91%|#########1| 45593/50000 [30:22:30<2:20:49,  1.92s/it]
Outer training loop:  91%|#########1| 45594/50000 [30:22:32<2:23:48,  1.96s/it]
Outer training loop:  91%|#########1| 45595/50000 [30:22:34<2:20:57,  1.92s/it]
Outer training loop:  91%|#########1| 45596/50000 [30:22:36<2:18:09,  1.88s/it]
Outer training loop:  91%|#########1| 45597/50000 [30:22:38<2:18:51,  1.89s/it]
Outer training loop:  91%|#########1| 45598/50000 [30:22:40<2:17:31,  1.87s/it]
Outer training loop:  91%|#########1| 45599/50000 [30:22:41<2:15:24,  1.85s/it]
Outer training loop:  91%|#########1| 45600/50000 [30:22:43<2:17:56,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  91%|#########1| 45601/50000 [30:22:51<4:23:25,  3.59s/it]
Outer training loop:  91%|#########1| 45602/50000 [30:22:53<3:45:32,  3.08s/it]
Outer training loop:  91%|#########1| 45603/50000 [30:22:55<3:16:40,  2.68s/it]
Outer training loop:  91%|#########1| 45604/50000 [30:22:56<2:56:28,  2.41s/it]
Outer training loop:  91%|#########1| 45605/50000 [30:22:58<2:47:46,  2.29s/it]
Outer training loop:  91%|#########1| 45606/50000 [30:23:00<2:38:12,  2.16s/it]
Outer training loop:  91%|#########1| 45607/50000 [30:23:02<2:32:42,  2.09s/it]
Outer training loop:  91%|#########1| 45608/50000 [30:23:04<2:27:15,  2.01s/it]
Outer training loop:  91%|#########1| 45609/50000 [30:23:06<2:29:38,  2.04s/it]
Outer training loop:  91%|#########1| 45610/50000 [30:23:08<2:24:27,  1.97s/it]
Outer training loop:  91%|#########1| 45611/50000 [30:23:10<2:23:26,  1.96s/it]
Outer training loop:  91%|#########1| 45612/50000 [30:23:12<2:19:52,  1.91s/it]
Outer training loop:  91%|#########1| 45613/50000 [30:23:14<2:20:05,  1.92s/it]
Outer training loop:  91%|#########1| 45614/50000 [30:23:15<2:18:21,  1.89s/it]
Outer training loop:  91%|#########1| 45615/50000 [30:23:17<2:17:57,  1.89s/it]
Outer training loop:  91%|#########1| 45616/50000 [30:23:19<2:20:09,  1.92s/it]
Outer training loop:  91%|#########1| 45617/50000 [30:23:21<2:16:43,  1.87s/it]
Outer training loop:  91%|#########1| 45618/50000 [30:23:23<2:13:32,  1.83s/it]
Outer training loop:  91%|#########1| 45619/50000 [30:23:25<2:16:13,  1.87s/it]
Outer training loop:  91%|#########1| 45620/50000 [30:23:27<2:21:27,  1.94s/it]
Outer training loop:  91%|#########1| 45621/50000 [30:23:29<2:19:48,  1.92s/it]
Outer training loop:  91%|#########1| 45622/50000 [30:23:30<2:16:59,  1.88s/it]
Outer training loop:  91%|#########1| 45623/50000 [30:23:32<2:19:57,  1.92s/it]
Outer training loop:  91%|#########1| 45624/50000 [30:23:34<2:22:19,  1.95s/it]
Outer training loop:  91%|#########1| 45625/50000 [30:23:36<2:19:28,  1.91s/it]
Outer training loop:  91%|#########1| 45626/50000 [30:23:38<2:22:06,  1.95s/it]
Outer training loop:  91%|#########1| 45627/50000 [30:23:40<2:20:25,  1.93s/it]
Outer training loop:  91%|#########1| 45628/50000 [30:23:42<2:16:32,  1.87s/it]
Outer training loop:  91%|#########1| 45629/50000 [30:23:44<2:20:44,  1.93s/it]
Outer training loop:  91%|#########1| 45630/50000 [30:23:46<2:18:25,  1.90s/it]
Outer training loop:  91%|#########1| 45631/50000 [30:23:48<2:16:41,  1.88s/it]
Outer training loop:  91%|#########1| 45632/50000 [30:23:50<2:16:50,  1.88s/it]
Outer training loop:  91%|#########1| 45633/50000 [30:23:52<2:20:43,  1.93s/it]
Outer training loop:  91%|#########1| 45634/50000 [30:23:53<2:18:18,  1.90s/it]
Outer training loop:  91%|#########1| 45635/50000 [30:23:56<2:22:09,  1.95s/it]
Outer training loop:  91%|#########1| 45636/50000 [30:23:58<2:26:30,  2.01s/it]
Outer training loop:  91%|#########1| 45637/50000 [30:23:59<2:21:28,  1.95s/it]
Outer training loop:  91%|#########1| 45638/50000 [30:24:01<2:19:01,  1.91s/it]
Outer training loop:  91%|#########1| 45639/50000 [30:24:03<2:22:09,  1.96s/it]
Outer training loop:  91%|#########1| 45640/50000 [30:24:05<2:22:21,  1.96s/it]
Outer training loop:  91%|#########1| 45641/50000 [30:24:07<2:18:19,  1.90s/it]
Outer training loop:  91%|#########1| 45642/50000 [30:24:09<2:23:53,  1.98s/it]
Outer training loop:  91%|#########1| 45643/50000 [30:24:11<2:21:16,  1.95s/it]
Outer training loop:  91%|#########1| 45644/50000 [30:24:13<2:20:00,  1.93s/it]
Outer training loop:  91%|#########1| 45645/50000 [30:24:15<2:24:03,  1.98s/it]
Outer training loop:  91%|#########1| 45646/50000 [30:24:17<2:18:14,  1.90s/it]
Outer training loop:  91%|#########1| 45647/50000 [30:24:19<2:14:40,  1.86s/it]
Outer training loop:  91%|#########1| 45648/50000 [30:24:20<2:16:08,  1.88s/it]
Outer training loop:  91%|#########1| 45649/50000 [30:24:23<2:19:54,  1.93s/it]
Outer training loop:  91%|#########1| 45650/50000 [30:24:24<2:15:30,  1.87s/it]
Outer training loop:  91%|#########1| 45651/50000 [30:24:26<2:16:27,  1.88s/it]
Outer training loop:  91%|#########1| 45652/50000 [30:24:28<2:16:54,  1.89s/it]
Outer training loop:  91%|#########1| 45653/50000 [30:24:30<2:13:58,  1.85s/it]
Outer training loop:  91%|#########1| 45654/50000 [30:24:32<2:14:13,  1.85s/it]
Outer training loop:  91%|#########1| 45655/50000 [30:24:34<2:18:12,  1.91s/it]
Outer training loop:  91%|#########1| 45656/50000 [30:24:36<2:18:37,  1.91s/it]
Outer training loop:  91%|#########1| 45657/50000 [30:24:38<2:17:25,  1.90s/it]
Outer training loop:  91%|#########1| 45658/50000 [30:24:39<2:16:19,  1.88s/it]
Outer training loop:  91%|#########1| 45659/50000 [30:24:41<2:17:39,  1.90s/it]
Outer training loop:  91%|#########1| 45660/50000 [30:24:43<2:14:43,  1.86s/it]
Outer training loop:  91%|#########1| 45661/50000 [30:24:45<2:11:05,  1.81s/it]
Outer training loop:  91%|#########1| 45662/50000 [30:24:47<2:12:21,  1.83s/it]
Outer training loop:  91%|#########1| 45663/50000 [30:24:49<2:13:47,  1.85s/it]
Outer training loop:  91%|#########1| 45664/50000 [30:24:50<2:15:12,  1.87s/it]
Outer training loop:  91%|#########1| 45665/50000 [30:24:52<2:15:30,  1.88s/it]
Outer training loop:  91%|#########1| 45666/50000 [30:24:54<2:16:52,  1.89s/it]
Outer training loop:  91%|#########1| 45667/50000 [30:24:56<2:12:54,  1.84s/it]
Outer training loop:  91%|#########1| 45668/50000 [30:24:58<2:12:01,  1.83s/it]
Outer training loop:  91%|#########1| 45669/50000 [30:25:00<2:15:06,  1.87s/it]
Outer training loop:  91%|#########1| 45670/50000 [30:25:02<2:13:30,  1.85s/it]
Outer training loop:  91%|#########1| 45671/50000 [30:25:03<2:11:42,  1.83s/it]
Outer training loop:  91%|#########1| 45672/50000 [30:25:05<2:17:04,  1.90s/it]
Outer training loop:  91%|#########1| 45673/50000 [30:25:07<2:17:50,  1.91s/it]
Outer training loop:  91%|#########1| 45674/50000 [30:25:09<2:20:58,  1.96s/it]
Outer training loop:  91%|#########1| 45675/50000 [30:25:11<2:18:39,  1.92s/it]
Outer training loop:  91%|#########1| 45676/50000 [30:25:13<2:18:10,  1.92s/it]
Outer training loop:  91%|#########1| 45677/50000 [30:25:15<2:18:11,  1.92s/it]
Outer training loop:  91%|#########1| 45678/50000 [30:25:17<2:13:18,  1.85s/it]
Outer training loop:  91%|#########1| 45679/50000 [30:25:19<2:12:02,  1.83s/it]
Outer training loop:  91%|#########1| 45680/50000 [30:25:21<2:17:16,  1.91s/it]
Outer training loop:  91%|#########1| 45681/50000 [30:25:23<2:16:05,  1.89s/it]
Outer training loop:  91%|#########1| 45682/50000 [30:25:24<2:14:28,  1.87s/it]
Outer training loop:  91%|#########1| 45683/50000 [30:25:26<2:18:42,  1.93s/it]
Outer training loop:  91%|#########1| 45684/50000 [30:25:28<2:19:42,  1.94s/it]
Outer training loop:  91%|#########1| 45685/50000 [30:25:30<2:14:09,  1.87s/it]
Outer training loop:  91%|#########1| 45686/50000 [30:25:32<2:21:17,  1.97s/it]
Outer training loop:  91%|#########1| 45687/50000 [30:25:34<2:18:40,  1.93s/it]
Outer training loop:  91%|#########1| 45688/50000 [30:25:36<2:18:33,  1.93s/it]
Outer training loop:  91%|#########1| 45689/50000 [30:25:38<2:17:31,  1.91s/it]
Outer training loop:  91%|#########1| 45690/50000 [30:25:40<2:19:37,  1.94s/it]
Outer training loop:  91%|#########1| 45691/50000 [30:25:42<2:16:44,  1.90s/it]
Outer training loop:  91%|#########1| 45692/50000 [30:25:44<2:17:29,  1.91s/it]
Outer training loop:  91%|#########1| 45693/50000 [30:25:46<2:18:15,  1.93s/it]
Outer training loop:  91%|#########1| 45694/50000 [30:25:47<2:16:04,  1.90s/it]
Outer training loop:  91%|#########1| 45695/50000 [30:25:49<2:16:49,  1.91s/it]
Outer training loop:  91%|#########1| 45696/50000 [30:25:52<2:22:27,  1.99s/it]
Outer training loop:  91%|#########1| 45697/50000 [30:25:53<2:19:25,  1.94s/it]
Outer training loop:  91%|#########1| 45698/50000 [30:25:55<2:16:31,  1.90s/it]
Outer training loop:  91%|#########1| 45699/50000 [30:25:57<2:15:12,  1.89s/it]
Outer training loop:  91%|#########1| 45700/50000 [30:25:59<2:21:16,  1.97s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  91%|#########1| 45701/50000 [30:26:07<4:20:52,  3.64s/it]
Outer training loop:  91%|#########1| 45702/50000 [30:26:09<3:42:51,  3.11s/it]
Outer training loop:  91%|#########1| 45703/50000 [30:26:10<3:15:12,  2.73s/it]
Outer training loop:  91%|#########1| 45704/50000 [30:26:13<3:00:09,  2.52s/it]
Outer training loop:  91%|#########1| 45705/50000 [30:26:14<2:45:13,  2.31s/it]
Outer training loop:  91%|#########1| 45706/50000 [30:26:16<2:34:42,  2.16s/it]
Outer training loop:  91%|#########1| 45707/50000 [30:26:18<2:28:26,  2.07s/it]
Outer training loop:  91%|#########1| 45708/50000 [30:26:20<2:27:55,  2.07s/it]
Outer training loop:  91%|#########1| 45709/50000 [30:26:22<2:22:45,  2.00s/it]
Outer training loop:  91%|#########1| 45710/50000 [30:26:24<2:16:55,  1.91s/it]
Outer training loop:  91%|#########1| 45711/50000 [30:26:25<2:15:34,  1.90s/it]
Outer training loop:  91%|#########1| 45712/50000 [30:26:28<2:19:20,  1.95s/it]
Outer training loop:  91%|#########1| 45713/50000 [30:26:29<2:15:20,  1.89s/it]
Outer training loop:  91%|#########1| 45714/50000 [30:26:31<2:14:54,  1.89s/it]
Outer training loop:  91%|#########1| 45715/50000 [30:26:33<2:19:10,  1.95s/it]
Outer training loop:  91%|#########1| 45716/50000 [30:26:35<2:16:00,  1.90s/it]
Outer training loop:  91%|#########1| 45717/50000 [30:26:37<2:14:30,  1.88s/it]
Outer training loop:  91%|#########1| 45718/50000 [30:26:39<2:14:53,  1.89s/it]
Outer training loop:  91%|#########1| 45719/50000 [30:26:41<2:18:06,  1.94s/it]
Outer training loop:  91%|#########1| 45720/50000 [30:26:43<2:17:43,  1.93s/it]
Outer training loop:  91%|#########1| 45721/50000 [30:26:45<2:20:44,  1.97s/it]
Outer training loop:  91%|#########1| 45722/50000 [30:26:47<2:21:11,  1.98s/it]
Outer training loop:  91%|#########1| 45723/50000 [30:26:49<2:19:36,  1.96s/it]
Outer training loop:  91%|#########1| 45724/50000 [30:26:51<2:19:43,  1.96s/it]
Outer training loop:  91%|#########1| 45725/50000 [30:26:53<2:20:58,  1.98s/it]
Outer training loop:  91%|#########1| 45726/50000 [30:26:55<2:18:32,  1.94s/it]
Outer training loop:  91%|#########1| 45727/50000 [30:26:57<2:18:07,  1.94s/it]
Outer training loop:  91%|#########1| 45728/50000 [30:26:59<2:21:36,  1.99s/it]
Outer training loop:  91%|#########1| 45729/50000 [30:27:01<2:21:42,  1.99s/it]
Outer training loop:  91%|#########1| 45730/50000 [30:27:03<2:20:56,  1.98s/it]
Outer training loop:  91%|#########1| 45731/50000 [30:27:04<2:17:43,  1.94s/it]
Outer training loop:  91%|#########1| 45732/50000 [30:27:07<2:21:57,  2.00s/it]
Outer training loop:  91%|#########1| 45733/50000 [30:27:08<2:18:22,  1.95s/it]
Outer training loop:  91%|#########1| 45734/50000 [30:27:10<2:16:05,  1.91s/it]
Outer training loop:  91%|#########1| 45735/50000 [30:27:12<2:20:57,  1.98s/it]
Outer training loop:  91%|#########1| 45736/50000 [30:27:14<2:17:15,  1.93s/it]
Outer training loop:  91%|#########1| 45737/50000 [30:27:16<2:14:13,  1.89s/it]
Outer training loop:  91%|#########1| 45738/50000 [30:27:18<2:18:50,  1.95s/it]
Outer training loop:  91%|#########1| 45739/50000 [30:27:20<2:16:47,  1.93s/it]
Outer training loop:  91%|#########1| 45740/50000 [30:27:22<2:13:28,  1.88s/it]
Outer training loop:  91%|#########1| 45741/50000 [30:27:24<2:13:46,  1.88s/it]
Outer training loop:  91%|#########1| 45742/50000 [30:27:26<2:14:45,  1.90s/it]
Outer training loop:  91%|#########1| 45743/50000 [30:27:27<2:14:19,  1.89s/it]
Outer training loop:  91%|#########1| 45744/50000 [30:27:29<2:12:44,  1.87s/it]
Outer training loop:  91%|#########1| 45745/50000 [30:27:31<2:16:08,  1.92s/it]
Outer training loop:  91%|#########1| 45746/50000 [30:27:33<2:16:45,  1.93s/it]
Outer training loop:  91%|#########1| 45747/50000 [30:27:35<2:15:49,  1.92s/it]
Outer training loop:  91%|#########1| 45748/50000 [30:27:37<2:16:23,  1.92s/it]
Outer training loop:  91%|#########1| 45749/50000 [30:27:39<2:16:16,  1.92s/it]
Outer training loop:  92%|#########1| 45750/50000 [30:27:41<2:12:54,  1.88s/it]
Outer training loop:  92%|#########1| 45751/50000 [30:27:43<2:17:08,  1.94s/it]
Outer training loop:  92%|#########1| 45752/50000 [30:27:45<2:16:51,  1.93s/it]
Outer training loop:  92%|#########1| 45753/50000 [30:27:47<2:14:19,  1.90s/it]
Outer training loop:  92%|#########1| 45754/50000 [30:27:49<2:16:25,  1.93s/it]
Outer training loop:  92%|#########1| 45755/50000 [30:27:51<2:21:41,  2.00s/it]
Outer training loop:  92%|#########1| 45756/50000 [30:27:53<2:17:42,  1.95s/it]
Outer training loop:  92%|#########1| 45757/50000 [30:27:54<2:17:04,  1.94s/it]
Outer training loop:  92%|#########1| 45758/50000 [30:27:56<2:16:20,  1.93s/it]
Outer training loop:  92%|#########1| 45759/50000 [30:27:58<2:15:22,  1.92s/it]
Outer training loop:  92%|#########1| 45760/50000 [30:28:00<2:13:02,  1.88s/it]
Outer training loop:  92%|#########1| 45761/50000 [30:28:02<2:14:32,  1.90s/it]
Outer training loop:  92%|#########1| 45762/50000 [30:28:04<2:12:48,  1.88s/it]
Outer training loop:  92%|#########1| 45763/50000 [30:28:06<2:12:56,  1.88s/it]
Outer training loop:  92%|#########1| 45764/50000 [30:28:08<2:10:44,  1.85s/it]
Outer training loop:  92%|#########1| 45765/50000 [30:28:10<2:14:26,  1.90s/it]
Outer training loop:  92%|#########1| 45766/50000 [30:28:11<2:13:16,  1.89s/it]
Outer training loop:  92%|#########1| 45767/50000 [30:28:13<2:14:14,  1.90s/it]
Outer training loop:  92%|#########1| 45768/50000 [30:28:15<2:16:02,  1.93s/it]
Outer training loop:  92%|#########1| 45769/50000 [30:28:17<2:13:13,  1.89s/it]
Outer training loop:  92%|#########1| 45770/50000 [30:28:19<2:14:21,  1.91s/it]
Outer training loop:  92%|#########1| 45771/50000 [30:28:21<2:21:27,  2.01s/it]
Outer training loop:  92%|#########1| 45772/50000 [30:28:23<2:18:20,  1.96s/it]
Outer training loop:  92%|#########1| 45773/50000 [30:28:25<2:15:37,  1.93s/it]
Outer training loop:  92%|#########1| 45774/50000 [30:28:27<2:13:01,  1.89s/it]
Outer training loop:  92%|#########1| 45775/50000 [30:28:29<2:16:44,  1.94s/it]
Outer training loop:  92%|#########1| 45776/50000 [30:28:31<2:15:26,  1.92s/it]
Outer training loop:  92%|#########1| 45777/50000 [30:28:33<2:12:43,  1.89s/it]
Outer training loop:  92%|#########1| 45778/50000 [30:28:35<2:17:57,  1.96s/it]
Outer training loop:  92%|#########1| 45779/50000 [30:28:37<2:17:09,  1.95s/it]
Outer training loop:  92%|#########1| 45780/50000 [30:28:38<2:13:08,  1.89s/it]
Outer training loop:  92%|#########1| 45781/50000 [30:28:40<2:11:15,  1.87s/it]
Outer training loop:  92%|#########1| 45782/50000 [30:28:42<2:17:28,  1.96s/it]
Outer training loop:  92%|#########1| 45783/50000 [30:28:44<2:13:34,  1.90s/it]
Outer training loop:  92%|#########1| 45784/50000 [30:28:46<2:11:44,  1.87s/it]
Outer training loop:  92%|#########1| 45785/50000 [30:28:48<2:15:06,  1.92s/it]
Outer training loop:  92%|#########1| 45786/50000 [30:28:50<2:16:55,  1.95s/it]
Outer training loop:  92%|#########1| 45787/50000 [30:28:52<2:12:24,  1.89s/it]
Outer training loop:  92%|#########1| 45788/50000 [30:28:53<2:10:05,  1.85s/it]
Outer training loop:  92%|#########1| 45789/50000 [30:28:55<2:13:01,  1.90s/it]
Outer training loop:  92%|#########1| 45790/50000 [30:28:57<2:09:47,  1.85s/it]
Outer training loop:  92%|#########1| 45791/50000 [30:28:59<2:07:55,  1.82s/it]
Outer training loop:  92%|#########1| 45792/50000 [30:29:01<2:09:08,  1.84s/it]
Outer training loop:  92%|#########1| 45793/50000 [30:29:03<2:07:18,  1.82s/it]
Outer training loop:  92%|#########1| 45794/50000 [30:29:04<2:08:14,  1.83s/it]
Outer training loop:  92%|#########1| 45795/50000 [30:29:07<2:14:51,  1.92s/it]
Outer training loop:  92%|#########1| 45796/50000 [30:29:09<2:13:32,  1.91s/it]
Outer training loop:  92%|#########1| 45797/50000 [30:29:10<2:09:25,  1.85s/it]
Outer training loop:  92%|#########1| 45798/50000 [30:29:12<2:10:07,  1.86s/it]
Outer training loop:  92%|#########1| 45799/50000 [30:29:14<2:14:12,  1.92s/it]
Outer training loop:  92%|#########1| 45800/50000 [30:29:16<2:12:07,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  2.00s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.98s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]

Outer training loop:  92%|#########1| 45801/50000 [30:29:23<4:08:46,  3.55s/it]
Outer training loop:  92%|#########1| 45802/50000 [30:29:25<3:33:58,  3.06s/it]
Outer training loop:  92%|#########1| 45803/50000 [30:29:27<3:09:49,  2.71s/it]
Outer training loop:  92%|#########1| 45804/50000 [30:29:29<2:50:57,  2.44s/it]
Outer training loop:  92%|#########1| 45805/50000 [30:29:31<2:36:12,  2.23s/it]
Outer training loop:  92%|#########1| 45806/50000 [30:29:33<2:29:23,  2.14s/it]
Outer training loop:  92%|#########1| 45807/50000 [30:29:35<2:23:12,  2.05s/it]
Outer training loop:  92%|#########1| 45808/50000 [30:29:36<2:19:49,  2.00s/it]
Outer training loop:  92%|#########1| 45809/50000 [30:29:38<2:15:55,  1.95s/it]
Outer training loop:  92%|#########1| 45810/50000 [30:29:40<2:18:41,  1.99s/it]
Outer training loop:  92%|#########1| 45811/50000 [30:29:42<2:15:00,  1.93s/it]
Outer training loop:  92%|#########1| 45812/50000 [30:29:44<2:11:29,  1.88s/it]
Outer training loop:  92%|#########1| 45813/50000 [30:29:46<2:14:26,  1.93s/it]
Outer training loop:  92%|#########1| 45814/50000 [30:29:48<2:11:14,  1.88s/it]
Outer training loop:  92%|#########1| 45815/50000 [30:29:49<2:09:19,  1.85s/it]
Outer training loop:  92%|#########1| 45816/50000 [30:29:51<2:11:40,  1.89s/it]
Outer training loop:  92%|#########1| 45817/50000 [30:29:53<2:11:23,  1.88s/it]
Outer training loop:  92%|#########1| 45818/50000 [30:29:55<2:15:17,  1.94s/it]
Outer training loop:  92%|#########1| 45819/50000 [30:29:57<2:17:02,  1.97s/it]
Outer training loop:  92%|#########1| 45820/50000 [30:29:59<2:15:14,  1.94s/it]
Outer training loop:  92%|#########1| 45821/50000 [30:30:01<2:11:29,  1.89s/it]
Outer training loop:  92%|#########1| 45822/50000 [30:30:03<2:11:59,  1.90s/it]
Outer training loop:  92%|#########1| 45823/50000 [30:30:05<2:14:56,  1.94s/it]
Outer training loop:  92%|#########1| 45824/50000 [30:30:07<2:11:43,  1.89s/it]
Outer training loop:  92%|#########1| 45825/50000 [30:30:09<2:11:25,  1.89s/it]
Outer training loop:  92%|#########1| 45826/50000 [30:30:11<2:15:40,  1.95s/it]
Outer training loop:  92%|#########1| 45827/50000 [30:30:13<2:14:01,  1.93s/it]
Outer training loop:  92%|#########1| 45828/50000 [30:30:15<2:12:19,  1.90s/it]
Outer training loop:  92%|#########1| 45829/50000 [30:30:16<2:12:53,  1.91s/it]
Outer training loop:  92%|#########1| 45830/50000 [30:30:19<2:16:58,  1.97s/it]
Outer training loop:  92%|#########1| 45831/50000 [30:30:21<2:16:39,  1.97s/it]
Outer training loop:  92%|#########1| 45832/50000 [30:30:22<2:16:26,  1.96s/it]
Outer training loop:  92%|#########1| 45833/50000 [30:30:25<2:19:33,  2.01s/it]
Outer training loop:  92%|#########1| 45834/50000 [30:30:27<2:18:53,  2.00s/it]
Outer training loop:  92%|#########1| 45835/50000 [30:30:29<2:17:45,  1.98s/it]
Outer training loop:  92%|#########1| 45836/50000 [30:30:31<2:20:54,  2.03s/it]
Outer training loop:  92%|#########1| 45837/50000 [30:30:32<2:16:52,  1.97s/it]
Outer training loop:  92%|#########1| 45838/50000 [30:30:34<2:14:37,  1.94s/it]
Outer training loop:  92%|#########1| 45839/50000 [30:30:36<2:14:41,  1.94s/it]
Outer training loop:  92%|#########1| 45840/50000 [30:30:38<2:14:50,  1.94s/it]
Outer training loop:  92%|#########1| 45841/50000 [30:30:40<2:13:51,  1.93s/it]
Outer training loop:  92%|#########1| 45842/50000 [30:30:42<2:08:54,  1.86s/it]
Outer training loop:  92%|#########1| 45843/50000 [30:30:44<2:16:01,  1.96s/it]
Outer training loop:  92%|#########1| 45844/50000 [30:30:46<2:11:59,  1.91s/it]
Outer training loop:  92%|#########1| 45845/50000 [30:30:48<2:09:42,  1.87s/it]
Outer training loop:  92%|#########1| 45846/50000 [30:30:50<2:13:50,  1.93s/it]
Outer training loop:  92%|#########1| 45847/50000 [30:30:51<2:10:36,  1.89s/it]
Outer training loop:  92%|#########1| 45848/50000 [30:30:53<2:08:41,  1.86s/it]
Outer training loop:  92%|#########1| 45849/50000 [30:30:55<2:11:59,  1.91s/it]
Outer training loop:  92%|#########1| 45850/50000 [30:30:57<2:12:13,  1.91s/it]
Outer training loop:  92%|#########1| 45851/50000 [30:30:59<2:12:28,  1.92s/it]
Outer training loop:  92%|#########1| 45852/50000 [30:31:01<2:10:23,  1.89s/it]
Outer training loop:  92%|#########1| 45853/50000 [30:31:03<2:11:51,  1.91s/it]
Outer training loop:  92%|#########1| 45854/50000 [30:31:05<2:09:10,  1.87s/it]
Outer training loop:  92%|#########1| 45855/50000 [30:31:07<2:09:21,  1.87s/it]
Outer training loop:  92%|#########1| 45856/50000 [30:31:08<2:09:47,  1.88s/it]
Outer training loop:  92%|#########1| 45857/50000 [30:31:10<2:08:43,  1.86s/it]
Outer training loop:  92%|#########1| 45858/50000 [30:31:12<2:06:47,  1.84s/it]
Outer training loop:  92%|#########1| 45859/50000 [30:31:14<2:09:10,  1.87s/it]
Outer training loop:  92%|#########1| 45860/50000 [30:31:16<2:06:02,  1.83s/it]
Outer training loop:  92%|#########1| 45861/50000 [30:31:18<2:05:18,  1.82s/it]
Outer training loop:  92%|#########1| 45862/50000 [30:31:19<2:05:13,  1.82s/it]
Outer training loop:  92%|#########1| 45863/50000 [30:31:21<2:08:40,  1.87s/it]
Outer training loop:  92%|#########1| 45864/50000 [30:31:23<2:06:40,  1.84s/it]
Outer training loop:  92%|#########1| 45865/50000 [30:31:25<2:05:40,  1.82s/it]
Outer training loop:  92%|#########1| 45866/50000 [30:31:27<2:08:50,  1.87s/it]
Outer training loop:  92%|#########1| 45867/50000 [30:31:29<2:09:56,  1.89s/it]
Outer training loop:  92%|#########1| 45868/50000 [30:31:31<2:10:42,  1.90s/it]
Outer training loop:  92%|#########1| 45869/50000 [30:31:33<2:11:35,  1.91s/it]
Outer training loop:  92%|#########1| 45870/50000 [30:31:35<2:15:28,  1.97s/it]
Outer training loop:  92%|#########1| 45871/50000 [30:31:37<2:13:56,  1.95s/it]
Outer training loop:  92%|#########1| 45872/50000 [30:31:39<2:14:10,  1.95s/it]
Outer training loop:  92%|#########1| 45873/50000 [30:31:41<2:16:05,  1.98s/it]
Outer training loop:  92%|#########1| 45874/50000 [30:31:43<2:14:47,  1.96s/it]
Outer training loop:  92%|#########1| 45875/50000 [30:31:44<2:13:46,  1.95s/it]
Outer training loop:  92%|#########1| 45876/50000 [30:31:46<2:11:33,  1.91s/it]
Outer training loop:  92%|#########1| 45877/50000 [30:31:48<2:15:37,  1.97s/it]
Outer training loop:  92%|#########1| 45878/50000 [30:31:50<2:14:02,  1.95s/it]
Outer training loop:  92%|#########1| 45879/50000 [30:31:52<2:14:06,  1.95s/it]
Outer training loop:  92%|#########1| 45880/50000 [30:31:54<2:15:36,  1.97s/it]
Outer training loop:  92%|#########1| 45881/50000 [30:31:56<2:15:39,  1.98s/it]
Outer training loop:  92%|#########1| 45882/50000 [30:31:58<2:12:31,  1.93s/it]
Outer training loop:  92%|#########1| 45883/50000 [30:32:00<2:14:17,  1.96s/it]
Outer training loop:  92%|#########1| 45884/50000 [30:32:02<2:11:27,  1.92s/it]
Outer training loop:  92%|#########1| 45885/50000 [30:32:04<2:07:51,  1.86s/it]
Outer training loop:  92%|#########1| 45886/50000 [30:32:06<2:08:06,  1.87s/it]
Outer training loop:  92%|#########1| 45887/50000 [30:32:07<2:07:03,  1.85s/it]
Outer training loop:  92%|#########1| 45888/50000 [30:32:09<2:07:54,  1.87s/it]
Outer training loop:  92%|#########1| 45889/50000 [30:32:11<2:06:35,  1.85s/it]
Outer training loop:  92%|#########1| 45890/50000 [30:32:13<2:10:10,  1.90s/it]
Outer training loop:  92%|#########1| 45891/50000 [30:32:15<2:07:50,  1.87s/it]
Outer training loop:  92%|#########1| 45892/50000 [30:32:17<2:06:20,  1.85s/it]
Outer training loop:  92%|#########1| 45893/50000 [30:32:19<2:09:33,  1.89s/it]
Outer training loop:  92%|#########1| 45894/50000 [30:32:20<2:07:06,  1.86s/it]
Outer training loop:  92%|#########1| 45895/50000 [30:32:22<2:04:28,  1.82s/it]
Outer training loop:  92%|#########1| 45896/50000 [30:32:24<2:06:59,  1.86s/it]
Outer training loop:  92%|#########1| 45897/50000 [30:32:26<2:07:16,  1.86s/it]
Outer training loop:  92%|#########1| 45898/50000 [30:32:28<2:09:45,  1.90s/it]
Outer training loop:  92%|#########1| 45899/50000 [30:32:30<2:09:41,  1.90s/it]
Outer training loop:  92%|#########1| 45900/50000 [30:32:32<2:11:37,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  92%|#########1| 45901/50000 [30:32:39<4:05:04,  3.59s/it]
Outer training loop:  92%|#########1| 45902/50000 [30:32:41<3:27:55,  3.04s/it]
Outer training loop:  92%|#########1| 45903/50000 [30:32:43<3:03:29,  2.69s/it]
Outer training loop:  92%|#########1| 45904/50000 [30:32:45<2:48:55,  2.47s/it]
Outer training loop:  92%|#########1| 45905/50000 [30:32:47<2:35:28,  2.28s/it]
Outer training loop:  92%|#########1| 45906/50000 [30:32:49<2:24:47,  2.12s/it]
Outer training loop:  92%|#########1| 45907/50000 [30:32:51<2:22:16,  2.09s/it]
Outer training loop:  92%|#########1| 45908/50000 [30:32:52<2:14:37,  1.97s/it]
Outer training loop:  92%|#########1| 45909/50000 [30:32:54<2:10:29,  1.91s/it]
Outer training loop:  92%|#########1| 45910/50000 [30:32:56<2:14:06,  1.97s/it]
Outer training loop:  92%|#########1| 45911/50000 [30:32:58<2:11:00,  1.92s/it]
Outer training loop:  92%|#########1| 45912/50000 [30:33:00<2:06:33,  1.86s/it]
Outer training loop:  92%|#########1| 45913/50000 [30:33:01<2:05:33,  1.84s/it]
Outer training loop:  92%|#########1| 45914/50000 [30:33:04<2:12:37,  1.95s/it]
Outer training loop:  92%|#########1| 45915/50000 [30:33:06<2:17:12,  2.02s/it]
Outer training loop:  92%|#########1| 45916/50000 [30:33:08<2:17:00,  2.01s/it]
Outer training loop:  92%|#########1| 45917/50000 [30:33:10<2:20:03,  2.06s/it]
Outer training loop:  92%|#########1| 45918/50000 [30:33:12<2:14:54,  1.98s/it]
Outer training loop:  92%|#########1| 45919/50000 [30:33:14<2:14:26,  1.98s/it]
Outer training loop:  92%|#########1| 45920/50000 [30:33:16<2:16:10,  2.00s/it]
Outer training loop:  92%|#########1| 45921/50000 [30:33:18<2:13:31,  1.96s/it]
Outer training loop:  92%|#########1| 45922/50000 [30:33:20<2:11:43,  1.94s/it]
Outer training loop:  92%|#########1| 45923/50000 [30:33:21<2:09:23,  1.90s/it]
Outer training loop:  92%|#########1| 45924/50000 [30:33:23<2:12:26,  1.95s/it]
Outer training loop:  92%|#########1| 45925/50000 [30:33:25<2:11:22,  1.93s/it]
Outer training loop:  92%|#########1| 45926/50000 [30:33:27<2:09:24,  1.91s/it]
Outer training loop:  92%|#########1| 45927/50000 [30:33:29<2:14:26,  1.98s/it]
Outer training loop:  92%|#########1| 45928/50000 [30:33:31<2:12:07,  1.95s/it]
Outer training loop:  92%|#########1| 45929/50000 [30:33:33<2:10:46,  1.93s/it]
Outer training loop:  92%|#########1| 45930/50000 [30:33:35<2:10:01,  1.92s/it]
Outer training loop:  92%|#########1| 45931/50000 [30:33:37<2:12:22,  1.95s/it]
Outer training loop:  92%|#########1| 45932/50000 [30:33:39<2:08:35,  1.90s/it]
Outer training loop:  92%|#########1| 45933/50000 [30:33:41<2:07:54,  1.89s/it]
Outer training loop:  92%|#########1| 45934/50000 [30:33:43<2:08:43,  1.90s/it]
Outer training loop:  92%|#########1| 45935/50000 [30:33:45<2:08:38,  1.90s/it]
Outer training loop:  92%|#########1| 45936/50000 [30:33:46<2:06:24,  1.87s/it]
Outer training loop:  92%|#########1| 45937/50000 [30:33:48<2:07:12,  1.88s/it]
Outer training loop:  92%|#########1| 45938/50000 [30:33:50<2:10:56,  1.93s/it]
Outer training loop:  92%|#########1| 45939/50000 [30:33:52<2:10:30,  1.93s/it]
Outer training loop:  92%|#########1| 45940/50000 [30:33:54<2:09:47,  1.92s/it]
Outer training loop:  92%|#########1| 45941/50000 [30:33:56<2:12:39,  1.96s/it]
Outer training loop:  92%|#########1| 45942/50000 [30:33:58<2:10:15,  1.93s/it]
Outer training loop:  92%|#########1| 45943/50000 [30:34:00<2:08:08,  1.90s/it]
Outer training loop:  92%|#########1| 45944/50000 [30:34:02<2:05:04,  1.85s/it]
Outer training loop:  92%|#########1| 45945/50000 [30:34:04<2:07:46,  1.89s/it]
Outer training loop:  92%|#########1| 45946/50000 [30:34:05<2:07:53,  1.89s/it]
Outer training loop:  92%|#########1| 45947/50000 [30:34:07<2:07:47,  1.89s/it]
Outer training loop:  92%|#########1| 45948/50000 [30:34:09<2:08:33,  1.90s/it]
Outer training loop:  92%|#########1| 45949/50000 [30:34:11<2:08:36,  1.90s/it]
Outer training loop:  92%|#########1| 45950/50000 [30:34:13<2:07:44,  1.89s/it]
Outer training loop:  92%|#########1| 45951/50000 [30:34:15<2:08:58,  1.91s/it]
Outer training loop:  92%|#########1| 45952/50000 [30:34:17<2:05:42,  1.86s/it]
Outer training loop:  92%|#########1| 45953/50000 [30:34:19<2:05:49,  1.87s/it]
Outer training loop:  92%|#########1| 45954/50000 [30:34:20<2:05:35,  1.86s/it]
Outer training loop:  92%|#########1| 45955/50000 [30:34:22<2:06:26,  1.88s/it]
Outer training loop:  92%|#########1| 45956/50000 [30:34:24<2:06:34,  1.88s/it]
Outer training loop:  92%|#########1| 45957/50000 [30:34:26<2:03:24,  1.83s/it]
Outer training loop:  92%|#########1| 45958/50000 [30:34:28<2:05:45,  1.87s/it]
Outer training loop:  92%|#########1| 45959/50000 [30:34:30<2:06:51,  1.88s/it]
Outer training loop:  92%|#########1| 45960/50000 [30:34:32<2:08:01,  1.90s/it]
Outer training loop:  92%|#########1| 45961/50000 [30:34:34<2:08:22,  1.91s/it]
Outer training loop:  92%|#########1| 45962/50000 [30:34:36<2:14:26,  2.00s/it]
Outer training loop:  92%|#########1| 45963/50000 [30:34:38<2:10:14,  1.94s/it]
Outer training loop:  92%|#########1| 45964/50000 [30:34:40<2:08:26,  1.91s/it]
Outer training loop:  92%|#########1| 45965/50000 [30:34:42<2:16:23,  2.03s/it]
Outer training loop:  92%|#########1| 45966/50000 [30:34:44<2:14:49,  2.01s/it]
Outer training loop:  92%|#########1| 45967/50000 [30:34:46<2:10:06,  1.94s/it]
Outer training loop:  92%|#########1| 45968/50000 [30:34:48<2:10:45,  1.95s/it]
Outer training loop:  92%|#########1| 45969/50000 [30:34:49<2:08:31,  1.91s/it]
Outer training loop:  92%|#########1| 45970/50000 [30:34:51<2:07:01,  1.89s/it]
Outer training loop:  92%|#########1| 45971/50000 [30:34:53<2:05:00,  1.86s/it]
Outer training loop:  92%|#########1| 45972/50000 [30:34:55<2:08:31,  1.91s/it]
Outer training loop:  92%|#########1| 45973/50000 [30:34:57<2:07:41,  1.90s/it]
Outer training loop:  92%|#########1| 45974/50000 [30:34:59<2:07:25,  1.90s/it]
Outer training loop:  92%|#########1| 45975/50000 [30:35:01<2:09:48,  1.93s/it]
Outer training loop:  92%|#########1| 45976/50000 [30:35:03<2:07:49,  1.91s/it]
Outer training loop:  92%|#########1| 45977/50000 [30:35:05<2:07:25,  1.90s/it]
Outer training loop:  92%|#########1| 45978/50000 [30:35:07<2:09:28,  1.93s/it]
Outer training loop:  92%|#########1| 45979/50000 [30:35:09<2:11:01,  1.96s/it]
Outer training loop:  92%|#########1| 45980/50000 [30:35:10<2:08:00,  1.91s/it]
Outer training loop:  92%|#########1| 45981/50000 [30:35:12<2:07:18,  1.90s/it]
Outer training loop:  92%|#########1| 45982/50000 [30:35:14<2:08:22,  1.92s/it]
Outer training loop:  92%|#########1| 45983/50000 [30:35:16<2:08:22,  1.92s/it]
Outer training loop:  92%|#########1| 45984/50000 [30:35:18<2:08:46,  1.92s/it]
Outer training loop:  92%|#########1| 45985/50000 [30:35:20<2:11:47,  1.97s/it]
Outer training loop:  92%|#########1| 45986/50000 [30:35:22<2:07:15,  1.90s/it]
Outer training loop:  92%|#########1| 45987/50000 [30:35:24<2:06:26,  1.89s/it]
Outer training loop:  92%|#########1| 45988/50000 [30:35:26<2:04:23,  1.86s/it]
Outer training loop:  92%|#########1| 45989/50000 [30:35:28<2:08:18,  1.92s/it]
Outer training loop:  92%|#########1| 45990/50000 [30:35:30<2:08:15,  1.92s/it]
Outer training loop:  92%|#########1| 45991/50000 [30:35:31<2:04:30,  1.86s/it]
Outer training loop:  92%|#########1| 45992/50000 [30:35:33<2:06:29,  1.89s/it]
Outer training loop:  92%|#########1| 45993/50000 [30:35:35<2:06:33,  1.90s/it]
Outer training loop:  92%|#########1| 45994/50000 [30:35:37<2:04:35,  1.87s/it]
Outer training loop:  92%|#########1| 45995/50000 [30:35:39<2:06:33,  1.90s/it]
Outer training loop:  92%|#########1| 45996/50000 [30:35:41<2:02:57,  1.84s/it]
Outer training loop:  92%|#########1| 45997/50000 [30:35:42<2:01:49,  1.83s/it]
Outer training loop:  92%|#########1| 45998/50000 [30:35:44<2:00:50,  1.81s/it]
Outer training loop:  92%|#########1| 45999/50000 [30:35:46<2:04:36,  1.87s/it]
Outer training loop:  92%|#########2| 46000/50000 [30:35:48<2:02:08,  1.83s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.03s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.98s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:42,  1.98s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:36,  1.94s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:33,  1.92s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:31,  1.95s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.92s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.91s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:26,  1.96s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.94s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:17,  1.88s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:20,  1.95s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:21,  1.99s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:16,  1.95s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.91s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.89s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:09,  1.96s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.91s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.89s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.90s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:02,  1.98s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:55,  1.90s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.89s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.90s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:51,  1.96s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:48,  1.95s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:44,  1.91s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:41,  1.87s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:37,  1.85s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:37,  1.87s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:40,  1.96s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:36,  1.93s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:27,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:24,  1.84s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:26,  1.92s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:23,  1.91s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:15,  1.84s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:08,  1.84s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.83s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<00:58,  1.79s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:56,  1.78s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.93s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:56,  1.90s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.87s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:52,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:51,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:49,  1.90s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:45,  1.99s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:44,  2.02s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:41,  1.97s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:38,  1.93s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.87s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.85s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.82s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:29,  1.99s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.92s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.91s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.89s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.85s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.87s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:14,  2.03s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.97s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.93s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.94s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.80s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.75s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:22,  1.76s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:34,  1.93s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:31,  1.91s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:24,  1.85s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:22,  1.84s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.89s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:16,  1.87s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.88s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:10,  1.86s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:05,  1.84s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:01,  1.87s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:57,  1.83s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:00,  1.92s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:55,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:50,  1.85s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:46,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:41,  1.82s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:44,  1.89s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:41,  1.92s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:35,  1.87s/it]

Evaluating policy:  40%|###9      | 33/83 [01:01<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:31,  1.94s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:28,  1.92s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:25,  1.89s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.85s/it]

Evaluating policy:  49%|####9     | 41/83 [01:16<01:17,  1.84s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:21,  1.99s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:17,  1.94s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:15,  1.93s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:10,  1.90s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:07,  1.87s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:07,  1.98s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.91s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.91s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.88s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.84s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.84s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:53,  1.98s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:50,  1.94s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.91s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.88s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.88s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:38,  1.84s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.81s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:37,  1.96s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:34,  1.93s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.86s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:19,  1.98s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:17,  1.91s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:15,  1.91s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.88s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.88s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.89s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.89s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.86s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.84s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:02,  2.02s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.98s/it]

Outer training loop:  92%|#########2| 46001/50000 [30:41:09<108:32:15, 97.71s/it]
Outer training loop:  92%|#########2| 46002/50000 [30:41:11<76:34:37, 68.95s/it] 
Outer training loop:  92%|#########2| 46003/50000 [30:41:13<54:14:18, 48.85s/it]
Outer training loop:  92%|#########2| 46004/50000 [30:41:15<38:32:08, 34.72s/it]
Outer training loop:  92%|#########2| 46005/50000 [30:41:17<27:35:13, 24.86s/it]
Outer training loop:  92%|#########2| 46006/50000 [30:41:18<19:53:20, 17.93s/it]
Outer training loop:  92%|#########2| 46007/50000 [30:41:20<14:34:19, 13.14s/it]
Outer training loop:  92%|#########2| 46008/50000 [30:41:22<10:48:38,  9.75s/it]
Outer training loop:  92%|#########2| 46009/50000 [30:41:24<8:10:42,  7.38s/it] 
Outer training loop:  92%|#########2| 46010/50000 [30:41:26<6:20:23,  5.72s/it]
Outer training loop:  92%|#########2| 46011/50000 [30:41:29<5:16:14,  4.76s/it]
Outer training loop:  92%|#########2| 46012/50000 [30:41:30<4:17:20,  3.87s/it]
Outer training loop:  92%|#########2| 46013/50000 [30:41:32<3:38:49,  3.29s/it]
Outer training loop:  92%|#########2| 46014/50000 [30:41:34<3:14:11,  2.92s/it]
Outer training loop:  92%|#########2| 46015/50000 [30:41:36<2:52:10,  2.59s/it]
Outer training loop:  92%|#########2| 46016/50000 [30:41:38<2:37:08,  2.37s/it]
Outer training loop:  92%|#########2| 46017/50000 [30:41:40<2:25:53,  2.20s/it]
Outer training loop:  92%|#########2| 46018/50000 [30:41:42<2:22:42,  2.15s/it]
Outer training loop:  92%|#########2| 46019/50000 [30:41:44<2:19:46,  2.11s/it]
Outer training loop:  92%|#########2| 46020/50000 [30:41:46<2:13:50,  2.02s/it]
Outer training loop:  92%|#########2| 46021/50000 [30:41:47<2:10:37,  1.97s/it]
Outer training loop:  92%|#########2| 46022/50000 [30:41:50<2:12:33,  2.00s/it]
Outer training loop:  92%|#########2| 46023/50000 [30:41:51<2:07:09,  1.92s/it]
Outer training loop:  92%|#########2| 46024/50000 [30:41:53<2:03:02,  1.86s/it]
Outer training loop:  92%|#########2| 46025/50000 [30:41:55<2:06:53,  1.92s/it]
Outer training loop:  92%|#########2| 46026/50000 [30:41:57<2:04:37,  1.88s/it]
Outer training loop:  92%|#########2| 46027/50000 [30:41:59<2:03:33,  1.87s/it]
Outer training loop:  92%|#########2| 46028/50000 [30:42:00<2:01:44,  1.84s/it]
Outer training loop:  92%|#########2| 46029/50000 [30:42:03<2:06:27,  1.91s/it]
Outer training loop:  92%|#########2| 46030/50000 [30:42:04<2:03:29,  1.87s/it]
Outer training loop:  92%|#########2| 46031/50000 [30:42:06<2:03:12,  1.86s/it]
Outer training loop:  92%|#########2| 46032/50000 [30:42:08<2:06:06,  1.91s/it]
Outer training loop:  92%|#########2| 46033/50000 [30:42:10<2:04:32,  1.88s/it]
Outer training loop:  92%|#########2| 46034/50000 [30:42:12<2:01:52,  1.84s/it]
Outer training loop:  92%|#########2| 46035/50000 [30:42:14<2:05:59,  1.91s/it]
Outer training loop:  92%|#########2| 46036/50000 [30:42:16<2:03:19,  1.87s/it]
Outer training loop:  92%|#########2| 46037/50000 [30:42:17<2:01:13,  1.84s/it]
Outer training loop:  92%|#########2| 46038/50000 [30:42:19<2:05:03,  1.89s/it]
Outer training loop:  92%|#########2| 46039/50000 [30:42:21<2:01:39,  1.84s/it]
Outer training loop:  92%|#########2| 46040/50000 [30:42:23<2:00:21,  1.82s/it]
Outer training loop:  92%|#########2| 46041/50000 [30:42:25<2:02:12,  1.85s/it]
Outer training loop:  92%|#########2| 46042/50000 [30:42:27<2:04:54,  1.89s/it]
Outer training loop:  92%|#########2| 46043/50000 [30:42:29<2:04:07,  1.88s/it]
Outer training loop:  92%|#########2| 46044/50000 [30:42:30<2:01:42,  1.85s/it]
Outer training loop:  92%|#########2| 46045/50000 [30:42:32<2:05:35,  1.91s/it]
Outer training loop:  92%|#########2| 46046/50000 [30:42:34<2:04:55,  1.90s/it]
Outer training loop:  92%|#########2| 46047/50000 [30:42:36<2:01:29,  1.84s/it]
Outer training loop:  92%|#########2| 46048/50000 [30:42:38<2:05:56,  1.91s/it]
Outer training loop:  92%|#########2| 46049/50000 [30:42:40<2:05:03,  1.90s/it]
Outer training loop:  92%|#########2| 46050/50000 [30:42:42<2:05:34,  1.91s/it]
Outer training loop:  92%|#########2| 46051/50000 [30:42:44<2:04:57,  1.90s/it]
Outer training loop:  92%|#########2| 46052/50000 [30:42:46<2:05:40,  1.91s/it]
Outer training loop:  92%|#########2| 46053/50000 [30:42:47<2:02:57,  1.87s/it]
Outer training loop:  92%|#########2| 46054/50000 [30:42:49<2:03:44,  1.88s/it]
Outer training loop:  92%|#########2| 46055/50000 [30:42:51<2:01:32,  1.85s/it]
Outer training loop:  92%|#########2| 46056/50000 [30:42:53<2:04:49,  1.90s/it]
Outer training loop:  92%|#########2| 46057/50000 [30:42:55<2:05:13,  1.91s/it]
Outer training loop:  92%|#########2| 46058/50000 [30:42:57<2:01:51,  1.85s/it]
Outer training loop:  92%|#########2| 46059/50000 [30:42:59<2:04:17,  1.89s/it]
Outer training loop:  92%|#########2| 46060/50000 [30:43:01<2:03:16,  1.88s/it]
Outer training loop:  92%|#########2| 46061/50000 [30:43:03<2:03:28,  1.88s/it]
Outer training loop:  92%|#########2| 46062/50000 [30:43:05<2:05:38,  1.91s/it]
Outer training loop:  92%|#########2| 46063/50000 [30:43:06<2:04:36,  1.90s/it]
Outer training loop:  92%|#########2| 46064/50000 [30:43:08<2:03:02,  1.88s/it]
Outer training loop:  92%|#########2| 46065/50000 [30:43:10<2:02:17,  1.86s/it]
Outer training loop:  92%|#########2| 46066/50000 [30:43:12<2:00:35,  1.84s/it]
Outer training loop:  92%|#########2| 46067/50000 [30:43:14<2:03:23,  1.88s/it]
Outer training loop:  92%|#########2| 46068/50000 [30:43:16<2:00:49,  1.84s/it]
Outer training loop:  92%|#########2| 46069/50000 [30:43:17<1:59:14,  1.82s/it]
Outer training loop:  92%|#########2| 46070/50000 [30:43:19<2:04:35,  1.90s/it]
Outer training loop:  92%|#########2| 46071/50000 [30:43:21<2:02:47,  1.88s/it]
Outer training loop:  92%|#########2| 46072/50000 [30:43:23<2:00:16,  1.84s/it]
Outer training loop:  92%|#########2| 46073/50000 [30:43:25<2:04:24,  1.90s/it]
Outer training loop:  92%|#########2| 46074/50000 [30:43:27<2:04:44,  1.91s/it]
Outer training loop:  92%|#########2| 46075/50000 [30:43:29<2:05:10,  1.91s/it]
Outer training loop:  92%|#########2| 46076/50000 [30:43:31<2:07:29,  1.95s/it]
Outer training loop:  92%|#########2| 46077/50000 [30:43:33<2:02:37,  1.88s/it]
Outer training loop:  92%|#########2| 46078/50000 [30:43:35<2:03:05,  1.88s/it]
Outer training loop:  92%|#########2| 46079/50000 [30:43:36<2:01:39,  1.86s/it]
Outer training loop:  92%|#########2| 46080/50000 [30:43:38<2:04:50,  1.91s/it]
Outer training loop:  92%|#########2| 46081/50000 [30:43:40<2:03:40,  1.89s/it]
Outer training loop:  92%|#########2| 46082/50000 [30:43:42<2:01:22,  1.86s/it]
Outer training loop:  92%|#########2| 46083/50000 [30:43:44<2:03:37,  1.89s/it]
Outer training loop:  92%|#########2| 46084/50000 [30:43:46<2:01:04,  1.86s/it]
Outer training loop:  92%|#########2| 46085/50000 [30:43:48<1:59:23,  1.83s/it]
Outer training loop:  92%|#########2| 46086/50000 [30:43:49<1:58:25,  1.82s/it]
Outer training loop:  92%|#########2| 46087/50000 [30:43:51<1:59:52,  1.84s/it]
Outer training loop:  92%|#########2| 46088/50000 [30:43:53<2:00:20,  1.85s/it]
Outer training loop:  92%|#########2| 46089/50000 [30:43:55<1:59:03,  1.83s/it]
Outer training loop:  92%|#########2| 46090/50000 [30:43:57<2:02:35,  1.88s/it]
Outer training loop:  92%|#########2| 46091/50000 [30:43:59<2:01:56,  1.87s/it]
Outer training loop:  92%|#########2| 46092/50000 [30:44:01<2:00:52,  1.86s/it]
Outer training loop:  92%|#########2| 46093/50000 [30:44:02<2:02:22,  1.88s/it]
Outer training loop:  92%|#########2| 46094/50000 [30:44:04<2:03:12,  1.89s/it]
Outer training loop:  92%|#########2| 46095/50000 [30:44:06<2:03:54,  1.90s/it]
Outer training loop:  92%|#########2| 46096/50000 [30:44:08<2:05:50,  1.93s/it]
Outer training loop:  92%|#########2| 46097/50000 [30:44:10<2:06:39,  1.95s/it]
Outer training loop:  92%|#########2| 46098/50000 [30:44:12<2:01:22,  1.87s/it]
Outer training loop:  92%|#########2| 46099/50000 [30:44:14<2:00:05,  1.85s/it]
Outer training loop:  92%|#########2| 46100/50000 [30:44:16<2:03:59,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.91s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  92%|#########2| 46101/50000 [30:44:23<3:54:02,  3.60s/it]
Outer training loop:  92%|#########2| 46102/50000 [30:44:25<3:22:43,  3.12s/it]
Outer training loop:  92%|#########2| 46103/50000 [30:44:27<2:57:50,  2.74s/it]
Outer training loop:  92%|#########2| 46104/50000 [30:44:29<2:43:33,  2.52s/it]
Outer training loop:  92%|#########2| 46105/50000 [30:44:31<2:31:08,  2.33s/it]
Outer training loop:  92%|#########2| 46106/50000 [30:44:33<2:22:04,  2.19s/it]
Outer training loop:  92%|#########2| 46107/50000 [30:44:35<2:15:08,  2.08s/it]
Outer training loop:  92%|#########2| 46108/50000 [30:44:37<2:15:01,  2.08s/it]
Outer training loop:  92%|#########2| 46109/50000 [30:44:39<2:08:26,  1.98s/it]
Outer training loop:  92%|#########2| 46110/50000 [30:44:41<2:07:06,  1.96s/it]
Outer training loop:  92%|#########2| 46111/50000 [30:44:43<2:08:21,  1.98s/it]
Outer training loop:  92%|#########2| 46112/50000 [30:44:44<2:05:23,  1.94s/it]
Outer training loop:  92%|#########2| 46113/50000 [30:44:46<2:01:12,  1.87s/it]
Outer training loop:  92%|#########2| 46114/50000 [30:44:48<2:02:49,  1.90s/it]
Outer training loop:  92%|#########2| 46115/50000 [30:44:50<2:01:18,  1.87s/it]
Outer training loop:  92%|#########2| 46116/50000 [30:44:52<2:00:56,  1.87s/it]
Outer training loop:  92%|#########2| 46117/50000 [30:44:54<2:01:30,  1.88s/it]
Outer training loop:  92%|#########2| 46118/50000 [30:44:56<2:06:13,  1.95s/it]
Outer training loop:  92%|#########2| 46119/50000 [30:44:58<2:06:23,  1.95s/it]
Outer training loop:  92%|#########2| 46120/50000 [30:45:00<2:03:38,  1.91s/it]
Outer training loop:  92%|#########2| 46121/50000 [30:45:02<2:07:13,  1.97s/it]
Outer training loop:  92%|#########2| 46122/50000 [30:45:03<2:04:37,  1.93s/it]
Outer training loop:  92%|#########2| 46123/50000 [30:45:05<2:04:09,  1.92s/it]
Outer training loop:  92%|#########2| 46124/50000 [30:45:07<2:01:41,  1.88s/it]
Outer training loop:  92%|#########2| 46125/50000 [30:45:09<2:03:55,  1.92s/it]
Outer training loop:  92%|#########2| 46126/50000 [30:45:11<2:01:56,  1.89s/it]
Outer training loop:  92%|#########2| 46127/50000 [30:45:13<2:02:22,  1.90s/it]
Outer training loop:  92%|#########2| 46128/50000 [30:45:15<2:03:16,  1.91s/it]
Outer training loop:  92%|#########2| 46129/50000 [30:45:17<2:01:14,  1.88s/it]
Outer training loop:  92%|#########2| 46130/50000 [30:45:19<2:00:45,  1.87s/it]
Outer training loop:  92%|#########2| 46131/50000 [30:45:20<2:02:34,  1.90s/it]
Outer training loop:  92%|#########2| 46132/50000 [30:45:23<2:05:28,  1.95s/it]
Outer training loop:  92%|#########2| 46133/50000 [30:45:24<2:03:26,  1.92s/it]
Outer training loop:  92%|#########2| 46134/50000 [30:45:26<2:03:50,  1.92s/it]
Outer training loop:  92%|#########2| 46135/50000 [30:45:28<2:05:40,  1.95s/it]
Outer training loop:  92%|#########2| 46136/50000 [30:45:30<2:03:47,  1.92s/it]
Outer training loop:  92%|#########2| 46137/50000 [30:45:32<2:02:32,  1.90s/it]
Outer training loop:  92%|#########2| 46138/50000 [30:45:34<2:06:04,  1.96s/it]
Outer training loop:  92%|#########2| 46139/50000 [30:45:36<2:03:24,  1.92s/it]
Outer training loop:  92%|#########2| 46140/50000 [30:45:38<2:00:38,  1.88s/it]
Outer training loop:  92%|#########2| 46141/50000 [30:45:40<2:03:25,  1.92s/it]
Outer training loop:  92%|#########2| 46142/50000 [30:45:42<2:00:22,  1.87s/it]
Outer training loop:  92%|#########2| 46143/50000 [30:45:44<2:02:44,  1.91s/it]
Outer training loop:  92%|#########2| 46144/50000 [30:45:45<1:59:06,  1.85s/it]
Outer training loop:  92%|#########2| 46145/50000 [30:45:47<2:02:50,  1.91s/it]
Outer training loop:  92%|#########2| 46146/50000 [30:45:49<2:01:04,  1.88s/it]
Outer training loop:  92%|#########2| 46147/50000 [30:45:51<2:00:17,  1.87s/it]
Outer training loop:  92%|#########2| 46148/50000 [30:45:53<2:04:38,  1.94s/it]
Outer training loop:  92%|#########2| 46149/50000 [30:45:55<2:05:11,  1.95s/it]
Outer training loop:  92%|#########2| 46150/50000 [30:45:57<2:01:27,  1.89s/it]
Outer training loop:  92%|#########2| 46151/50000 [30:45:59<2:06:08,  1.97s/it]
Outer training loop:  92%|#########2| 46152/50000 [30:46:01<2:03:22,  1.92s/it]
Outer training loop:  92%|#########2| 46153/50000 [30:46:03<2:01:33,  1.90s/it]
Outer training loop:  92%|#########2| 46154/50000 [30:46:05<2:03:03,  1.92s/it]
Outer training loop:  92%|#########2| 46155/50000 [30:46:06<2:01:31,  1.90s/it]
Outer training loop:  92%|#########2| 46156/50000 [30:46:08<2:01:04,  1.89s/it]
Outer training loop:  92%|#########2| 46157/50000 [30:46:10<1:59:46,  1.87s/it]
Outer training loop:  92%|#########2| 46158/50000 [30:46:12<2:00:17,  1.88s/it]
Outer training loop:  92%|#########2| 46159/50000 [30:46:14<2:00:21,  1.88s/it]
Outer training loop:  92%|#########2| 46160/50000 [30:46:16<1:56:30,  1.82s/it]
Outer training loop:  92%|#########2| 46161/50000 [30:46:18<1:59:12,  1.86s/it]
Outer training loop:  92%|#########2| 46162/50000 [30:46:19<1:59:30,  1.87s/it]
Outer training loop:  92%|#########2| 46163/50000 [30:46:21<2:00:13,  1.88s/it]
Outer training loop:  92%|#########2| 46164/50000 [30:46:23<1:57:41,  1.84s/it]
Outer training loop:  92%|#########2| 46165/50000 [30:46:25<2:01:17,  1.90s/it]
Outer training loop:  92%|#########2| 46166/50000 [30:46:27<2:01:12,  1.90s/it]
Outer training loop:  92%|#########2| 46167/50000 [30:46:29<2:00:42,  1.89s/it]
Outer training loop:  92%|#########2| 46168/50000 [30:46:31<2:03:01,  1.93s/it]
Outer training loop:  92%|#########2| 46169/50000 [30:46:33<2:01:07,  1.90s/it]
Outer training loop:  92%|#########2| 46170/50000 [30:46:35<1:59:20,  1.87s/it]
Outer training loop:  92%|#########2| 46171/50000 [30:46:37<2:02:05,  1.91s/it]
Outer training loop:  92%|#########2| 46172/50000 [30:46:38<1:58:19,  1.85s/it]
Outer training loop:  92%|#########2| 46173/50000 [30:46:40<1:57:21,  1.84s/it]
Outer training loop:  92%|#########2| 46174/50000 [30:46:42<1:56:14,  1.82s/it]
Outer training loop:  92%|#########2| 46175/50000 [30:46:44<2:01:06,  1.90s/it]
Outer training loop:  92%|#########2| 46176/50000 [30:46:46<2:01:17,  1.90s/it]
Outer training loop:  92%|#########2| 46177/50000 [30:46:48<2:02:29,  1.92s/it]
Outer training loop:  92%|#########2| 46178/50000 [30:46:50<2:02:48,  1.93s/it]
Outer training loop:  92%|#########2| 46179/50000 [30:46:52<2:02:38,  1.93s/it]
Outer training loop:  92%|#########2| 46180/50000 [30:46:53<1:57:15,  1.84s/it]
Outer training loop:  92%|#########2| 46181/50000 [30:46:55<2:01:16,  1.91s/it]
Outer training loop:  92%|#########2| 46182/50000 [30:46:57<2:00:01,  1.89s/it]
Outer training loop:  92%|#########2| 46183/50000 [30:46:59<1:58:28,  1.86s/it]
Outer training loop:  92%|#########2| 46184/50000 [30:47:01<2:01:07,  1.90s/it]
Outer training loop:  92%|#########2| 46185/50000 [30:47:03<2:00:33,  1.90s/it]
Outer training loop:  92%|#########2| 46186/50000 [30:47:05<1:58:44,  1.87s/it]
Outer training loop:  92%|#########2| 46187/50000 [30:47:06<1:56:27,  1.83s/it]
Outer training loop:  92%|#########2| 46188/50000 [30:47:08<2:00:31,  1.90s/it]
Outer training loop:  92%|#########2| 46189/50000 [30:47:10<2:00:13,  1.89s/it]
Outer training loop:  92%|#########2| 46190/50000 [30:47:12<1:55:53,  1.83s/it]
Outer training loop:  92%|#########2| 46191/50000 [30:47:14<1:58:31,  1.87s/it]
Outer training loop:  92%|#########2| 46192/50000 [30:47:16<1:57:00,  1.84s/it]
Outer training loop:  92%|#########2| 46193/50000 [30:47:18<1:59:33,  1.88s/it]
Outer training loop:  92%|#########2| 46194/50000 [30:47:20<2:00:41,  1.90s/it]
Outer training loop:  92%|#########2| 46195/50000 [30:47:22<1:59:39,  1.89s/it]
Outer training loop:  92%|#########2| 46196/50000 [30:47:23<1:58:49,  1.87s/it]
Outer training loop:  92%|#########2| 46197/50000 [30:47:25<1:59:34,  1.89s/it]
Outer training loop:  92%|#########2| 46198/50000 [30:47:27<2:03:48,  1.95s/it]
Outer training loop:  92%|#########2| 46199/50000 [30:47:29<2:01:10,  1.91s/it]
Outer training loop:  92%|#########2| 46200/50000 [30:47:31<1:59:12,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  92%|#########2| 46201/50000 [30:47:38<3:44:29,  3.55s/it]
Outer training loop:  92%|#########2| 46202/50000 [30:47:41<3:16:20,  3.10s/it]
Outer training loop:  92%|#########2| 46203/50000 [30:47:42<2:52:42,  2.73s/it]
Outer training loop:  92%|#########2| 46204/50000 [30:47:44<2:35:47,  2.46s/it]
Outer training loop:  92%|#########2| 46205/50000 [30:47:46<2:21:45,  2.24s/it]
Outer training loop:  92%|#########2| 46206/50000 [30:47:48<2:17:09,  2.17s/it]
Outer training loop:  92%|#########2| 46207/50000 [30:47:50<2:09:54,  2.05s/it]
Outer training loop:  92%|#########2| 46208/50000 [30:47:52<2:05:31,  1.99s/it]
Outer training loop:  92%|#########2| 46209/50000 [30:47:54<2:05:39,  1.99s/it]
Outer training loop:  92%|#########2| 46210/50000 [30:47:55<2:02:31,  1.94s/it]
Outer training loop:  92%|#########2| 46211/50000 [30:47:57<2:02:16,  1.94s/it]
Outer training loop:  92%|#########2| 46212/50000 [30:47:59<1:59:19,  1.89s/it]
Outer training loop:  92%|#########2| 46213/50000 [30:48:01<2:01:01,  1.92s/it]
Outer training loop:  92%|#########2| 46214/50000 [30:48:03<1:57:36,  1.86s/it]
Outer training loop:  92%|#########2| 46215/50000 [30:48:05<1:59:13,  1.89s/it]
Outer training loop:  92%|#########2| 46216/50000 [30:48:07<2:01:12,  1.92s/it]
Outer training loop:  92%|#########2| 46217/50000 [30:48:09<1:58:50,  1.88s/it]
Outer training loop:  92%|#########2| 46218/50000 [30:48:10<1:57:29,  1.86s/it]
Outer training loop:  92%|#########2| 46219/50000 [30:48:12<2:00:07,  1.91s/it]
Outer training loop:  92%|#########2| 46220/50000 [30:48:14<2:00:37,  1.91s/it]
Outer training loop:  92%|#########2| 46221/50000 [30:48:16<2:00:39,  1.92s/it]
Outer training loop:  92%|#########2| 46222/50000 [30:48:18<1:56:59,  1.86s/it]
Outer training loop:  92%|#########2| 46223/50000 [30:48:20<2:02:15,  1.94s/it]
Outer training loop:  92%|#########2| 46224/50000 [30:48:22<1:57:39,  1.87s/it]
Outer training loop:  92%|#########2| 46225/50000 [30:48:24<1:55:28,  1.84s/it]
Outer training loop:  92%|#########2| 46226/50000 [30:48:26<1:59:11,  1.89s/it]
Outer training loop:  92%|#########2| 46227/50000 [30:48:27<1:59:01,  1.89s/it]
Outer training loop:  92%|#########2| 46228/50000 [30:48:29<1:57:45,  1.87s/it]
Outer training loop:  92%|#########2| 46229/50000 [30:48:31<1:53:48,  1.81s/it]
Outer training loop:  92%|#########2| 46230/50000 [30:48:33<1:58:17,  1.88s/it]
Outer training loop:  92%|#########2| 46231/50000 [30:48:35<2:00:22,  1.92s/it]
Outer training loop:  92%|#########2| 46232/50000 [30:48:37<1:59:30,  1.90s/it]
Outer training loop:  92%|#########2| 46233/50000 [30:48:39<2:01:46,  1.94s/it]
Outer training loop:  92%|#########2| 46234/50000 [30:48:41<1:59:52,  1.91s/it]
Outer training loop:  92%|#########2| 46235/50000 [30:48:43<2:00:04,  1.91s/it]
Outer training loop:  92%|#########2| 46236/50000 [30:48:45<2:04:17,  1.98s/it]
Outer training loop:  92%|#########2| 46237/50000 [30:48:47<1:59:53,  1.91s/it]
Outer training loop:  92%|#########2| 46238/50000 [30:48:48<1:58:55,  1.90s/it]
Outer training loop:  92%|#########2| 46239/50000 [30:48:51<2:02:23,  1.95s/it]
Outer training loop:  92%|#########2| 46240/50000 [30:48:53<2:03:18,  1.97s/it]
Outer training loop:  92%|#########2| 46241/50000 [30:48:54<2:01:08,  1.93s/it]
Outer training loop:  92%|#########2| 46242/50000 [30:48:56<2:01:33,  1.94s/it]
Outer training loop:  92%|#########2| 46243/50000 [30:48:58<1:59:29,  1.91s/it]
Outer training loop:  92%|#########2| 46244/50000 [30:49:00<1:57:30,  1.88s/it]
Outer training loop:  92%|#########2| 46245/50000 [30:49:02<1:55:27,  1.84s/it]
Outer training loop:  92%|#########2| 46246/50000 [30:49:04<1:58:56,  1.90s/it]
Outer training loop:  92%|#########2| 46247/50000 [30:49:06<1:58:40,  1.90s/it]
Outer training loop:  92%|#########2| 46248/50000 [30:49:07<1:57:24,  1.88s/it]
Outer training loop:  92%|#########2| 46249/50000 [30:49:10<2:02:08,  1.95s/it]
Outer training loop:  92%|#########2| 46250/50000 [30:49:11<1:56:56,  1.87s/it]
Outer training loop:  93%|#########2| 46251/50000 [30:49:13<1:55:53,  1.85s/it]
Outer training loop:  93%|#########2| 46252/50000 [30:49:15<1:54:30,  1.83s/it]
Outer training loop:  93%|#########2| 46253/50000 [30:49:17<1:57:39,  1.88s/it]
Outer training loop:  93%|#########2| 46254/50000 [30:49:19<1:55:55,  1.86s/it]
Outer training loop:  93%|#########2| 46255/50000 [30:49:20<1:54:44,  1.84s/it]
Outer training loop:  93%|#########2| 46256/50000 [30:49:22<1:57:05,  1.88s/it]
Outer training loop:  93%|#########2| 46257/50000 [30:49:24<1:55:57,  1.86s/it]
Outer training loop:  93%|#########2| 46258/50000 [30:49:26<1:56:08,  1.86s/it]
Outer training loop:  93%|#########2| 46259/50000 [30:49:28<1:55:18,  1.85s/it]
Outer training loop:  93%|#########2| 46260/50000 [30:49:30<1:56:39,  1.87s/it]
Outer training loop:  93%|#########2| 46261/50000 [30:49:32<1:55:56,  1.86s/it]
Outer training loop:  93%|#########2| 46262/50000 [30:49:33<1:53:43,  1.83s/it]
Outer training loop:  93%|#########2| 46263/50000 [30:49:36<1:58:13,  1.90s/it]
Outer training loop:  93%|#########2| 46264/50000 [30:49:37<1:55:27,  1.85s/it]
Outer training loop:  93%|#########2| 46265/50000 [30:49:39<1:53:44,  1.83s/it]
Outer training loop:  93%|#########2| 46266/50000 [30:49:41<1:54:20,  1.84s/it]
Outer training loop:  93%|#########2| 46267/50000 [30:49:43<1:58:32,  1.91s/it]
Outer training loop:  93%|#########2| 46268/50000 [30:49:45<1:57:08,  1.88s/it]
Outer training loop:  93%|#########2| 46269/50000 [30:49:47<1:56:01,  1.87s/it]
Outer training loop:  93%|#########2| 46270/50000 [30:49:49<1:58:11,  1.90s/it]
Outer training loop:  93%|#########2| 46271/50000 [30:49:50<1:57:33,  1.89s/it]
Outer training loop:  93%|#########2| 46272/50000 [30:49:52<1:55:45,  1.86s/it]
Outer training loop:  93%|#########2| 46273/50000 [30:49:54<1:56:19,  1.87s/it]
Outer training loop:  93%|#########2| 46274/50000 [30:49:56<1:59:49,  1.93s/it]
Outer training loop:  93%|#########2| 46275/50000 [30:49:58<1:56:46,  1.88s/it]
Outer training loop:  93%|#########2| 46276/50000 [30:50:00<1:56:22,  1.87s/it]
Outer training loop:  93%|#########2| 46277/50000 [30:50:02<1:58:26,  1.91s/it]
Outer training loop:  93%|#########2| 46278/50000 [30:50:04<1:55:32,  1.86s/it]
Outer training loop:  93%|#########2| 46279/50000 [30:50:05<1:54:29,  1.85s/it]
Outer training loop:  93%|#########2| 46280/50000 [30:50:07<1:53:35,  1.83s/it]
Outer training loop:  93%|#########2| 46281/50000 [30:50:09<1:56:00,  1.87s/it]
Outer training loop:  93%|#########2| 46282/50000 [30:50:11<1:54:51,  1.85s/it]
Outer training loop:  93%|#########2| 46283/50000 [30:50:13<1:54:45,  1.85s/it]
Outer training loop:  93%|#########2| 46284/50000 [30:50:15<1:56:51,  1.89s/it]
Outer training loop:  93%|#########2| 46285/50000 [30:50:17<1:54:31,  1.85s/it]
Outer training loop:  93%|#########2| 46286/50000 [30:50:18<1:53:25,  1.83s/it]
Outer training loop:  93%|#########2| 46287/50000 [30:50:20<1:55:20,  1.86s/it]
Outer training loop:  93%|#########2| 46288/50000 [30:50:22<1:56:24,  1.88s/it]
Outer training loop:  93%|#########2| 46289/50000 [30:50:24<1:55:57,  1.87s/it]
Outer training loop:  93%|#########2| 46290/50000 [30:50:26<2:00:44,  1.95s/it]
Outer training loop:  93%|#########2| 46291/50000 [30:50:28<2:01:07,  1.96s/it]
Outer training loop:  93%|#########2| 46292/50000 [30:50:30<1:58:19,  1.91s/it]
Outer training loop:  93%|#########2| 46293/50000 [30:50:32<1:56:00,  1.88s/it]
Outer training loop:  93%|#########2| 46294/50000 [30:50:34<1:58:04,  1.91s/it]
Outer training loop:  93%|#########2| 46295/50000 [30:50:36<1:56:28,  1.89s/it]
Outer training loop:  93%|#########2| 46296/50000 [30:50:37<1:55:44,  1.87s/it]
Outer training loop:  93%|#########2| 46297/50000 [30:50:39<1:55:30,  1.87s/it]
Outer training loop:  93%|#########2| 46298/50000 [30:50:41<1:58:04,  1.91s/it]
Outer training loop:  93%|#########2| 46299/50000 [30:50:43<1:58:17,  1.92s/it]
Outer training loop:  93%|#########2| 46300/50000 [30:50:45<1:58:16,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  93%|#########2| 46301/50000 [30:50:53<3:42:46,  3.61s/it]
Outer training loop:  93%|#########2| 46302/50000 [30:50:55<3:14:08,  3.15s/it]
Outer training loop:  93%|#########2| 46303/50000 [30:50:57<2:49:07,  2.74s/it]
Outer training loop:  93%|#########2| 46304/50000 [30:50:59<2:33:00,  2.48s/it]
Outer training loop:  93%|#########2| 46305/50000 [30:51:00<2:23:11,  2.33s/it]
Outer training loop:  93%|#########2| 46306/50000 [30:51:02<2:14:04,  2.18s/it]
Outer training loop:  93%|#########2| 46307/50000 [30:51:04<2:10:45,  2.12s/it]
Outer training loop:  93%|#########2| 46308/50000 [30:51:06<2:04:47,  2.03s/it]
Outer training loop:  93%|#########2| 46309/50000 [30:51:08<2:06:04,  2.05s/it]
Outer training loop:  93%|#########2| 46310/50000 [30:51:10<2:04:24,  2.02s/it]
Outer training loop:  93%|#########2| 46311/50000 [30:51:12<2:02:04,  1.99s/it]
Outer training loop:  93%|#########2| 46312/50000 [30:51:14<2:05:01,  2.03s/it]
Outer training loop:  93%|#########2| 46313/50000 [30:51:16<2:02:27,  1.99s/it]
Outer training loop:  93%|#########2| 46314/50000 [30:51:18<1:59:07,  1.94s/it]
Outer training loop:  93%|#########2| 46315/50000 [30:51:20<2:00:52,  1.97s/it]
Outer training loop:  93%|#########2| 46316/50000 [30:51:22<1:56:38,  1.90s/it]
Outer training loop:  93%|#########2| 46317/50000 [30:51:24<1:59:31,  1.95s/it]
Outer training loop:  93%|#########2| 46318/50000 [30:51:26<2:03:15,  2.01s/it]
Outer training loop:  93%|#########2| 46319/50000 [30:51:28<2:00:51,  1.97s/it]
Outer training loop:  93%|#########2| 46320/50000 [30:51:30<1:56:57,  1.91s/it]
Outer training loop:  93%|#########2| 46321/50000 [30:51:32<1:59:50,  1.95s/it]
Outer training loop:  93%|#########2| 46322/50000 [30:51:33<1:57:22,  1.91s/it]
Outer training loop:  93%|#########2| 46323/50000 [30:51:35<1:57:09,  1.91s/it]
Outer training loop:  93%|#########2| 46324/50000 [30:51:37<1:56:41,  1.90s/it]
Outer training loop:  93%|#########2| 46325/50000 [30:51:39<1:57:54,  1.93s/it]
Outer training loop:  93%|#########2| 46326/50000 [30:51:41<1:56:51,  1.91s/it]
Outer training loop:  93%|#########2| 46327/50000 [30:51:43<1:56:16,  1.90s/it]
Outer training loop:  93%|#########2| 46328/50000 [30:51:45<1:57:15,  1.92s/it]
Outer training loop:  93%|#########2| 46329/50000 [30:51:47<1:56:17,  1.90s/it]
Outer training loop:  93%|#########2| 46330/50000 [30:51:49<1:56:16,  1.90s/it]
Outer training loop:  93%|#########2| 46331/50000 [30:51:51<1:56:14,  1.90s/it]
Outer training loop:  93%|#########2| 46332/50000 [30:51:53<1:59:25,  1.95s/it]
Outer training loop:  93%|#########2| 46333/50000 [30:51:54<1:56:52,  1.91s/it]
Outer training loop:  93%|#########2| 46334/50000 [30:51:56<1:53:45,  1.86s/it]
Outer training loop:  93%|#########2| 46335/50000 [30:51:58<1:57:32,  1.92s/it]
Outer training loop:  93%|#########2| 46336/50000 [30:52:00<1:57:06,  1.92s/it]
Outer training loop:  93%|#########2| 46337/50000 [30:52:02<1:59:06,  1.95s/it]
Outer training loop:  93%|#########2| 46338/50000 [30:52:04<2:02:05,  2.00s/it]
Outer training loop:  93%|#########2| 46339/50000 [30:52:06<1:59:00,  1.95s/it]
Outer training loop:  93%|#########2| 46340/50000 [30:52:08<1:58:24,  1.94s/it]
Outer training loop:  93%|#########2| 46341/50000 [30:52:10<1:55:52,  1.90s/it]
Outer training loop:  93%|#########2| 46342/50000 [30:52:12<1:59:04,  1.95s/it]
Outer training loop:  93%|#########2| 46343/50000 [30:52:14<1:56:49,  1.92s/it]
Outer training loop:  93%|#########2| 46344/50000 [30:52:16<1:53:34,  1.86s/it]
Outer training loop:  93%|#########2| 46345/50000 [30:52:18<1:55:43,  1.90s/it]
Outer training loop:  93%|#########2| 46346/50000 [30:52:19<1:54:21,  1.88s/it]
Outer training loop:  93%|#########2| 46347/50000 [30:52:21<1:54:31,  1.88s/it]
Outer training loop:  93%|#########2| 46348/50000 [30:52:23<1:53:35,  1.87s/it]
Outer training loop:  93%|#########2| 46349/50000 [30:52:25<1:58:39,  1.95s/it]
Outer training loop:  93%|#########2| 46350/50000 [30:52:27<1:55:54,  1.91s/it]
Outer training loop:  93%|#########2| 46351/50000 [30:52:29<1:55:39,  1.90s/it]
Outer training loop:  93%|#########2| 46352/50000 [30:52:31<1:56:41,  1.92s/it]
Outer training loop:  93%|#########2| 46353/50000 [30:52:33<1:54:40,  1.89s/it]
Outer training loop:  93%|#########2| 46354/50000 [30:52:35<1:56:02,  1.91s/it]
Outer training loop:  93%|#########2| 46355/50000 [30:52:37<1:57:15,  1.93s/it]
Outer training loop:  93%|#########2| 46356/50000 [30:52:39<1:56:48,  1.92s/it]
Outer training loop:  93%|#########2| 46357/50000 [30:52:40<1:54:09,  1.88s/it]
Outer training loop:  93%|#########2| 46358/50000 [30:52:42<1:52:48,  1.86s/it]
Outer training loop:  93%|#########2| 46359/50000 [30:52:44<1:51:59,  1.85s/it]
Outer training loop:  93%|#########2| 46360/50000 [30:52:46<1:54:36,  1.89s/it]
Outer training loop:  93%|#########2| 46361/50000 [30:52:48<1:53:34,  1.87s/it]
Outer training loop:  93%|#########2| 46362/50000 [30:52:50<1:54:05,  1.88s/it]
Outer training loop:  93%|#########2| 46363/50000 [30:52:52<1:57:08,  1.93s/it]
Outer training loop:  93%|#########2| 46364/50000 [30:52:53<1:52:18,  1.85s/it]
Outer training loop:  93%|#########2| 46365/50000 [30:52:55<1:54:44,  1.89s/it]
Outer training loop:  93%|#########2| 46366/50000 [30:52:57<1:53:04,  1.87s/it]
Outer training loop:  93%|#########2| 46367/50000 [30:52:59<1:56:13,  1.92s/it]
Outer training loop:  93%|#########2| 46368/50000 [30:53:01<1:54:47,  1.90s/it]
Outer training loop:  93%|#########2| 46369/50000 [30:53:03<1:54:38,  1.89s/it]
Outer training loop:  93%|#########2| 46370/50000 [30:53:05<1:56:56,  1.93s/it]
Outer training loop:  93%|#########2| 46371/50000 [30:53:07<1:54:00,  1.88s/it]
Outer training loop:  93%|#########2| 46372/50000 [30:53:09<1:52:36,  1.86s/it]
Outer training loop:  93%|#########2| 46373/50000 [30:53:10<1:51:49,  1.85s/it]
Outer training loop:  93%|#########2| 46374/50000 [30:53:12<1:53:10,  1.87s/it]
Outer training loop:  93%|#########2| 46375/50000 [30:53:14<1:53:09,  1.87s/it]
Outer training loop:  93%|#########2| 46376/50000 [30:53:16<1:51:09,  1.84s/it]
Outer training loop:  93%|#########2| 46377/50000 [30:53:18<1:52:44,  1.87s/it]
Outer training loop:  93%|#########2| 46378/50000 [30:53:20<1:51:18,  1.84s/it]
Outer training loop:  93%|#########2| 46379/50000 [30:53:21<1:49:49,  1.82s/it]
Outer training loop:  93%|#########2| 46380/50000 [30:53:23<1:53:26,  1.88s/it]
Outer training loop:  93%|#########2| 46381/50000 [30:53:25<1:52:57,  1.87s/it]
Outer training loop:  93%|#########2| 46382/50000 [30:53:27<1:52:54,  1.87s/it]
Outer training loop:  93%|#########2| 46383/50000 [30:53:29<1:51:30,  1.85s/it]
Outer training loop:  93%|#########2| 46384/50000 [30:53:31<1:52:49,  1.87s/it]
Outer training loop:  93%|#########2| 46385/50000 [30:53:33<1:53:06,  1.88s/it]
Outer training loop:  93%|#########2| 46386/50000 [30:53:35<1:51:01,  1.84s/it]
Outer training loop:  93%|#########2| 46387/50000 [30:53:37<1:57:56,  1.96s/it]
Outer training loop:  93%|#########2| 46388/50000 [30:53:39<1:55:50,  1.92s/it]
Outer training loop:  93%|#########2| 46389/50000 [30:53:40<1:51:50,  1.86s/it]
Outer training loop:  93%|#########2| 46390/50000 [30:53:42<1:47:58,  1.79s/it]
Outer training loop:  93%|#########2| 46391/50000 [30:53:44<1:53:18,  1.88s/it]
Outer training loop:  93%|#########2| 46392/50000 [30:53:46<1:51:16,  1.85s/it]
Outer training loop:  93%|#########2| 46393/50000 [30:53:48<1:50:45,  1.84s/it]
Outer training loop:  93%|#########2| 46394/50000 [30:53:50<1:56:38,  1.94s/it]
Outer training loop:  93%|#########2| 46395/50000 [30:53:52<1:53:31,  1.89s/it]
Outer training loop:  93%|#########2| 46396/50000 [30:53:53<1:50:53,  1.85s/it]
Outer training loop:  93%|#########2| 46397/50000 [30:53:55<1:50:40,  1.84s/it]
Outer training loop:  93%|#########2| 46398/50000 [30:53:57<1:55:19,  1.92s/it]
Outer training loop:  93%|#########2| 46399/50000 [30:53:59<1:53:39,  1.89s/it]
Outer training loop:  93%|#########2| 46400/50000 [30:54:01<1:52:33,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.90s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  93%|#########2| 46401/50000 [30:54:08<3:32:26,  3.54s/it]
Outer training loop:  93%|#########2| 46402/50000 [30:54:10<3:05:11,  3.09s/it]
Outer training loop:  93%|#########2| 46403/50000 [30:54:12<2:43:46,  2.73s/it]
Outer training loop:  93%|#########2| 46404/50000 [30:54:14<2:28:37,  2.48s/it]
Outer training loop:  93%|#########2| 46405/50000 [30:54:16<2:21:37,  2.36s/it]
Outer training loop:  93%|#########2| 46406/50000 [30:54:18<2:12:46,  2.22s/it]
Outer training loop:  93%|#########2| 46407/50000 [30:54:20<2:06:01,  2.10s/it]
Outer training loop:  93%|#########2| 46408/50000 [30:54:22<2:05:25,  2.10s/it]
Outer training loop:  93%|#########2| 46409/50000 [30:54:24<2:00:20,  2.01s/it]
Outer training loop:  93%|#########2| 46410/50000 [30:54:26<1:57:05,  1.96s/it]
Outer training loop:  93%|#########2| 46411/50000 [30:54:28<1:54:52,  1.92s/it]
Outer training loop:  93%|#########2| 46412/50000 [30:54:29<1:54:01,  1.91s/it]
Outer training loop:  93%|#########2| 46413/50000 [30:54:31<1:50:40,  1.85s/it]
Outer training loop:  93%|#########2| 46414/50000 [30:54:33<1:50:46,  1.85s/it]
Outer training loop:  93%|#########2| 46415/50000 [30:54:35<1:52:17,  1.88s/it]
Outer training loop:  93%|#########2| 46416/50000 [30:54:37<1:54:28,  1.92s/it]
Outer training loop:  93%|#########2| 46417/50000 [30:54:39<1:50:13,  1.85s/it]
Outer training loop:  93%|#########2| 46418/50000 [30:54:40<1:49:49,  1.84s/it]
Outer training loop:  93%|#########2| 46419/50000 [30:54:42<1:53:10,  1.90s/it]
Outer training loop:  93%|#########2| 46420/50000 [30:54:44<1:54:20,  1.92s/it]
Outer training loop:  93%|#########2| 46421/50000 [30:54:46<1:55:12,  1.93s/it]
Outer training loop:  93%|#########2| 46422/50000 [30:54:49<1:59:18,  2.00s/it]
Outer training loop:  93%|#########2| 46423/50000 [30:54:50<1:56:19,  1.95s/it]
Outer training loop:  93%|#########2| 46424/50000 [30:54:52<1:54:38,  1.92s/it]
Outer training loop:  93%|#########2| 46425/50000 [30:54:54<1:50:30,  1.85s/it]
Outer training loop:  93%|#########2| 46426/50000 [30:54:56<1:53:52,  1.91s/it]
Outer training loop:  93%|#########2| 46427/50000 [30:54:58<1:53:51,  1.91s/it]
Outer training loop:  93%|#########2| 46428/50000 [30:55:00<1:52:50,  1.90s/it]
Outer training loop:  93%|#########2| 46429/50000 [30:55:02<1:54:52,  1.93s/it]
Outer training loop:  93%|#########2| 46430/50000 [30:55:04<1:53:42,  1.91s/it]
Outer training loop:  93%|#########2| 46431/50000 [30:55:05<1:51:36,  1.88s/it]
Outer training loop:  93%|#########2| 46432/50000 [30:55:07<1:54:37,  1.93s/it]
Outer training loop:  93%|#########2| 46433/50000 [30:55:09<1:54:11,  1.92s/it]
Outer training loop:  93%|#########2| 46434/50000 [30:55:11<1:56:06,  1.95s/it]
Outer training loop:  93%|#########2| 46435/50000 [30:55:13<1:53:10,  1.90s/it]
Outer training loop:  93%|#########2| 46436/50000 [30:55:15<1:53:43,  1.91s/it]
Outer training loop:  93%|#########2| 46437/50000 [30:55:17<1:51:14,  1.87s/it]
Outer training loop:  93%|#########2| 46438/50000 [30:55:19<1:54:01,  1.92s/it]
Outer training loop:  93%|#########2| 46439/50000 [30:55:21<1:55:40,  1.95s/it]
Outer training loop:  93%|#########2| 46440/50000 [30:55:23<1:52:43,  1.90s/it]
Outer training loop:  93%|#########2| 46441/50000 [30:55:25<1:52:26,  1.90s/it]
Outer training loop:  93%|#########2| 46442/50000 [30:55:27<1:55:39,  1.95s/it]
Outer training loop:  93%|#########2| 46443/50000 [30:55:29<1:56:46,  1.97s/it]
Outer training loop:  93%|#########2| 46444/50000 [30:55:30<1:53:02,  1.91s/it]
Outer training loop:  93%|#########2| 46445/50000 [30:55:32<1:53:22,  1.91s/it]
Outer training loop:  93%|#########2| 46446/50000 [30:55:34<1:55:49,  1.96s/it]
Outer training loop:  93%|#########2| 46447/50000 [30:55:36<1:53:35,  1.92s/it]
Outer training loop:  93%|#########2| 46448/50000 [30:55:38<1:51:52,  1.89s/it]
Outer training loop:  93%|#########2| 46449/50000 [30:55:40<1:55:04,  1.94s/it]
Outer training loop:  93%|#########2| 46450/50000 [30:55:42<1:51:53,  1.89s/it]
Outer training loop:  93%|#########2| 46451/50000 [30:55:44<1:52:55,  1.91s/it]
Outer training loop:  93%|#########2| 46452/50000 [30:55:46<1:54:00,  1.93s/it]
Outer training loop:  93%|#########2| 46453/50000 [30:55:48<1:52:47,  1.91s/it]
Outer training loop:  93%|#########2| 46454/50000 [30:55:50<1:52:51,  1.91s/it]
Outer training loop:  93%|#########2| 46455/50000 [30:55:52<1:52:39,  1.91s/it]
Outer training loop:  93%|#########2| 46456/50000 [30:55:54<1:54:09,  1.93s/it]
Outer training loop:  93%|#########2| 46457/50000 [30:55:55<1:51:35,  1.89s/it]
Outer training loop:  93%|#########2| 46458/50000 [30:55:57<1:49:45,  1.86s/it]
Outer training loop:  93%|#########2| 46459/50000 [30:55:59<1:53:52,  1.93s/it]
Outer training loop:  93%|#########2| 46460/50000 [30:56:01<1:50:52,  1.88s/it]
Outer training loop:  93%|#########2| 46461/50000 [30:56:03<1:52:00,  1.90s/it]
Outer training loop:  93%|#########2| 46462/50000 [30:56:05<1:50:26,  1.87s/it]
Outer training loop:  93%|#########2| 46463/50000 [30:56:07<1:52:56,  1.92s/it]
Outer training loop:  93%|#########2| 46464/50000 [30:56:09<1:50:08,  1.87s/it]
Outer training loop:  93%|#########2| 46465/50000 [30:56:10<1:50:12,  1.87s/it]
Outer training loop:  93%|#########2| 46466/50000 [30:56:12<1:53:46,  1.93s/it]
Outer training loop:  93%|#########2| 46467/50000 [30:56:14<1:52:34,  1.91s/it]
Outer training loop:  93%|#########2| 46468/50000 [30:56:16<1:51:50,  1.90s/it]
Outer training loop:  93%|#########2| 46469/50000 [30:56:18<1:49:53,  1.87s/it]
Outer training loop:  93%|#########2| 46470/50000 [30:56:20<1:54:14,  1.94s/it]
Outer training loop:  93%|#########2| 46471/50000 [30:56:22<1:52:19,  1.91s/it]
Outer training loop:  93%|#########2| 46472/50000 [30:56:24<1:49:55,  1.87s/it]
Outer training loop:  93%|#########2| 46473/50000 [30:56:26<1:48:56,  1.85s/it]
Outer training loop:  93%|#########2| 46474/50000 [30:56:28<1:52:25,  1.91s/it]
Outer training loop:  93%|#########2| 46475/50000 [30:56:29<1:49:58,  1.87s/it]
Outer training loop:  93%|#########2| 46476/50000 [30:56:31<1:47:29,  1.83s/it]
Outer training loop:  93%|#########2| 46477/50000 [30:56:33<1:49:21,  1.86s/it]
Outer training loop:  93%|#########2| 46478/50000 [30:56:35<1:51:09,  1.89s/it]
Outer training loop:  93%|#########2| 46479/50000 [30:56:37<1:49:49,  1.87s/it]
Outer training loop:  93%|#########2| 46480/50000 [30:56:39<1:53:39,  1.94s/it]
Outer training loop:  93%|#########2| 46481/50000 [30:56:41<1:50:35,  1.89s/it]
Outer training loop:  93%|#########2| 46482/50000 [30:56:42<1:48:29,  1.85s/it]
Outer training loop:  93%|#########2| 46483/50000 [30:56:45<1:52:23,  1.92s/it]
Outer training loop:  93%|#########2| 46484/50000 [30:56:47<1:53:53,  1.94s/it]
Outer training loop:  93%|#########2| 46485/50000 [30:56:48<1:51:54,  1.91s/it]
Outer training loop:  93%|#########2| 46486/50000 [30:56:51<1:55:51,  1.98s/it]
Outer training loop:  93%|#########2| 46487/50000 [30:56:52<1:54:49,  1.96s/it]
Outer training loop:  93%|#########2| 46488/50000 [30:56:54<1:51:56,  1.91s/it]
Outer training loop:  93%|#########2| 46489/50000 [30:56:56<1:51:13,  1.90s/it]
Outer training loop:  93%|#########2| 46490/50000 [30:56:58<1:54:22,  1.96s/it]
Outer training loop:  93%|#########2| 46491/50000 [30:57:00<1:52:50,  1.93s/it]
Outer training loop:  93%|#########2| 46492/50000 [30:57:02<1:52:02,  1.92s/it]
Outer training loop:  93%|#########2| 46493/50000 [30:57:04<1:49:38,  1.88s/it]
Outer training loop:  93%|#########2| 46494/50000 [30:57:06<1:52:28,  1.92s/it]
Outer training loop:  93%|#########2| 46495/50000 [30:57:08<1:51:58,  1.92s/it]
Outer training loop:  93%|#########2| 46496/50000 [30:57:09<1:50:21,  1.89s/it]
Outer training loop:  93%|#########2| 46497/50000 [30:57:12<1:54:32,  1.96s/it]
Outer training loop:  93%|#########2| 46498/50000 [30:57:13<1:51:55,  1.92s/it]
Outer training loop:  93%|#########2| 46499/50000 [30:57:15<1:51:27,  1.91s/it]
Outer training loop:  93%|#########3| 46500/50000 [30:57:17<1:52:37,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:28,  1.81s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:27,  1.83s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:31,  1.90s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.89s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:18,  1.85s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:18,  1.87s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:20,  1.93s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.88s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:12,  1.87s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:09,  1.85s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:07,  1.87s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:02,  1.83s/it]

Evaluating policy:  20%|##        | 17/83 [00:31<02:01,  1.83s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<02:04,  1.91s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:57,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:54,  1.85s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:52,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:46<01:48,  1.87s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:44,  1.87s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:46,  1.94s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:39,  1.91s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.88s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:35,  1.95s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:30,  1.88s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:28,  1.87s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:20,  1.78s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:19,  1.80s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:21,  1.89s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.87s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:17,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:14,  1.90s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:09,  1.87s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:10,  1.97s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:08,  1.95s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:05,  1.91s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:02,  1.90s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:58,  1.90s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:57,  1.90s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:52,  1.88s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:53,  2.00s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:51,  1.98s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:48,  1.94s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.89s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.88s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.88s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:38,  1.85s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.97s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.93s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:34,  1.93s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:32,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.87s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.87s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:21,  2.00s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.92s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.90s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.93s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.92s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.93s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.89s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.94s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.96s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:04,  2.09s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:02,  2.05s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.93s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:18,  1.70s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:22,  1.76s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:23,  1.80s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.81s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:19,  1.79s/it]

Evaluating policy:   7%|7         | 6/83 [00:10<02:20,  1.82s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:21,  1.86s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:20,  1.87s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:30,  2.04s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:24,  1.98s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:17,  1.97s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:14,  1.95s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.91s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.91s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.90s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:09,  1.99s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:03,  1.93s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.93s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:58,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:59,  1.96s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:56,  1.94s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:54,  1.93s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.93s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:50,  1.98s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:46,  1.94s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:44,  1.93s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:34,  1.85s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:38,  1.97s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.88s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:24,  1.92s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:22,  1.93s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:19,  1.90s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:18,  1.90s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:15,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:13,  1.90s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:18,  2.06s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:13,  1.98s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:11,  1.99s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:07,  1.94s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:34<01:05,  1.94s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:01,  1.87s/it]

Evaluating policy:  61%|######1   | 51/83 [01:38<01:04,  2.00s/it]

Evaluating policy:  63%|######2   | 52/83 [01:40<01:00,  1.96s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:57,  1.93s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:54,  1.87s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:51,  1.85s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:51,  1.90s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:46,  1.87s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:47,  1.98s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:55<00:44,  1.94s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:06<00:30,  1.81s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:31,  1.97s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:28,  1.93s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:21,  1.83s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:19<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.80s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.85s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:12,  1.84s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:12,  2.01s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.92s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.90s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.84s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.86s/it]

Outer training loop:  93%|#########3| 46501/50000 [31:02:41<95:36:32, 98.37s/it]
Outer training loop:  93%|#########3| 46502/50000 [31:02:43<67:28:03, 69.44s/it]
Outer training loop:  93%|#########3| 46503/50000 [31:02:44<47:43:54, 49.14s/it]
Outer training loop:  93%|#########3| 46504/50000 [31:02:47<34:04:11, 35.08s/it]
Outer training loop:  93%|#########3| 46505/50000 [31:02:49<24:24:44, 25.15s/it]
Outer training loop:  93%|#########3| 46506/50000 [31:02:50<17:36:27, 18.14s/it]
Outer training loop:  93%|#########3| 46507/50000 [31:02:52<12:51:59, 13.26s/it]
Outer training loop:  93%|#########3| 46508/50000 [31:02:55<9:39:22,  9.95s/it] 
Outer training loop:  93%|#########3| 46509/50000 [31:02:56<7:15:15,  7.48s/it]
Outer training loop:  93%|#########3| 46510/50000 [31:02:58<5:38:19,  5.82s/it]
Outer training loop:  93%|#########3| 46511/50000 [31:03:00<4:29:23,  4.63s/it]
Outer training loop:  93%|#########3| 46512/50000 [31:03:02<3:43:12,  3.84s/it]
Outer training loop:  93%|#########3| 46513/50000 [31:03:04<3:07:40,  3.23s/it]
Outer training loop:  93%|#########3| 46514/50000 [31:03:06<2:42:18,  2.79s/it]
Outer training loop:  93%|#########3| 46515/50000 [31:03:08<2:32:14,  2.62s/it]
Outer training loop:  93%|#########3| 46516/50000 [31:03:10<2:20:53,  2.43s/it]
Outer training loop:  93%|#########3| 46517/50000 [31:03:12<2:10:32,  2.25s/it]
Outer training loop:  93%|#########3| 46518/50000 [31:03:13<2:03:04,  2.12s/it]
Outer training loop:  93%|#########3| 46519/50000 [31:03:15<2:00:59,  2.09s/it]
Outer training loop:  93%|#########3| 46520/50000 [31:03:17<1:56:52,  2.02s/it]
Outer training loop:  93%|#########3| 46521/50000 [31:03:19<1:55:35,  1.99s/it]
Outer training loop:  93%|#########3| 46522/50000 [31:03:21<1:57:37,  2.03s/it]
Outer training loop:  93%|#########3| 46523/50000 [31:03:23<1:54:39,  1.98s/it]
Outer training loop:  93%|#########3| 46524/50000 [31:03:25<1:54:00,  1.97s/it]
Outer training loop:  93%|#########3| 46525/50000 [31:03:27<1:51:05,  1.92s/it]
Outer training loop:  93%|#########3| 46526/50000 [31:03:29<1:56:25,  2.01s/it]
Outer training loop:  93%|#########3| 46527/50000 [31:03:31<1:54:33,  1.98s/it]
Outer training loop:  93%|#########3| 46528/50000 [31:03:33<1:51:37,  1.93s/it]
Outer training loop:  93%|#########3| 46529/50000 [31:03:35<1:54:08,  1.97s/it]
Outer training loop:  93%|#########3| 46530/50000 [31:03:37<1:50:08,  1.90s/it]
Outer training loop:  93%|#########3| 46531/50000 [31:03:39<1:47:25,  1.86s/it]
Outer training loop:  93%|#########3| 46532/50000 [31:03:41<1:52:41,  1.95s/it]
Outer training loop:  93%|#########3| 46533/50000 [31:03:43<1:51:12,  1.92s/it]
Outer training loop:  93%|#########3| 46534/50000 [31:03:45<1:52:16,  1.94s/it]
Outer training loop:  93%|#########3| 46535/50000 [31:03:46<1:49:19,  1.89s/it]
Outer training loop:  93%|#########3| 46536/50000 [31:03:48<1:51:32,  1.93s/it]
Outer training loop:  93%|#########3| 46537/50000 [31:03:50<1:48:47,  1.88s/it]
Outer training loop:  93%|#########3| 46538/50000 [31:03:52<1:46:38,  1.85s/it]
Outer training loop:  93%|#########3| 46539/50000 [31:03:54<1:44:34,  1.81s/it]
Outer training loop:  93%|#########3| 46540/50000 [31:03:56<1:47:34,  1.87s/it]
Outer training loop:  93%|#########3| 46541/50000 [31:03:58<1:49:46,  1.90s/it]
Outer training loop:  93%|#########3| 46542/50000 [31:03:59<1:49:24,  1.90s/it]
Outer training loop:  93%|#########3| 46543/50000 [31:04:01<1:51:19,  1.93s/it]
Outer training loop:  93%|#########3| 46544/50000 [31:04:03<1:51:10,  1.93s/it]
Outer training loop:  93%|#########3| 46545/50000 [31:04:05<1:52:31,  1.95s/it]
Outer training loop:  93%|#########3| 46546/50000 [31:04:07<1:53:53,  1.98s/it]
Outer training loop:  93%|#########3| 46547/50000 [31:04:09<1:50:47,  1.93s/it]
Outer training loop:  93%|#########3| 46548/50000 [31:04:11<1:48:47,  1.89s/it]
Outer training loop:  93%|#########3| 46549/50000 [31:04:13<1:46:04,  1.84s/it]
Outer training loop:  93%|#########3| 46550/50000 [31:04:15<1:48:25,  1.89s/it]
Outer training loop:  93%|#########3| 46551/50000 [31:04:16<1:45:36,  1.84s/it]
Outer training loop:  93%|#########3| 46552/50000 [31:04:18<1:45:17,  1.83s/it]
Outer training loop:  93%|#########3| 46553/50000 [31:04:20<1:50:22,  1.92s/it]
Outer training loop:  93%|#########3| 46554/50000 [31:04:22<1:47:07,  1.87s/it]
Outer training loop:  93%|#########3| 46555/50000 [31:04:24<1:46:52,  1.86s/it]
Outer training loop:  93%|#########3| 46556/50000 [31:04:26<1:51:41,  1.95s/it]
Outer training loop:  93%|#########3| 46557/50000 [31:04:28<1:48:38,  1.89s/it]
Outer training loop:  93%|#########3| 46558/50000 [31:04:30<1:45:34,  1.84s/it]
Outer training loop:  93%|#########3| 46559/50000 [31:04:31<1:44:21,  1.82s/it]
Outer training loop:  93%|#########3| 46560/50000 [31:04:33<1:48:05,  1.89s/it]
Outer training loop:  93%|#########3| 46561/50000 [31:04:35<1:50:00,  1.92s/it]
Outer training loop:  93%|#########3| 46562/50000 [31:04:37<1:49:03,  1.90s/it]
Outer training loop:  93%|#########3| 46563/50000 [31:04:39<1:50:38,  1.93s/it]
Outer training loop:  93%|#########3| 46564/50000 [31:04:41<1:49:57,  1.92s/it]
Outer training loop:  93%|#########3| 46565/50000 [31:04:43<1:51:49,  1.95s/it]
Outer training loop:  93%|#########3| 46566/50000 [31:04:45<1:50:07,  1.92s/it]
Outer training loop:  93%|#########3| 46567/50000 [31:04:47<1:53:19,  1.98s/it]
Outer training loop:  93%|#########3| 46568/50000 [31:04:49<1:51:01,  1.94s/it]
Outer training loop:  93%|#########3| 46569/50000 [31:04:51<1:50:08,  1.93s/it]
Outer training loop:  93%|#########3| 46570/50000 [31:04:53<1:50:44,  1.94s/it]
Outer training loop:  93%|#########3| 46571/50000 [31:04:55<1:47:41,  1.88s/it]
Outer training loop:  93%|#########3| 46572/50000 [31:04:56<1:46:19,  1.86s/it]
Outer training loop:  93%|#########3| 46573/50000 [31:04:59<1:48:50,  1.91s/it]
Outer training loop:  93%|#########3| 46574/50000 [31:05:00<1:49:33,  1.92s/it]
Outer training loop:  93%|#########3| 46575/50000 [31:05:02<1:47:50,  1.89s/it]
Outer training loop:  93%|#########3| 46576/50000 [31:05:05<1:54:36,  2.01s/it]
Outer training loop:  93%|#########3| 46577/50000 [31:05:06<1:51:21,  1.95s/it]
Outer training loop:  93%|#########3| 46578/50000 [31:05:08<1:49:43,  1.92s/it]
Outer training loop:  93%|#########3| 46579/50000 [31:05:10<1:49:10,  1.91s/it]
Outer training loop:  93%|#########3| 46580/50000 [31:05:12<1:50:35,  1.94s/it]
Outer training loop:  93%|#########3| 46581/50000 [31:05:14<1:51:54,  1.96s/it]
Outer training loop:  93%|#########3| 46582/50000 [31:05:16<1:48:51,  1.91s/it]
Outer training loop:  93%|#########3| 46583/50000 [31:05:18<1:53:27,  1.99s/it]
Outer training loop:  93%|#########3| 46584/50000 [31:05:20<1:50:58,  1.95s/it]
Outer training loop:  93%|#########3| 46585/50000 [31:05:22<1:49:43,  1.93s/it]
Outer training loop:  93%|#########3| 46586/50000 [31:05:24<1:52:13,  1.97s/it]
Outer training loop:  93%|#########3| 46587/50000 [31:05:26<1:49:31,  1.93s/it]
Outer training loop:  93%|#########3| 46588/50000 [31:05:28<1:47:44,  1.89s/it]
Outer training loop:  93%|#########3| 46589/50000 [31:05:29<1:45:29,  1.86s/it]
Outer training loop:  93%|#########3| 46590/50000 [31:05:31<1:48:39,  1.91s/it]
Outer training loop:  93%|#########3| 46591/50000 [31:05:33<1:47:39,  1.89s/it]
Outer training loop:  93%|#########3| 46592/50000 [31:05:35<1:49:17,  1.92s/it]
Outer training loop:  93%|#########3| 46593/50000 [31:05:37<1:50:17,  1.94s/it]
Outer training loop:  93%|#########3| 46594/50000 [31:05:39<1:47:23,  1.89s/it]
Outer training loop:  93%|#########3| 46595/50000 [31:05:41<1:47:18,  1.89s/it]
Outer training loop:  93%|#########3| 46596/50000 [31:05:43<1:47:50,  1.90s/it]
Outer training loop:  93%|#########3| 46597/50000 [31:05:45<1:49:12,  1.93s/it]
Outer training loop:  93%|#########3| 46598/50000 [31:05:46<1:45:25,  1.86s/it]
Outer training loop:  93%|#########3| 46599/50000 [31:05:48<1:46:21,  1.88s/it]
Outer training loop:  93%|#########3| 46600/50000 [31:05:51<1:51:09,  1.96s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  93%|#########3| 46601/50000 [31:05:58<3:27:27,  3.66s/it]
Outer training loop:  93%|#########3| 46602/50000 [31:06:00<2:55:28,  3.10s/it]
Outer training loop:  93%|#########3| 46603/50000 [31:06:02<2:32:19,  2.69s/it]
Outer training loop:  93%|#########3| 46604/50000 [31:06:04<2:21:14,  2.50s/it]
Outer training loop:  93%|#########3| 46605/50000 [31:06:06<2:09:12,  2.28s/it]
Outer training loop:  93%|#########3| 46606/50000 [31:06:07<2:03:15,  2.18s/it]
Outer training loop:  93%|#########3| 46607/50000 [31:06:09<2:00:00,  2.12s/it]
Outer training loop:  93%|#########3| 46608/50000 [31:06:11<1:55:33,  2.04s/it]
Outer training loop:  93%|#########3| 46609/50000 [31:06:13<1:54:35,  2.03s/it]
Outer training loop:  93%|#########3| 46610/50000 [31:06:15<1:54:12,  2.02s/it]
Outer training loop:  93%|#########3| 46611/50000 [31:06:17<1:53:28,  2.01s/it]
Outer training loop:  93%|#########3| 46612/50000 [31:06:19<1:54:18,  2.02s/it]
Outer training loop:  93%|#########3| 46613/50000 [31:06:21<1:51:07,  1.97s/it]
Outer training loop:  93%|#########3| 46614/50000 [31:06:23<1:53:53,  2.02s/it]
Outer training loop:  93%|#########3| 46615/50000 [31:06:25<1:51:15,  1.97s/it]
Outer training loop:  93%|#########3| 46616/50000 [31:06:27<1:51:45,  1.98s/it]
Outer training loop:  93%|#########3| 46617/50000 [31:06:29<1:53:51,  2.02s/it]
Outer training loop:  93%|#########3| 46618/50000 [31:06:31<1:51:47,  1.98s/it]
Outer training loop:  93%|#########3| 46619/50000 [31:06:33<1:48:36,  1.93s/it]
Outer training loop:  93%|#########3| 46620/50000 [31:06:35<1:44:25,  1.85s/it]
Outer training loop:  93%|#########3| 46621/50000 [31:06:37<1:47:06,  1.90s/it]
Outer training loop:  93%|#########3| 46622/50000 [31:06:39<1:47:40,  1.91s/it]
Outer training loop:  93%|#########3| 46623/50000 [31:06:40<1:46:05,  1.88s/it]
Outer training loop:  93%|#########3| 46624/50000 [31:06:43<1:50:58,  1.97s/it]
Outer training loop:  93%|#########3| 46625/50000 [31:06:45<1:50:00,  1.96s/it]
Outer training loop:  93%|#########3| 46626/50000 [31:06:46<1:48:14,  1.92s/it]
Outer training loop:  93%|#########3| 46627/50000 [31:06:48<1:46:28,  1.89s/it]
Outer training loop:  93%|#########3| 46628/50000 [31:06:50<1:51:59,  1.99s/it]
Outer training loop:  93%|#########3| 46629/50000 [31:06:52<1:48:48,  1.94s/it]
Outer training loop:  93%|#########3| 46630/50000 [31:06:54<1:46:09,  1.89s/it]
Outer training loop:  93%|#########3| 46631/50000 [31:06:56<1:49:32,  1.95s/it]
Outer training loop:  93%|#########3| 46632/50000 [31:06:58<1:47:25,  1.91s/it]
Outer training loop:  93%|#########3| 46633/50000 [31:07:00<1:45:27,  1.88s/it]
Outer training loop:  93%|#########3| 46634/50000 [31:07:02<1:45:11,  1.87s/it]
Outer training loop:  93%|#########3| 46635/50000 [31:07:04<1:47:00,  1.91s/it]
Outer training loop:  93%|#########3| 46636/50000 [31:07:05<1:45:51,  1.89s/it]
Outer training loop:  93%|#########3| 46637/50000 [31:07:07<1:45:31,  1.88s/it]
Outer training loop:  93%|#########3| 46638/50000 [31:07:09<1:47:28,  1.92s/it]
Outer training loop:  93%|#########3| 46639/50000 [31:07:11<1:45:22,  1.88s/it]
Outer training loop:  93%|#########3| 46640/50000 [31:07:13<1:45:20,  1.88s/it]
Outer training loop:  93%|#########3| 46641/50000 [31:07:15<1:44:15,  1.86s/it]
Outer training loop:  93%|#########3| 46642/50000 [31:07:17<1:48:50,  1.94s/it]
Outer training loop:  93%|#########3| 46643/50000 [31:07:19<1:47:40,  1.92s/it]
Outer training loop:  93%|#########3| 46644/50000 [31:07:21<1:45:14,  1.88s/it]
Outer training loop:  93%|#########3| 46645/50000 [31:07:23<1:46:25,  1.90s/it]
Outer training loop:  93%|#########3| 46646/50000 [31:07:24<1:45:09,  1.88s/it]
Outer training loop:  93%|#########3| 46647/50000 [31:07:26<1:43:53,  1.86s/it]
Outer training loop:  93%|#########3| 46648/50000 [31:07:28<1:46:53,  1.91s/it]
Outer training loop:  93%|#########3| 46649/50000 [31:07:30<1:45:29,  1.89s/it]
Outer training loop:  93%|#########3| 46650/50000 [31:07:32<1:43:35,  1.86s/it]
Outer training loop:  93%|#########3| 46651/50000 [31:07:34<1:46:01,  1.90s/it]
Outer training loop:  93%|#########3| 46652/50000 [31:07:36<1:44:34,  1.87s/it]
Outer training loop:  93%|#########3| 46653/50000 [31:07:37<1:43:24,  1.85s/it]
Outer training loop:  93%|#########3| 46654/50000 [31:07:40<1:48:04,  1.94s/it]
Outer training loop:  93%|#########3| 46655/50000 [31:07:41<1:44:50,  1.88s/it]
Outer training loop:  93%|#########3| 46656/50000 [31:07:43<1:43:50,  1.86s/it]
Outer training loop:  93%|#########3| 46657/50000 [31:07:45<1:42:04,  1.83s/it]
Outer training loop:  93%|#########3| 46658/50000 [31:07:47<1:44:51,  1.88s/it]
Outer training loop:  93%|#########3| 46659/50000 [31:07:49<1:44:50,  1.88s/it]
Outer training loop:  93%|#########3| 46660/50000 [31:07:51<1:42:28,  1.84s/it]
Outer training loop:  93%|#########3| 46661/50000 [31:07:52<1:42:42,  1.85s/it]
Outer training loop:  93%|#########3| 46662/50000 [31:07:54<1:46:36,  1.92s/it]
Outer training loop:  93%|#########3| 46663/50000 [31:07:56<1:44:04,  1.87s/it]
Outer training loop:  93%|#########3| 46664/50000 [31:07:58<1:45:49,  1.90s/it]
Outer training loop:  93%|#########3| 46665/50000 [31:08:00<1:42:15,  1.84s/it]
Outer training loop:  93%|#########3| 46666/50000 [31:08:02<1:41:09,  1.82s/it]
Outer training loop:  93%|#########3| 46667/50000 [31:08:04<1:41:37,  1.83s/it]
Outer training loop:  93%|#########3| 46668/50000 [31:08:05<1:43:02,  1.86s/it]
Outer training loop:  93%|#########3| 46669/50000 [31:08:07<1:42:57,  1.85s/it]
Outer training loop:  93%|#########3| 46670/50000 [31:08:09<1:39:54,  1.80s/it]
Outer training loop:  93%|#########3| 46671/50000 [31:08:11<1:39:59,  1.80s/it]
Outer training loop:  93%|#########3| 46672/50000 [31:08:13<1:45:09,  1.90s/it]
Outer training loop:  93%|#########3| 46673/50000 [31:08:15<1:44:20,  1.88s/it]
Outer training loop:  93%|#########3| 46674/50000 [31:08:17<1:42:52,  1.86s/it]
Outer training loop:  93%|#########3| 46675/50000 [31:08:19<1:45:27,  1.90s/it]
Outer training loop:  93%|#########3| 46676/50000 [31:08:20<1:45:23,  1.90s/it]
Outer training loop:  93%|#########3| 46677/50000 [31:08:22<1:43:57,  1.88s/it]
Outer training loop:  93%|#########3| 46678/50000 [31:08:24<1:41:45,  1.84s/it]
Outer training loop:  93%|#########3| 46679/50000 [31:08:26<1:45:06,  1.90s/it]
Outer training loop:  93%|#########3| 46680/50000 [31:08:28<1:43:19,  1.87s/it]
Outer training loop:  93%|#########3| 46681/50000 [31:08:30<1:41:06,  1.83s/it]
Outer training loop:  93%|#########3| 46682/50000 [31:08:32<1:42:18,  1.85s/it]
Outer training loop:  93%|#########3| 46683/50000 [31:08:33<1:42:51,  1.86s/it]
Outer training loop:  93%|#########3| 46684/50000 [31:08:35<1:43:27,  1.87s/it]
Outer training loop:  93%|#########3| 46685/50000 [31:08:37<1:44:14,  1.89s/it]
Outer training loop:  93%|#########3| 46686/50000 [31:08:39<1:45:50,  1.92s/it]
Outer training loop:  93%|#########3| 46687/50000 [31:08:41<1:44:02,  1.88s/it]
Outer training loop:  93%|#########3| 46688/50000 [31:08:43<1:43:56,  1.88s/it]
Outer training loop:  93%|#########3| 46689/50000 [31:08:45<1:45:47,  1.92s/it]
Outer training loop:  93%|#########3| 46690/50000 [31:08:47<1:43:56,  1.88s/it]
Outer training loop:  93%|#########3| 46691/50000 [31:08:49<1:43:42,  1.88s/it]
Outer training loop:  93%|#########3| 46692/50000 [31:08:50<1:41:45,  1.85s/it]
Outer training loop:  93%|#########3| 46693/50000 [31:08:52<1:43:30,  1.88s/it]
Outer training loop:  93%|#########3| 46694/50000 [31:08:54<1:42:49,  1.87s/it]
Outer training loop:  93%|#########3| 46695/50000 [31:08:56<1:42:07,  1.85s/it]
Outer training loop:  93%|#########3| 46696/50000 [31:08:58<1:44:38,  1.90s/it]
Outer training loop:  93%|#########3| 46697/50000 [31:09:00<1:46:17,  1.93s/it]
Outer training loop:  93%|#########3| 46698/50000 [31:09:02<1:44:15,  1.89s/it]
Outer training loop:  93%|#########3| 46699/50000 [31:09:03<1:40:24,  1.82s/it]
Outer training loop:  93%|#########3| 46700/50000 [31:09:05<1:43:13,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.77s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  93%|#########3| 46701/50000 [31:09:13<3:13:38,  3.52s/it]
Outer training loop:  93%|#########3| 46702/50000 [31:09:15<2:44:23,  2.99s/it]
Outer training loop:  93%|#########3| 46703/50000 [31:09:16<2:25:19,  2.64s/it]
Outer training loop:  93%|#########3| 46704/50000 [31:09:18<2:14:24,  2.45s/it]
Outer training loop:  93%|#########3| 46705/50000 [31:09:20<2:05:34,  2.29s/it]
Outer training loop:  93%|#########3| 46706/50000 [31:09:22<1:56:39,  2.12s/it]
Outer training loop:  93%|#########3| 46707/50000 [31:09:24<1:50:09,  2.01s/it]
Outer training loop:  93%|#########3| 46708/50000 [31:09:26<1:50:02,  2.01s/it]
Outer training loop:  93%|#########3| 46709/50000 [31:09:28<1:51:09,  2.03s/it]
Outer training loop:  93%|#########3| 46710/50000 [31:09:30<1:47:17,  1.96s/it]
Outer training loop:  93%|#########3| 46711/50000 [31:09:31<1:43:55,  1.90s/it]
Outer training loop:  93%|#########3| 46712/50000 [31:09:34<1:48:11,  1.97s/it]
Outer training loop:  93%|#########3| 46713/50000 [31:09:35<1:46:03,  1.94s/it]
Outer training loop:  93%|#########3| 46714/50000 [31:09:37<1:42:40,  1.87s/it]
Outer training loop:  93%|#########3| 46715/50000 [31:09:39<1:44:31,  1.91s/it]
Outer training loop:  93%|#########3| 46716/50000 [31:09:41<1:45:32,  1.93s/it]
Outer training loop:  93%|#########3| 46717/50000 [31:09:43<1:42:50,  1.88s/it]
Outer training loop:  93%|#########3| 46718/50000 [31:09:45<1:43:04,  1.88s/it]
Outer training loop:  93%|#########3| 46719/50000 [31:09:47<1:46:35,  1.95s/it]
Outer training loop:  93%|#########3| 46720/50000 [31:09:49<1:45:29,  1.93s/it]
Outer training loop:  93%|#########3| 46721/50000 [31:09:51<1:46:57,  1.96s/it]
Outer training loop:  93%|#########3| 46722/50000 [31:09:53<1:48:49,  1.99s/it]
Outer training loop:  93%|#########3| 46723/50000 [31:09:55<1:46:51,  1.96s/it]
Outer training loop:  93%|#########3| 46724/50000 [31:09:56<1:43:01,  1.89s/it]
Outer training loop:  93%|#########3| 46725/50000 [31:09:58<1:44:21,  1.91s/it]
Outer training loop:  93%|#########3| 46726/50000 [31:10:00<1:45:48,  1.94s/it]
Outer training loop:  93%|#########3| 46727/50000 [31:10:02<1:42:10,  1.87s/it]
Outer training loop:  93%|#########3| 46728/50000 [31:10:04<1:42:39,  1.88s/it]
Outer training loop:  93%|#########3| 46729/50000 [31:10:06<1:45:44,  1.94s/it]
Outer training loop:  93%|#########3| 46730/50000 [31:10:08<1:41:26,  1.86s/it]
Outer training loop:  93%|#########3| 46731/50000 [31:10:10<1:41:22,  1.86s/it]
Outer training loop:  93%|#########3| 46732/50000 [31:10:11<1:40:02,  1.84s/it]
Outer training loop:  93%|#########3| 46733/50000 [31:10:13<1:41:49,  1.87s/it]
Outer training loop:  93%|#########3| 46734/50000 [31:10:15<1:41:29,  1.86s/it]
Outer training loop:  93%|#########3| 46735/50000 [31:10:17<1:41:04,  1.86s/it]
Outer training loop:  93%|#########3| 46736/50000 [31:10:19<1:39:59,  1.84s/it]
Outer training loop:  93%|#########3| 46737/50000 [31:10:21<1:42:53,  1.89s/it]
Outer training loop:  93%|#########3| 46738/50000 [31:10:23<1:42:36,  1.89s/it]
Outer training loop:  93%|#########3| 46739/50000 [31:10:25<1:44:58,  1.93s/it]
Outer training loop:  93%|#########3| 46740/50000 [31:10:27<1:47:27,  1.98s/it]
Outer training loop:  93%|#########3| 46741/50000 [31:10:29<1:45:36,  1.94s/it]
Outer training loop:  93%|#########3| 46742/50000 [31:10:30<1:42:50,  1.89s/it]
Outer training loop:  93%|#########3| 46743/50000 [31:10:32<1:44:07,  1.92s/it]
Outer training loop:  93%|#########3| 46744/50000 [31:10:34<1:43:44,  1.91s/it]
Outer training loop:  93%|#########3| 46745/50000 [31:10:36<1:41:35,  1.87s/it]
Outer training loop:  93%|#########3| 46746/50000 [31:10:38<1:40:30,  1.85s/it]
Outer training loop:  93%|#########3| 46747/50000 [31:10:40<1:44:13,  1.92s/it]
Outer training loop:  93%|#########3| 46748/50000 [31:10:42<1:44:14,  1.92s/it]
Outer training loop:  93%|#########3| 46749/50000 [31:10:44<1:44:10,  1.92s/it]
Outer training loop:  94%|#########3| 46750/50000 [31:10:46<1:47:24,  1.98s/it]
Outer training loop:  94%|#########3| 46751/50000 [31:10:48<1:44:08,  1.92s/it]
Outer training loop:  94%|#########3| 46752/50000 [31:10:50<1:40:59,  1.87s/it]
Outer training loop:  94%|#########3| 46753/50000 [31:10:52<1:45:19,  1.95s/it]
Outer training loop:  94%|#########3| 46754/50000 [31:10:53<1:43:28,  1.91s/it]
Outer training loop:  94%|#########3| 46755/50000 [31:10:55<1:43:41,  1.92s/it]
Outer training loop:  94%|#########3| 46756/50000 [31:10:57<1:42:21,  1.89s/it]
Outer training loop:  94%|#########3| 46757/50000 [31:10:59<1:46:49,  1.98s/it]
Outer training loop:  94%|#########3| 46758/50000 [31:11:01<1:45:09,  1.95s/it]
Outer training loop:  94%|#########3| 46759/50000 [31:11:03<1:46:02,  1.96s/it]
Outer training loop:  94%|#########3| 46760/50000 [31:11:05<1:45:55,  1.96s/it]
Outer training loop:  94%|#########3| 46761/50000 [31:11:07<1:45:11,  1.95s/it]
Outer training loop:  94%|#########3| 46762/50000 [31:11:09<1:44:38,  1.94s/it]
Outer training loop:  94%|#########3| 46763/50000 [31:11:11<1:50:46,  2.05s/it]
Outer training loop:  94%|#########3| 46764/50000 [31:11:13<1:48:33,  2.01s/it]
Outer training loop:  94%|#########3| 46765/50000 [31:11:15<1:46:26,  1.97s/it]
Outer training loop:  94%|#########3| 46766/50000 [31:11:17<1:49:37,  2.03s/it]
Outer training loop:  94%|#########3| 46767/50000 [31:11:19<1:47:04,  1.99s/it]
Outer training loop:  94%|#########3| 46768/50000 [31:11:21<1:46:28,  1.98s/it]
Outer training loop:  94%|#########3| 46769/50000 [31:11:23<1:46:12,  1.97s/it]
Outer training loop:  94%|#########3| 46770/50000 [31:11:25<1:49:58,  2.04s/it]
Outer training loop:  94%|#########3| 46771/50000 [31:11:27<1:47:32,  2.00s/it]
Outer training loop:  94%|#########3| 46772/50000 [31:11:29<1:47:29,  2.00s/it]
Outer training loop:  94%|#########3| 46773/50000 [31:11:31<1:47:42,  2.00s/it]
Outer training loop:  94%|#########3| 46774/50000 [31:11:33<1:46:32,  1.98s/it]
Outer training loop:  94%|#########3| 46775/50000 [31:11:35<1:46:26,  1.98s/it]
Outer training loop:  94%|#########3| 46776/50000 [31:11:37<1:46:42,  1.99s/it]
Outer training loop:  94%|#########3| 46777/50000 [31:11:39<1:44:36,  1.95s/it]
Outer training loop:  94%|#########3| 46778/50000 [31:11:41<1:41:20,  1.89s/it]
Outer training loop:  94%|#########3| 46779/50000 [31:11:43<1:40:16,  1.87s/it]
Outer training loop:  94%|#########3| 46780/50000 [31:11:45<1:44:22,  1.95s/it]
Outer training loop:  94%|#########3| 46781/50000 [31:11:47<1:43:55,  1.94s/it]
Outer training loop:  94%|#########3| 46782/50000 [31:11:49<1:42:55,  1.92s/it]
Outer training loop:  94%|#########3| 46783/50000 [31:11:51<1:44:34,  1.95s/it]
Outer training loop:  94%|#########3| 46784/50000 [31:11:52<1:42:37,  1.91s/it]
Outer training loop:  94%|#########3| 46785/50000 [31:11:54<1:44:54,  1.96s/it]
Outer training loop:  94%|#########3| 46786/50000 [31:11:56<1:45:26,  1.97s/it]
Outer training loop:  94%|#########3| 46787/50000 [31:11:58<1:43:26,  1.93s/it]
Outer training loop:  94%|#########3| 46788/50000 [31:12:00<1:41:13,  1.89s/it]
Outer training loop:  94%|#########3| 46789/50000 [31:12:02<1:44:22,  1.95s/it]
Outer training loop:  94%|#########3| 46790/50000 [31:12:04<1:43:20,  1.93s/it]
Outer training loop:  94%|#########3| 46791/50000 [31:12:06<1:43:39,  1.94s/it]
Outer training loop:  94%|#########3| 46792/50000 [31:12:08<1:46:17,  1.99s/it]
Outer training loop:  94%|#########3| 46793/50000 [31:12:10<1:43:36,  1.94s/it]
Outer training loop:  94%|#########3| 46794/50000 [31:12:12<1:41:31,  1.90s/it]
Outer training loop:  94%|#########3| 46795/50000 [31:12:14<1:41:51,  1.91s/it]
Outer training loop:  94%|#########3| 46796/50000 [31:12:16<1:44:02,  1.95s/it]
Outer training loop:  94%|#########3| 46797/50000 [31:12:18<1:43:32,  1.94s/it]
Outer training loop:  94%|#########3| 46798/50000 [31:12:19<1:40:54,  1.89s/it]
Outer training loop:  94%|#########3| 46799/50000 [31:12:21<1:41:18,  1.90s/it]
Outer training loop:  94%|#########3| 46800/50000 [31:12:23<1:42:53,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  94%|#########3| 46801/50000 [31:12:31<3:10:20,  3.57s/it]
Outer training loop:  94%|#########3| 46802/50000 [31:12:33<2:41:46,  3.04s/it]
Outer training loop:  94%|#########3| 46803/50000 [31:12:34<2:22:15,  2.67s/it]
Outer training loop:  94%|#########3| 46804/50000 [31:12:36<2:11:29,  2.47s/it]
Outer training loop:  94%|#########3| 46805/50000 [31:12:38<1:59:46,  2.25s/it]
Outer training loop:  94%|#########3| 46806/50000 [31:12:40<1:56:06,  2.18s/it]
Outer training loop:  94%|#########3| 46807/50000 [31:12:42<1:52:11,  2.11s/it]
Outer training loop:  94%|#########3| 46808/50000 [31:12:44<1:49:47,  2.06s/it]
Outer training loop:  94%|#########3| 46809/50000 [31:12:46<1:47:14,  2.02s/it]
Outer training loop:  94%|#########3| 46810/50000 [31:12:48<1:48:49,  2.05s/it]
Outer training loop:  94%|#########3| 46811/50000 [31:12:50<1:47:22,  2.02s/it]
Outer training loop:  94%|#########3| 46812/50000 [31:12:52<1:43:05,  1.94s/it]
Outer training loop:  94%|#########3| 46813/50000 [31:12:54<1:42:07,  1.92s/it]
Outer training loop:  94%|#########3| 46814/50000 [31:12:56<1:43:12,  1.94s/it]
Outer training loop:  94%|#########3| 46815/50000 [31:12:57<1:40:23,  1.89s/it]
Outer training loop:  94%|#########3| 46816/50000 [31:12:59<1:37:58,  1.85s/it]
Outer training loop:  94%|#########3| 46817/50000 [31:13:01<1:39:51,  1.88s/it]
Outer training loop:  94%|#########3| 46818/50000 [31:13:03<1:39:56,  1.88s/it]
Outer training loop:  94%|#########3| 46819/50000 [31:13:05<1:42:14,  1.93s/it]
Outer training loop:  94%|#########3| 46820/50000 [31:13:07<1:41:33,  1.92s/it]
Outer training loop:  94%|#########3| 46821/50000 [31:13:09<1:39:37,  1.88s/it]
Outer training loop:  94%|#########3| 46822/50000 [31:13:11<1:40:02,  1.89s/it]
Outer training loop:  94%|#########3| 46823/50000 [31:13:13<1:41:35,  1.92s/it]
Outer training loop:  94%|#########3| 46824/50000 [31:13:15<1:43:25,  1.95s/it]
Outer training loop:  94%|#########3| 46825/50000 [31:13:16<1:39:59,  1.89s/it]
Outer training loop:  94%|#########3| 46826/50000 [31:13:18<1:37:23,  1.84s/it]
Outer training loop:  94%|#########3| 46827/50000 [31:13:20<1:38:43,  1.87s/it]
Outer training loop:  94%|#########3| 46828/50000 [31:13:22<1:41:03,  1.91s/it]
Outer training loop:  94%|#########3| 46829/50000 [31:13:24<1:40:31,  1.90s/it]
Outer training loop:  94%|#########3| 46830/50000 [31:13:26<1:40:18,  1.90s/it]
Outer training loop:  94%|#########3| 46831/50000 [31:13:28<1:43:23,  1.96s/it]
Outer training loop:  94%|#########3| 46832/50000 [31:13:30<1:40:28,  1.90s/it]
Outer training loop:  94%|#########3| 46833/50000 [31:13:32<1:39:45,  1.89s/it]
Outer training loop:  94%|#########3| 46834/50000 [31:13:33<1:38:16,  1.86s/it]
Outer training loop:  94%|#########3| 46835/50000 [31:13:36<1:43:20,  1.96s/it]
Outer training loop:  94%|#########3| 46836/50000 [31:13:37<1:38:56,  1.88s/it]
Outer training loop:  94%|#########3| 46837/50000 [31:13:39<1:38:13,  1.86s/it]
Outer training loop:  94%|#########3| 46838/50000 [31:13:41<1:39:09,  1.88s/it]
Outer training loop:  94%|#########3| 46839/50000 [31:13:43<1:38:44,  1.87s/it]
Outer training loop:  94%|#########3| 46840/50000 [31:13:45<1:37:50,  1.86s/it]
Outer training loop:  94%|#########3| 46841/50000 [31:13:47<1:40:51,  1.92s/it]
Outer training loop:  94%|#########3| 46842/50000 [31:13:48<1:38:31,  1.87s/it]
Outer training loop:  94%|#########3| 46843/50000 [31:13:50<1:37:34,  1.85s/it]
Outer training loop:  94%|#########3| 46844/50000 [31:13:52<1:36:01,  1.83s/it]
Outer training loop:  94%|#########3| 46845/50000 [31:13:54<1:38:54,  1.88s/it]
Outer training loop:  94%|#########3| 46846/50000 [31:13:56<1:38:06,  1.87s/it]
Outer training loop:  94%|#########3| 46847/50000 [31:13:58<1:39:25,  1.89s/it]
Outer training loop:  94%|#########3| 46848/50000 [31:14:00<1:42:12,  1.95s/it]
Outer training loop:  94%|#########3| 46849/50000 [31:14:02<1:40:34,  1.91s/it]
Outer training loop:  94%|#########3| 46850/50000 [31:14:03<1:37:23,  1.86s/it]
Outer training loop:  94%|#########3| 46851/50000 [31:14:05<1:38:03,  1.87s/it]
Outer training loop:  94%|#########3| 46852/50000 [31:14:07<1:39:10,  1.89s/it]
Outer training loop:  94%|#########3| 46853/50000 [31:14:09<1:37:03,  1.85s/it]
Outer training loop:  94%|#########3| 46854/50000 [31:14:11<1:37:46,  1.86s/it]
Outer training loop:  94%|#########3| 46855/50000 [31:14:13<1:40:17,  1.91s/it]
Outer training loop:  94%|#########3| 46856/50000 [31:14:15<1:41:03,  1.93s/it]
Outer training loop:  94%|#########3| 46857/50000 [31:14:17<1:38:12,  1.87s/it]
Outer training loop:  94%|#########3| 46858/50000 [31:14:19<1:38:42,  1.88s/it]
Outer training loop:  94%|#########3| 46859/50000 [31:14:21<1:40:52,  1.93s/it]
Outer training loop:  94%|#########3| 46860/50000 [31:14:22<1:38:45,  1.89s/it]
Outer training loop:  94%|#########3| 46861/50000 [31:14:24<1:38:04,  1.87s/it]
Outer training loop:  94%|#########3| 46862/50000 [31:14:26<1:41:46,  1.95s/it]
Outer training loop:  94%|#########3| 46863/50000 [31:14:28<1:39:31,  1.90s/it]
Outer training loop:  94%|#########3| 46864/50000 [31:14:30<1:38:59,  1.89s/it]
Outer training loop:  94%|#########3| 46865/50000 [31:14:32<1:42:50,  1.97s/it]
Outer training loop:  94%|#########3| 46866/50000 [31:14:34<1:40:14,  1.92s/it]
Outer training loop:  94%|#########3| 46867/50000 [31:14:36<1:39:02,  1.90s/it]
Outer training loop:  94%|#########3| 46868/50000 [31:14:38<1:38:36,  1.89s/it]
Outer training loop:  94%|#########3| 46869/50000 [31:14:40<1:41:52,  1.95s/it]
Outer training loop:  94%|#########3| 46870/50000 [31:14:42<1:40:31,  1.93s/it]
Outer training loop:  94%|#########3| 46871/50000 [31:14:43<1:37:33,  1.87s/it]
Outer training loop:  94%|#########3| 46872/50000 [31:14:45<1:38:53,  1.90s/it]
Outer training loop:  94%|#########3| 46873/50000 [31:14:47<1:37:25,  1.87s/it]
Outer training loop:  94%|#########3| 46874/50000 [31:14:49<1:37:32,  1.87s/it]
Outer training loop:  94%|#########3| 46875/50000 [31:14:51<1:36:41,  1.86s/it]
Outer training loop:  94%|#########3| 46876/50000 [31:14:53<1:38:07,  1.88s/it]
Outer training loop:  94%|#########3| 46877/50000 [31:14:55<1:35:18,  1.83s/it]
Outer training loop:  94%|#########3| 46878/50000 [31:14:56<1:32:46,  1.78s/it]
Outer training loop:  94%|#########3| 46879/50000 [31:14:58<1:35:47,  1.84s/it]
Outer training loop:  94%|#########3| 46880/50000 [31:15:00<1:34:25,  1.82s/it]
Outer training loop:  94%|#########3| 46881/50000 [31:15:02<1:34:03,  1.81s/it]
Outer training loop:  94%|#########3| 46882/50000 [31:15:04<1:36:50,  1.86s/it]
Outer training loop:  94%|#########3| 46883/50000 [31:15:06<1:35:16,  1.83s/it]
Outer training loop:  94%|#########3| 46884/50000 [31:15:07<1:34:41,  1.82s/it]
Outer training loop:  94%|#########3| 46885/50000 [31:15:09<1:38:10,  1.89s/it]
Outer training loop:  94%|#########3| 46886/50000 [31:15:11<1:36:52,  1.87s/it]
Outer training loop:  94%|#########3| 46887/50000 [31:15:13<1:38:04,  1.89s/it]
Outer training loop:  94%|#########3| 46888/50000 [31:15:15<1:40:52,  1.94s/it]
Outer training loop:  94%|#########3| 46889/50000 [31:15:17<1:39:40,  1.92s/it]
Outer training loop:  94%|#########3| 46890/50000 [31:15:19<1:39:17,  1.92s/it]
Outer training loop:  94%|#########3| 46891/50000 [31:15:21<1:39:52,  1.93s/it]
Outer training loop:  94%|#########3| 46892/50000 [31:15:23<1:38:40,  1.91s/it]
Outer training loop:  94%|#########3| 46893/50000 [31:15:25<1:36:30,  1.86s/it]
Outer training loop:  94%|#########3| 46894/50000 [31:15:26<1:37:23,  1.88s/it]
Outer training loop:  94%|#########3| 46895/50000 [31:15:28<1:38:00,  1.89s/it]
Outer training loop:  94%|#########3| 46896/50000 [31:15:30<1:37:52,  1.89s/it]
Outer training loop:  94%|#########3| 46897/50000 [31:15:32<1:36:58,  1.88s/it]
Outer training loop:  94%|#########3| 46898/50000 [31:15:34<1:40:52,  1.95s/it]
Outer training loop:  94%|#########3| 46899/50000 [31:15:36<1:38:31,  1.91s/it]
Outer training loop:  94%|#########3| 46900/50000 [31:15:38<1:37:44,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.75s/it]

Outer training loop:  94%|#########3| 46901/50000 [31:15:45<3:01:52,  3.52s/it]
Outer training loop:  94%|#########3| 46902/50000 [31:15:47<2:37:56,  3.06s/it]
Outer training loop:  94%|#########3| 46903/50000 [31:15:49<2:22:53,  2.77s/it]
Outer training loop:  94%|#########3| 46904/50000 [31:15:51<2:09:28,  2.51s/it]
Outer training loop:  94%|#########3| 46905/50000 [31:15:53<1:55:58,  2.25s/it]
Outer training loop:  94%|#########3| 46906/50000 [31:15:55<1:51:49,  2.17s/it]
Outer training loop:  94%|#########3| 46907/50000 [31:15:56<1:44:04,  2.02s/it]
Outer training loop:  94%|#########3| 46908/50000 [31:15:58<1:40:59,  1.96s/it]
Outer training loop:  94%|#########3| 46909/50000 [31:16:00<1:42:20,  1.99s/it]
Outer training loop:  94%|#########3| 46910/50000 [31:16:02<1:38:34,  1.91s/it]
Outer training loop:  94%|#########3| 46911/50000 [31:16:04<1:37:53,  1.90s/it]
Outer training loop:  94%|#########3| 46912/50000 [31:16:06<1:39:11,  1.93s/it]
Outer training loop:  94%|#########3| 46913/50000 [31:16:08<1:37:20,  1.89s/it]
Outer training loop:  94%|#########3| 46914/50000 [31:16:10<1:36:30,  1.88s/it]
Outer training loop:  94%|#########3| 46915/50000 [31:16:12<1:39:44,  1.94s/it]
Outer training loop:  94%|#########3| 46916/50000 [31:16:14<1:39:19,  1.93s/it]
Outer training loop:  94%|#########3| 46917/50000 [31:16:15<1:38:24,  1.92s/it]
Outer training loop:  94%|#########3| 46918/50000 [31:16:18<1:41:08,  1.97s/it]
Outer training loop:  94%|#########3| 46919/50000 [31:16:20<1:40:35,  1.96s/it]
Outer training loop:  94%|#########3| 46920/50000 [31:16:21<1:37:06,  1.89s/it]
Outer training loop:  94%|#########3| 46921/50000 [31:16:23<1:38:54,  1.93s/it]
Outer training loop:  94%|#########3| 46922/50000 [31:16:25<1:40:24,  1.96s/it]
Outer training loop:  94%|#########3| 46923/50000 [31:16:27<1:36:11,  1.88s/it]
Outer training loop:  94%|#########3| 46924/50000 [31:16:29<1:36:00,  1.87s/it]
Outer training loop:  94%|#########3| 46925/50000 [31:16:31<1:37:11,  1.90s/it]
Outer training loop:  94%|#########3| 46926/50000 [31:16:33<1:34:38,  1.85s/it]
Outer training loop:  94%|#########3| 46927/50000 [31:16:34<1:33:30,  1.83s/it]
Outer training loop:  94%|#########3| 46928/50000 [31:16:36<1:36:32,  1.89s/it]
Outer training loop:  94%|#########3| 46929/50000 [31:16:38<1:34:59,  1.86s/it]
Outer training loop:  94%|#########3| 46930/50000 [31:16:40<1:35:41,  1.87s/it]
Outer training loop:  94%|#########3| 46931/50000 [31:16:42<1:36:48,  1.89s/it]
Outer training loop:  94%|#########3| 46932/50000 [31:16:44<1:34:27,  1.85s/it]
Outer training loop:  94%|#########3| 46933/50000 [31:16:46<1:35:43,  1.87s/it]
Outer training loop:  94%|#########3| 46934/50000 [31:16:47<1:33:58,  1.84s/it]
Outer training loop:  94%|#########3| 46935/50000 [31:16:49<1:36:07,  1.88s/it]
Outer training loop:  94%|#########3| 46936/50000 [31:16:51<1:36:24,  1.89s/it]
Outer training loop:  94%|#########3| 46937/50000 [31:16:53<1:35:46,  1.88s/it]
Outer training loop:  94%|#########3| 46938/50000 [31:16:55<1:34:22,  1.85s/it]
Outer training loop:  94%|#########3| 46939/50000 [31:16:57<1:36:17,  1.89s/it]
Outer training loop:  94%|#########3| 46940/50000 [31:16:59<1:35:19,  1.87s/it]
Outer training loop:  94%|#########3| 46941/50000 [31:17:01<1:34:32,  1.85s/it]
Outer training loop:  94%|#########3| 46942/50000 [31:17:03<1:36:48,  1.90s/it]
Outer training loop:  94%|#########3| 46943/50000 [31:17:04<1:37:22,  1.91s/it]
Outer training loop:  94%|#########3| 46944/50000 [31:17:06<1:38:37,  1.94s/it]
Outer training loop:  94%|#########3| 46945/50000 [31:17:09<1:40:33,  1.98s/it]
Outer training loop:  94%|#########3| 46946/50000 [31:17:10<1:37:44,  1.92s/it]
Outer training loop:  94%|#########3| 46947/50000 [31:17:12<1:37:17,  1.91s/it]
Outer training loop:  94%|#########3| 46948/50000 [31:17:14<1:35:57,  1.89s/it]
Outer training loop:  94%|#########3| 46949/50000 [31:17:16<1:38:10,  1.93s/it]
Outer training loop:  94%|#########3| 46950/50000 [31:17:18<1:35:30,  1.88s/it]
Outer training loop:  94%|#########3| 46951/50000 [31:17:20<1:37:01,  1.91s/it]
Outer training loop:  94%|#########3| 46952/50000 [31:17:22<1:37:48,  1.93s/it]
Outer training loop:  94%|#########3| 46953/50000 [31:17:24<1:39:22,  1.96s/it]
Outer training loop:  94%|#########3| 46954/50000 [31:17:26<1:37:20,  1.92s/it]
Outer training loop:  94%|#########3| 46955/50000 [31:17:27<1:33:54,  1.85s/it]
Outer training loop:  94%|#########3| 46956/50000 [31:17:29<1:35:31,  1.88s/it]
Outer training loop:  94%|#########3| 46957/50000 [31:17:31<1:35:12,  1.88s/it]
Outer training loop:  94%|#########3| 46958/50000 [31:17:33<1:35:04,  1.88s/it]
Outer training loop:  94%|#########3| 46959/50000 [31:17:35<1:38:27,  1.94s/it]
Outer training loop:  94%|#########3| 46960/50000 [31:17:37<1:37:04,  1.92s/it]
Outer training loop:  94%|#########3| 46961/50000 [31:17:39<1:35:56,  1.89s/it]
Outer training loop:  94%|#########3| 46962/50000 [31:17:41<1:37:52,  1.93s/it]
Outer training loop:  94%|#########3| 46963/50000 [31:17:43<1:37:32,  1.93s/it]
Outer training loop:  94%|#########3| 46964/50000 [31:17:45<1:35:35,  1.89s/it]
Outer training loop:  94%|#########3| 46965/50000 [31:17:46<1:33:11,  1.84s/it]
Outer training loop:  94%|#########3| 46966/50000 [31:17:48<1:36:41,  1.91s/it]
Outer training loop:  94%|#########3| 46967/50000 [31:17:50<1:36:49,  1.92s/it]
Outer training loop:  94%|#########3| 46968/50000 [31:17:52<1:35:28,  1.89s/it]
Outer training loop:  94%|#########3| 46969/50000 [31:17:54<1:38:30,  1.95s/it]
Outer training loop:  94%|#########3| 46970/50000 [31:17:56<1:36:28,  1.91s/it]
Outer training loop:  94%|#########3| 46971/50000 [31:17:58<1:34:58,  1.88s/it]
Outer training loop:  94%|#########3| 46972/50000 [31:18:00<1:33:31,  1.85s/it]
Outer training loop:  94%|#########3| 46973/50000 [31:18:02<1:35:03,  1.88s/it]
Outer training loop:  94%|#########3| 46974/50000 [31:18:03<1:35:18,  1.89s/it]
Outer training loop:  94%|#########3| 46975/50000 [31:18:05<1:36:19,  1.91s/it]
Outer training loop:  94%|#########3| 46976/50000 [31:18:07<1:37:18,  1.93s/it]
Outer training loop:  94%|#########3| 46977/50000 [31:18:09<1:35:19,  1.89s/it]
Outer training loop:  94%|#########3| 46978/50000 [31:18:11<1:35:42,  1.90s/it]
Outer training loop:  94%|#########3| 46979/50000 [31:18:13<1:35:02,  1.89s/it]
Outer training loop:  94%|#########3| 46980/50000 [31:18:15<1:37:21,  1.93s/it]
Outer training loop:  94%|#########3| 46981/50000 [31:18:17<1:35:09,  1.89s/it]
Outer training loop:  94%|#########3| 46982/50000 [31:18:19<1:35:01,  1.89s/it]
Outer training loop:  94%|#########3| 46983/50000 [31:18:21<1:38:06,  1.95s/it]
Outer training loop:  94%|#########3| 46984/50000 [31:18:23<1:36:20,  1.92s/it]
Outer training loop:  94%|#########3| 46985/50000 [31:18:24<1:33:57,  1.87s/it]
Outer training loop:  94%|#########3| 46986/50000 [31:18:26<1:37:05,  1.93s/it]
Outer training loop:  94%|#########3| 46987/50000 [31:18:28<1:37:19,  1.94s/it]
Outer training loop:  94%|#########3| 46988/50000 [31:18:30<1:35:56,  1.91s/it]
Outer training loop:  94%|#########3| 46989/50000 [31:18:32<1:34:47,  1.89s/it]
Outer training loop:  94%|#########3| 46990/50000 [31:18:34<1:36:40,  1.93s/it]
Outer training loop:  94%|#########3| 46991/50000 [31:18:36<1:36:45,  1.93s/it]
Outer training loop:  94%|#########3| 46992/50000 [31:18:38<1:35:29,  1.90s/it]
Outer training loop:  94%|#########3| 46993/50000 [31:18:40<1:35:20,  1.90s/it]
Outer training loop:  94%|#########3| 46994/50000 [31:18:42<1:32:45,  1.85s/it]
Outer training loop:  94%|#########3| 46995/50000 [31:18:43<1:33:52,  1.87s/it]
Outer training loop:  94%|#########3| 46996/50000 [31:18:45<1:33:41,  1.87s/it]
Outer training loop:  94%|#########3| 46997/50000 [31:18:47<1:31:30,  1.83s/it]
Outer training loop:  94%|#########3| 46998/50000 [31:18:49<1:32:39,  1.85s/it]
Outer training loop:  94%|#########3| 46999/50000 [31:18:51<1:35:01,  1.90s/it]
Outer training loop:  94%|#########3| 47000/50000 [31:18:53<1:33:33,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.98s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.89s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:33,  1.87s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:32,  1.89s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:37,  1.97s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:36,  1.98s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:33,  1.97s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:31,  1.96s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:28,  1.95s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.95s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:26,  2.01s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:24,  2.01s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:20,  1.99s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:17,  1.96s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:18,  2.01s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:11,  1.93s/it]

Evaluating policy:  19%|#9        | 16/83 [00:31<02:05,  1.87s/it]

Evaluating policy:  20%|##        | 17/83 [00:33<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:35<02:02,  1.88s/it]

Evaluating policy:  23%|##2       | 19/83 [00:37<02:05,  1.96s/it]

Evaluating policy:  24%|##4       | 20/83 [00:39<02:01,  1.93s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:57,  1.89s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:56,  1.90s/it]

Evaluating policy:  28%|##7       | 23/83 [00:45<02:00,  2.01s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:55,  1.95s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:49,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:47,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:48,  1.98s/it]

Evaluating policy:  35%|###4      | 29/83 [00:56<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:58<01:41,  1.91s/it]

Evaluating policy:  37%|###7      | 31/83 [01:00<01:39,  1.92s/it]

Evaluating policy:  39%|###8      | 32/83 [01:02<01:35,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:06<01:36,  1.97s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:25,  1.85s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:21,  1.80s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:20,  1.83s/it]

Evaluating policy:  48%|####8     | 40/83 [01:17<01:22,  1.91s/it]

Evaluating policy:  49%|####9     | 41/83 [01:19<01:19,  1.90s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:18,  1.91s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:09,  1.84s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:06,  1.81s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:09,  1.93s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:05,  1.87s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.89s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:01,  1.85s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:55,  1.85s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:57,  1.97s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:54,  1.94s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:51,  1.92s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:50,  1.95s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:47,  1.91s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:42,  1.86s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:57<00:44,  2.01s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:40,  1.93s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:37,  1.89s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:33,  1.84s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:06<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:29,  1.82s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:27,  1.82s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:25,  1.79s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:25,  1.95s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.85s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.86s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.86s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.86s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.84s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.86s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:08,  2.03s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.94s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.87s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.78s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.77s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.79s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:23,  1.81s/it]

Evaluating policy:   6%|6         | 5/83 [00:08<02:17,  1.77s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:32,  1.98s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.92s/it]

Evaluating policy:  10%|9         | 8/83 [00:14<02:21,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:16<02:19,  1.89s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:20,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.90s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.88s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:10,  1.86s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.88s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:11,  1.94s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:11,  1.97s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.94s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:04,  1.95s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<02:00,  1.91s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:56,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:53,  1.89s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:55,  1.95s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.88s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:42,  1.86s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:44,  1.94s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:42,  1.93s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.85s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:34,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:32,  1.86s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:34,  1.97s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:30,  1.93s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:28,  1.92s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:19,  1.86s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:21,  1.95s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:19,  1.95s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:16,  1.91s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.91s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:11,  1.98s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:04,  1.90s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.87s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.87s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:56,  1.83s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:54,  1.82s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:55,  1.92s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:53,  1.90s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:51,  1.92s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.89s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.83s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.82s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:41,  1.98s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:38,  1.93s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.90s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:33,  1.83s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:32,  1.88s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.88s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.85s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.88s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:26,  2.01s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:23,  1.95s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.91s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.88s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.86s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.84s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.82s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:10,  1.81s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:08,  1.80s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  2.00s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.95s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.88s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.87s/it]

Outer training loop:  94%|#########4| 47001/50000 [31:24:16<81:48:06, 98.19s/it]
Outer training loop:  94%|#########4| 47002/50000 [31:24:18<57:41:34, 69.28s/it]
Outer training loop:  94%|#########4| 47003/50000 [31:24:19<40:49:44, 49.04s/it]
Outer training loop:  94%|#########4| 47004/50000 [31:24:21<29:01:37, 34.88s/it]
Outer training loop:  94%|#########4| 47005/50000 [31:24:23<20:45:19, 24.95s/it]
Outer training loop:  94%|#########4| 47006/50000 [31:24:25<14:59:21, 18.02s/it]
Outer training loop:  94%|#########4| 47007/50000 [31:24:27<10:57:54, 13.19s/it]
Outer training loop:  94%|#########4| 47008/50000 [31:24:29<8:16:41,  9.96s/it] 
Outer training loop:  94%|#########4| 47009/50000 [31:24:31<6:13:57,  7.50s/it]
Outer training loop:  94%|#########4| 47010/50000 [31:24:33<4:48:43,  5.79s/it]
Outer training loop:  94%|#########4| 47011/50000 [31:24:35<3:51:48,  4.65s/it]
Outer training loop:  94%|#########4| 47012/50000 [31:24:37<3:11:14,  3.84s/it]
Outer training loop:  94%|#########4| 47013/50000 [31:24:38<2:40:15,  3.22s/it]
Outer training loop:  94%|#########4| 47014/50000 [31:24:40<2:22:11,  2.86s/it]
Outer training loop:  94%|#########4| 47015/50000 [31:24:42<2:06:02,  2.53s/it]
Outer training loop:  94%|#########4| 47016/50000 [31:24:44<1:55:56,  2.33s/it]
Outer training loop:  94%|#########4| 47017/50000 [31:24:46<1:48:27,  2.18s/it]
Outer training loop:  94%|#########4| 47018/50000 [31:24:48<1:46:16,  2.14s/it]
Outer training loop:  94%|#########4| 47019/50000 [31:24:50<1:44:47,  2.11s/it]
Outer training loop:  94%|#########4| 47020/50000 [31:24:52<1:39:25,  2.00s/it]
Outer training loop:  94%|#########4| 47021/50000 [31:24:54<1:40:16,  2.02s/it]
Outer training loop:  94%|#########4| 47022/50000 [31:24:56<1:36:36,  1.95s/it]
Outer training loop:  94%|#########4| 47023/50000 [31:24:57<1:32:58,  1.87s/it]
Outer training loop:  94%|#########4| 47024/50000 [31:24:59<1:31:48,  1.85s/it]
Outer training loop:  94%|#########4| 47025/50000 [31:25:01<1:34:03,  1.90s/it]
Outer training loop:  94%|#########4| 47026/50000 [31:25:03<1:31:01,  1.84s/it]
Outer training loop:  94%|#########4| 47027/50000 [31:25:05<1:31:31,  1.85s/it]
Outer training loop:  94%|#########4| 47028/50000 [31:25:07<1:31:21,  1.84s/it]
Outer training loop:  94%|#########4| 47029/50000 [31:25:08<1:31:22,  1.85s/it]
Outer training loop:  94%|#########4| 47030/50000 [31:25:10<1:30:37,  1.83s/it]
Outer training loop:  94%|#########4| 47031/50000 [31:25:12<1:30:25,  1.83s/it]
Outer training loop:  94%|#########4| 47032/50000 [31:25:14<1:33:14,  1.88s/it]
Outer training loop:  94%|#########4| 47033/50000 [31:25:16<1:31:24,  1.85s/it]
Outer training loop:  94%|#########4| 47034/50000 [31:25:18<1:33:13,  1.89s/it]
Outer training loop:  94%|#########4| 47035/50000 [31:25:20<1:34:47,  1.92s/it]
Outer training loop:  94%|#########4| 47036/50000 [31:25:22<1:32:45,  1.88s/it]
Outer training loop:  94%|#########4| 47037/50000 [31:25:23<1:32:04,  1.86s/it]
Outer training loop:  94%|#########4| 47038/50000 [31:25:25<1:35:28,  1.93s/it]
Outer training loop:  94%|#########4| 47039/50000 [31:25:27<1:34:35,  1.92s/it]
Outer training loop:  94%|#########4| 47040/50000 [31:25:29<1:33:01,  1.89s/it]
Outer training loop:  94%|#########4| 47041/50000 [31:25:31<1:31:21,  1.85s/it]
Outer training loop:  94%|#########4| 47042/50000 [31:25:33<1:32:38,  1.88s/it]
Outer training loop:  94%|#########4| 47043/50000 [31:25:35<1:33:37,  1.90s/it]
Outer training loop:  94%|#########4| 47044/50000 [31:25:37<1:33:22,  1.90s/it]
Outer training loop:  94%|#########4| 47045/50000 [31:25:39<1:34:14,  1.91s/it]
Outer training loop:  94%|#########4| 47046/50000 [31:25:41<1:33:48,  1.91s/it]
Outer training loop:  94%|#########4| 47047/50000 [31:25:42<1:32:25,  1.88s/it]
Outer training loop:  94%|#########4| 47048/50000 [31:25:44<1:33:36,  1.90s/it]
Outer training loop:  94%|#########4| 47049/50000 [31:25:46<1:31:24,  1.86s/it]
Outer training loop:  94%|#########4| 47050/50000 [31:25:48<1:31:53,  1.87s/it]
Outer training loop:  94%|#########4| 47051/50000 [31:25:50<1:33:33,  1.90s/it]
Outer training loop:  94%|#########4| 47052/50000 [31:25:52<1:32:46,  1.89s/it]
Outer training loop:  94%|#########4| 47053/50000 [31:25:54<1:30:18,  1.84s/it]
Outer training loop:  94%|#########4| 47054/50000 [31:25:55<1:30:32,  1.84s/it]
Outer training loop:  94%|#########4| 47055/50000 [31:25:57<1:31:24,  1.86s/it]
Outer training loop:  94%|#########4| 47056/50000 [31:25:59<1:30:28,  1.84s/it]
Outer training loop:  94%|#########4| 47057/50000 [31:26:01<1:32:28,  1.89s/it]
Outer training loop:  94%|#########4| 47058/50000 [31:26:03<1:33:57,  1.92s/it]
Outer training loop:  94%|#########4| 47059/50000 [31:26:05<1:33:45,  1.91s/it]
Outer training loop:  94%|#########4| 47060/50000 [31:26:07<1:31:38,  1.87s/it]
Outer training loop:  94%|#########4| 47061/50000 [31:26:09<1:34:13,  1.92s/it]
Outer training loop:  94%|#########4| 47062/50000 [31:26:11<1:33:32,  1.91s/it]
Outer training loop:  94%|#########4| 47063/50000 [31:26:12<1:31:52,  1.88s/it]
Outer training loop:  94%|#########4| 47064/50000 [31:26:14<1:29:00,  1.82s/it]
Outer training loop:  94%|#########4| 47065/50000 [31:26:16<1:32:37,  1.89s/it]
Outer training loop:  94%|#########4| 47066/50000 [31:26:18<1:29:47,  1.84s/it]
Outer training loop:  94%|#########4| 47067/50000 [31:26:20<1:30:59,  1.86s/it]
Outer training loop:  94%|#########4| 47068/50000 [31:26:22<1:33:08,  1.91s/it]
Outer training loop:  94%|#########4| 47069/50000 [31:26:24<1:31:12,  1.87s/it]
Outer training loop:  94%|#########4| 47070/50000 [31:26:25<1:31:06,  1.87s/it]
Outer training loop:  94%|#########4| 47071/50000 [31:26:28<1:33:51,  1.92s/it]
Outer training loop:  94%|#########4| 47072/50000 [31:26:29<1:31:50,  1.88s/it]
Outer training loop:  94%|#########4| 47073/50000 [31:26:31<1:31:07,  1.87s/it]
Outer training loop:  94%|#########4| 47074/50000 [31:26:33<1:29:59,  1.85s/it]
Outer training loop:  94%|#########4| 47075/50000 [31:26:35<1:33:20,  1.91s/it]
Outer training loop:  94%|#########4| 47076/50000 [31:26:37<1:33:00,  1.91s/it]
Outer training loop:  94%|#########4| 47077/50000 [31:26:39<1:30:30,  1.86s/it]
Outer training loop:  94%|#########4| 47078/50000 [31:26:41<1:32:47,  1.91s/it]
Outer training loop:  94%|#########4| 47079/50000 [31:26:43<1:32:35,  1.90s/it]
Outer training loop:  94%|#########4| 47080/50000 [31:26:44<1:32:07,  1.89s/it]
Outer training loop:  94%|#########4| 47081/50000 [31:26:46<1:31:35,  1.88s/it]
Outer training loop:  94%|#########4| 47082/50000 [31:26:48<1:33:54,  1.93s/it]
Outer training loop:  94%|#########4| 47083/50000 [31:26:50<1:32:36,  1.90s/it]
Outer training loop:  94%|#########4| 47084/50000 [31:26:52<1:34:17,  1.94s/it]
Outer training loop:  94%|#########4| 47085/50000 [31:26:54<1:34:51,  1.95s/it]
Outer training loop:  94%|#########4| 47086/50000 [31:26:56<1:33:51,  1.93s/it]
Outer training loop:  94%|#########4| 47087/50000 [31:26:58<1:32:33,  1.91s/it]
Outer training loop:  94%|#########4| 47088/50000 [31:27:00<1:33:19,  1.92s/it]
Outer training loop:  94%|#########4| 47089/50000 [31:27:02<1:33:09,  1.92s/it]
Outer training loop:  94%|#########4| 47090/50000 [31:27:04<1:31:48,  1.89s/it]
Outer training loop:  94%|#########4| 47091/50000 [31:27:06<1:32:21,  1.91s/it]
Outer training loop:  94%|#########4| 47092/50000 [31:27:08<1:33:27,  1.93s/it]
Outer training loop:  94%|#########4| 47093/50000 [31:27:09<1:31:31,  1.89s/it]
Outer training loop:  94%|#########4| 47094/50000 [31:27:11<1:31:24,  1.89s/it]
Outer training loop:  94%|#########4| 47095/50000 [31:27:13<1:30:10,  1.86s/it]
Outer training loop:  94%|#########4| 47096/50000 [31:27:15<1:32:04,  1.90s/it]
Outer training loop:  94%|#########4| 47097/50000 [31:27:17<1:31:01,  1.88s/it]
Outer training loop:  94%|#########4| 47098/50000 [31:27:19<1:29:38,  1.85s/it]
Outer training loop:  94%|#########4| 47099/50000 [31:27:21<1:32:52,  1.92s/it]
Outer training loop:  94%|#########4| 47100/50000 [31:27:23<1:31:30,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.90s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.95s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  94%|#########4| 47101/50000 [31:27:30<2:52:12,  3.56s/it]
Outer training loop:  94%|#########4| 47102/50000 [31:27:32<2:26:39,  3.04s/it]
Outer training loop:  94%|#########4| 47103/50000 [31:27:34<2:12:19,  2.74s/it]
Outer training loop:  94%|#########4| 47104/50000 [31:27:36<2:00:25,  2.49s/it]
Outer training loop:  94%|#########4| 47105/50000 [31:27:38<1:49:51,  2.28s/it]
Outer training loop:  94%|#########4| 47106/50000 [31:27:39<1:43:53,  2.15s/it]
Outer training loop:  94%|#########4| 47107/50000 [31:27:41<1:41:58,  2.12s/it]
Outer training loop:  94%|#########4| 47108/50000 [31:27:43<1:39:10,  2.06s/it]
Outer training loop:  94%|#########4| 47109/50000 [31:27:45<1:34:29,  1.96s/it]
Outer training loop:  94%|#########4| 47110/50000 [31:27:47<1:35:33,  1.98s/it]
Outer training loop:  94%|#########4| 47111/50000 [31:27:49<1:33:58,  1.95s/it]
Outer training loop:  94%|#########4| 47112/50000 [31:27:51<1:32:09,  1.91s/it]
Outer training loop:  94%|#########4| 47113/50000 [31:27:53<1:34:49,  1.97s/it]
Outer training loop:  94%|#########4| 47114/50000 [31:27:55<1:32:45,  1.93s/it]
Outer training loop:  94%|#########4| 47115/50000 [31:27:57<1:31:08,  1.90s/it]
Outer training loop:  94%|#########4| 47116/50000 [31:27:59<1:34:21,  1.96s/it]
Outer training loop:  94%|#########4| 47117/50000 [31:28:00<1:31:29,  1.90s/it]
Outer training loop:  94%|#########4| 47118/50000 [31:28:02<1:30:30,  1.88s/it]
Outer training loop:  94%|#########4| 47119/50000 [31:28:04<1:29:23,  1.86s/it]
Outer training loop:  94%|#########4| 47120/50000 [31:28:06<1:31:22,  1.90s/it]
Outer training loop:  94%|#########4| 47121/50000 [31:28:08<1:29:51,  1.87s/it]
Outer training loop:  94%|#########4| 47122/50000 [31:28:10<1:30:06,  1.88s/it]
Outer training loop:  94%|#########4| 47123/50000 [31:28:12<1:29:23,  1.86s/it]
Outer training loop:  94%|#########4| 47124/50000 [31:28:14<1:32:14,  1.92s/it]
Outer training loop:  94%|#########4| 47125/50000 [31:28:16<1:30:39,  1.89s/it]
Outer training loop:  94%|#########4| 47126/50000 [31:28:17<1:27:58,  1.84s/it]
Outer training loop:  94%|#########4| 47127/50000 [31:28:19<1:31:44,  1.92s/it]
Outer training loop:  94%|#########4| 47128/50000 [31:28:21<1:30:53,  1.90s/it]
Outer training loop:  94%|#########4| 47129/50000 [31:28:23<1:29:04,  1.86s/it]
Outer training loop:  94%|#########4| 47130/50000 [31:28:25<1:31:19,  1.91s/it]
Outer training loop:  94%|#########4| 47131/50000 [31:28:27<1:33:23,  1.95s/it]
Outer training loop:  94%|#########4| 47132/50000 [31:28:29<1:32:08,  1.93s/it]
Outer training loop:  94%|#########4| 47133/50000 [31:28:31<1:30:44,  1.90s/it]
Outer training loop:  94%|#########4| 47134/50000 [31:28:33<1:32:13,  1.93s/it]
Outer training loop:  94%|#########4| 47135/50000 [31:28:35<1:31:57,  1.93s/it]
Outer training loop:  94%|#########4| 47136/50000 [31:28:37<1:30:59,  1.91s/it]
Outer training loop:  94%|#########4| 47137/50000 [31:28:39<1:32:59,  1.95s/it]
Outer training loop:  94%|#########4| 47138/50000 [31:28:40<1:31:25,  1.92s/it]
Outer training loop:  94%|#########4| 47139/50000 [31:28:42<1:29:47,  1.88s/it]
Outer training loop:  94%|#########4| 47140/50000 [31:28:44<1:32:49,  1.95s/it]
Outer training loop:  94%|#########4| 47141/50000 [31:28:46<1:31:38,  1.92s/it]
Outer training loop:  94%|#########4| 47142/50000 [31:28:48<1:29:37,  1.88s/it]
Outer training loop:  94%|#########4| 47143/50000 [31:28:50<1:29:37,  1.88s/it]
Outer training loop:  94%|#########4| 47144/50000 [31:28:52<1:32:51,  1.95s/it]
Outer training loop:  94%|#########4| 47145/50000 [31:28:54<1:31:06,  1.91s/it]
Outer training loop:  94%|#########4| 47146/50000 [31:28:56<1:30:51,  1.91s/it]
Outer training loop:  94%|#########4| 47147/50000 [31:28:58<1:33:47,  1.97s/it]
Outer training loop:  94%|#########4| 47148/50000 [31:29:00<1:31:39,  1.93s/it]
Outer training loop:  94%|#########4| 47149/50000 [31:29:02<1:30:54,  1.91s/it]
Outer training loop:  94%|#########4| 47150/50000 [31:29:03<1:29:57,  1.89s/it]
Outer training loop:  94%|#########4| 47151/50000 [31:29:06<1:35:19,  2.01s/it]
Outer training loop:  94%|#########4| 47152/50000 [31:29:07<1:32:42,  1.95s/it]
Outer training loop:  94%|#########4| 47153/50000 [31:29:09<1:31:55,  1.94s/it]
Outer training loop:  94%|#########4| 47154/50000 [31:29:11<1:30:46,  1.91s/it]
Outer training loop:  94%|#########4| 47155/50000 [31:29:13<1:32:39,  1.95s/it]
Outer training loop:  94%|#########4| 47156/50000 [31:29:15<1:31:18,  1.93s/it]
Outer training loop:  94%|#########4| 47157/50000 [31:29:17<1:29:25,  1.89s/it]
Outer training loop:  94%|#########4| 47158/50000 [31:29:19<1:29:38,  1.89s/it]
Outer training loop:  94%|#########4| 47159/50000 [31:29:21<1:29:29,  1.89s/it]
Outer training loop:  94%|#########4| 47160/50000 [31:29:23<1:27:59,  1.86s/it]
Outer training loop:  94%|#########4| 47161/50000 [31:29:24<1:26:44,  1.83s/it]
Outer training loop:  94%|#########4| 47162/50000 [31:29:26<1:28:09,  1.86s/it]
Outer training loop:  94%|#########4| 47163/50000 [31:29:28<1:29:37,  1.90s/it]
Outer training loop:  94%|#########4| 47164/50000 [31:29:30<1:27:50,  1.86s/it]
Outer training loop:  94%|#########4| 47165/50000 [31:29:32<1:30:03,  1.91s/it]
Outer training loop:  94%|#########4| 47166/50000 [31:29:34<1:28:54,  1.88s/it]
Outer training loop:  94%|#########4| 47167/50000 [31:29:36<1:28:55,  1.88s/it]
Outer training loop:  94%|#########4| 47168/50000 [31:29:38<1:30:25,  1.92s/it]
Outer training loop:  94%|#########4| 47169/50000 [31:29:40<1:29:08,  1.89s/it]
Outer training loop:  94%|#########4| 47170/50000 [31:29:41<1:28:43,  1.88s/it]
Outer training loop:  94%|#########4| 47171/50000 [31:29:43<1:29:38,  1.90s/it]
Outer training loop:  94%|#########4| 47172/50000 [31:29:45<1:31:56,  1.95s/it]
Outer training loop:  94%|#########4| 47173/50000 [31:29:47<1:28:24,  1.88s/it]
Outer training loop:  94%|#########4| 47174/50000 [31:29:49<1:27:05,  1.85s/it]
Outer training loop:  94%|#########4| 47175/50000 [31:29:51<1:29:57,  1.91s/it]
Outer training loop:  94%|#########4| 47176/50000 [31:29:53<1:28:08,  1.87s/it]
Outer training loop:  94%|#########4| 47177/50000 [31:29:55<1:28:09,  1.87s/it]
Outer training loop:  94%|#########4| 47178/50000 [31:29:56<1:26:42,  1.84s/it]
Outer training loop:  94%|#########4| 47179/50000 [31:29:58<1:29:19,  1.90s/it]
Outer training loop:  94%|#########4| 47180/50000 [31:30:00<1:28:09,  1.88s/it]
Outer training loop:  94%|#########4| 47181/50000 [31:30:02<1:29:57,  1.91s/it]
Outer training loop:  94%|#########4| 47182/50000 [31:30:04<1:30:42,  1.93s/it]
Outer training loop:  94%|#########4| 47183/50000 [31:30:06<1:29:01,  1.90s/it]
Outer training loop:  94%|#########4| 47184/50000 [31:30:08<1:27:18,  1.86s/it]
Outer training loop:  94%|#########4| 47185/50000 [31:30:10<1:27:00,  1.85s/it]
Outer training loop:  94%|#########4| 47186/50000 [31:30:12<1:29:29,  1.91s/it]
Outer training loop:  94%|#########4| 47187/50000 [31:30:14<1:29:41,  1.91s/it]
Outer training loop:  94%|#########4| 47188/50000 [31:30:15<1:27:56,  1.88s/it]
Outer training loop:  94%|#########4| 47189/50000 [31:30:17<1:29:32,  1.91s/it]
Outer training loop:  94%|#########4| 47190/50000 [31:30:19<1:29:20,  1.91s/it]
Outer training loop:  94%|#########4| 47191/50000 [31:30:21<1:28:44,  1.90s/it]
Outer training loop:  94%|#########4| 47192/50000 [31:30:23<1:28:04,  1.88s/it]
Outer training loop:  94%|#########4| 47193/50000 [31:30:25<1:28:03,  1.88s/it]
Outer training loop:  94%|#########4| 47194/50000 [31:30:27<1:29:04,  1.90s/it]
Outer training loop:  94%|#########4| 47195/50000 [31:30:29<1:27:25,  1.87s/it]
Outer training loop:  94%|#########4| 47196/50000 [31:30:31<1:28:01,  1.88s/it]
Outer training loop:  94%|#########4| 47197/50000 [31:30:32<1:25:48,  1.84s/it]
Outer training loop:  94%|#########4| 47198/50000 [31:30:34<1:26:12,  1.85s/it]
Outer training loop:  94%|#########4| 47199/50000 [31:30:36<1:30:40,  1.94s/it]
Outer training loop:  94%|#########4| 47200/50000 [31:30:38<1:29:14,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.74s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.80s/it]

Outer training loop:  94%|#########4| 47201/50000 [31:30:46<2:45:32,  3.55s/it]
Outer training loop:  94%|#########4| 47202/50000 [31:30:47<2:21:11,  3.03s/it]
Outer training loop:  94%|#########4| 47203/50000 [31:30:49<2:08:46,  2.76s/it]
Outer training loop:  94%|#########4| 47204/50000 [31:30:51<1:56:46,  2.51s/it]
Outer training loop:  94%|#########4| 47205/50000 [31:30:53<1:46:45,  2.29s/it]
Outer training loop:  94%|#########4| 47206/50000 [31:30:55<1:42:59,  2.21s/it]
Outer training loop:  94%|#########4| 47207/50000 [31:30:57<1:37:00,  2.08s/it]
Outer training loop:  94%|#########4| 47208/50000 [31:30:59<1:33:10,  2.00s/it]
Outer training loop:  94%|#########4| 47209/50000 [31:31:01<1:33:27,  2.01s/it]
Outer training loop:  94%|#########4| 47210/50000 [31:31:03<1:30:52,  1.95s/it]
Outer training loop:  94%|#########4| 47211/50000 [31:31:04<1:28:02,  1.89s/it]
Outer training loop:  94%|#########4| 47212/50000 [31:31:06<1:27:51,  1.89s/it]
Outer training loop:  94%|#########4| 47213/50000 [31:31:08<1:29:23,  1.92s/it]
Outer training loop:  94%|#########4| 47214/50000 [31:31:10<1:27:39,  1.89s/it]
Outer training loop:  94%|#########4| 47215/50000 [31:31:12<1:25:51,  1.85s/it]
Outer training loop:  94%|#########4| 47216/50000 [31:31:14<1:28:32,  1.91s/it]
Outer training loop:  94%|#########4| 47217/50000 [31:31:16<1:26:06,  1.86s/it]
Outer training loop:  94%|#########4| 47218/50000 [31:31:17<1:24:20,  1.82s/it]
Outer training loop:  94%|#########4| 47219/50000 [31:31:19<1:24:26,  1.82s/it]
Outer training loop:  94%|#########4| 47220/50000 [31:31:21<1:28:27,  1.91s/it]
Outer training loop:  94%|#########4| 47221/50000 [31:31:23<1:26:42,  1.87s/it]
Outer training loop:  94%|#########4| 47222/50000 [31:31:25<1:25:55,  1.86s/it]
Outer training loop:  94%|#########4| 47223/50000 [31:31:27<1:28:20,  1.91s/it]
Outer training loop:  94%|#########4| 47224/50000 [31:31:29<1:28:22,  1.91s/it]
Outer training loop:  94%|#########4| 47225/50000 [31:31:31<1:27:03,  1.88s/it]
Outer training loop:  94%|#########4| 47226/50000 [31:31:32<1:25:52,  1.86s/it]
Outer training loop:  94%|#########4| 47227/50000 [31:31:34<1:28:08,  1.91s/it]
Outer training loop:  94%|#########4| 47228/50000 [31:31:36<1:28:56,  1.93s/it]
Outer training loop:  94%|#########4| 47229/50000 [31:31:38<1:27:56,  1.90s/it]
Outer training loop:  94%|#########4| 47230/50000 [31:31:40<1:29:04,  1.93s/it]
Outer training loop:  94%|#########4| 47231/50000 [31:31:42<1:29:50,  1.95s/it]
Outer training loop:  94%|#########4| 47232/50000 [31:31:44<1:29:11,  1.93s/it]
Outer training loop:  94%|#########4| 47233/50000 [31:31:46<1:32:11,  2.00s/it]
Outer training loop:  94%|#########4| 47234/50000 [31:31:48<1:28:39,  1.92s/it]
Outer training loop:  94%|#########4| 47235/50000 [31:31:50<1:28:07,  1.91s/it]
Outer training loop:  94%|#########4| 47236/50000 [31:31:52<1:30:20,  1.96s/it]
Outer training loop:  94%|#########4| 47237/50000 [31:31:54<1:29:08,  1.94s/it]
Outer training loop:  94%|#########4| 47238/50000 [31:31:56<1:26:54,  1.89s/it]
Outer training loop:  94%|#########4| 47239/50000 [31:31:58<1:28:08,  1.92s/it]
Outer training loop:  94%|#########4| 47240/50000 [31:32:00<1:28:31,  1.92s/it]
Outer training loop:  94%|#########4| 47241/50000 [31:32:02<1:28:15,  1.92s/it]
Outer training loop:  94%|#########4| 47242/50000 [31:32:03<1:26:10,  1.87s/it]
Outer training loop:  94%|#########4| 47243/50000 [31:32:05<1:29:57,  1.96s/it]
Outer training loop:  94%|#########4| 47244/50000 [31:32:07<1:28:37,  1.93s/it]
Outer training loop:  94%|#########4| 47245/50000 [31:32:09<1:28:28,  1.93s/it]
Outer training loop:  94%|#########4| 47246/50000 [31:32:11<1:31:16,  1.99s/it]
Outer training loop:  94%|#########4| 47247/50000 [31:32:13<1:28:51,  1.94s/it]
Outer training loop:  94%|#########4| 47248/50000 [31:32:15<1:24:47,  1.85s/it]
Outer training loop:  94%|#########4| 47249/50000 [31:32:17<1:28:24,  1.93s/it]
Outer training loop:  94%|#########4| 47250/50000 [31:32:19<1:26:23,  1.88s/it]
Outer training loop:  95%|#########4| 47251/50000 [31:32:21<1:25:11,  1.86s/it]
Outer training loop:  95%|#########4| 47252/50000 [31:32:22<1:23:47,  1.83s/it]
Outer training loop:  95%|#########4| 47253/50000 [31:32:24<1:25:39,  1.87s/it]
Outer training loop:  95%|#########4| 47254/50000 [31:32:26<1:26:09,  1.88s/it]
Outer training loop:  95%|#########4| 47255/50000 [31:32:28<1:26:02,  1.88s/it]
Outer training loop:  95%|#########4| 47256/50000 [31:32:30<1:27:03,  1.90s/it]
Outer training loop:  95%|#########4| 47257/50000 [31:32:32<1:27:16,  1.91s/it]
Outer training loop:  95%|#########4| 47258/50000 [31:32:34<1:26:28,  1.89s/it]
Outer training loop:  95%|#########4| 47259/50000 [31:32:36<1:28:49,  1.94s/it]
Outer training loop:  95%|#########4| 47260/50000 [31:32:38<1:25:51,  1.88s/it]
Outer training loop:  95%|#########4| 47261/50000 [31:32:39<1:24:48,  1.86s/it]
Outer training loop:  95%|#########4| 47262/50000 [31:32:41<1:26:54,  1.90s/it]
Outer training loop:  95%|#########4| 47263/50000 [31:32:43<1:26:25,  1.89s/it]
Outer training loop:  95%|#########4| 47264/50000 [31:32:45<1:24:33,  1.85s/it]
Outer training loop:  95%|#########4| 47265/50000 [31:32:47<1:23:40,  1.84s/it]
Outer training loop:  95%|#########4| 47266/50000 [31:32:49<1:26:11,  1.89s/it]
Outer training loop:  95%|#########4| 47267/50000 [31:32:51<1:24:42,  1.86s/it]
Outer training loop:  95%|#########4| 47268/50000 [31:32:52<1:24:23,  1.85s/it]
Outer training loop:  95%|#########4| 47269/50000 [31:32:54<1:22:34,  1.81s/it]
Outer training loop:  95%|#########4| 47270/50000 [31:32:56<1:25:17,  1.87s/it]
Outer training loop:  95%|#########4| 47271/50000 [31:32:58<1:23:01,  1.83s/it]
Outer training loop:  95%|#########4| 47272/50000 [31:33:00<1:21:28,  1.79s/it]
Outer training loop:  95%|#########4| 47273/50000 [31:33:02<1:25:46,  1.89s/it]
Outer training loop:  95%|#########4| 47274/50000 [31:33:04<1:24:18,  1.86s/it]
Outer training loop:  95%|#########4| 47275/50000 [31:33:05<1:25:18,  1.88s/it]
Outer training loop:  95%|#########4| 47276/50000 [31:33:07<1:24:49,  1.87s/it]
Outer training loop:  95%|#########4| 47277/50000 [31:33:09<1:27:17,  1.92s/it]
Outer training loop:  95%|#########4| 47278/50000 [31:33:11<1:27:42,  1.93s/it]
Outer training loop:  95%|#########4| 47279/50000 [31:33:13<1:25:01,  1.87s/it]
Outer training loop:  95%|#########4| 47280/50000 [31:33:15<1:25:05,  1.88s/it]
Outer training loop:  95%|#########4| 47281/50000 [31:33:17<1:24:34,  1.87s/it]
Outer training loop:  95%|#########4| 47282/50000 [31:33:19<1:25:51,  1.90s/it]
Outer training loop:  95%|#########4| 47283/50000 [31:33:21<1:24:45,  1.87s/it]
Outer training loop:  95%|#########4| 47284/50000 [31:33:23<1:26:39,  1.91s/it]
Outer training loop:  95%|#########4| 47285/50000 [31:33:24<1:26:05,  1.90s/it]
Outer training loop:  95%|#########4| 47286/50000 [31:33:26<1:27:20,  1.93s/it]
Outer training loop:  95%|#########4| 47287/50000 [31:33:28<1:29:10,  1.97s/it]
Outer training loop:  95%|#########4| 47288/50000 [31:33:30<1:28:37,  1.96s/it]
Outer training loop:  95%|#########4| 47289/50000 [31:33:32<1:25:40,  1.90s/it]
Outer training loop:  95%|#########4| 47290/50000 [31:33:34<1:28:25,  1.96s/it]
Outer training loop:  95%|#########4| 47291/50000 [31:33:36<1:28:10,  1.95s/it]
Outer training loop:  95%|#########4| 47292/50000 [31:33:38<1:25:58,  1.90s/it]
Outer training loop:  95%|#########4| 47293/50000 [31:33:40<1:28:59,  1.97s/it]
Outer training loop:  95%|#########4| 47294/50000 [31:33:42<1:27:45,  1.95s/it]
Outer training loop:  95%|#########4| 47295/50000 [31:33:44<1:25:34,  1.90s/it]
Outer training loop:  95%|#########4| 47296/50000 [31:33:46<1:23:58,  1.86s/it]
Outer training loop:  95%|#########4| 47297/50000 [31:33:48<1:25:30,  1.90s/it]
Outer training loop:  95%|#########4| 47298/50000 [31:33:49<1:24:55,  1.89s/it]
Outer training loop:  95%|#########4| 47299/50000 [31:33:51<1:24:26,  1.88s/it]
Outer training loop:  95%|#########4| 47300/50000 [31:33:53<1:26:55,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.71s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  95%|#########4| 47301/50000 [31:34:01<2:40:47,  3.57s/it]
Outer training loop:  95%|#########4| 47302/50000 [31:34:03<2:16:54,  3.04s/it]
Outer training loop:  95%|#########4| 47303/50000 [31:34:04<2:00:29,  2.68s/it]
Outer training loop:  95%|#########4| 47304/50000 [31:34:06<1:47:37,  2.40s/it]
Outer training loop:  95%|#########4| 47305/50000 [31:34:08<1:43:35,  2.31s/it]
Outer training loop:  95%|#########4| 47306/50000 [31:34:10<1:38:23,  2.19s/it]
Outer training loop:  95%|#########4| 47307/50000 [31:34:12<1:32:36,  2.06s/it]
Outer training loop:  95%|#########4| 47308/50000 [31:34:14<1:33:45,  2.09s/it]
Outer training loop:  95%|#########4| 47309/50000 [31:34:16<1:30:27,  2.02s/it]
Outer training loop:  95%|#########4| 47310/50000 [31:34:18<1:27:46,  1.96s/it]
Outer training loop:  95%|#########4| 47311/50000 [31:34:20<1:27:43,  1.96s/it]
Outer training loop:  95%|#########4| 47312/50000 [31:34:22<1:27:49,  1.96s/it]
Outer training loop:  95%|#########4| 47313/50000 [31:34:23<1:26:07,  1.92s/it]
Outer training loop:  95%|#########4| 47314/50000 [31:34:25<1:26:39,  1.94s/it]
Outer training loop:  95%|#########4| 47315/50000 [31:34:27<1:28:06,  1.97s/it]
Outer training loop:  95%|#########4| 47316/50000 [31:34:29<1:25:27,  1.91s/it]
Outer training loop:  95%|#########4| 47317/50000 [31:34:31<1:24:32,  1.89s/it]
Outer training loop:  95%|#########4| 47318/50000 [31:34:33<1:27:09,  1.95s/it]
Outer training loop:  95%|#########4| 47319/50000 [31:34:35<1:25:01,  1.90s/it]
Outer training loop:  95%|#########4| 47320/50000 [31:34:37<1:24:20,  1.89s/it]
Outer training loop:  95%|#########4| 47321/50000 [31:34:39<1:22:32,  1.85s/it]
Outer training loop:  95%|#########4| 47322/50000 [31:34:41<1:25:39,  1.92s/it]
Outer training loop:  95%|#########4| 47323/50000 [31:34:43<1:24:55,  1.90s/it]
Outer training loop:  95%|#########4| 47324/50000 [31:34:44<1:23:04,  1.86s/it]
Outer training loop:  95%|#########4| 47325/50000 [31:34:47<1:28:01,  1.97s/it]
Outer training loop:  95%|#########4| 47326/50000 [31:34:48<1:24:41,  1.90s/it]
Outer training loop:  95%|#########4| 47327/50000 [31:34:50<1:25:15,  1.91s/it]
Outer training loop:  95%|#########4| 47328/50000 [31:34:52<1:23:46,  1.88s/it]
Outer training loop:  95%|#########4| 47329/50000 [31:34:54<1:25:21,  1.92s/it]
Outer training loop:  95%|#########4| 47330/50000 [31:34:56<1:24:26,  1.90s/it]
Outer training loop:  95%|#########4| 47331/50000 [31:34:58<1:24:43,  1.90s/it]
Outer training loop:  95%|#########4| 47332/50000 [31:35:00<1:25:41,  1.93s/it]
Outer training loop:  95%|#########4| 47333/50000 [31:35:02<1:24:07,  1.89s/it]
Outer training loop:  95%|#########4| 47334/50000 [31:35:03<1:23:19,  1.88s/it]
Outer training loop:  95%|#########4| 47335/50000 [31:35:05<1:24:26,  1.90s/it]
Outer training loop:  95%|#########4| 47336/50000 [31:35:08<1:29:18,  2.01s/it]
Outer training loop:  95%|#########4| 47337/50000 [31:35:10<1:28:03,  1.98s/it]
Outer training loop:  95%|#########4| 47338/50000 [31:35:11<1:25:34,  1.93s/it]
Outer training loop:  95%|#########4| 47339/50000 [31:35:13<1:27:31,  1.97s/it]
Outer training loop:  95%|#########4| 47340/50000 [31:35:15<1:24:44,  1.91s/it]
Outer training loop:  95%|#########4| 47341/50000 [31:35:17<1:23:43,  1.89s/it]
Outer training loop:  95%|#########4| 47342/50000 [31:35:19<1:24:27,  1.91s/it]
Outer training loop:  95%|#########4| 47343/50000 [31:35:21<1:25:37,  1.93s/it]
Outer training loop:  95%|#########4| 47344/50000 [31:35:23<1:26:30,  1.95s/it]
Outer training loop:  95%|#########4| 47345/50000 [31:35:25<1:30:30,  2.05s/it]
Outer training loop:  95%|#########4| 47346/50000 [31:35:27<1:26:41,  1.96s/it]
Outer training loop:  95%|#########4| 47347/50000 [31:35:29<1:26:06,  1.95s/it]
Outer training loop:  95%|#########4| 47348/50000 [31:35:31<1:23:53,  1.90s/it]
Outer training loop:  95%|#########4| 47349/50000 [31:35:33<1:26:50,  1.97s/it]
Outer training loop:  95%|#########4| 47350/50000 [31:35:35<1:27:22,  1.98s/it]
Outer training loop:  95%|#########4| 47351/50000 [31:35:37<1:26:10,  1.95s/it]
Outer training loop:  95%|#########4| 47352/50000 [31:35:39<1:27:23,  1.98s/it]
Outer training loop:  95%|#########4| 47353/50000 [31:35:41<1:25:52,  1.95s/it]
Outer training loop:  95%|#########4| 47354/50000 [31:35:43<1:25:38,  1.94s/it]
Outer training loop:  95%|#########4| 47355/50000 [31:35:44<1:23:37,  1.90s/it]
Outer training loop:  95%|#########4| 47356/50000 [31:35:46<1:25:05,  1.93s/it]
Outer training loop:  95%|#########4| 47357/50000 [31:35:48<1:24:20,  1.91s/it]
Outer training loop:  95%|#########4| 47358/50000 [31:35:50<1:23:19,  1.89s/it]
Outer training loop:  95%|#########4| 47359/50000 [31:35:52<1:23:40,  1.90s/it]
Outer training loop:  95%|#########4| 47360/50000 [31:35:54<1:23:01,  1.89s/it]
Outer training loop:  95%|#########4| 47361/50000 [31:35:56<1:23:05,  1.89s/it]
Outer training loop:  95%|#########4| 47362/50000 [31:35:58<1:22:34,  1.88s/it]
Outer training loop:  95%|#########4| 47363/50000 [31:36:00<1:22:42,  1.88s/it]
Outer training loop:  95%|#########4| 47364/50000 [31:36:01<1:22:00,  1.87s/it]
Outer training loop:  95%|#########4| 47365/50000 [31:36:03<1:21:24,  1.85s/it]
Outer training loop:  95%|#########4| 47366/50000 [31:36:05<1:25:49,  1.95s/it]
Outer training loop:  95%|#########4| 47367/50000 [31:36:07<1:24:13,  1.92s/it]
Outer training loop:  95%|#########4| 47368/50000 [31:36:09<1:22:57,  1.89s/it]
Outer training loop:  95%|#########4| 47369/50000 [31:36:11<1:21:48,  1.87s/it]
Outer training loop:  95%|#########4| 47370/50000 [31:36:13<1:24:54,  1.94s/it]
Outer training loop:  95%|#########4| 47371/50000 [31:36:15<1:23:05,  1.90s/it]
Outer training loop:  95%|#########4| 47372/50000 [31:36:17<1:22:15,  1.88s/it]
Outer training loop:  95%|#########4| 47373/50000 [31:36:19<1:26:18,  1.97s/it]
Outer training loop:  95%|#########4| 47374/50000 [31:36:21<1:26:50,  1.98s/it]
Outer training loop:  95%|#########4| 47375/50000 [31:36:23<1:27:47,  2.01s/it]
Outer training loop:  95%|#########4| 47376/50000 [31:36:25<1:29:00,  2.04s/it]
Outer training loop:  95%|#########4| 47377/50000 [31:36:27<1:25:08,  1.95s/it]
Outer training loop:  95%|#########4| 47378/50000 [31:36:29<1:24:38,  1.94s/it]
Outer training loop:  95%|#########4| 47379/50000 [31:36:31<1:24:54,  1.94s/it]
Outer training loop:  95%|#########4| 47380/50000 [31:36:33<1:25:45,  1.96s/it]
Outer training loop:  95%|#########4| 47381/50000 [31:36:34<1:25:04,  1.95s/it]
Outer training loop:  95%|#########4| 47382/50000 [31:36:36<1:23:18,  1.91s/it]
Outer training loop:  95%|#########4| 47383/50000 [31:36:38<1:24:08,  1.93s/it]
Outer training loop:  95%|#########4| 47384/50000 [31:36:40<1:21:32,  1.87s/it]
Outer training loop:  95%|#########4| 47385/50000 [31:36:42<1:20:38,  1.85s/it]
Outer training loop:  95%|#########4| 47386/50000 [31:36:44<1:20:26,  1.85s/it]
Outer training loop:  95%|#########4| 47387/50000 [31:36:46<1:23:56,  1.93s/it]
Outer training loop:  95%|#########4| 47388/50000 [31:36:48<1:21:47,  1.88s/it]
Outer training loop:  95%|#########4| 47389/50000 [31:36:49<1:20:55,  1.86s/it]
Outer training loop:  95%|#########4| 47390/50000 [31:36:51<1:23:07,  1.91s/it]
Outer training loop:  95%|#########4| 47391/50000 [31:36:53<1:21:37,  1.88s/it]
Outer training loop:  95%|#########4| 47392/50000 [31:36:55<1:20:55,  1.86s/it]
Outer training loop:  95%|#########4| 47393/50000 [31:36:57<1:22:19,  1.89s/it]
Outer training loop:  95%|#########4| 47394/50000 [31:36:59<1:20:59,  1.86s/it]
Outer training loop:  95%|#########4| 47395/50000 [31:37:01<1:21:09,  1.87s/it]
Outer training loop:  95%|#########4| 47396/50000 [31:37:03<1:21:39,  1.88s/it]
Outer training loop:  95%|#########4| 47397/50000 [31:37:05<1:24:31,  1.95s/it]
Outer training loop:  95%|#########4| 47398/50000 [31:37:07<1:23:43,  1.93s/it]
Outer training loop:  95%|#########4| 47399/50000 [31:37:08<1:21:35,  1.88s/it]
Outer training loop:  95%|#########4| 47400/50000 [31:37:10<1:24:34,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.88s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  95%|#########4| 47401/50000 [31:37:18<2:34:19,  3.56s/it]
Outer training loop:  95%|#########4| 47402/50000 [31:37:20<2:13:08,  3.07s/it]
Outer training loop:  95%|#########4| 47403/50000 [31:37:22<1:59:33,  2.76s/it]
Outer training loop:  95%|#########4| 47404/50000 [31:37:24<1:50:50,  2.56s/it]
Outer training loop:  95%|#########4| 47405/50000 [31:37:26<1:44:08,  2.41s/it]
Outer training loop:  95%|#########4| 47406/50000 [31:37:28<1:36:29,  2.23s/it]
Outer training loop:  95%|#########4| 47407/50000 [31:37:30<1:33:04,  2.15s/it]
Outer training loop:  95%|#########4| 47408/50000 [31:37:32<1:30:29,  2.09s/it]
Outer training loop:  95%|#########4| 47409/50000 [31:37:33<1:27:15,  2.02s/it]
Outer training loop:  95%|#########4| 47410/50000 [31:37:36<1:28:46,  2.06s/it]
Outer training loop:  95%|#########4| 47411/50000 [31:37:37<1:25:40,  1.99s/it]
Outer training loop:  95%|#########4| 47412/50000 [31:37:39<1:23:32,  1.94s/it]
Outer training loop:  95%|#########4| 47413/50000 [31:37:41<1:21:54,  1.90s/it]
Outer training loop:  95%|#########4| 47414/50000 [31:37:43<1:24:04,  1.95s/it]
Outer training loop:  95%|#########4| 47415/50000 [31:37:45<1:22:53,  1.92s/it]
Outer training loop:  95%|#########4| 47416/50000 [31:37:47<1:21:06,  1.88s/it]
Outer training loop:  95%|#########4| 47417/50000 [31:37:49<1:23:02,  1.93s/it]
Outer training loop:  95%|#########4| 47418/50000 [31:37:51<1:23:34,  1.94s/it]
Outer training loop:  95%|#########4| 47419/50000 [31:37:53<1:22:43,  1.92s/it]
Outer training loop:  95%|#########4| 47420/50000 [31:37:55<1:24:53,  1.97s/it]
Outer training loop:  95%|#########4| 47421/50000 [31:37:57<1:23:22,  1.94s/it]
Outer training loop:  95%|#########4| 47422/50000 [31:37:59<1:23:32,  1.94s/it]
Outer training loop:  95%|#########4| 47423/50000 [31:38:01<1:25:45,  2.00s/it]
Outer training loop:  95%|#########4| 47424/50000 [31:38:03<1:23:17,  1.94s/it]
Outer training loop:  95%|#########4| 47425/50000 [31:38:04<1:22:03,  1.91s/it]
Outer training loop:  95%|#########4| 47426/50000 [31:38:06<1:21:37,  1.90s/it]
Outer training loop:  95%|#########4| 47427/50000 [31:38:08<1:23:45,  1.95s/it]
Outer training loop:  95%|#########4| 47428/50000 [31:38:10<1:22:32,  1.93s/it]
Outer training loop:  95%|#########4| 47429/50000 [31:38:12<1:22:02,  1.91s/it]
Outer training loop:  95%|#########4| 47430/50000 [31:38:14<1:24:07,  1.96s/it]
Outer training loop:  95%|#########4| 47431/50000 [31:38:16<1:20:38,  1.88s/it]
Outer training loop:  95%|#########4| 47432/50000 [31:38:18<1:18:34,  1.84s/it]
Outer training loop:  95%|#########4| 47433/50000 [31:38:19<1:18:18,  1.83s/it]
Outer training loop:  95%|#########4| 47434/50000 [31:38:21<1:20:16,  1.88s/it]
Outer training loop:  95%|#########4| 47435/50000 [31:38:23<1:21:23,  1.90s/it]
Outer training loop:  95%|#########4| 47436/50000 [31:38:25<1:21:45,  1.91s/it]
Outer training loop:  95%|#########4| 47437/50000 [31:38:27<1:24:22,  1.98s/it]
Outer training loop:  95%|#########4| 47438/50000 [31:38:29<1:22:41,  1.94s/it]
Outer training loop:  95%|#########4| 47439/50000 [31:38:31<1:22:21,  1.93s/it]
Outer training loop:  95%|#########4| 47440/50000 [31:38:33<1:20:20,  1.88s/it]
Outer training loop:  95%|#########4| 47441/50000 [31:38:35<1:22:53,  1.94s/it]
Outer training loop:  95%|#########4| 47442/50000 [31:38:37<1:20:52,  1.90s/it]
Outer training loop:  95%|#########4| 47443/50000 [31:38:39<1:20:36,  1.89s/it]
Outer training loop:  95%|#########4| 47444/50000 [31:38:41<1:23:35,  1.96s/it]
Outer training loop:  95%|#########4| 47445/50000 [31:38:43<1:22:58,  1.95s/it]
Outer training loop:  95%|#########4| 47446/50000 [31:38:45<1:23:23,  1.96s/it]
Outer training loop:  95%|#########4| 47447/50000 [31:38:47<1:23:49,  1.97s/it]
Outer training loop:  95%|#########4| 47448/50000 [31:38:49<1:21:50,  1.92s/it]
Outer training loop:  95%|#########4| 47449/50000 [31:38:50<1:22:10,  1.93s/it]
Outer training loop:  95%|#########4| 47450/50000 [31:38:52<1:19:28,  1.87s/it]
Outer training loop:  95%|#########4| 47451/50000 [31:38:54<1:20:29,  1.89s/it]
Outer training loop:  95%|#########4| 47452/50000 [31:38:56<1:19:44,  1.88s/it]
Outer training loop:  95%|#########4| 47453/50000 [31:38:58<1:17:41,  1.83s/it]
Outer training loop:  95%|#########4| 47454/50000 [31:39:00<1:19:50,  1.88s/it]
Outer training loop:  95%|#########4| 47455/50000 [31:39:02<1:19:34,  1.88s/it]
Outer training loop:  95%|#########4| 47456/50000 [31:39:03<1:19:22,  1.87s/it]
Outer training loop:  95%|#########4| 47457/50000 [31:39:05<1:19:57,  1.89s/it]
Outer training loop:  95%|#########4| 47458/50000 [31:39:07<1:21:24,  1.92s/it]
Outer training loop:  95%|#########4| 47459/50000 [31:39:09<1:19:20,  1.87s/it]
Outer training loop:  95%|#########4| 47460/50000 [31:39:11<1:19:20,  1.87s/it]
Outer training loop:  95%|#########4| 47461/50000 [31:39:13<1:21:21,  1.92s/it]
Outer training loop:  95%|#########4| 47462/50000 [31:39:15<1:21:25,  1.92s/it]
Outer training loop:  95%|#########4| 47463/50000 [31:39:17<1:19:12,  1.87s/it]
Outer training loop:  95%|#########4| 47464/50000 [31:39:19<1:21:58,  1.94s/it]
Outer training loop:  95%|#########4| 47465/50000 [31:39:21<1:19:36,  1.88s/it]
Outer training loop:  95%|#########4| 47466/50000 [31:39:22<1:18:28,  1.86s/it]
Outer training loop:  95%|#########4| 47467/50000 [31:39:24<1:18:07,  1.85s/it]
Outer training loop:  95%|#########4| 47468/50000 [31:39:26<1:21:40,  1.94s/it]
Outer training loop:  95%|#########4| 47469/50000 [31:39:28<1:20:35,  1.91s/it]
Outer training loop:  95%|#########4| 47470/50000 [31:39:30<1:18:34,  1.86s/it]
Outer training loop:  95%|#########4| 47471/50000 [31:39:32<1:20:12,  1.90s/it]
Outer training loop:  95%|#########4| 47472/50000 [31:39:34<1:22:15,  1.95s/it]
Outer training loop:  95%|#########4| 47473/50000 [31:39:36<1:20:45,  1.92s/it]
Outer training loop:  95%|#########4| 47474/50000 [31:39:38<1:18:56,  1.88s/it]
Outer training loop:  95%|#########4| 47475/50000 [31:39:40<1:20:10,  1.90s/it]
Outer training loop:  95%|#########4| 47476/50000 [31:39:41<1:19:28,  1.89s/it]
Outer training loop:  95%|#########4| 47477/50000 [31:39:43<1:18:18,  1.86s/it]
Outer training loop:  95%|#########4| 47478/50000 [31:39:45<1:21:03,  1.93s/it]
Outer training loop:  95%|#########4| 47479/50000 [31:39:47<1:18:13,  1.86s/it]
Outer training loop:  95%|#########4| 47480/50000 [31:39:49<1:17:20,  1.84s/it]
Outer training loop:  95%|#########4| 47481/50000 [31:39:51<1:18:29,  1.87s/it]
Outer training loop:  95%|#########4| 47482/50000 [31:39:53<1:20:19,  1.91s/it]
Outer training loop:  95%|#########4| 47483/50000 [31:39:55<1:19:44,  1.90s/it]
Outer training loop:  95%|#########4| 47484/50000 [31:39:56<1:18:05,  1.86s/it]
Outer training loop:  95%|#########4| 47485/50000 [31:39:58<1:19:49,  1.90s/it]
Outer training loop:  95%|#########4| 47486/50000 [31:40:00<1:19:02,  1.89s/it]
Outer training loop:  95%|#########4| 47487/50000 [31:40:02<1:16:42,  1.83s/it]
Outer training loop:  95%|#########4| 47488/50000 [31:40:04<1:18:40,  1.88s/it]
Outer training loop:  95%|#########4| 47489/50000 [31:40:06<1:17:55,  1.86s/it]
Outer training loop:  95%|#########4| 47490/50000 [31:40:07<1:15:42,  1.81s/it]
Outer training loop:  95%|#########4| 47491/50000 [31:40:09<1:14:51,  1.79s/it]
Outer training loop:  95%|#########4| 47492/50000 [31:40:11<1:17:44,  1.86s/it]
Outer training loop:  95%|#########4| 47493/50000 [31:40:13<1:17:46,  1.86s/it]
Outer training loop:  95%|#########4| 47494/50000 [31:40:15<1:17:52,  1.86s/it]
Outer training loop:  95%|#########4| 47495/50000 [31:40:17<1:19:34,  1.91s/it]
Outer training loop:  95%|#########4| 47496/50000 [31:40:19<1:19:41,  1.91s/it]
Outer training loop:  95%|#########4| 47497/50000 [31:40:21<1:17:48,  1.87s/it]
Outer training loop:  95%|#########4| 47498/50000 [31:40:23<1:19:23,  1.90s/it]
Outer training loop:  95%|#########4| 47499/50000 [31:40:25<1:19:31,  1.91s/it]
Outer training loop:  95%|#########5| 47500/50000 [31:40:27<1:20:17,  1.93s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.76s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.73s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:39,  1.95s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:35,  1.92s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:32,  1.91s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:32,  1.93s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.90s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:23,  1.87s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:21,  1.86s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:22,  1.93s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.88s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:17,  1.94s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:11,  1.91s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:09,  1.94s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:01,  1.90s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:59,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<02:01,  1.96s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:55,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.91s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:51,  1.96s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:49,  1.95s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.91s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:39,  1.85s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:36,  1.82s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:40,  1.93s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.93s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:29,  1.83s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:28,  1.84s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:27,  1.86s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:28,  1.92s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:20,  1.84s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.82s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:15,  1.84s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:11,  1.89s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:07,  1.83s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:05,  1.81s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:03,  1.82s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:00,  1.79s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<00:59,  1.79s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.92s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.88s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:53,  1.85s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.83s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.86s/it]

Evaluating policy:  70%|######9   | 58/83 [01:48<00:46,  1.85s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:47,  1.99s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:44,  1.95s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:42,  1.91s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:36,  1.82s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:30,  1.78s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:30,  1.93s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:29,  1.96s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.91s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.89s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:20,  1.82s/it]

Evaluating policy:  88%|########7 | 73/83 [02:17<00:18,  1.84s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.86s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:16,  2.05s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:14,  2.02s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.97s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.93s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.96s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.97s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.91s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:21,  1.73s/it]

Evaluating policy:   2%|2         | 2/83 [00:04<02:35,  1.93s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:30,  1.88s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:32,  1.96s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.92s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.92s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:23,  1.92s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.96s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:17,  1.89s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:14,  1.87s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:12,  1.87s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:14,  1.92s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:09,  1.90s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:07,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:08,  1.95s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<01:58,  1.85s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:56,  1.85s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:53,  1.83s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:54,  1.87s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.85s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:48,  1.84s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:46,  1.84s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:43,  1.82s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.89s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:40,  1.84s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:38,  1.82s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:36,  1.81s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:33,  1.79s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.90s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:34,  1.89s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:32,  1.93s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:31,  1.94s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:29,  1.94s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:28,  1.97s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.91s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:22,  1.93s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:20,  1.92s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:18,  2.01s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:15,  1.99s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:10,  1.91s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:07,  1.88s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.87s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:00,  1.84s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:03,  1.98s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<01:00,  1.94s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:57,  1.92s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.86s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:51,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.88s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.88s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:47,  1.99s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:44,  1.94s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:42,  1.91s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.89s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.85s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.84s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:32,  2.01s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:29,  1.95s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.92s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.91s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.87s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.86s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.86s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:14,  2.05s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.98s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.90s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.94s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.89s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.91s/it]

Outer training loop:  95%|#########5| 47501/50000 [31:45:48<67:57:27, 97.90s/it]
Outer training loop:  95%|#########5| 47502/50000 [31:45:50<47:56:11, 69.08s/it]
Outer training loop:  95%|#########5| 47503/50000 [31:45:52<33:58:17, 48.98s/it]
Outer training loop:  95%|#########5| 47504/50000 [31:45:55<24:16:37, 35.01s/it]
Outer training loop:  95%|#########5| 47505/50000 [31:45:57<17:22:08, 25.06s/it]
Outer training loop:  95%|#########5| 47506/50000 [31:45:58<12:32:09, 18.10s/it]
Outer training loop:  95%|#########5| 47507/50000 [31:46:01<9:13:48, 13.33s/it] 
Outer training loop:  95%|#########5| 47508/50000 [31:46:02<6:50:37,  9.89s/it]
Outer training loop:  95%|#########5| 47509/50000 [31:46:04<5:10:35,  7.48s/it]
Outer training loop:  95%|#########5| 47510/50000 [31:46:06<4:01:18,  5.81s/it]
Outer training loop:  95%|#########5| 47511/50000 [31:46:08<3:13:05,  4.65s/it]
Outer training loop:  95%|#########5| 47512/50000 [31:46:10<2:39:13,  3.84s/it]
Outer training loop:  95%|#########5| 47513/50000 [31:46:12<2:16:10,  3.29s/it]
Outer training loop:  95%|#########5| 47514/50000 [31:46:14<2:00:05,  2.90s/it]
Outer training loop:  95%|#########5| 47515/50000 [31:46:16<1:48:11,  2.61s/it]
Outer training loop:  95%|#########5| 47516/50000 [31:46:18<1:38:52,  2.39s/it]
Outer training loop:  95%|#########5| 47517/50000 [31:46:20<1:37:34,  2.36s/it]
Outer training loop:  95%|#########5| 47518/50000 [31:46:22<1:31:56,  2.22s/it]
Outer training loop:  95%|#########5| 47519/50000 [31:46:24<1:27:13,  2.11s/it]
Outer training loop:  95%|#########5| 47520/50000 [31:46:26<1:25:45,  2.07s/it]
Outer training loop:  95%|#########5| 47521/50000 [31:46:28<1:26:02,  2.08s/it]
Outer training loop:  95%|#########5| 47522/50000 [31:46:30<1:23:19,  2.02s/it]
Outer training loop:  95%|#########5| 47523/50000 [31:46:32<1:22:22,  2.00s/it]
Outer training loop:  95%|#########5| 47524/50000 [31:46:34<1:24:56,  2.06s/it]
Outer training loop:  95%|#########5| 47525/50000 [31:46:36<1:22:49,  2.01s/it]
Outer training loop:  95%|#########5| 47526/50000 [31:46:38<1:22:44,  2.01s/it]
Outer training loop:  95%|#########5| 47527/50000 [31:46:40<1:20:00,  1.94s/it]
Outer training loop:  95%|#########5| 47528/50000 [31:46:42<1:23:18,  2.02s/it]
Outer training loop:  95%|#########5| 47529/50000 [31:46:44<1:21:09,  1.97s/it]
Outer training loop:  95%|#########5| 47530/50000 [31:46:46<1:20:05,  1.95s/it]
Outer training loop:  95%|#########5| 47531/50000 [31:46:48<1:21:25,  1.98s/it]
Outer training loop:  95%|#########5| 47532/50000 [31:46:50<1:20:04,  1.95s/it]
Outer training loop:  95%|#########5| 47533/50000 [31:46:52<1:20:18,  1.95s/it]
Outer training loop:  95%|#########5| 47534/50000 [31:46:54<1:21:13,  1.98s/it]
Outer training loop:  95%|#########5| 47535/50000 [31:46:55<1:19:39,  1.94s/it]
Outer training loop:  95%|#########5| 47536/50000 [31:46:57<1:18:41,  1.92s/it]
Outer training loop:  95%|#########5| 47537/50000 [31:46:59<1:19:09,  1.93s/it]
Outer training loop:  95%|#########5| 47538/50000 [31:47:01<1:18:22,  1.91s/it]
Outer training loop:  95%|#########5| 47539/50000 [31:47:03<1:16:21,  1.86s/it]
Outer training loop:  95%|#########5| 47540/50000 [31:47:05<1:19:21,  1.94s/it]
Outer training loop:  95%|#########5| 47541/50000 [31:47:07<1:18:35,  1.92s/it]
Outer training loop:  95%|#########5| 47542/50000 [31:47:09<1:16:56,  1.88s/it]
Outer training loop:  95%|#########5| 47543/50000 [31:47:11<1:17:02,  1.88s/it]
Outer training loop:  95%|#########5| 47544/50000 [31:47:13<1:19:35,  1.94s/it]
Outer training loop:  95%|#########5| 47545/50000 [31:47:14<1:17:17,  1.89s/it]
Outer training loop:  95%|#########5| 47546/50000 [31:47:16<1:17:29,  1.89s/it]
Outer training loop:  95%|#########5| 47547/50000 [31:47:18<1:16:47,  1.88s/it]
Outer training loop:  95%|#########5| 47548/50000 [31:47:20<1:19:30,  1.95s/it]
Outer training loop:  95%|#########5| 47549/50000 [31:47:22<1:18:20,  1.92s/it]
Outer training loop:  95%|#########5| 47550/50000 [31:47:24<1:16:56,  1.88s/it]
Outer training loop:  95%|#########5| 47551/50000 [31:47:26<1:20:11,  1.96s/it]
Outer training loop:  95%|#########5| 47552/50000 [31:47:28<1:18:57,  1.94s/it]
Outer training loop:  95%|#########5| 47553/50000 [31:47:30<1:20:03,  1.96s/it]
Outer training loop:  95%|#########5| 47554/50000 [31:47:32<1:20:52,  1.98s/it]
Outer training loop:  95%|#########5| 47555/50000 [31:47:34<1:19:30,  1.95s/it]
Outer training loop:  95%|#########5| 47556/50000 [31:47:36<1:18:32,  1.93s/it]
Outer training loop:  95%|#########5| 47557/50000 [31:47:38<1:17:01,  1.89s/it]
Outer training loop:  95%|#########5| 47558/50000 [31:47:40<1:18:24,  1.93s/it]
Outer training loop:  95%|#########5| 47559/50000 [31:47:41<1:17:46,  1.91s/it]
Outer training loop:  95%|#########5| 47560/50000 [31:47:43<1:17:27,  1.90s/it]
Outer training loop:  95%|#########5| 47561/50000 [31:47:45<1:17:40,  1.91s/it]
Outer training loop:  95%|#########5| 47562/50000 [31:47:47<1:17:06,  1.90s/it]
Outer training loop:  95%|#########5| 47563/50000 [31:47:49<1:17:43,  1.91s/it]
Outer training loop:  95%|#########5| 47564/50000 [31:47:51<1:16:13,  1.88s/it]
Outer training loop:  95%|#########5| 47565/50000 [31:47:53<1:17:22,  1.91s/it]
Outer training loop:  95%|#########5| 47566/50000 [31:47:55<1:16:43,  1.89s/it]
Outer training loop:  95%|#########5| 47567/50000 [31:47:57<1:15:54,  1.87s/it]
Outer training loop:  95%|#########5| 47568/50000 [31:47:59<1:17:38,  1.92s/it]
Outer training loop:  95%|#########5| 47569/50000 [31:48:01<1:18:07,  1.93s/it]
Outer training loop:  95%|#########5| 47570/50000 [31:48:02<1:16:24,  1.89s/it]
Outer training loop:  95%|#########5| 47571/50000 [31:48:04<1:17:27,  1.91s/it]
Outer training loop:  95%|#########5| 47572/50000 [31:48:06<1:15:26,  1.86s/it]
Outer training loop:  95%|#########5| 47573/50000 [31:48:08<1:14:44,  1.85s/it]
Outer training loop:  95%|#########5| 47574/50000 [31:48:10<1:15:38,  1.87s/it]
Outer training loop:  95%|#########5| 47575/50000 [31:48:12<1:15:26,  1.87s/it]
Outer training loop:  95%|#########5| 47576/50000 [31:48:13<1:15:08,  1.86s/it]
Outer training loop:  95%|#########5| 47577/50000 [31:48:15<1:12:50,  1.80s/it]
Outer training loop:  95%|#########5| 47578/50000 [31:48:17<1:14:29,  1.85s/it]
Outer training loop:  95%|#########5| 47579/50000 [31:48:19<1:13:42,  1.83s/it]
Outer training loop:  95%|#########5| 47580/50000 [31:48:21<1:14:33,  1.85s/it]
Outer training loop:  95%|#########5| 47581/50000 [31:48:23<1:16:01,  1.89s/it]
Outer training loop:  95%|#########5| 47582/50000 [31:48:25<1:14:52,  1.86s/it]
Outer training loop:  95%|#########5| 47583/50000 [31:48:26<1:13:51,  1.83s/it]
Outer training loop:  95%|#########5| 47584/50000 [31:48:28<1:14:18,  1.85s/it]
Outer training loop:  95%|#########5| 47585/50000 [31:48:30<1:17:06,  1.92s/it]
Outer training loop:  95%|#########5| 47586/50000 [31:48:32<1:15:55,  1.89s/it]
Outer training loop:  95%|#########5| 47587/50000 [31:48:34<1:15:10,  1.87s/it]
Outer training loop:  95%|#########5| 47588/50000 [31:48:36<1:15:15,  1.87s/it]
Outer training loop:  95%|#########5| 47589/50000 [31:48:38<1:16:57,  1.92s/it]
Outer training loop:  95%|#########5| 47590/50000 [31:48:40<1:17:15,  1.92s/it]
Outer training loop:  95%|#########5| 47591/50000 [31:48:42<1:15:41,  1.89s/it]
Outer training loop:  95%|#########5| 47592/50000 [31:48:44<1:17:14,  1.92s/it]
Outer training loop:  95%|#########5| 47593/50000 [31:48:45<1:15:29,  1.88s/it]
Outer training loop:  95%|#########5| 47594/50000 [31:48:47<1:14:45,  1.86s/it]
Outer training loop:  95%|#########5| 47595/50000 [31:48:49<1:16:52,  1.92s/it]
Outer training loop:  95%|#########5| 47596/50000 [31:48:51<1:16:43,  1.91s/it]
Outer training loop:  95%|#########5| 47597/50000 [31:48:53<1:15:55,  1.90s/it]
Outer training loop:  95%|#########5| 47598/50000 [31:48:55<1:14:21,  1.86s/it]
Outer training loop:  95%|#########5| 47599/50000 [31:48:57<1:16:31,  1.91s/it]
Outer training loop:  95%|#########5| 47600/50000 [31:48:59<1:15:38,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.99s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.90s/it]

Outer training loop:  95%|#########5| 47601/50000 [31:49:06<2:25:05,  3.63s/it]
Outer training loop:  95%|#########5| 47602/50000 [31:49:08<2:03:20,  3.09s/it]
Outer training loop:  95%|#########5| 47603/50000 [31:49:10<1:51:03,  2.78s/it]
Outer training loop:  95%|#########5| 47604/50000 [31:49:12<1:38:34,  2.47s/it]
Outer training loop:  95%|#########5| 47605/50000 [31:49:14<1:31:27,  2.29s/it]
Outer training loop:  95%|#########5| 47606/50000 [31:49:16<1:28:24,  2.22s/it]
Outer training loop:  95%|#########5| 47607/50000 [31:49:18<1:22:59,  2.08s/it]
Outer training loop:  95%|#########5| 47608/50000 [31:49:20<1:20:58,  2.03s/it]
Outer training loop:  95%|#########5| 47609/50000 [31:49:21<1:18:09,  1.96s/it]
Outer training loop:  95%|#########5| 47610/50000 [31:49:23<1:18:17,  1.97s/it]
Outer training loop:  95%|#########5| 47611/50000 [31:49:25<1:18:36,  1.97s/it]
Outer training loop:  95%|#########5| 47612/50000 [31:49:27<1:15:54,  1.91s/it]
Outer training loop:  95%|#########5| 47613/50000 [31:49:29<1:17:22,  1.94s/it]
Outer training loop:  95%|#########5| 47614/50000 [31:49:31<1:16:13,  1.92s/it]
Outer training loop:  95%|#########5| 47615/50000 [31:49:33<1:12:34,  1.83s/it]
Outer training loop:  95%|#########5| 47616/50000 [31:49:35<1:15:37,  1.90s/it]
Outer training loop:  95%|#########5| 47617/50000 [31:49:36<1:13:51,  1.86s/it]
Outer training loop:  95%|#########5| 47618/50000 [31:49:38<1:13:01,  1.84s/it]
Outer training loop:  95%|#########5| 47619/50000 [31:49:40<1:11:38,  1.81s/it]
Outer training loop:  95%|#########5| 47620/50000 [31:49:42<1:13:27,  1.85s/it]
Outer training loop:  95%|#########5| 47621/50000 [31:49:44<1:14:01,  1.87s/it]
Outer training loop:  95%|#########5| 47622/50000 [31:49:46<1:13:15,  1.85s/it]
Outer training loop:  95%|#########5| 47623/50000 [31:49:47<1:14:15,  1.87s/it]
Outer training loop:  95%|#########5| 47624/50000 [31:49:50<1:19:05,  2.00s/it]
Outer training loop:  95%|#########5| 47625/50000 [31:49:52<1:17:02,  1.95s/it]
Outer training loop:  95%|#########5| 47626/50000 [31:49:53<1:14:56,  1.89s/it]
Outer training loop:  95%|#########5| 47627/50000 [31:49:55<1:15:21,  1.91s/it]
Outer training loop:  95%|#########5| 47628/50000 [31:49:57<1:14:30,  1.88s/it]
Outer training loop:  95%|#########5| 47629/50000 [31:49:59<1:14:01,  1.87s/it]
Outer training loop:  95%|#########5| 47630/50000 [31:50:01<1:13:06,  1.85s/it]
Outer training loop:  95%|#########5| 47631/50000 [31:50:03<1:16:00,  1.92s/it]
Outer training loop:  95%|#########5| 47632/50000 [31:50:05<1:15:42,  1.92s/it]
Outer training loop:  95%|#########5| 47633/50000 [31:50:07<1:15:01,  1.90s/it]
Outer training loop:  95%|#########5| 47634/50000 [31:50:09<1:18:08,  1.98s/it]
Outer training loop:  95%|#########5| 47635/50000 [31:50:11<1:16:13,  1.93s/it]
Outer training loop:  95%|#########5| 47636/50000 [31:50:13<1:15:50,  1.92s/it]
Outer training loop:  95%|#########5| 47637/50000 [31:50:15<1:16:30,  1.94s/it]
Outer training loop:  95%|#########5| 47638/50000 [31:50:16<1:15:17,  1.91s/it]
Outer training loop:  95%|#########5| 47639/50000 [31:50:18<1:12:51,  1.85s/it]
Outer training loop:  95%|#########5| 47640/50000 [31:50:20<1:13:54,  1.88s/it]
Outer training loop:  95%|#########5| 47641/50000 [31:50:22<1:15:33,  1.92s/it]
Outer training loop:  95%|#########5| 47642/50000 [31:50:24<1:13:25,  1.87s/it]
Outer training loop:  95%|#########5| 47643/50000 [31:50:26<1:13:18,  1.87s/it]
Outer training loop:  95%|#########5| 47644/50000 [31:50:28<1:14:32,  1.90s/it]
Outer training loop:  95%|#########5| 47645/50000 [31:50:29<1:13:02,  1.86s/it]
Outer training loop:  95%|#########5| 47646/50000 [31:50:31<1:13:22,  1.87s/it]
Outer training loop:  95%|#########5| 47647/50000 [31:50:33<1:14:41,  1.90s/it]
Outer training loop:  95%|#########5| 47648/50000 [31:50:35<1:14:13,  1.89s/it]
Outer training loop:  95%|#########5| 47649/50000 [31:50:37<1:13:43,  1.88s/it]
Outer training loop:  95%|#########5| 47650/50000 [31:50:39<1:16:27,  1.95s/it]
Outer training loop:  95%|#########5| 47651/50000 [31:50:41<1:18:26,  2.00s/it]
Outer training loop:  95%|#########5| 47652/50000 [31:50:43<1:16:37,  1.96s/it]
Outer training loop:  95%|#########5| 47653/50000 [31:50:45<1:14:40,  1.91s/it]
Outer training loop:  95%|#########5| 47654/50000 [31:50:47<1:16:43,  1.96s/it]
Outer training loop:  95%|#########5| 47655/50000 [31:50:49<1:15:23,  1.93s/it]
Outer training loop:  95%|#########5| 47656/50000 [31:50:51<1:14:26,  1.91s/it]
Outer training loop:  95%|#########5| 47657/50000 [31:50:53<1:15:18,  1.93s/it]
Outer training loop:  95%|#########5| 47658/50000 [31:50:54<1:14:03,  1.90s/it]
Outer training loop:  95%|#########5| 47659/50000 [31:50:56<1:13:51,  1.89s/it]
Outer training loop:  95%|#########5| 47660/50000 [31:50:58<1:14:46,  1.92s/it]
Outer training loop:  95%|#########5| 47661/50000 [31:51:00<1:16:17,  1.96s/it]
Outer training loop:  95%|#########5| 47662/50000 [31:51:02<1:14:18,  1.91s/it]
Outer training loop:  95%|#########5| 47663/50000 [31:51:04<1:13:52,  1.90s/it]
Outer training loop:  95%|#########5| 47664/50000 [31:51:06<1:15:48,  1.95s/it]
Outer training loop:  95%|#########5| 47665/50000 [31:51:08<1:14:48,  1.92s/it]
Outer training loop:  95%|#########5| 47666/50000 [31:51:10<1:14:46,  1.92s/it]
Outer training loop:  95%|#########5| 47667/50000 [31:51:12<1:16:06,  1.96s/it]
Outer training loop:  95%|#########5| 47668/50000 [31:51:14<1:18:10,  2.01s/it]
Outer training loop:  95%|#########5| 47669/50000 [31:51:16<1:15:09,  1.93s/it]
Outer training loop:  95%|#########5| 47670/50000 [31:51:18<1:12:48,  1.87s/it]
Outer training loop:  95%|#########5| 47671/50000 [31:51:20<1:14:32,  1.92s/it]
Outer training loop:  95%|#########5| 47672/50000 [31:51:22<1:14:36,  1.92s/it]
Outer training loop:  95%|#########5| 47673/50000 [31:51:23<1:12:49,  1.88s/it]
Outer training loop:  95%|#########5| 47674/50000 [31:51:25<1:15:36,  1.95s/it]
Outer training loop:  95%|#########5| 47675/50000 [31:51:27<1:16:13,  1.97s/it]
Outer training loop:  95%|#########5| 47676/50000 [31:51:29<1:14:52,  1.93s/it]
Outer training loop:  95%|#########5| 47677/50000 [31:51:31<1:14:51,  1.93s/it]
Outer training loop:  95%|#########5| 47678/50000 [31:51:33<1:16:13,  1.97s/it]
Outer training loop:  95%|#########5| 47679/50000 [31:51:35<1:15:47,  1.96s/it]
Outer training loop:  95%|#########5| 47680/50000 [31:51:37<1:13:07,  1.89s/it]
Outer training loop:  95%|#########5| 47681/50000 [31:51:39<1:15:57,  1.97s/it]
Outer training loop:  95%|#########5| 47682/50000 [31:51:41<1:15:28,  1.95s/it]
Outer training loop:  95%|#########5| 47683/50000 [31:51:43<1:13:06,  1.89s/it]
Outer training loop:  95%|#########5| 47684/50000 [31:51:44<1:11:24,  1.85s/it]
Outer training loop:  95%|#########5| 47685/50000 [31:51:47<1:14:53,  1.94s/it]
Outer training loop:  95%|#########5| 47686/50000 [31:51:49<1:13:57,  1.92s/it]
Outer training loop:  95%|#########5| 47687/50000 [31:51:50<1:14:31,  1.93s/it]
Outer training loop:  95%|#########5| 47688/50000 [31:51:53<1:15:59,  1.97s/it]
Outer training loop:  95%|#########5| 47689/50000 [31:51:54<1:15:11,  1.95s/it]
Outer training loop:  95%|#########5| 47690/50000 [31:51:56<1:14:30,  1.94s/it]
Outer training loop:  95%|#########5| 47691/50000 [31:51:58<1:16:32,  1.99s/it]
Outer training loop:  95%|#########5| 47692/50000 [31:52:00<1:15:13,  1.96s/it]
Outer training loop:  95%|#########5| 47693/50000 [31:52:02<1:12:48,  1.89s/it]
Outer training loop:  95%|#########5| 47694/50000 [31:52:04<1:16:01,  1.98s/it]
Outer training loop:  95%|#########5| 47695/50000 [31:52:06<1:13:49,  1.92s/it]
Outer training loop:  95%|#########5| 47696/50000 [31:52:08<1:13:37,  1.92s/it]
Outer training loop:  95%|#########5| 47697/50000 [31:52:10<1:12:35,  1.89s/it]
Outer training loop:  95%|#########5| 47698/50000 [31:52:12<1:15:44,  1.97s/it]
Outer training loop:  95%|#########5| 47699/50000 [31:52:14<1:14:28,  1.94s/it]
Outer training loop:  95%|#########5| 47700/50000 [31:52:16<1:16:01,  1.98s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.86s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.81s/it]

Outer training loop:  95%|#########5| 47701/50000 [31:52:23<2:19:20,  3.64s/it]
Outer training loop:  95%|#########5| 47702/50000 [31:52:25<2:01:06,  3.16s/it]
Outer training loop:  95%|#########5| 47703/50000 [31:52:28<1:48:30,  2.83s/it]
Outer training loop:  95%|#########5| 47704/50000 [31:52:29<1:37:43,  2.55s/it]
Outer training loop:  95%|#########5| 47705/50000 [31:52:31<1:30:58,  2.38s/it]
Outer training loop:  95%|#########5| 47706/50000 [31:52:33<1:24:45,  2.22s/it]
Outer training loop:  95%|#########5| 47707/50000 [31:52:35<1:21:00,  2.12s/it]
Outer training loop:  95%|#########5| 47708/50000 [31:52:37<1:18:49,  2.06s/it]
Outer training loop:  95%|#########5| 47709/50000 [31:52:39<1:19:48,  2.09s/it]
Outer training loop:  95%|#########5| 47710/50000 [31:52:41<1:18:19,  2.05s/it]
Outer training loop:  95%|#########5| 47711/50000 [31:52:43<1:18:34,  2.06s/it]
Outer training loop:  95%|#########5| 47712/50000 [31:52:45<1:19:05,  2.07s/it]
Outer training loop:  95%|#########5| 47713/50000 [31:52:47<1:15:37,  1.98s/it]
Outer training loop:  95%|#########5| 47714/50000 [31:52:49<1:14:59,  1.97s/it]
Outer training loop:  95%|#########5| 47715/50000 [31:52:51<1:14:36,  1.96s/it]
Outer training loop:  95%|#########5| 47716/50000 [31:52:53<1:14:39,  1.96s/it]
Outer training loop:  95%|#########5| 47717/50000 [31:52:55<1:12:22,  1.90s/it]
Outer training loop:  95%|#########5| 47718/50000 [31:52:57<1:11:41,  1.88s/it]
Outer training loop:  95%|#########5| 47719/50000 [31:52:59<1:13:16,  1.93s/it]
Outer training loop:  95%|#########5| 47720/50000 [31:53:01<1:14:53,  1.97s/it]
Outer training loop:  95%|#########5| 47721/50000 [31:53:02<1:12:42,  1.91s/it]
Outer training loop:  95%|#########5| 47722/50000 [31:53:05<1:16:19,  2.01s/it]
Outer training loop:  95%|#########5| 47723/50000 [31:53:07<1:15:11,  1.98s/it]
Outer training loop:  95%|#########5| 47724/50000 [31:53:08<1:13:37,  1.94s/it]
Outer training loop:  95%|#########5| 47725/50000 [31:53:11<1:15:06,  1.98s/it]
Outer training loop:  95%|#########5| 47726/50000 [31:53:13<1:17:18,  2.04s/it]
Outer training loop:  95%|#########5| 47727/50000 [31:53:15<1:17:07,  2.04s/it]
Outer training loop:  95%|#########5| 47728/50000 [31:53:17<1:14:42,  1.97s/it]
Outer training loop:  95%|#########5| 47729/50000 [31:53:19<1:16:53,  2.03s/it]
Outer training loop:  95%|#########5| 47730/50000 [31:53:21<1:16:29,  2.02s/it]
Outer training loop:  95%|#########5| 47731/50000 [31:53:23<1:15:19,  1.99s/it]
Outer training loop:  95%|#########5| 47732/50000 [31:53:25<1:18:16,  2.07s/it]
Outer training loop:  95%|#########5| 47733/50000 [31:53:27<1:15:14,  1.99s/it]
Outer training loop:  95%|#########5| 47734/50000 [31:53:29<1:14:33,  1.97s/it]
Outer training loop:  95%|#########5| 47735/50000 [31:53:31<1:16:28,  2.03s/it]
Outer training loop:  95%|#########5| 47736/50000 [31:53:33<1:14:21,  1.97s/it]
Outer training loop:  95%|#########5| 47737/50000 [31:53:34<1:12:13,  1.92s/it]
Outer training loop:  95%|#########5| 47738/50000 [31:53:36<1:11:48,  1.90s/it]
Outer training loop:  95%|#########5| 47739/50000 [31:53:38<1:13:38,  1.95s/it]
Outer training loop:  95%|#########5| 47740/50000 [31:53:40<1:12:23,  1.92s/it]
Outer training loop:  95%|#########5| 47741/50000 [31:53:42<1:12:47,  1.93s/it]
Outer training loop:  95%|#########5| 47742/50000 [31:53:44<1:14:08,  1.97s/it]
Outer training loop:  95%|#########5| 47743/50000 [31:53:46<1:13:19,  1.95s/it]
Outer training loop:  95%|#########5| 47744/50000 [31:53:48<1:11:53,  1.91s/it]
Outer training loop:  95%|#########5| 47745/50000 [31:53:50<1:09:50,  1.86s/it]
Outer training loop:  95%|#########5| 47746/50000 [31:53:52<1:13:28,  1.96s/it]
Outer training loop:  95%|#########5| 47747/50000 [31:53:54<1:14:21,  1.98s/it]
Outer training loop:  95%|#########5| 47748/50000 [31:53:56<1:11:30,  1.91s/it]
Outer training loop:  95%|#########5| 47749/50000 [31:53:58<1:12:32,  1.93s/it]
Outer training loop:  96%|#########5| 47750/50000 [31:53:59<1:11:29,  1.91s/it]
Outer training loop:  96%|#########5| 47751/50000 [31:54:01<1:11:13,  1.90s/it]
Outer training loop:  96%|#########5| 47752/50000 [31:54:03<1:12:00,  1.92s/it]
Outer training loop:  96%|#########5| 47753/50000 [31:54:06<1:15:20,  2.01s/it]
Outer training loop:  96%|#########5| 47754/50000 [31:54:08<1:15:25,  2.02s/it]
Outer training loop:  96%|#########5| 47755/50000 [31:54:10<1:16:19,  2.04s/it]
Outer training loop:  96%|#########5| 47756/50000 [31:54:12<1:16:41,  2.05s/it]
Outer training loop:  96%|#########5| 47757/50000 [31:54:14<1:13:53,  1.98s/it]
Outer training loop:  96%|#########5| 47758/50000 [31:54:15<1:11:13,  1.91s/it]
Outer training loop:  96%|#########5| 47759/50000 [31:54:17<1:10:27,  1.89s/it]
Outer training loop:  96%|#########5| 47760/50000 [31:54:19<1:13:28,  1.97s/it]
Outer training loop:  96%|#########5| 47761/50000 [31:54:21<1:14:01,  1.98s/it]
Outer training loop:  96%|#########5| 47762/50000 [31:54:23<1:13:39,  1.97s/it]
Outer training loop:  96%|#########5| 47763/50000 [31:54:25<1:14:36,  2.00s/it]
Outer training loop:  96%|#########5| 47764/50000 [31:54:27<1:13:30,  1.97s/it]
Outer training loop:  96%|#########5| 47765/50000 [31:54:29<1:13:34,  1.98s/it]
Outer training loop:  96%|#########5| 47766/50000 [31:54:31<1:13:29,  1.97s/it]
Outer training loop:  96%|#########5| 47767/50000 [31:54:33<1:12:22,  1.94s/it]
Outer training loop:  96%|#########5| 47768/50000 [31:54:35<1:12:54,  1.96s/it]
Outer training loop:  96%|#########5| 47769/50000 [31:54:37<1:12:40,  1.95s/it]
Outer training loop:  96%|#########5| 47770/50000 [31:54:39<1:13:48,  1.99s/it]
Outer training loop:  96%|#########5| 47771/50000 [31:54:41<1:13:31,  1.98s/it]
Outer training loop:  96%|#########5| 47772/50000 [31:54:43<1:13:17,  1.97s/it]
Outer training loop:  96%|#########5| 47773/50000 [31:54:45<1:13:02,  1.97s/it]
Outer training loop:  96%|#########5| 47774/50000 [31:54:47<1:10:39,  1.90s/it]
Outer training loop:  96%|#########5| 47775/50000 [31:54:49<1:10:43,  1.91s/it]
Outer training loop:  96%|#########5| 47776/50000 [31:54:51<1:12:22,  1.95s/it]
Outer training loop:  96%|#########5| 47777/50000 [31:54:53<1:11:48,  1.94s/it]
Outer training loop:  96%|#########5| 47778/50000 [31:54:54<1:11:30,  1.93s/it]
Outer training loop:  96%|#########5| 47779/50000 [31:54:56<1:10:26,  1.90s/it]
Outer training loop:  96%|#########5| 47780/50000 [31:54:58<1:12:33,  1.96s/it]
Outer training loop:  96%|#########5| 47781/50000 [31:55:00<1:10:32,  1.91s/it]
Outer training loop:  96%|#########5| 47782/50000 [31:55:02<1:10:09,  1.90s/it]
Outer training loop:  96%|#########5| 47783/50000 [31:55:04<1:12:15,  1.96s/it]
Outer training loop:  96%|#########5| 47784/50000 [31:55:06<1:12:43,  1.97s/it]
Outer training loop:  96%|#########5| 47785/50000 [31:55:08<1:12:25,  1.96s/it]
Outer training loop:  96%|#########5| 47786/50000 [31:55:10<1:10:55,  1.92s/it]
Outer training loop:  96%|#########5| 47787/50000 [31:55:12<1:13:08,  1.98s/it]
Outer training loop:  96%|#########5| 47788/50000 [31:55:14<1:11:55,  1.95s/it]
Outer training loop:  96%|#########5| 47789/50000 [31:55:16<1:10:38,  1.92s/it]
Outer training loop:  96%|#########5| 47790/50000 [31:55:18<1:11:51,  1.95s/it]
Outer training loop:  96%|#########5| 47791/50000 [31:55:20<1:10:26,  1.91s/it]
Outer training loop:  96%|#########5| 47792/50000 [31:55:22<1:10:39,  1.92s/it]
Outer training loop:  96%|#########5| 47793/50000 [31:55:23<1:09:42,  1.90s/it]
Outer training loop:  96%|#########5| 47794/50000 [31:55:25<1:11:10,  1.94s/it]
Outer training loop:  96%|#########5| 47795/50000 [31:55:27<1:11:11,  1.94s/it]
Outer training loop:  96%|#########5| 47796/50000 [31:55:29<1:08:38,  1.87s/it]
Outer training loop:  96%|#########5| 47797/50000 [31:55:31<1:12:40,  1.98s/it]
Outer training loop:  96%|#########5| 47798/50000 [31:55:33<1:10:11,  1.91s/it]
Outer training loop:  96%|#########5| 47799/50000 [31:55:35<1:09:09,  1.89s/it]
Outer training loop:  96%|#########5| 47800/50000 [31:55:37<1:07:32,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.79s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.76s/it]

Outer training loop:  96%|#########5| 47801/50000 [31:55:44<2:08:10,  3.50s/it]
Outer training loop:  96%|#########5| 47802/50000 [31:55:46<1:51:14,  3.04s/it]
Outer training loop:  96%|#########5| 47803/50000 [31:55:48<1:38:55,  2.70s/it]
Outer training loop:  96%|#########5| 47804/50000 [31:55:50<1:29:07,  2.44s/it]
Outer training loop:  96%|#########5| 47805/50000 [31:55:52<1:26:42,  2.37s/it]
Outer training loop:  96%|#########5| 47806/50000 [31:55:54<1:20:27,  2.20s/it]
Outer training loop:  96%|#########5| 47807/50000 [31:55:56<1:16:21,  2.09s/it]
Outer training loop:  96%|#########5| 47808/50000 [31:55:58<1:16:03,  2.08s/it]
Outer training loop:  96%|#########5| 47809/50000 [31:55:59<1:13:18,  2.01s/it]
Outer training loop:  96%|#########5| 47810/50000 [31:56:01<1:11:37,  1.96s/it]
Outer training loop:  96%|#########5| 47811/50000 [31:56:03<1:10:24,  1.93s/it]
Outer training loop:  96%|#########5| 47812/50000 [31:56:05<1:12:56,  2.00s/it]
Outer training loop:  96%|#########5| 47813/50000 [31:56:07<1:11:14,  1.95s/it]
Outer training loop:  96%|#########5| 47814/50000 [31:56:09<1:09:28,  1.91s/it]
Outer training loop:  96%|#########5| 47815/50000 [31:56:11<1:11:16,  1.96s/it]
Outer training loop:  96%|#########5| 47816/50000 [31:56:13<1:10:33,  1.94s/it]
Outer training loop:  96%|#########5| 47817/50000 [31:56:15<1:09:21,  1.91s/it]
Outer training loop:  96%|#########5| 47818/50000 [31:56:17<1:07:41,  1.86s/it]
Outer training loop:  96%|#########5| 47819/50000 [31:56:19<1:10:23,  1.94s/it]
Outer training loop:  96%|#########5| 47820/50000 [31:56:21<1:11:02,  1.96s/it]
Outer training loop:  96%|#########5| 47821/50000 [31:56:22<1:09:09,  1.90s/it]
Outer training loop:  96%|#########5| 47822/50000 [31:56:25<1:11:30,  1.97s/it]
Outer training loop:  96%|#########5| 47823/50000 [31:56:26<1:10:48,  1.95s/it]
Outer training loop:  96%|#########5| 47824/50000 [31:56:28<1:11:18,  1.97s/it]
Outer training loop:  96%|#########5| 47825/50000 [31:56:30<1:09:22,  1.91s/it]
Outer training loop:  96%|#########5| 47826/50000 [31:56:32<1:10:03,  1.93s/it]
Outer training loop:  96%|#########5| 47827/50000 [31:56:34<1:09:11,  1.91s/it]
Outer training loop:  96%|#########5| 47828/50000 [31:56:36<1:08:29,  1.89s/it]
Outer training loop:  96%|#########5| 47829/50000 [31:56:38<1:10:59,  1.96s/it]
Outer training loop:  96%|#########5| 47830/50000 [31:56:40<1:11:43,  1.98s/it]
Outer training loop:  96%|#########5| 47831/50000 [31:56:42<1:11:25,  1.98s/it]
Outer training loop:  96%|#########5| 47832/50000 [31:56:44<1:10:04,  1.94s/it]
Outer training loop:  96%|#########5| 47833/50000 [31:56:46<1:10:27,  1.95s/it]
Outer training loop:  96%|#########5| 47834/50000 [31:56:48<1:09:31,  1.93s/it]
Outer training loop:  96%|#########5| 47835/50000 [31:56:50<1:09:10,  1.92s/it]
Outer training loop:  96%|#########5| 47836/50000 [31:56:52<1:09:11,  1.92s/it]
Outer training loop:  96%|#########5| 47837/50000 [31:56:54<1:09:17,  1.92s/it]
Outer training loop:  96%|#########5| 47838/50000 [31:56:55<1:08:55,  1.91s/it]
Outer training loop:  96%|#########5| 47839/50000 [31:56:57<1:07:53,  1.89s/it]
Outer training loop:  96%|#########5| 47840/50000 [31:56:59<1:10:27,  1.96s/it]
Outer training loop:  96%|#########5| 47841/50000 [31:57:01<1:10:22,  1.96s/it]
Outer training loop:  96%|#########5| 47842/50000 [31:57:03<1:09:47,  1.94s/it]
Outer training loop:  96%|#########5| 47843/50000 [31:57:05<1:10:17,  1.96s/it]
Outer training loop:  96%|#########5| 47844/50000 [31:57:07<1:10:41,  1.97s/it]
Outer training loop:  96%|#########5| 47845/50000 [31:57:09<1:08:47,  1.92s/it]
Outer training loop:  96%|#########5| 47846/50000 [31:57:11<1:09:59,  1.95s/it]
Outer training loop:  96%|#########5| 47847/50000 [31:57:13<1:08:45,  1.92s/it]
Outer training loop:  96%|#########5| 47848/50000 [31:57:15<1:07:25,  1.88s/it]
Outer training loop:  96%|#########5| 47849/50000 [31:57:17<1:07:25,  1.88s/it]
Outer training loop:  96%|#########5| 47850/50000 [31:57:19<1:08:45,  1.92s/it]
Outer training loop:  96%|#########5| 47851/50000 [31:57:21<1:09:50,  1.95s/it]
Outer training loop:  96%|#########5| 47852/50000 [31:57:22<1:08:16,  1.91s/it]
Outer training loop:  96%|#########5| 47853/50000 [31:57:24<1:09:53,  1.95s/it]
Outer training loop:  96%|#########5| 47854/50000 [31:57:26<1:08:54,  1.93s/it]
Outer training loop:  96%|#########5| 47855/50000 [31:57:28<1:10:03,  1.96s/it]
Outer training loop:  96%|#########5| 47856/50000 [31:57:31<1:12:20,  2.02s/it]
Outer training loop:  96%|#########5| 47857/50000 [31:57:32<1:09:50,  1.96s/it]
Outer training loop:  96%|#########5| 47858/50000 [31:57:34<1:10:17,  1.97s/it]
Outer training loop:  96%|#########5| 47859/50000 [31:57:36<1:09:35,  1.95s/it]
Outer training loop:  96%|#########5| 47860/50000 [31:57:38<1:11:40,  2.01s/it]
Outer training loop:  96%|#########5| 47861/50000 [31:57:40<1:10:30,  1.98s/it]
Outer training loop:  96%|#########5| 47862/50000 [31:57:42<1:09:14,  1.94s/it]
Outer training loop:  96%|#########5| 47863/50000 [31:57:44<1:08:44,  1.93s/it]
Outer training loop:  96%|#########5| 47864/50000 [31:57:46<1:10:37,  1.98s/it]
Outer training loop:  96%|#########5| 47865/50000 [31:57:48<1:08:55,  1.94s/it]
Outer training loop:  96%|#########5| 47866/50000 [31:57:50<1:08:20,  1.92s/it]
Outer training loop:  96%|#########5| 47867/50000 [31:57:52<1:09:53,  1.97s/it]
Outer training loop:  96%|#########5| 47868/50000 [31:57:54<1:08:36,  1.93s/it]
Outer training loop:  96%|#########5| 47869/50000 [31:57:56<1:07:57,  1.91s/it]
Outer training loop:  96%|#########5| 47870/50000 [31:57:58<1:07:54,  1.91s/it]
Outer training loop:  96%|#########5| 47871/50000 [31:58:00<1:10:06,  1.98s/it]
Outer training loop:  96%|#########5| 47872/50000 [31:58:01<1:08:26,  1.93s/it]
Outer training loop:  96%|#########5| 47873/50000 [31:58:03<1:07:15,  1.90s/it]
Outer training loop:  96%|#########5| 47874/50000 [31:58:05<1:09:48,  1.97s/it]
Outer training loop:  96%|#########5| 47875/50000 [31:58:07<1:07:20,  1.90s/it]
Outer training loop:  96%|#########5| 47876/50000 [31:58:09<1:05:44,  1.86s/it]
Outer training loop:  96%|#########5| 47877/50000 [31:58:11<1:07:38,  1.91s/it]
Outer training loop:  96%|#########5| 47878/50000 [31:58:13<1:07:55,  1.92s/it]
Outer training loop:  96%|#########5| 47879/50000 [31:58:15<1:06:37,  1.88s/it]
Outer training loop:  96%|#########5| 47880/50000 [31:58:17<1:07:12,  1.90s/it]
Outer training loop:  96%|#########5| 47881/50000 [31:58:18<1:06:06,  1.87s/it]
Outer training loop:  96%|#########5| 47882/50000 [31:58:20<1:06:01,  1.87s/it]
Outer training loop:  96%|#########5| 47883/50000 [31:58:22<1:07:26,  1.91s/it]
Outer training loop:  96%|#########5| 47884/50000 [31:58:24<1:06:26,  1.88s/it]
Outer training loop:  96%|#########5| 47885/50000 [31:58:26<1:06:51,  1.90s/it]
Outer training loop:  96%|#########5| 47886/50000 [31:58:28<1:08:33,  1.95s/it]
Outer training loop:  96%|#########5| 47887/50000 [31:58:30<1:07:44,  1.92s/it]
Outer training loop:  96%|#########5| 47888/50000 [31:58:32<1:06:28,  1.89s/it]
Outer training loop:  96%|#########5| 47889/50000 [31:58:34<1:05:59,  1.88s/it]
Outer training loop:  96%|#########5| 47890/50000 [31:58:36<1:07:26,  1.92s/it]
Outer training loop:  96%|#########5| 47891/50000 [31:58:38<1:07:36,  1.92s/it]
Outer training loop:  96%|#########5| 47892/50000 [31:58:39<1:06:31,  1.89s/it]
Outer training loop:  96%|#########5| 47893/50000 [31:58:41<1:07:38,  1.93s/it]
Outer training loop:  96%|#########5| 47894/50000 [31:58:44<1:10:00,  1.99s/it]
Outer training loop:  96%|#########5| 47895/50000 [31:58:45<1:08:36,  1.96s/it]
Outer training loop:  96%|#########5| 47896/50000 [31:58:47<1:07:03,  1.91s/it]
Outer training loop:  96%|#########5| 47897/50000 [31:58:49<1:09:07,  1.97s/it]
Outer training loop:  96%|#########5| 47898/50000 [31:58:51<1:09:14,  1.98s/it]
Outer training loop:  96%|#########5| 47899/50000 [31:58:53<1:07:30,  1.93s/it]
Outer training loop:  96%|#########5| 47900/50000 [31:58:55<1:10:20,  2.01s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  96%|#########5| 47901/50000 [31:59:03<2:06:25,  3.61s/it]
Outer training loop:  96%|#########5| 47902/50000 [31:59:05<1:47:25,  3.07s/it]
Outer training loop:  96%|#########5| 47903/50000 [31:59:06<1:34:15,  2.70s/it]
Outer training loop:  96%|#########5| 47904/50000 [31:59:08<1:27:15,  2.50s/it]
Outer training loop:  96%|#########5| 47905/50000 [31:59:10<1:21:39,  2.34s/it]
Outer training loop:  96%|#########5| 47906/50000 [31:59:12<1:15:28,  2.16s/it]
Outer training loop:  96%|#########5| 47907/50000 [31:59:14<1:11:11,  2.04s/it]
Outer training loop:  96%|#########5| 47908/50000 [31:59:16<1:10:51,  2.03s/it]
Outer training loop:  96%|#########5| 47909/50000 [31:59:18<1:09:23,  1.99s/it]
Outer training loop:  96%|#########5| 47910/50000 [31:59:20<1:08:07,  1.96s/it]
Outer training loop:  96%|#########5| 47911/50000 [31:59:22<1:07:31,  1.94s/it]
Outer training loop:  96%|#########5| 47912/50000 [31:59:23<1:06:40,  1.92s/it]
Outer training loop:  96%|#########5| 47913/50000 [31:59:25<1:05:45,  1.89s/it]
Outer training loop:  96%|#########5| 47914/50000 [31:59:27<1:08:01,  1.96s/it]
Outer training loop:  96%|#########5| 47915/50000 [31:59:29<1:06:50,  1.92s/it]
Outer training loop:  96%|#########5| 47916/50000 [31:59:31<1:05:32,  1.89s/it]
Outer training loop:  96%|#########5| 47917/50000 [31:59:33<1:07:36,  1.95s/it]
Outer training loop:  96%|#########5| 47918/50000 [31:59:35<1:07:20,  1.94s/it]
Outer training loop:  96%|#########5| 47919/50000 [31:59:37<1:04:34,  1.86s/it]
Outer training loop:  96%|#########5| 47920/50000 [31:59:38<1:03:17,  1.83s/it]
Outer training loop:  96%|#########5| 47921/50000 [31:59:41<1:06:35,  1.92s/it]
Outer training loop:  96%|#########5| 47922/50000 [31:59:42<1:05:33,  1.89s/it]
Outer training loop:  96%|#########5| 47923/50000 [31:59:44<1:04:20,  1.86s/it]
Outer training loop:  96%|#########5| 47924/50000 [31:59:46<1:05:55,  1.91s/it]
Outer training loop:  96%|#########5| 47925/50000 [31:59:48<1:03:54,  1.85s/it]
Outer training loop:  96%|#########5| 47926/50000 [31:59:50<1:04:43,  1.87s/it]
Outer training loop:  96%|#########5| 47927/50000 [31:59:52<1:06:28,  1.92s/it]
Outer training loop:  96%|#########5| 47928/50000 [31:59:54<1:05:51,  1.91s/it]
Outer training loop:  96%|#########5| 47929/50000 [31:59:56<1:04:18,  1.86s/it]
Outer training loop:  96%|#########5| 47930/50000 [31:59:57<1:03:18,  1.83s/it]
Outer training loop:  96%|#########5| 47931/50000 [31:59:59<1:04:13,  1.86s/it]
Outer training loop:  96%|#########5| 47932/50000 [32:00:01<1:02:56,  1.83s/it]
Outer training loop:  96%|#########5| 47933/50000 [32:00:03<1:01:49,  1.79s/it]
Outer training loop:  96%|#########5| 47934/50000 [32:00:05<1:04:16,  1.87s/it]
Outer training loop:  96%|#########5| 47935/50000 [32:00:07<1:04:15,  1.87s/it]
Outer training loop:  96%|#########5| 47936/50000 [32:00:08<1:04:02,  1.86s/it]
Outer training loop:  96%|#########5| 47937/50000 [32:00:10<1:04:08,  1.87s/it]
Outer training loop:  96%|#########5| 47938/50000 [32:00:12<1:05:13,  1.90s/it]
Outer training loop:  96%|#########5| 47939/50000 [32:00:14<1:03:32,  1.85s/it]
Outer training loop:  96%|#########5| 47940/50000 [32:00:16<1:03:28,  1.85s/it]
Outer training loop:  96%|#########5| 47941/50000 [32:00:18<1:05:23,  1.91s/it]
Outer training loop:  96%|#########5| 47942/50000 [32:00:20<1:07:18,  1.96s/it]
Outer training loop:  96%|#########5| 47943/50000 [32:00:22<1:05:19,  1.91s/it]
Outer training loop:  96%|#########5| 47944/50000 [32:00:24<1:04:04,  1.87s/it]
Outer training loop:  96%|#########5| 47945/50000 [32:00:26<1:06:26,  1.94s/it]
Outer training loop:  96%|#########5| 47946/50000 [32:00:28<1:07:00,  1.96s/it]
Outer training loop:  96%|#########5| 47947/50000 [32:00:30<1:05:40,  1.92s/it]
Outer training loop:  96%|#########5| 47948/50000 [32:00:31<1:05:11,  1.91s/it]
Outer training loop:  96%|#########5| 47949/50000 [32:00:33<1:06:36,  1.95s/it]
Outer training loop:  96%|#########5| 47950/50000 [32:00:35<1:06:53,  1.96s/it]
Outer training loop:  96%|#########5| 47951/50000 [32:00:37<1:05:31,  1.92s/it]
Outer training loop:  96%|#########5| 47952/50000 [32:00:39<1:07:27,  1.98s/it]
Outer training loop:  96%|#########5| 47953/50000 [32:00:41<1:07:46,  1.99s/it]
Outer training loop:  96%|#########5| 47954/50000 [32:00:43<1:06:48,  1.96s/it]
Outer training loop:  96%|#########5| 47955/50000 [32:00:45<1:06:27,  1.95s/it]
Outer training loop:  96%|#########5| 47956/50000 [32:00:47<1:05:23,  1.92s/it]
Outer training loop:  96%|#########5| 47957/50000 [32:00:49<1:05:23,  1.92s/it]
Outer training loop:  96%|#########5| 47958/50000 [32:00:51<1:05:06,  1.91s/it]
Outer training loop:  96%|#########5| 47959/50000 [32:00:53<1:06:34,  1.96s/it]
Outer training loop:  96%|#########5| 47960/50000 [32:00:55<1:05:02,  1.91s/it]
Outer training loop:  96%|#########5| 47961/50000 [32:00:57<1:04:35,  1.90s/it]
Outer training loop:  96%|#########5| 47962/50000 [32:00:58<1:03:21,  1.87s/it]
Outer training loop:  96%|#########5| 47963/50000 [32:01:00<1:04:26,  1.90s/it]
Outer training loop:  96%|#########5| 47964/50000 [32:01:02<1:04:02,  1.89s/it]
Outer training loop:  96%|#########5| 47965/50000 [32:01:04<1:02:57,  1.86s/it]
Outer training loop:  96%|#########5| 47966/50000 [32:01:06<1:03:54,  1.89s/it]
Outer training loop:  96%|#########5| 47967/50000 [32:01:08<1:03:13,  1.87s/it]
Outer training loop:  96%|#########5| 47968/50000 [32:01:10<1:02:03,  1.83s/it]
Outer training loop:  96%|#########5| 47969/50000 [32:01:11<1:02:40,  1.85s/it]
Outer training loop:  96%|#########5| 47970/50000 [32:01:13<1:04:05,  1.89s/it]
Outer training loop:  96%|#########5| 47971/50000 [32:01:15<1:03:22,  1.87s/it]
Outer training loop:  96%|#########5| 47972/50000 [32:01:17<1:02:32,  1.85s/it]
Outer training loop:  96%|#########5| 47973/50000 [32:01:19<1:04:32,  1.91s/it]
Outer training loop:  96%|#########5| 47974/50000 [32:01:21<1:04:34,  1.91s/it]
Outer training loop:  96%|#########5| 47975/50000 [32:01:23<1:03:11,  1.87s/it]
Outer training loop:  96%|#########5| 47976/50000 [32:01:25<1:06:09,  1.96s/it]
Outer training loop:  96%|#########5| 47977/50000 [32:01:27<1:04:10,  1.90s/it]
Outer training loop:  96%|#########5| 47978/50000 [32:01:29<1:03:22,  1.88s/it]
Outer training loop:  96%|#########5| 47979/50000 [32:01:30<1:02:56,  1.87s/it]
Outer training loop:  96%|#########5| 47980/50000 [32:01:32<1:04:40,  1.92s/it]
Outer training loop:  96%|#########5| 47981/50000 [32:01:34<1:04:25,  1.91s/it]
Outer training loop:  96%|#########5| 47982/50000 [32:01:36<1:03:18,  1.88s/it]
Outer training loop:  96%|#########5| 47983/50000 [32:01:38<1:03:37,  1.89s/it]
Outer training loop:  96%|#########5| 47984/50000 [32:01:40<1:03:39,  1.89s/it]
Outer training loop:  96%|#########5| 47985/50000 [32:01:42<1:03:18,  1.89s/it]
Outer training loop:  96%|#########5| 47986/50000 [32:01:44<1:04:53,  1.93s/it]
Outer training loop:  96%|#########5| 47987/50000 [32:01:46<1:03:52,  1.90s/it]
Outer training loop:  96%|#########5| 47988/50000 [32:01:48<1:03:13,  1.89s/it]
Outer training loop:  96%|#########5| 47989/50000 [32:01:49<1:02:51,  1.88s/it]
Outer training loop:  96%|#########5| 47990/50000 [32:01:52<1:05:11,  1.95s/it]
Outer training loop:  96%|#########5| 47991/50000 [32:01:54<1:06:22,  1.98s/it]
Outer training loop:  96%|#########5| 47992/50000 [32:01:55<1:05:37,  1.96s/it]
Outer training loop:  96%|#########5| 47993/50000 [32:01:58<1:07:10,  2.01s/it]
Outer training loop:  96%|#########5| 47994/50000 [32:01:59<1:05:23,  1.96s/it]
Outer training loop:  96%|#########5| 47995/50000 [32:02:01<1:04:09,  1.92s/it]
Outer training loop:  96%|#########5| 47996/50000 [32:02:03<1:06:00,  1.98s/it]
Outer training loop:  96%|#########5| 47997/50000 [32:02:05<1:04:25,  1.93s/it]
Outer training loop:  96%|#########5| 47998/50000 [32:02:07<1:02:34,  1.88s/it]
Outer training loop:  96%|#########5| 47999/50000 [32:02:09<1:02:18,  1.87s/it]
Outer training loop:  96%|#########6| 48000/50000 [32:02:11<1:03:43,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.85s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:31,  1.84s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:33,  1.90s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:29,  1.87s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:28,  1.88s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.89s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:29,  1.94s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:21,  1.94s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:19,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:16,  1.92s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:13,  1.93s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:01,  1.82s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:01,  1.84s/it]

Evaluating policy:  22%|##1       | 18/83 [00:33<01:57,  1.81s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:56,  1.86s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:54,  1.84s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:51,  1.83s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.85s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:53,  1.92s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:50,  1.91s/it]

Evaluating policy:  31%|###1      | 26/83 [00:48<01:46,  1.86s/it]

Evaluating policy:  33%|###2      | 27/83 [00:50<01:44,  1.87s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:41,  1.85s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.84s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:32,  1.85s/it]

Evaluating policy:  41%|####      | 34/83 [01:03<01:28,  1.81s/it]

Evaluating policy:  42%|####2     | 35/83 [01:05<01:30,  1.89s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:28,  1.88s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.85s/it]

Evaluating policy:  48%|####8     | 40/83 [01:14<01:18,  1.82s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:21,  1.94s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:15,  1.88s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:13,  1.89s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:12,  1.90s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:08,  1.85s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.84s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:08,  1.96s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:06,  1.94s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:03,  1.94s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:01,  1.93s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:59,  1.92s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:56,  1.90s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:54,  1.87s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:52,  1.88s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:54,  2.01s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:53,  2.05s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:49,  1.99s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:44,  1.91s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.89s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.87s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.98s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:36,  1.94s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:34,  1.91s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.87s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.87s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:28,  1.87s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.85s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.87s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.85s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:22,  2.04s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:19,  1.98s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.95s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.95s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.96s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.92s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.89s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.90s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:04,  2.05s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.89s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:29,  1.82s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:22,  1.78s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.89s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:27,  1.89s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:23,  1.89s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:35,  2.07s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:28,  2.01s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:23,  1.96s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:21,  2.00s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:17,  1.97s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:14,  1.95s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:12,  1.94s/it]

Evaluating policy:  19%|#9        | 16/83 [00:31<02:16,  2.04s/it]

Evaluating policy:  20%|##        | 17/83 [00:33<02:11,  2.00s/it]

Evaluating policy:  22%|##1       | 18/83 [00:35<02:09,  1.99s/it]

Evaluating policy:  23%|##2       | 19/83 [00:37<02:03,  1.93s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:01,  1.93s/it]

Evaluating policy:  25%|##5       | 21/83 [00:41<02:03,  1.99s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:58,  1.94s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:56,  1.94s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:52,  1.90s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:49,  1.89s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:51,  1.96s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:46,  1.90s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:56<01:40,  1.86s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:37,  1.84s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:36,  1.85s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.94s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:31,  1.87s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:29,  1.87s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:31,  1.94s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:27,  1.91s/it]

Evaluating policy:  46%|####5     | 38/83 [01:13<01:29,  1.98s/it]

Evaluating policy:  47%|####6     | 39/83 [01:15<01:24,  1.92s/it]

Evaluating policy:  48%|####8     | 40/83 [01:17<01:21,  1.89s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:18,  1.88s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:16,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:13,  1.85s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:17,  2.00s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:13,  1.94s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:09,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:30<01:07,  1.89s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:32<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:02,  1.84s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:00,  1.82s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:02,  1.94s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<00:59,  1.93s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:57,  1.90s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:55,  1.90s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:47<00:51,  1.89s/it]

Evaluating policy:  69%|######8   | 57/83 [01:49<00:47,  1.84s/it]

Evaluating policy:  70%|######9   | 58/83 [01:51<00:49,  1.97s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:53<00:46,  1.93s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:43,  1.90s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:40,  1.85s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:39,  1.86s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:36,  1.83s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:34,  1.82s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:32,  1.79s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:30,  1.78s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:31,  1.96s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.91s/it]

Evaluating policy:  83%|########3 | 69/83 [02:11<00:26,  1.90s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:24,  1.89s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.87s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.84s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.83s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:16,  1.81s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:14,  1.82s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.99s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.93s/it]

Evaluating policy:  94%|#########3| 78/83 [02:28<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.82s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.82s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.82s/it]

Outer training loop:  96%|#########6| 48001/50000 [32:07:34<54:30:44, 98.17s/it]
Outer training loop:  96%|#########6| 48002/50000 [32:07:36<38:27:33, 69.30s/it]
Outer training loop:  96%|#########6| 48003/50000 [32:07:38<27:17:31, 49.20s/it]
Outer training loop:  96%|#########6| 48004/50000 [32:07:40<19:23:36, 34.98s/it]
Outer training loop:  96%|#########6| 48005/50000 [32:07:41<13:52:13, 25.03s/it]
Outer training loop:  96%|#########6| 48006/50000 [32:07:43<9:59:45, 18.05s/it] 
Outer training loop:  96%|#########6| 48007/50000 [32:07:45<7:19:06, 13.22s/it]
Outer training loop:  96%|#########6| 48008/50000 [32:07:47<5:24:41,  9.78s/it]
Outer training loop:  96%|#########6| 48009/50000 [32:07:49<4:05:43,  7.41s/it]
Outer training loop:  96%|#########6| 48010/50000 [32:07:51<3:13:24,  5.83s/it]
Outer training loop:  96%|#########6| 48011/50000 [32:07:53<2:33:59,  4.65s/it]
Outer training loop:  96%|#########6| 48012/50000 [32:07:55<2:06:54,  3.83s/it]
Outer training loop:  96%|#########6| 48013/50000 [32:07:57<1:48:15,  3.27s/it]
Outer training loop:  96%|#########6| 48014/50000 [32:07:58<1:33:11,  2.82s/it]
Outer training loop:  96%|#########6| 48015/50000 [32:08:00<1:22:36,  2.50s/it]
Outer training loop:  96%|#########6| 48016/50000 [32:08:02<1:15:48,  2.29s/it]
Outer training loop:  96%|#########6| 48017/50000 [32:08:04<1:13:06,  2.21s/it]
Outer training loop:  96%|#########6| 48018/50000 [32:08:06<1:09:30,  2.10s/it]
Outer training loop:  96%|#########6| 48019/50000 [32:08:08<1:06:47,  2.02s/it]
Outer training loop:  96%|#########6| 48020/50000 [32:08:10<1:05:14,  1.98s/it]
Outer training loop:  96%|#########6| 48021/50000 [32:08:12<1:05:41,  1.99s/it]
Outer training loop:  96%|#########6| 48022/50000 [32:08:14<1:05:48,  2.00s/it]
Outer training loop:  96%|#########6| 48023/50000 [32:08:15<1:03:41,  1.93s/it]
Outer training loop:  96%|#########6| 48024/50000 [32:08:17<1:04:21,  1.95s/it]
Outer training loop:  96%|#########6| 48025/50000 [32:08:19<1:02:41,  1.90s/it]
Outer training loop:  96%|#########6| 48026/50000 [32:08:21<1:02:59,  1.91s/it]
Outer training loop:  96%|#########6| 48027/50000 [32:08:23<1:04:46,  1.97s/it]
Outer training loop:  96%|#########6| 48028/50000 [32:08:25<1:03:22,  1.93s/it]
Outer training loop:  96%|#########6| 48029/50000 [32:08:27<1:03:36,  1.94s/it]
Outer training loop:  96%|#########6| 48030/50000 [32:08:29<1:03:58,  1.95s/it]
Outer training loop:  96%|#########6| 48031/50000 [32:08:31<1:04:40,  1.97s/it]
Outer training loop:  96%|#########6| 48032/50000 [32:08:33<1:03:20,  1.93s/it]
Outer training loop:  96%|#########6| 48033/50000 [32:08:35<1:02:44,  1.91s/it]
Outer training loop:  96%|#########6| 48034/50000 [32:08:37<1:04:32,  1.97s/it]
Outer training loop:  96%|#########6| 48035/50000 [32:08:39<1:03:36,  1.94s/it]
Outer training loop:  96%|#########6| 48036/50000 [32:08:41<1:02:34,  1.91s/it]
Outer training loop:  96%|#########6| 48037/50000 [32:08:43<1:03:12,  1.93s/it]
Outer training loop:  96%|#########6| 48038/50000 [32:08:44<1:01:58,  1.90s/it]
Outer training loop:  96%|#########6| 48039/50000 [32:08:46<1:00:31,  1.85s/it]
Outer training loop:  96%|#########6| 48040/50000 [32:08:48<59:41,  1.83s/it]  
Outer training loop:  96%|#########6| 48041/50000 [32:08:50<1:01:57,  1.90s/it]
Outer training loop:  96%|#########6| 48042/50000 [32:08:52<1:00:59,  1.87s/it]
Outer training loop:  96%|#########6| 48043/50000 [32:08:53<59:43,  1.83s/it]  
Outer training loop:  96%|#########6| 48044/50000 [32:08:55<1:00:37,  1.86s/it]
Outer training loop:  96%|#########6| 48045/50000 [32:08:57<1:02:30,  1.92s/it]
Outer training loop:  96%|#########6| 48046/50000 [32:08:59<1:01:19,  1.88s/it]
Outer training loop:  96%|#########6| 48047/50000 [32:09:01<1:02:14,  1.91s/it]
Outer training loop:  96%|#########6| 48048/50000 [32:09:03<1:01:04,  1.88s/it]
Outer training loop:  96%|#########6| 48049/50000 [32:09:05<1:00:12,  1.85s/it]
Outer training loop:  96%|#########6| 48050/50000 [32:09:07<59:31,  1.83s/it]  
Outer training loop:  96%|#########6| 48051/50000 [32:09:09<1:01:51,  1.90s/it]
Outer training loop:  96%|#########6| 48052/50000 [32:09:10<1:01:02,  1.88s/it]
Outer training loop:  96%|#########6| 48053/50000 [32:09:12<1:00:53,  1.88s/it]
Outer training loop:  96%|#########6| 48054/50000 [32:09:14<59:32,  1.84s/it]  
Outer training loop:  96%|#########6| 48055/50000 [32:09:16<1:00:36,  1.87s/it]
Outer training loop:  96%|#########6| 48056/50000 [32:09:18<1:00:09,  1.86s/it]
Outer training loop:  96%|#########6| 48057/50000 [32:09:20<59:59,  1.85s/it]  
Outer training loop:  96%|#########6| 48058/50000 [32:09:21<58:19,  1.80s/it]
Outer training loop:  96%|#########6| 48059/50000 [32:09:23<1:00:26,  1.87s/it]
Outer training loop:  96%|#########6| 48060/50000 [32:09:25<1:00:17,  1.86s/it]
Outer training loop:  96%|#########6| 48061/50000 [32:09:27<59:36,  1.84s/it]  
Outer training loop:  96%|#########6| 48062/50000 [32:09:29<59:00,  1.83s/it]
Outer training loop:  96%|#########6| 48063/50000 [32:09:31<1:01:01,  1.89s/it]
Outer training loop:  96%|#########6| 48064/50000 [32:09:33<59:32,  1.85s/it]  
Outer training loop:  96%|#########6| 48065/50000 [32:09:34<59:22,  1.84s/it]
Outer training loop:  96%|#########6| 48066/50000 [32:09:36<58:19,  1.81s/it]
Outer training loop:  96%|#########6| 48067/50000 [32:09:38<1:01:22,  1.91s/it]
Outer training loop:  96%|#########6| 48068/50000 [32:09:40<1:00:39,  1.88s/it]
Outer training loop:  96%|#########6| 48069/50000 [32:09:42<1:00:05,  1.87s/it]
Outer training loop:  96%|#########6| 48070/50000 [32:09:44<1:00:55,  1.89s/it]
Outer training loop:  96%|#########6| 48071/50000 [32:09:46<1:00:12,  1.87s/it]
Outer training loop:  96%|#########6| 48072/50000 [32:09:48<1:00:39,  1.89s/it]
Outer training loop:  96%|#########6| 48073/50000 [32:09:50<1:00:01,  1.87s/it]
Outer training loop:  96%|#########6| 48074/50000 [32:09:52<1:01:52,  1.93s/it]
Outer training loop:  96%|#########6| 48075/50000 [32:09:53<1:00:58,  1.90s/it]
Outer training loop:  96%|#########6| 48076/50000 [32:09:55<1:00:52,  1.90s/it]
Outer training loop:  96%|#########6| 48077/50000 [32:09:57<1:03:08,  1.97s/it]
Outer training loop:  96%|#########6| 48078/50000 [32:09:59<1:01:16,  1.91s/it]
Outer training loop:  96%|#########6| 48079/50000 [32:10:01<1:00:27,  1.89s/it]
Outer training loop:  96%|#########6| 48080/50000 [32:10:03<1:01:37,  1.93s/it]
Outer training loop:  96%|#########6| 48081/50000 [32:10:05<1:00:24,  1.89s/it]
Outer training loop:  96%|#########6| 48082/50000 [32:10:07<59:02,  1.85s/it]  
Outer training loop:  96%|#########6| 48083/50000 [32:10:09<59:17,  1.86s/it]
Outer training loop:  96%|#########6| 48084/50000 [32:10:11<1:00:44,  1.90s/it]
Outer training loop:  96%|#########6| 48085/50000 [32:10:12<1:00:43,  1.90s/it]
Outer training loop:  96%|#########6| 48086/50000 [32:10:14<58:58,  1.85s/it]  
Outer training loop:  96%|#########6| 48087/50000 [32:10:16<1:01:43,  1.94s/it]
Outer training loop:  96%|#########6| 48088/50000 [32:10:18<1:00:45,  1.91s/it]
Outer training loop:  96%|#########6| 48089/50000 [32:10:20<59:49,  1.88s/it]  
Outer training loop:  96%|#########6| 48090/50000 [32:10:22<59:12,  1.86s/it]
Outer training loop:  96%|#########6| 48091/50000 [32:10:24<1:00:31,  1.90s/it]
Outer training loop:  96%|#########6| 48092/50000 [32:10:26<1:00:39,  1.91s/it]
Outer training loop:  96%|#########6| 48093/50000 [32:10:27<59:27,  1.87s/it]  
Outer training loop:  96%|#########6| 48094/50000 [32:10:29<58:23,  1.84s/it]
Outer training loop:  96%|#########6| 48095/50000 [32:10:31<59:40,  1.88s/it]
Outer training loop:  96%|#########6| 48096/50000 [32:10:33<1:00:18,  1.90s/it]
Outer training loop:  96%|#########6| 48097/50000 [32:10:35<1:00:44,  1.92s/it]
Outer training loop:  96%|#########6| 48098/50000 [32:10:37<1:00:39,  1.91s/it]
Outer training loop:  96%|#########6| 48099/50000 [32:10:39<1:00:00,  1.89s/it]
Outer training loop:  96%|#########6| 48100/50000 [32:10:41<59:08,  1.87s/it]  

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.80s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  96%|#########6| 48101/50000 [32:10:48<1:51:38,  3.53s/it]
Outer training loop:  96%|#########6| 48102/50000 [32:10:50<1:37:27,  3.08s/it]
Outer training loop:  96%|#########6| 48103/50000 [32:10:52<1:25:45,  2.71s/it]
Outer training loop:  96%|#########6| 48104/50000 [32:10:54<1:17:54,  2.47s/it]
Outer training loop:  96%|#########6| 48105/50000 [32:10:56<1:14:25,  2.36s/it]
Outer training loop:  96%|#########6| 48106/50000 [32:10:58<1:10:38,  2.24s/it]
Outer training loop:  96%|#########6| 48107/50000 [32:11:00<1:07:14,  2.13s/it]
Outer training loop:  96%|#########6| 48108/50000 [32:11:02<1:04:55,  2.06s/it]
Outer training loop:  96%|#########6| 48109/50000 [32:11:03<1:02:28,  1.98s/it]
Outer training loop:  96%|#########6| 48110/50000 [32:11:05<1:01:58,  1.97s/it]
Outer training loop:  96%|#########6| 48111/50000 [32:11:07<59:23,  1.89s/it]  
Outer training loop:  96%|#########6| 48112/50000 [32:11:09<1:00:28,  1.92s/it]
Outer training loop:  96%|#########6| 48113/50000 [32:11:11<1:00:12,  1.91s/it]
Outer training loop:  96%|#########6| 48114/50000 [32:11:13<1:00:14,  1.92s/it]
Outer training loop:  96%|#########6| 48115/50000 [32:11:15<1:01:12,  1.95s/it]
Outer training loop:  96%|#########6| 48116/50000 [32:11:17<1:00:47,  1.94s/it]
Outer training loop:  96%|#########6| 48117/50000 [32:11:19<1:00:39,  1.93s/it]
Outer training loop:  96%|#########6| 48118/50000 [32:11:21<59:30,  1.90s/it]  
Outer training loop:  96%|#########6| 48119/50000 [32:11:23<1:02:39,  2.00s/it]
Outer training loop:  96%|#########6| 48120/50000 [32:11:25<1:01:06,  1.95s/it]
Outer training loop:  96%|#########6| 48121/50000 [32:11:27<59:57,  1.91s/it]  
Outer training loop:  96%|#########6| 48122/50000 [32:11:29<1:01:23,  1.96s/it]
Outer training loop:  96%|#########6| 48123/50000 [32:11:30<1:00:06,  1.92s/it]
Outer training loop:  96%|#########6| 48124/50000 [32:11:32<58:58,  1.89s/it]  
Outer training loop:  96%|#########6| 48125/50000 [32:11:34<59:20,  1.90s/it]
Outer training loop:  96%|#########6| 48126/50000 [32:11:36<1:00:21,  1.93s/it]
Outer training loop:  96%|#########6| 48127/50000 [32:11:38<57:16,  1.84s/it]  
Outer training loop:  96%|#########6| 48128/50000 [32:11:40<58:01,  1.86s/it]
Outer training loop:  96%|#########6| 48129/50000 [32:11:42<1:00:16,  1.93s/it]
Outer training loop:  96%|#########6| 48130/50000 [32:11:44<1:01:15,  1.97s/it]
Outer training loop:  96%|#########6| 48131/50000 [32:11:46<1:00:22,  1.94s/it]
Outer training loop:  96%|#########6| 48132/50000 [32:11:48<1:01:27,  1.97s/it]
Outer training loop:  96%|#########6| 48133/50000 [32:11:50<1:00:23,  1.94s/it]
Outer training loop:  96%|#########6| 48134/50000 [32:11:51<59:10,  1.90s/it]  
Outer training loop:  96%|#########6| 48135/50000 [32:11:53<58:52,  1.89s/it]
Outer training loop:  96%|#########6| 48136/50000 [32:11:55<1:00:54,  1.96s/it]
Outer training loop:  96%|#########6| 48137/50000 [32:11:57<59:27,  1.91s/it]  
Outer training loop:  96%|#########6| 48138/50000 [32:11:59<1:00:01,  1.93s/it]
Outer training loop:  96%|#########6| 48139/50000 [32:12:01<1:01:29,  1.98s/it]
Outer training loop:  96%|#########6| 48140/50000 [32:12:03<59:35,  1.92s/it]  
Outer training loop:  96%|#########6| 48141/50000 [32:12:05<1:00:29,  1.95s/it]
Outer training loop:  96%|#########6| 48142/50000 [32:12:07<59:15,  1.91s/it]  
Outer training loop:  96%|#########6| 48143/50000 [32:12:09<59:56,  1.94s/it]
Outer training loop:  96%|#########6| 48144/50000 [32:12:11<1:00:22,  1.95s/it]
Outer training loop:  96%|#########6| 48145/50000 [32:12:13<59:49,  1.93s/it]  
Outer training loop:  96%|#########6| 48146/50000 [32:12:15<58:56,  1.91s/it]
Outer training loop:  96%|#########6| 48147/50000 [32:12:16<57:32,  1.86s/it]
Outer training loop:  96%|#########6| 48148/50000 [32:12:18<57:09,  1.85s/it]
Outer training loop:  96%|#########6| 48149/50000 [32:12:20<58:21,  1.89s/it]
Outer training loop:  96%|#########6| 48150/50000 [32:12:22<1:00:46,  1.97s/it]
Outer training loop:  96%|#########6| 48151/50000 [32:12:24<59:01,  1.92s/it]  
Outer training loop:  96%|#########6| 48152/50000 [32:12:26<59:22,  1.93s/it]
Outer training loop:  96%|#########6| 48153/50000 [32:12:28<1:01:06,  1.99s/it]
Outer training loop:  96%|#########6| 48154/50000 [32:12:30<1:00:24,  1.96s/it]
Outer training loop:  96%|#########6| 48155/50000 [32:12:32<1:00:01,  1.95s/it]
Outer training loop:  96%|#########6| 48156/50000 [32:12:34<58:11,  1.89s/it]  
Outer training loop:  96%|#########6| 48157/50000 [32:12:36<59:13,  1.93s/it]
Outer training loop:  96%|#########6| 48158/50000 [32:12:38<57:44,  1.88s/it]
Outer training loop:  96%|#########6| 48159/50000 [32:12:39<56:33,  1.84s/it]
Outer training loop:  96%|#########6| 48160/50000 [32:12:41<58:37,  1.91s/it]
Outer training loop:  96%|#########6| 48161/50000 [32:12:43<57:20,  1.87s/it]
Outer training loop:  96%|#########6| 48162/50000 [32:12:45<57:17,  1.87s/it]
Outer training loop:  96%|#########6| 48163/50000 [32:12:47<59:33,  1.95s/it]
Outer training loop:  96%|#########6| 48164/50000 [32:12:49<58:29,  1.91s/it]
Outer training loop:  96%|#########6| 48165/50000 [32:12:51<59:13,  1.94s/it]
Outer training loop:  96%|#########6| 48166/50000 [32:12:53<57:58,  1.90s/it]
Outer training loop:  96%|#########6| 48167/50000 [32:12:55<58:55,  1.93s/it]
Outer training loop:  96%|#########6| 48168/50000 [32:12:57<58:36,  1.92s/it]
Outer training loop:  96%|#########6| 48169/50000 [32:12:59<59:03,  1.94s/it]
Outer training loop:  96%|#########6| 48170/50000 [32:13:01<59:15,  1.94s/it]
Outer training loop:  96%|#########6| 48171/50000 [32:13:03<59:02,  1.94s/it]
Outer training loop:  96%|#########6| 48172/50000 [32:13:04<58:31,  1.92s/it]
Outer training loop:  96%|#########6| 48173/50000 [32:13:07<1:00:33,  1.99s/it]
Outer training loop:  96%|#########6| 48174/50000 [32:13:08<58:48,  1.93s/it]  
Outer training loop:  96%|#########6| 48175/50000 [32:13:10<58:29,  1.92s/it]
Outer training loop:  96%|#########6| 48176/50000 [32:13:12<59:16,  1.95s/it]
Outer training loop:  96%|#########6| 48177/50000 [32:13:14<57:14,  1.88s/it]
Outer training loop:  96%|#########6| 48178/50000 [32:13:16<56:26,  1.86s/it]
Outer training loop:  96%|#########6| 48179/50000 [32:13:18<57:10,  1.88s/it]
Outer training loop:  96%|#########6| 48180/50000 [32:13:20<56:01,  1.85s/it]
Outer training loop:  96%|#########6| 48181/50000 [32:13:21<55:01,  1.82s/it]
Outer training loop:  96%|#########6| 48182/50000 [32:13:23<55:09,  1.82s/it]
Outer training loop:  96%|#########6| 48183/50000 [32:13:25<56:53,  1.88s/it]
Outer training loop:  96%|#########6| 48184/50000 [32:13:27<56:58,  1.88s/it]
Outer training loop:  96%|#########6| 48185/50000 [32:13:29<56:12,  1.86s/it]
Outer training loop:  96%|#########6| 48186/50000 [32:13:31<57:20,  1.90s/it]
Outer training loop:  96%|#########6| 48187/50000 [32:13:33<57:48,  1.91s/it]
Outer training loop:  96%|#########6| 48188/50000 [32:13:35<58:36,  1.94s/it]
Outer training loop:  96%|#########6| 48189/50000 [32:13:37<58:22,  1.93s/it]
Outer training loop:  96%|#########6| 48190/50000 [32:13:39<59:29,  1.97s/it]
Outer training loop:  96%|#########6| 48191/50000 [32:13:41<58:03,  1.93s/it]
Outer training loop:  96%|#########6| 48192/50000 [32:13:43<58:37,  1.95s/it]
Outer training loop:  96%|#########6| 48193/50000 [32:13:45<58:30,  1.94s/it]
Outer training loop:  96%|#########6| 48194/50000 [32:13:46<57:42,  1.92s/it]
Outer training loop:  96%|#########6| 48195/50000 [32:13:48<57:23,  1.91s/it]
Outer training loop:  96%|#########6| 48196/50000 [32:13:50<59:02,  1.96s/it]
Outer training loop:  96%|#########6| 48197/50000 [32:13:52<58:56,  1.96s/it]
Outer training loop:  96%|#########6| 48198/50000 [32:13:54<57:09,  1.90s/it]
Outer training loop:  96%|#########6| 48199/50000 [32:13:56<56:35,  1.89s/it]
Outer training loop:  96%|#########6| 48200/50000 [32:13:58<58:56,  1.96s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.78s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.91s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  96%|#########6| 48201/50000 [32:14:06<1:50:16,  3.68s/it]
Outer training loop:  96%|#########6| 48202/50000 [32:14:07<1:32:47,  3.10s/it]
Outer training loop:  96%|#########6| 48203/50000 [32:14:09<1:20:53,  2.70s/it]
Outer training loop:  96%|#########6| 48204/50000 [32:14:11<1:16:28,  2.56s/it]
Outer training loop:  96%|#########6| 48205/50000 [32:14:13<1:10:18,  2.35s/it]
Outer training loop:  96%|#########6| 48206/50000 [32:14:15<1:05:33,  2.19s/it]
Outer training loop:  96%|#########6| 48207/50000 [32:14:17<1:04:52,  2.17s/it]
Outer training loop:  96%|#########6| 48208/50000 [32:14:19<1:02:27,  2.09s/it]
Outer training loop:  96%|#########6| 48209/50000 [32:14:21<1:01:20,  2.05s/it]
Outer training loop:  96%|#########6| 48210/50000 [32:14:23<58:51,  1.97s/it]  
Outer training loop:  96%|#########6| 48211/50000 [32:14:25<59:55,  2.01s/it]
Outer training loop:  96%|#########6| 48212/50000 [32:14:27<57:26,  1.93s/it]
Outer training loop:  96%|#########6| 48213/50000 [32:14:29<57:18,  1.92s/it]
Outer training loop:  96%|#########6| 48214/50000 [32:14:31<57:58,  1.95s/it]
Outer training loop:  96%|#########6| 48215/50000 [32:14:33<56:51,  1.91s/it]
Outer training loop:  96%|#########6| 48216/50000 [32:14:35<58:33,  1.97s/it]
Outer training loop:  96%|#########6| 48217/50000 [32:14:36<57:09,  1.92s/it]
Outer training loop:  96%|#########6| 48218/50000 [32:14:38<57:07,  1.92s/it]
Outer training loop:  96%|#########6| 48219/50000 [32:14:40<55:59,  1.89s/it]
Outer training loop:  96%|#########6| 48220/50000 [32:14:42<56:11,  1.89s/it]
Outer training loop:  96%|#########6| 48221/50000 [32:14:44<58:10,  1.96s/it]
Outer training loop:  96%|#########6| 48222/50000 [32:14:46<57:17,  1.93s/it]
Outer training loop:  96%|#########6| 48223/50000 [32:14:48<56:09,  1.90s/it]
Outer training loop:  96%|#########6| 48224/50000 [32:14:50<58:29,  1.98s/it]
Outer training loop:  96%|#########6| 48225/50000 [32:14:52<57:30,  1.94s/it]
Outer training loop:  96%|#########6| 48226/50000 [32:14:54<55:26,  1.88s/it]
Outer training loop:  96%|#########6| 48227/50000 [32:14:56<55:43,  1.89s/it]
Outer training loop:  96%|#########6| 48228/50000 [32:14:58<58:14,  1.97s/it]
Outer training loop:  96%|#########6| 48229/50000 [32:15:00<57:16,  1.94s/it]
Outer training loop:  96%|#########6| 48230/50000 [32:15:01<56:05,  1.90s/it]
Outer training loop:  96%|#########6| 48231/50000 [32:15:03<57:16,  1.94s/it]
Outer training loop:  96%|#########6| 48232/50000 [32:15:05<57:07,  1.94s/it]
Outer training loop:  96%|#########6| 48233/50000 [32:15:07<55:16,  1.88s/it]
Outer training loop:  96%|#########6| 48234/50000 [32:15:09<56:14,  1.91s/it]
Outer training loop:  96%|#########6| 48235/50000 [32:15:11<56:19,  1.91s/it]
Outer training loop:  96%|#########6| 48236/50000 [32:15:13<55:08,  1.88s/it]
Outer training loop:  96%|#########6| 48237/50000 [32:15:15<56:03,  1.91s/it]
Outer training loop:  96%|#########6| 48238/50000 [32:15:17<56:50,  1.94s/it]
Outer training loop:  96%|#########6| 48239/50000 [32:15:19<55:20,  1.89s/it]
Outer training loop:  96%|#########6| 48240/50000 [32:15:20<55:44,  1.90s/it]
Outer training loop:  96%|#########6| 48241/50000 [32:15:22<56:10,  1.92s/it]
Outer training loop:  96%|#########6| 48242/50000 [32:15:24<55:44,  1.90s/it]
Outer training loop:  96%|#########6| 48243/50000 [32:15:26<55:21,  1.89s/it]
Outer training loop:  96%|#########6| 48244/50000 [32:15:28<56:47,  1.94s/it]
Outer training loop:  96%|#########6| 48245/50000 [32:15:30<56:22,  1.93s/it]
Outer training loop:  96%|#########6| 48246/50000 [32:15:32<56:35,  1.94s/it]
Outer training loop:  96%|#########6| 48247/50000 [32:15:34<56:35,  1.94s/it]
Outer training loop:  96%|#########6| 48248/50000 [32:15:36<56:00,  1.92s/it]
Outer training loop:  96%|#########6| 48249/50000 [32:15:38<54:57,  1.88s/it]
Outer training loop:  96%|#########6| 48250/50000 [32:15:39<53:42,  1.84s/it]
Outer training loop:  97%|#########6| 48251/50000 [32:15:41<55:03,  1.89s/it]
Outer training loop:  97%|#########6| 48252/50000 [32:15:43<53:55,  1.85s/it]
Outer training loop:  97%|#########6| 48253/50000 [32:15:45<53:18,  1.83s/it]
Outer training loop:  97%|#########6| 48254/50000 [32:15:47<54:27,  1.87s/it]
Outer training loop:  97%|#########6| 48255/50000 [32:15:49<53:46,  1.85s/it]
Outer training loop:  97%|#########6| 48256/50000 [32:15:51<53:33,  1.84s/it]
Outer training loop:  97%|#########6| 48257/50000 [32:15:53<55:20,  1.90s/it]
Outer training loop:  97%|#########6| 48258/50000 [32:15:54<53:14,  1.83s/it]
Outer training loop:  97%|#########6| 48259/50000 [32:15:56<52:27,  1.81s/it]
Outer training loop:  97%|#########6| 48260/50000 [32:15:58<52:04,  1.80s/it]
Outer training loop:  97%|#########6| 48261/50000 [32:16:00<54:10,  1.87s/it]
Outer training loop:  97%|#########6| 48262/50000 [32:16:02<54:01,  1.87s/it]
Outer training loop:  97%|#########6| 48263/50000 [32:16:04<54:09,  1.87s/it]
Outer training loop:  97%|#########6| 48264/50000 [32:16:06<55:42,  1.93s/it]
Outer training loop:  97%|#########6| 48265/50000 [32:16:07<54:29,  1.88s/it]
Outer training loop:  97%|#########6| 48266/50000 [32:16:09<54:54,  1.90s/it]
Outer training loop:  97%|#########6| 48267/50000 [32:16:11<56:22,  1.95s/it]
Outer training loop:  97%|#########6| 48268/50000 [32:16:13<54:38,  1.89s/it]
Outer training loop:  97%|#########6| 48269/50000 [32:16:15<53:52,  1.87s/it]
Outer training loop:  97%|#########6| 48270/50000 [32:16:17<54:35,  1.89s/it]
Outer training loop:  97%|#########6| 48271/50000 [32:16:19<53:58,  1.87s/it]
Outer training loop:  97%|#########6| 48272/50000 [32:16:21<53:48,  1.87s/it]
Outer training loop:  97%|#########6| 48273/50000 [32:16:22<53:32,  1.86s/it]
Outer training loop:  97%|#########6| 48274/50000 [32:16:25<55:10,  1.92s/it]
Outer training loop:  97%|#########6| 48275/50000 [32:16:26<54:32,  1.90s/it]
Outer training loop:  97%|#########6| 48276/50000 [32:16:28<53:46,  1.87s/it]
Outer training loop:  97%|#########6| 48277/50000 [32:16:30<55:47,  1.94s/it]
Outer training loop:  97%|#########6| 48278/50000 [32:16:32<54:53,  1.91s/it]
Outer training loop:  97%|#########6| 48279/50000 [32:16:34<54:16,  1.89s/it]
Outer training loop:  97%|#########6| 48280/50000 [32:16:36<55:58,  1.95s/it]
Outer training loop:  97%|#########6| 48281/50000 [32:16:38<55:20,  1.93s/it]
Outer training loop:  97%|#########6| 48282/50000 [32:16:40<54:10,  1.89s/it]
Outer training loop:  97%|#########6| 48283/50000 [32:16:42<53:39,  1.88s/it]
Outer training loop:  97%|#########6| 48284/50000 [32:16:44<54:38,  1.91s/it]
Outer training loop:  97%|#########6| 48285/50000 [32:16:45<53:39,  1.88s/it]
Outer training loop:  97%|#########6| 48286/50000 [32:16:47<52:44,  1.85s/it]
Outer training loop:  97%|#########6| 48287/50000 [32:16:49<54:27,  1.91s/it]
Outer training loop:  97%|#########6| 48288/50000 [32:16:51<53:44,  1.88s/it]
Outer training loop:  97%|#########6| 48289/50000 [32:16:53<52:45,  1.85s/it]
Outer training loop:  97%|#########6| 48290/50000 [32:16:55<52:34,  1.84s/it]
Outer training loop:  97%|#########6| 48291/50000 [32:16:57<54:22,  1.91s/it]
Outer training loop:  97%|#########6| 48292/50000 [32:16:59<54:04,  1.90s/it]
Outer training loop:  97%|#########6| 48293/50000 [32:17:00<52:56,  1.86s/it]
Outer training loop:  97%|#########6| 48294/50000 [32:17:02<52:01,  1.83s/it]
Outer training loop:  97%|#########6| 48295/50000 [32:17:04<53:55,  1.90s/it]
Outer training loop:  97%|#########6| 48296/50000 [32:17:06<53:26,  1.88s/it]
Outer training loop:  97%|#########6| 48297/50000 [32:17:08<52:09,  1.84s/it]
Outer training loop:  97%|#########6| 48298/50000 [32:17:10<53:50,  1.90s/it]
Outer training loop:  97%|#########6| 48299/50000 [32:17:12<52:44,  1.86s/it]
Outer training loop:  97%|#########6| 48300/50000 [32:17:14<53:40,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.79s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  97%|#########6| 48301/50000 [32:17:21<1:41:09,  3.57s/it]
Outer training loop:  97%|#########6| 48302/50000 [32:17:23<1:27:02,  3.08s/it]
Outer training loop:  97%|#########6| 48303/50000 [32:17:25<1:16:16,  2.70s/it]
Outer training loop:  97%|#########6| 48304/50000 [32:17:27<1:08:59,  2.44s/it]
Outer training loop:  97%|#########6| 48305/50000 [32:17:29<1:05:38,  2.32s/it]
Outer training loop:  97%|#########6| 48306/50000 [32:17:30<1:01:03,  2.16s/it]
Outer training loop:  97%|#########6| 48307/50000 [32:17:32<57:46,  2.05s/it]  
Outer training loop:  97%|#########6| 48308/50000 [32:17:34<56:00,  1.99s/it]
Outer training loop:  97%|#########6| 48309/50000 [32:17:36<56:12,  1.99s/it]
Outer training loop:  97%|#########6| 48310/50000 [32:17:38<54:22,  1.93s/it]
Outer training loop:  97%|#########6| 48311/50000 [32:17:40<54:45,  1.95s/it]
Outer training loop:  97%|#########6| 48312/50000 [32:17:42<55:06,  1.96s/it]
Outer training loop:  97%|#########6| 48313/50000 [32:17:44<55:42,  1.98s/it]
Outer training loop:  97%|#########6| 48314/50000 [32:17:46<52:52,  1.88s/it]
Outer training loop:  97%|#########6| 48315/50000 [32:17:47<52:36,  1.87s/it]
Outer training loop:  97%|#########6| 48316/50000 [32:17:49<52:53,  1.88s/it]
Outer training loop:  97%|#########6| 48317/50000 [32:17:51<51:44,  1.84s/it]
Outer training loop:  97%|#########6| 48318/50000 [32:17:53<51:36,  1.84s/it]
Outer training loop:  97%|#########6| 48319/50000 [32:17:55<51:48,  1.85s/it]
Outer training loop:  97%|#########6| 48320/50000 [32:17:57<53:24,  1.91s/it]
Outer training loop:  97%|#########6| 48321/50000 [32:17:59<53:35,  1.92s/it]
Outer training loop:  97%|#########6| 48322/50000 [32:18:00<52:02,  1.86s/it]
Outer training loop:  97%|#########6| 48323/50000 [32:18:02<53:44,  1.92s/it]
Outer training loop:  97%|#########6| 48324/50000 [32:18:04<52:56,  1.90s/it]
Outer training loop:  97%|#########6| 48325/50000 [32:18:06<52:17,  1.87s/it]
Outer training loop:  97%|#########6| 48326/50000 [32:18:08<53:44,  1.93s/it]
Outer training loop:  97%|#########6| 48327/50000 [32:18:10<53:14,  1.91s/it]
Outer training loop:  97%|#########6| 48328/50000 [32:18:12<51:59,  1.87s/it]
Outer training loop:  97%|#########6| 48329/50000 [32:18:14<51:11,  1.84s/it]
Outer training loop:  97%|#########6| 48330/50000 [32:18:16<52:23,  1.88s/it]
Outer training loop:  97%|#########6| 48331/50000 [32:18:17<52:17,  1.88s/it]
Outer training loop:  97%|#########6| 48332/50000 [32:18:19<51:30,  1.85s/it]
Outer training loop:  97%|#########6| 48333/50000 [32:18:21<53:44,  1.93s/it]
Outer training loop:  97%|#########6| 48334/50000 [32:18:23<52:04,  1.88s/it]
Outer training loop:  97%|#########6| 48335/50000 [32:18:25<51:08,  1.84s/it]
Outer training loop:  97%|#########6| 48336/50000 [32:18:27<50:39,  1.83s/it]
Outer training loop:  97%|#########6| 48337/50000 [32:18:29<52:15,  1.89s/it]
Outer training loop:  97%|#########6| 48338/50000 [32:18:30<51:30,  1.86s/it]
Outer training loop:  97%|#########6| 48339/50000 [32:18:32<51:22,  1.86s/it]
Outer training loop:  97%|#########6| 48340/50000 [32:18:34<50:55,  1.84s/it]
Outer training loop:  97%|#########6| 48341/50000 [32:18:36<52:59,  1.92s/it]
Outer training loop:  97%|#########6| 48342/50000 [32:18:38<52:31,  1.90s/it]
Outer training loop:  97%|#########6| 48343/50000 [32:18:40<51:17,  1.86s/it]
Outer training loop:  97%|#########6| 48344/50000 [32:18:42<53:30,  1.94s/it]
Outer training loop:  97%|#########6| 48345/50000 [32:18:44<52:17,  1.90s/it]
Outer training loop:  97%|#########6| 48346/50000 [32:18:46<51:16,  1.86s/it]
Outer training loop:  97%|#########6| 48347/50000 [32:18:48<52:21,  1.90s/it]
Outer training loop:  97%|#########6| 48348/50000 [32:18:49<52:34,  1.91s/it]
Outer training loop:  97%|#########6| 48349/50000 [32:18:51<51:00,  1.85s/it]
Outer training loop:  97%|#########6| 48350/50000 [32:18:53<50:00,  1.82s/it]
Outer training loop:  97%|#########6| 48351/50000 [32:18:55<51:57,  1.89s/it]
Outer training loop:  97%|#########6| 48352/50000 [32:18:57<49:46,  1.81s/it]
Outer training loop:  97%|#########6| 48353/50000 [32:18:58<50:07,  1.83s/it]
Outer training loop:  97%|#########6| 48354/50000 [32:19:00<50:04,  1.83s/it]
Outer training loop:  97%|#########6| 48355/50000 [32:19:02<51:03,  1.86s/it]
Outer training loop:  97%|#########6| 48356/50000 [32:19:04<50:22,  1.84s/it]
Outer training loop:  97%|#########6| 48357/50000 [32:19:06<50:16,  1.84s/it]
Outer training loop:  97%|#########6| 48358/50000 [32:19:08<52:34,  1.92s/it]
Outer training loop:  97%|#########6| 48359/50000 [32:19:10<52:52,  1.93s/it]
Outer training loop:  97%|#########6| 48360/50000 [32:19:12<51:31,  1.89s/it]
Outer training loop:  97%|#########6| 48361/50000 [32:19:14<53:44,  1.97s/it]
Outer training loop:  97%|#########6| 48362/50000 [32:19:16<52:24,  1.92s/it]
Outer training loop:  97%|#########6| 48363/50000 [32:19:18<52:45,  1.93s/it]
Outer training loop:  97%|#########6| 48364/50000 [32:19:19<51:08,  1.88s/it]
Outer training loop:  97%|#########6| 48365/50000 [32:19:21<52:37,  1.93s/it]
Outer training loop:  97%|#########6| 48366/50000 [32:19:23<51:44,  1.90s/it]
Outer training loop:  97%|#########6| 48367/50000 [32:19:25<51:54,  1.91s/it]
Outer training loop:  97%|#########6| 48368/50000 [32:19:27<52:46,  1.94s/it]
Outer training loop:  97%|#########6| 48369/50000 [32:19:29<52:07,  1.92s/it]
Outer training loop:  97%|#########6| 48370/50000 [32:19:31<50:28,  1.86s/it]
Outer training loop:  97%|#########6| 48371/50000 [32:19:33<51:34,  1.90s/it]
Outer training loop:  97%|#########6| 48372/50000 [32:19:35<52:15,  1.93s/it]
Outer training loop:  97%|#########6| 48373/50000 [32:19:37<51:40,  1.91s/it]
Outer training loop:  97%|#########6| 48374/50000 [32:19:39<52:34,  1.94s/it]
Outer training loop:  97%|#########6| 48375/50000 [32:19:41<51:42,  1.91s/it]
Outer training loop:  97%|#########6| 48376/50000 [32:19:42<50:58,  1.88s/it]
Outer training loop:  97%|#########6| 48377/50000 [32:19:44<52:16,  1.93s/it]
Outer training loop:  97%|#########6| 48378/50000 [32:19:46<50:54,  1.88s/it]
Outer training loop:  97%|#########6| 48379/50000 [32:19:48<49:17,  1.82s/it]
Outer training loop:  97%|#########6| 48380/50000 [32:19:50<51:28,  1.91s/it]
Outer training loop:  97%|#########6| 48381/50000 [32:19:52<50:52,  1.89s/it]
Outer training loop:  97%|#########6| 48382/50000 [32:19:54<51:04,  1.89s/it]
Outer training loop:  97%|#########6| 48383/50000 [32:19:56<50:52,  1.89s/it]
Outer training loop:  97%|#########6| 48384/50000 [32:19:58<51:43,  1.92s/it]
Outer training loop:  97%|#########6| 48385/50000 [32:19:59<51:20,  1.91s/it]
Outer training loop:  97%|#########6| 48386/50000 [32:20:01<50:24,  1.87s/it]
Outer training loop:  97%|#########6| 48387/50000 [32:20:03<49:34,  1.84s/it]
Outer training loop:  97%|#########6| 48388/50000 [32:20:05<52:00,  1.94s/it]
Outer training loop:  97%|#########6| 48389/50000 [32:20:07<52:08,  1.94s/it]
Outer training loop:  97%|#########6| 48390/50000 [32:20:09<50:41,  1.89s/it]
Outer training loop:  97%|#########6| 48391/50000 [32:20:11<51:33,  1.92s/it]
Outer training loop:  97%|#########6| 48392/50000 [32:20:13<50:49,  1.90s/it]
Outer training loop:  97%|#########6| 48393/50000 [32:20:15<50:07,  1.87s/it]
Outer training loop:  97%|#########6| 48394/50000 [32:20:17<51:39,  1.93s/it]
Outer training loop:  97%|#########6| 48395/50000 [32:20:18<50:55,  1.90s/it]
Outer training loop:  97%|#########6| 48396/50000 [32:20:20<50:24,  1.89s/it]
Outer training loop:  97%|#########6| 48397/50000 [32:20:22<49:54,  1.87s/it]
Outer training loop:  97%|#########6| 48398/50000 [32:20:24<51:14,  1.92s/it]
Outer training loop:  97%|#########6| 48399/50000 [32:20:26<50:53,  1.91s/it]
Outer training loop:  97%|#########6| 48400/50000 [32:20:28<49:55,  1.87s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.92s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.92s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.86s/it]

Outer training loop:  97%|#########6| 48401/50000 [32:20:35<1:34:09,  3.53s/it]
Outer training loop:  97%|#########6| 48402/50000 [32:20:37<1:22:12,  3.09s/it]
Outer training loop:  97%|#########6| 48403/50000 [32:20:39<1:11:46,  2.70s/it]
Outer training loop:  97%|#########6| 48404/50000 [32:20:41<1:06:11,  2.49s/it]
Outer training loop:  97%|#########6| 48405/50000 [32:20:43<1:01:12,  2.30s/it]
Outer training loop:  97%|#########6| 48406/50000 [32:20:45<57:17,  2.16s/it]  
Outer training loop:  97%|#########6| 48407/50000 [32:20:47<54:11,  2.04s/it]
Outer training loop:  97%|#########6| 48408/50000 [32:20:48<53:32,  2.02s/it]
Outer training loop:  97%|#########6| 48409/50000 [32:20:50<52:47,  1.99s/it]
Outer training loop:  97%|#########6| 48410/50000 [32:20:52<53:08,  2.01s/it]
Outer training loop:  97%|#########6| 48411/50000 [32:20:54<52:24,  1.98s/it]
Outer training loop:  97%|#########6| 48412/50000 [32:20:56<52:59,  2.00s/it]
Outer training loop:  97%|#########6| 48413/50000 [32:20:58<51:20,  1.94s/it]
Outer training loop:  97%|#########6| 48414/50000 [32:21:00<50:01,  1.89s/it]
Outer training loop:  97%|#########6| 48415/50000 [32:21:02<50:16,  1.90s/it]
Outer training loop:  97%|#########6| 48416/50000 [32:21:04<50:29,  1.91s/it]
Outer training loop:  97%|#########6| 48417/50000 [32:21:06<50:37,  1.92s/it]
Outer training loop:  97%|#########6| 48418/50000 [32:21:08<49:24,  1.87s/it]
Outer training loop:  97%|#########6| 48419/50000 [32:21:10<51:05,  1.94s/it]
Outer training loop:  97%|#########6| 48420/50000 [32:21:11<49:21,  1.87s/it]
Outer training loop:  97%|#########6| 48421/50000 [32:21:13<49:24,  1.88s/it]
Outer training loop:  97%|#########6| 48422/50000 [32:21:15<51:02,  1.94s/it]
Outer training loop:  97%|#########6| 48423/50000 [32:21:17<50:10,  1.91s/it]
Outer training loop:  97%|#########6| 48424/50000 [32:21:19<49:42,  1.89s/it]
Outer training loop:  97%|#########6| 48425/50000 [32:21:21<50:14,  1.91s/it]
Outer training loop:  97%|#########6| 48426/50000 [32:21:23<49:37,  1.89s/it]
Outer training loop:  97%|#########6| 48427/50000 [32:21:25<48:31,  1.85s/it]
Outer training loop:  97%|#########6| 48428/50000 [32:21:26<48:05,  1.84s/it]
Outer training loop:  97%|#########6| 48429/50000 [32:21:28<49:56,  1.91s/it]
Outer training loop:  97%|#########6| 48430/50000 [32:21:30<48:55,  1.87s/it]
Outer training loop:  97%|#########6| 48431/50000 [32:21:32<48:55,  1.87s/it]
Outer training loop:  97%|#########6| 48432/50000 [32:21:34<50:57,  1.95s/it]
Outer training loop:  97%|#########6| 48433/50000 [32:21:36<49:17,  1.89s/it]
Outer training loop:  97%|#########6| 48434/50000 [32:21:38<48:13,  1.85s/it]
Outer training loop:  97%|#########6| 48435/50000 [32:21:40<48:09,  1.85s/it]
Outer training loop:  97%|#########6| 48436/50000 [32:21:42<50:47,  1.95s/it]
Outer training loop:  97%|#########6| 48437/50000 [32:21:44<49:34,  1.90s/it]
Outer training loop:  97%|#########6| 48438/50000 [32:21:45<48:42,  1.87s/it]
Outer training loop:  97%|#########6| 48439/50000 [32:21:47<48:56,  1.88s/it]
Outer training loop:  97%|#########6| 48440/50000 [32:21:49<49:39,  1.91s/it]
Outer training loop:  97%|#########6| 48441/50000 [32:21:51<49:40,  1.91s/it]
Outer training loop:  97%|#########6| 48442/50000 [32:21:53<49:20,  1.90s/it]
Outer training loop:  97%|#########6| 48443/50000 [32:21:55<50:37,  1.95s/it]
Outer training loop:  97%|#########6| 48444/50000 [32:21:57<50:03,  1.93s/it]
Outer training loop:  97%|#########6| 48445/50000 [32:21:59<49:28,  1.91s/it]
Outer training loop:  97%|#########6| 48446/50000 [32:22:01<50:44,  1.96s/it]
Outer training loop:  97%|#########6| 48447/50000 [32:22:03<49:53,  1.93s/it]
Outer training loop:  97%|#########6| 48448/50000 [32:22:05<50:17,  1.94s/it]
Outer training loop:  97%|#########6| 48449/50000 [32:22:07<50:44,  1.96s/it]
Outer training loop:  97%|#########6| 48450/50000 [32:22:09<49:12,  1.90s/it]
Outer training loop:  97%|#########6| 48451/50000 [32:22:11<49:54,  1.93s/it]
Outer training loop:  97%|#########6| 48452/50000 [32:22:12<49:20,  1.91s/it]
Outer training loop:  97%|#########6| 48453/50000 [32:22:15<50:51,  1.97s/it]
Outer training loop:  97%|#########6| 48454/50000 [32:22:16<50:40,  1.97s/it]
Outer training loop:  97%|#########6| 48455/50000 [32:22:18<49:36,  1.93s/it]
Outer training loop:  97%|#########6| 48456/50000 [32:22:20<50:31,  1.96s/it]
Outer training loop:  97%|#########6| 48457/50000 [32:22:22<49:10,  1.91s/it]
Outer training loop:  97%|#########6| 48458/50000 [32:22:24<48:09,  1.87s/it]
Outer training loop:  97%|#########6| 48459/50000 [32:22:26<48:15,  1.88s/it]
Outer training loop:  97%|#########6| 48460/50000 [32:22:28<50:47,  1.98s/it]
Outer training loop:  97%|#########6| 48461/50000 [32:22:30<49:34,  1.93s/it]
Outer training loop:  97%|#########6| 48462/50000 [32:22:32<48:00,  1.87s/it]
Outer training loop:  97%|#########6| 48463/50000 [32:22:34<49:38,  1.94s/it]
Outer training loop:  97%|#########6| 48464/50000 [32:22:36<49:36,  1.94s/it]
Outer training loop:  97%|#########6| 48465/50000 [32:22:37<48:57,  1.91s/it]
Outer training loop:  97%|#########6| 48466/50000 [32:22:40<49:58,  1.95s/it]
Outer training loop:  97%|#########6| 48467/50000 [32:22:41<49:04,  1.92s/it]
Outer training loop:  97%|#########6| 48468/50000 [32:22:43<49:20,  1.93s/it]
Outer training loop:  97%|#########6| 48469/50000 [32:22:45<48:19,  1.89s/it]
Outer training loop:  97%|#########6| 48470/50000 [32:22:47<50:10,  1.97s/it]
Outer training loop:  97%|#########6| 48471/50000 [32:22:49<49:58,  1.96s/it]
Outer training loop:  97%|#########6| 48472/50000 [32:22:51<49:17,  1.94s/it]
Outer training loop:  97%|#########6| 48473/50000 [32:22:53<50:21,  1.98s/it]
Outer training loop:  97%|#########6| 48474/50000 [32:22:55<48:59,  1.93s/it]
Outer training loop:  97%|#########6| 48475/50000 [32:22:57<47:13,  1.86s/it]
Outer training loop:  97%|#########6| 48476/50000 [32:22:59<48:39,  1.92s/it]
Outer training loop:  97%|#########6| 48477/50000 [32:23:00<47:04,  1.85s/it]
Outer training loop:  97%|#########6| 48478/50000 [32:23:02<46:47,  1.84s/it]
Outer training loop:  97%|#########6| 48479/50000 [32:23:04<48:58,  1.93s/it]
Outer training loop:  97%|#########6| 48480/50000 [32:23:06<48:23,  1.91s/it]
Outer training loop:  97%|#########6| 48481/50000 [32:23:08<46:45,  1.85s/it]
Outer training loop:  97%|#########6| 48482/50000 [32:23:10<46:56,  1.86s/it]
Outer training loop:  97%|#########6| 48483/50000 [32:23:12<47:41,  1.89s/it]
Outer training loop:  97%|#########6| 48484/50000 [32:23:14<48:29,  1.92s/it]
Outer training loop:  97%|#########6| 48485/50000 [32:23:16<48:31,  1.92s/it]
Outer training loop:  97%|#########6| 48486/50000 [32:23:18<48:50,  1.94s/it]
Outer training loop:  97%|#########6| 48487/50000 [32:23:20<49:31,  1.96s/it]
Outer training loop:  97%|#########6| 48488/50000 [32:23:22<49:02,  1.95s/it]
Outer training loop:  97%|#########6| 48489/50000 [32:23:23<47:28,  1.89s/it]
Outer training loop:  97%|#########6| 48490/50000 [32:23:25<49:07,  1.95s/it]
Outer training loop:  97%|#########6| 48491/50000 [32:23:27<48:39,  1.94s/it]
Outer training loop:  97%|#########6| 48492/50000 [32:23:29<47:44,  1.90s/it]
Outer training loop:  97%|#########6| 48493/50000 [32:23:31<48:37,  1.94s/it]
Outer training loop:  97%|#########6| 48494/50000 [32:23:33<47:00,  1.87s/it]
Outer training loop:  97%|#########6| 48495/50000 [32:23:35<47:58,  1.91s/it]
Outer training loop:  97%|#########6| 48496/50000 [32:23:37<47:08,  1.88s/it]
Outer training loop:  97%|#########6| 48497/50000 [32:23:39<48:40,  1.94s/it]
Outer training loop:  97%|#########6| 48498/50000 [32:23:41<47:46,  1.91s/it]
Outer training loop:  97%|#########6| 48499/50000 [32:23:43<47:50,  1.91s/it]
Outer training loop:  97%|#########7| 48500/50000 [32:23:44<47:44,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.75s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.89s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:25,  1.77s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:25,  1.80s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:30,  1.89s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:27,  1.86s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:25,  1.87s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:27,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:30,  1.98s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:18,  1.90s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:24,  2.00s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:20,  1.97s/it]

Evaluating policy:  16%|#5        | 13/83 [00:25<02:14,  1.93s/it]

Evaluating policy:  17%|#6        | 14/83 [00:27<02:12,  1.92s/it]

Evaluating policy:  18%|#8        | 15/83 [00:29<02:15,  1.99s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:08,  1.92s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:06,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:02,  1.89s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:06,  1.98s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:03,  1.95s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:57,  1.90s/it]

Evaluating policy:  27%|##6       | 22/83 [00:42<01:55,  1.89s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<01:54,  1.90s/it]

Evaluating policy:  29%|##8       | 24/83 [00:46<01:56,  1.98s/it]

Evaluating policy:  30%|###       | 25/83 [00:48<01:53,  1.97s/it]

Evaluating policy:  31%|###1      | 26/83 [00:50<01:49,  1.93s/it]

Evaluating policy:  33%|###2      | 27/83 [00:52<01:48,  1.93s/it]

Evaluating policy:  34%|###3      | 28/83 [00:54<01:46,  1.93s/it]

Evaluating policy:  35%|###4      | 29/83 [00:56<01:45,  1.96s/it]

Evaluating policy:  36%|###6      | 30/83 [00:58<01:44,  1.97s/it]

Evaluating policy:  37%|###7      | 31/83 [01:00<01:41,  1.95s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:35,  1.90s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:30,  1.85s/it]

Evaluating policy:  42%|####2     | 35/83 [01:07<01:32,  1.92s/it]

Evaluating policy:  43%|####3     | 36/83 [01:09<01:29,  1.90s/it]

Evaluating policy:  45%|####4     | 37/83 [01:11<01:24,  1.83s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:21,  1.81s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:20,  1.82s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:16,  1.78s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:20,  1.91s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:17,  1.89s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:12,  1.85s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:09,  1.83s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:06,  1.80s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:04,  1.80s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:06,  1.91s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:03,  1.88s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.86s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:54,  1.82s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:52,  1.79s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:54,  1.93s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:52,  1.93s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:49,  1.92s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:44,  1.85s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:42,  1.87s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:40,  1.86s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:38,  1.84s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:39,  1.95s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.89s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:33,  1.85s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:30,  1.81s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.83s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:27,  1.83s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:25,  1.83s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.87s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:21,  1.97s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.90s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:17,  1.91s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.92s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:13,  1.90s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:11,  1.89s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.91s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.92s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:04,  2.05s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:02,  2.02s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  1.92s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:23,  1.75s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:23,  1.77s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:25,  1.81s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:25,  1.85s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:21,  1.82s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:21,  1.84s/it]

Evaluating policy:   8%|8         | 7/83 [00:12<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:32,  2.03s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:25,  1.96s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:20,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:19,  1.93s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:14,  1.92s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:05,  1.85s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:04,  1.86s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:05,  1.91s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:03,  1.90s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.89s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:56,  1.87s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:58,  1.94s/it]

Evaluating policy:  28%|##7       | 23/83 [00:44<02:01,  2.02s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:56,  1.97s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.92s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:50,  1.97s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:47,  1.95s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.91s/it]

Evaluating policy:  36%|###6      | 30/83 [00:57<01:40,  1.90s/it]

Evaluating policy:  37%|###7      | 31/83 [00:59<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:01<01:38,  1.94s/it]

Evaluating policy:  40%|###9      | 33/83 [01:03<01:36,  1.93s/it]

Evaluating policy:  41%|####      | 34/83 [01:05<01:34,  1.93s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:27,  1.90s/it]

Evaluating policy:  46%|####5     | 38/83 [01:12<01:28,  1.97s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:23,  1.90s/it]

Evaluating policy:  48%|####8     | 40/83 [01:16<01:22,  1.91s/it]

Evaluating policy:  49%|####9     | 41/83 [01:18<01:19,  1.89s/it]

Evaluating policy:  51%|#####     | 42/83 [01:20<01:18,  1.92s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:22<01:15,  1.89s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:24<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:26<01:16,  2.02s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:28<01:11,  1.94s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:08,  1.91s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.87s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:03,  1.86s/it]

Evaluating policy:  60%|######    | 50/83 [01:35<01:01,  1.86s/it]

Evaluating policy:  61%|######1   | 51/83 [01:37<01:00,  1.88s/it]

Evaluating policy:  63%|######2   | 52/83 [01:39<01:02,  2.00s/it]

Evaluating policy:  64%|######3   | 53/83 [01:41<00:58,  1.95s/it]

Evaluating policy:  65%|######5   | 54/83 [01:43<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:45<00:52,  1.86s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:48,  1.87s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:52<00:48,  2.01s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:54<00:45,  1.97s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:56<00:42,  1.93s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:58<00:39,  1.90s/it]

Evaluating policy:  76%|#######5  | 63/83 [02:00<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:02<00:35,  1.86s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:04<00:33,  1.86s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:31,  1.86s/it]

Evaluating policy:  81%|########  | 67/83 [02:08<00:32,  2.02s/it]

Evaluating policy:  82%|########1 | 68/83 [02:10<00:29,  1.97s/it]

Evaluating policy:  83%|########3 | 69/83 [02:12<00:27,  1.94s/it]

Evaluating policy:  84%|########4 | 70/83 [02:13<00:25,  1.93s/it]

Evaluating policy:  86%|########5 | 71/83 [02:15<00:22,  1.89s/it]

Evaluating policy:  87%|########6 | 72/83 [02:17<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:19<00:18,  1.88s/it]

Evaluating policy:  89%|########9 | 74/83 [02:21<00:16,  1.85s/it]

Evaluating policy:  90%|######### | 75/83 [02:23<00:14,  1.86s/it]

Evaluating policy:  92%|#########1| 76/83 [02:25<00:14,  2.02s/it]

Evaluating policy:  93%|#########2| 77/83 [02:27<00:11,  1.95s/it]

Evaluating policy:  94%|#########3| 78/83 [02:29<00:09,  1.90s/it]

Evaluating policy:  95%|#########5| 79/83 [02:30<00:07,  1.88s/it]

Evaluating policy:  96%|#########6| 80/83 [02:32<00:05,  1.87s/it]

Evaluating policy:  98%|#########7| 81/83 [02:34<00:03,  1.83s/it]

Evaluating policy:  99%|#########8| 82/83 [02:36<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 83/83 [02:38<00:00,  1.87s/it]

Outer training loop:  97%|#########7| 48501/50000 [32:29:08<40:58:52, 98.42s/it]
Outer training loop:  97%|#########7| 48502/50000 [32:29:10<28:53:18, 69.43s/it]
Outer training loop:  97%|#########7| 48503/50000 [32:29:12<20:31:13, 49.35s/it]
Outer training loop:  97%|#########7| 48504/50000 [32:29:14<14:34:46, 35.08s/it]
Outer training loop:  97%|#########7| 48505/50000 [32:29:16<10:25:25, 25.10s/it]
Outer training loop:  97%|#########7| 48506/50000 [32:29:18<7:30:57, 18.11s/it] 
Outer training loop:  97%|#########7| 48507/50000 [32:29:20<5:31:14, 13.31s/it]
Outer training loop:  97%|#########7| 48508/50000 [32:29:22<4:05:32,  9.87s/it]
Outer training loop:  97%|#########7| 48509/50000 [32:29:24<3:05:50,  7.48s/it]
Outer training loop:  97%|#########7| 48510/50000 [32:29:26<2:25:22,  5.85s/it]
Outer training loop:  97%|#########7| 48511/50000 [32:29:28<1:55:13,  4.64s/it]
Outer training loop:  97%|#########7| 48512/50000 [32:29:29<1:33:29,  3.77s/it]
Outer training loop:  97%|#########7| 48513/50000 [32:29:31<1:20:43,  3.26s/it]
Outer training loop:  97%|#########7| 48514/50000 [32:29:33<1:09:24,  2.80s/it]
Outer training loop:  97%|#########7| 48515/50000 [32:29:35<1:02:20,  2.52s/it]
Outer training loop:  97%|#########7| 48516/50000 [32:29:37<57:41,  2.33s/it]  
Outer training loop:  97%|#########7| 48517/50000 [32:29:39<55:13,  2.23s/it]
Outer training loop:  97%|#########7| 48518/50000 [32:29:41<52:00,  2.11s/it]
Outer training loop:  97%|#########7| 48519/50000 [32:29:42<50:04,  2.03s/it]
Outer training loop:  97%|#########7| 48520/50000 [32:29:45<50:41,  2.06s/it]
Outer training loop:  97%|#########7| 48521/50000 [32:29:46<49:24,  2.00s/it]
Outer training loop:  97%|#########7| 48522/50000 [32:29:48<47:25,  1.92s/it]
Outer training loop:  97%|#########7| 48523/50000 [32:29:50<49:11,  2.00s/it]
Outer training loop:  97%|#########7| 48524/50000 [32:29:52<48:37,  1.98s/it]
Outer training loop:  97%|#########7| 48525/50000 [32:29:54<46:58,  1.91s/it]
Outer training loop:  97%|#########7| 48526/50000 [32:29:56<46:08,  1.88s/it]
Outer training loop:  97%|#########7| 48527/50000 [32:29:58<47:23,  1.93s/it]
Outer training loop:  97%|#########7| 48528/50000 [32:30:00<46:00,  1.88s/it]
Outer training loop:  97%|#########7| 48529/50000 [32:30:01<45:36,  1.86s/it]
Outer training loop:  97%|#########7| 48530/50000 [32:30:03<45:56,  1.87s/it]
Outer training loop:  97%|#########7| 48531/50000 [32:30:05<46:25,  1.90s/it]
Outer training loop:  97%|#########7| 48532/50000 [32:30:07<45:54,  1.88s/it]
Outer training loop:  97%|#########7| 48533/50000 [32:30:09<47:21,  1.94s/it]
Outer training loop:  97%|#########7| 48534/50000 [32:30:11<45:43,  1.87s/it]
Outer training loop:  97%|#########7| 48535/50000 [32:30:13<46:26,  1.90s/it]
Outer training loop:  97%|#########7| 48536/50000 [32:30:15<45:41,  1.87s/it]
Outer training loop:  97%|#########7| 48537/50000 [32:30:17<46:30,  1.91s/it]
Outer training loop:  97%|#########7| 48538/50000 [32:30:19<46:43,  1.92s/it]
Outer training loop:  97%|#########7| 48539/50000 [32:30:21<46:54,  1.93s/it]
Outer training loop:  97%|#########7| 48540/50000 [32:30:23<47:51,  1.97s/it]
Outer training loop:  97%|#########7| 48541/50000 [32:30:25<49:04,  2.02s/it]
Outer training loop:  97%|#########7| 48542/50000 [32:30:27<48:02,  1.98s/it]
Outer training loop:  97%|#########7| 48543/50000 [32:30:29<46:58,  1.93s/it]
Outer training loop:  97%|#########7| 48544/50000 [32:30:31<48:09,  1.98s/it]
Outer training loop:  97%|#########7| 48545/50000 [32:30:33<47:12,  1.95s/it]
Outer training loop:  97%|#########7| 48546/50000 [32:30:34<46:50,  1.93s/it]
Outer training loop:  97%|#########7| 48547/50000 [32:30:36<47:55,  1.98s/it]
Outer training loop:  97%|#########7| 48548/50000 [32:30:38<47:00,  1.94s/it]
Outer training loop:  97%|#########7| 48549/50000 [32:30:40<46:19,  1.92s/it]
Outer training loop:  97%|#########7| 48550/50000 [32:30:42<46:53,  1.94s/it]
Outer training loop:  97%|#########7| 48551/50000 [32:30:44<46:10,  1.91s/it]
Outer training loop:  97%|#########7| 48552/50000 [32:30:46<45:41,  1.89s/it]
Outer training loop:  97%|#########7| 48553/50000 [32:30:48<47:15,  1.96s/it]
Outer training loop:  97%|#########7| 48554/50000 [32:30:50<47:43,  1.98s/it]
Outer training loop:  97%|#########7| 48555/50000 [32:30:52<46:19,  1.92s/it]
Outer training loop:  97%|#########7| 48556/50000 [32:30:54<45:25,  1.89s/it]
Outer training loop:  97%|#########7| 48557/50000 [32:30:56<47:04,  1.96s/it]
Outer training loop:  97%|#########7| 48558/50000 [32:30:58<46:17,  1.93s/it]
Outer training loop:  97%|#########7| 48559/50000 [32:30:59<45:48,  1.91s/it]
Outer training loop:  97%|#########7| 48560/50000 [32:31:02<47:23,  1.97s/it]
Outer training loop:  97%|#########7| 48561/50000 [32:31:03<45:39,  1.90s/it]
Outer training loop:  97%|#########7| 48562/50000 [32:31:05<46:35,  1.94s/it]
Outer training loop:  97%|#########7| 48563/50000 [32:31:07<46:48,  1.95s/it]
Outer training loop:  97%|#########7| 48564/50000 [32:31:09<46:17,  1.93s/it]
Outer training loop:  97%|#########7| 48565/50000 [32:31:11<45:29,  1.90s/it]
Outer training loop:  97%|#########7| 48566/50000 [32:31:13<45:42,  1.91s/it]
Outer training loop:  97%|#########7| 48567/50000 [32:31:15<46:13,  1.94s/it]
Outer training loop:  97%|#########7| 48568/50000 [32:31:17<45:26,  1.90s/it]
Outer training loop:  97%|#########7| 48569/50000 [32:31:19<44:21,  1.86s/it]
Outer training loop:  97%|#########7| 48570/50000 [32:31:21<44:55,  1.89s/it]
Outer training loop:  97%|#########7| 48571/50000 [32:31:22<44:29,  1.87s/it]
Outer training loop:  97%|#########7| 48572/50000 [32:31:24<44:46,  1.88s/it]
Outer training loop:  97%|#########7| 48573/50000 [32:31:26<46:47,  1.97s/it]
Outer training loop:  97%|#########7| 48574/50000 [32:31:28<46:03,  1.94s/it]
Outer training loop:  97%|#########7| 48575/50000 [32:31:30<45:04,  1.90s/it]
Outer training loop:  97%|#########7| 48576/50000 [32:31:32<45:10,  1.90s/it]
Outer training loop:  97%|#########7| 48577/50000 [32:31:34<46:10,  1.95s/it]
Outer training loop:  97%|#########7| 48578/50000 [32:31:36<45:09,  1.91s/it]
Outer training loop:  97%|#########7| 48579/50000 [32:31:38<43:15,  1.83s/it]
Outer training loop:  97%|#########7| 48580/50000 [32:31:39<43:39,  1.84s/it]
Outer training loop:  97%|#########7| 48581/50000 [32:31:42<45:36,  1.93s/it]
Outer training loop:  97%|#########7| 48582/50000 [32:31:43<44:31,  1.88s/it]
Outer training loop:  97%|#########7| 48583/50000 [32:31:45<43:10,  1.83s/it]
Outer training loop:  97%|#########7| 48584/50000 [32:31:47<42:44,  1.81s/it]
Outer training loop:  97%|#########7| 48585/50000 [32:31:49<44:07,  1.87s/it]
Outer training loop:  97%|#########7| 48586/50000 [32:31:51<43:55,  1.86s/it]
Outer training loop:  97%|#########7| 48587/50000 [32:31:53<44:00,  1.87s/it]
Outer training loop:  97%|#########7| 48588/50000 [32:31:55<45:05,  1.92s/it]
Outer training loop:  97%|#########7| 48589/50000 [32:31:56<43:31,  1.85s/it]
Outer training loop:  97%|#########7| 48590/50000 [32:31:58<43:01,  1.83s/it]
Outer training loop:  97%|#########7| 48591/50000 [32:32:00<44:50,  1.91s/it]
Outer training loop:  97%|#########7| 48592/50000 [32:32:02<45:40,  1.95s/it]
Outer training loop:  97%|#########7| 48593/50000 [32:32:04<44:25,  1.89s/it]
Outer training loop:  97%|#########7| 48594/50000 [32:32:06<44:04,  1.88s/it]
Outer training loop:  97%|#########7| 48595/50000 [32:32:08<45:15,  1.93s/it]
Outer training loop:  97%|#########7| 48596/50000 [32:32:10<45:32,  1.95s/it]
Outer training loop:  97%|#########7| 48597/50000 [32:32:12<44:26,  1.90s/it]
Outer training loop:  97%|#########7| 48598/50000 [32:32:14<44:43,  1.91s/it]
Outer training loop:  97%|#########7| 48599/50000 [32:32:15<44:04,  1.89s/it]
Outer training loop:  97%|#########7| 48600/50000 [32:32:17<43:10,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.79s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  97%|#########7| 48601/50000 [32:32:25<1:24:00,  3.60s/it]
Outer training loop:  97%|#########7| 48602/50000 [32:32:27<1:11:16,  3.06s/it]
Outer training loop:  97%|#########7| 48603/50000 [32:32:29<1:02:58,  2.70s/it]
Outer training loop:  97%|#########7| 48604/50000 [32:32:30<56:56,  2.45s/it]  
Outer training loop:  97%|#########7| 48605/50000 [32:32:32<53:43,  2.31s/it]
Outer training loop:  97%|#########7| 48606/50000 [32:32:34<50:19,  2.17s/it]
Outer training loop:  97%|#########7| 48607/50000 [32:32:36<47:46,  2.06s/it]
Outer training loop:  97%|#########7| 48608/50000 [32:32:38<47:01,  2.03s/it]
Outer training loop:  97%|#########7| 48609/50000 [32:32:40<45:04,  1.94s/it]
Outer training loop:  97%|#########7| 48610/50000 [32:32:41<44:01,  1.90s/it]
Outer training loop:  97%|#########7| 48611/50000 [32:32:44<45:23,  1.96s/it]
Outer training loop:  97%|#########7| 48612/50000 [32:32:45<44:05,  1.91s/it]
Outer training loop:  97%|#########7| 48613/50000 [32:32:47<43:15,  1.87s/it]
Outer training loop:  97%|#########7| 48614/50000 [32:32:49<42:37,  1.85s/it]
Outer training loop:  97%|#########7| 48615/50000 [32:32:51<44:02,  1.91s/it]
Outer training loop:  97%|#########7| 48616/50000 [32:32:53<43:20,  1.88s/it]
Outer training loop:  97%|#########7| 48617/50000 [32:32:55<43:37,  1.89s/it]
Outer training loop:  97%|#########7| 48618/50000 [32:32:57<44:27,  1.93s/it]
Outer training loop:  97%|#########7| 48619/50000 [32:32:59<43:38,  1.90s/it]
Outer training loop:  97%|#########7| 48620/50000 [32:33:00<43:00,  1.87s/it]
Outer training loop:  97%|#########7| 48621/50000 [32:33:02<43:51,  1.91s/it]
Outer training loop:  97%|#########7| 48622/50000 [32:33:04<43:34,  1.90s/it]
Outer training loop:  97%|#########7| 48623/50000 [32:33:06<43:49,  1.91s/it]
Outer training loop:  97%|#########7| 48624/50000 [32:33:08<44:37,  1.95s/it]
Outer training loop:  97%|#########7| 48625/50000 [32:33:10<44:03,  1.92s/it]
Outer training loop:  97%|#########7| 48626/50000 [32:33:12<43:24,  1.90s/it]
Outer training loop:  97%|#########7| 48627/50000 [32:33:14<42:48,  1.87s/it]
Outer training loop:  97%|#########7| 48628/50000 [32:33:16<42:48,  1.87s/it]
Outer training loop:  97%|#########7| 48629/50000 [32:33:17<42:17,  1.85s/it]
Outer training loop:  97%|#########7| 48630/50000 [32:33:19<42:23,  1.86s/it]
Outer training loop:  97%|#########7| 48631/50000 [32:33:21<43:34,  1.91s/it]
Outer training loop:  97%|#########7| 48632/50000 [32:33:23<42:55,  1.88s/it]
Outer training loop:  97%|#########7| 48633/50000 [32:33:25<41:47,  1.83s/it]
Outer training loop:  97%|#########7| 48634/50000 [32:33:27<42:11,  1.85s/it]
Outer training loop:  97%|#########7| 48635/50000 [32:33:29<43:51,  1.93s/it]
Outer training loop:  97%|#########7| 48636/50000 [32:33:31<42:34,  1.87s/it]
Outer training loop:  97%|#########7| 48637/50000 [32:33:32<41:42,  1.84s/it]
Outer training loop:  97%|#########7| 48638/50000 [32:33:35<44:55,  1.98s/it]
Outer training loop:  97%|#########7| 48639/50000 [32:33:37<44:07,  1.95s/it]
Outer training loop:  97%|#########7| 48640/50000 [32:33:38<43:15,  1.91s/it]
Outer training loop:  97%|#########7| 48641/50000 [32:33:40<43:50,  1.94s/it]
Outer training loop:  97%|#########7| 48642/50000 [32:33:42<41:58,  1.85s/it]
Outer training loop:  97%|#########7| 48643/50000 [32:33:44<42:06,  1.86s/it]
Outer training loop:  97%|#########7| 48644/50000 [32:33:46<42:45,  1.89s/it]
Outer training loop:  97%|#########7| 48645/50000 [32:33:48<43:41,  1.93s/it]
Outer training loop:  97%|#########7| 48646/50000 [32:33:50<42:51,  1.90s/it]
Outer training loop:  97%|#########7| 48647/50000 [32:33:52<42:14,  1.87s/it]
Outer training loop:  97%|#########7| 48648/50000 [32:33:54<43:35,  1.93s/it]
Outer training loop:  97%|#########7| 48649/50000 [32:33:55<42:56,  1.91s/it]
Outer training loop:  97%|#########7| 48650/50000 [32:33:57<42:36,  1.89s/it]
Outer training loop:  97%|#########7| 48651/50000 [32:33:59<42:49,  1.90s/it]
Outer training loop:  97%|#########7| 48652/50000 [32:34:01<42:50,  1.91s/it]
Outer training loop:  97%|#########7| 48653/50000 [32:34:03<42:26,  1.89s/it]
Outer training loop:  97%|#########7| 48654/50000 [32:34:05<41:54,  1.87s/it]
Outer training loop:  97%|#########7| 48655/50000 [32:34:07<42:05,  1.88s/it]
Outer training loop:  97%|#########7| 48656/50000 [32:34:09<43:13,  1.93s/it]
Outer training loop:  97%|#########7| 48657/50000 [32:34:11<42:00,  1.88s/it]
Outer training loop:  97%|#########7| 48658/50000 [32:34:12<42:19,  1.89s/it]
Outer training loop:  97%|#########7| 48659/50000 [32:34:14<43:08,  1.93s/it]
Outer training loop:  97%|#########7| 48660/50000 [32:34:16<42:36,  1.91s/it]
Outer training loop:  97%|#########7| 48661/50000 [32:34:18<42:11,  1.89s/it]
Outer training loop:  97%|#########7| 48662/50000 [32:34:20<42:38,  1.91s/it]
Outer training loop:  97%|#########7| 48663/50000 [32:34:22<41:54,  1.88s/it]
Outer training loop:  97%|#########7| 48664/50000 [32:34:24<41:30,  1.86s/it]
Outer training loop:  97%|#########7| 48665/50000 [32:34:26<41:31,  1.87s/it]
Outer training loop:  97%|#########7| 48666/50000 [32:34:28<42:41,  1.92s/it]
Outer training loop:  97%|#########7| 48667/50000 [32:34:30<42:31,  1.91s/it]
Outer training loop:  97%|#########7| 48668/50000 [32:34:31<41:45,  1.88s/it]
Outer training loop:  97%|#########7| 48669/50000 [32:34:33<42:35,  1.92s/it]
Outer training loop:  97%|#########7| 48670/50000 [32:34:35<42:42,  1.93s/it]
Outer training loop:  97%|#########7| 48671/50000 [32:34:37<41:12,  1.86s/it]
Outer training loop:  97%|#########7| 48672/50000 [32:34:39<42:00,  1.90s/it]
Outer training loop:  97%|#########7| 48673/50000 [32:34:41<41:43,  1.89s/it]
Outer training loop:  97%|#########7| 48674/50000 [32:34:43<41:12,  1.87s/it]
Outer training loop:  97%|#########7| 48675/50000 [32:34:44<40:27,  1.83s/it]
Outer training loop:  97%|#########7| 48676/50000 [32:34:46<41:44,  1.89s/it]
Outer training loop:  97%|#########7| 48677/50000 [32:34:48<41:40,  1.89s/it]
Outer training loop:  97%|#########7| 48678/50000 [32:34:50<41:27,  1.88s/it]
Outer training loop:  97%|#########7| 48679/50000 [32:34:52<41:46,  1.90s/it]
Outer training loop:  97%|#########7| 48680/50000 [32:34:54<40:41,  1.85s/it]
Outer training loop:  97%|#########7| 48681/50000 [32:34:56<40:57,  1.86s/it]
Outer training loop:  97%|#########7| 48682/50000 [32:34:58<42:04,  1.92s/it]
Outer training loop:  97%|#########7| 48683/50000 [32:35:00<41:10,  1.88s/it]
Outer training loop:  97%|#########7| 48684/50000 [32:35:01<40:25,  1.84s/it]
Outer training loop:  97%|#########7| 48685/50000 [32:35:03<39:43,  1.81s/it]
Outer training loop:  97%|#########7| 48686/50000 [32:35:05<41:05,  1.88s/it]
Outer training loop:  97%|#########7| 48687/50000 [32:35:07<40:05,  1.83s/it]
Outer training loop:  97%|#########7| 48688/50000 [32:35:09<41:18,  1.89s/it]
Outer training loop:  97%|#########7| 48689/50000 [32:35:11<40:59,  1.88s/it]
Outer training loop:  97%|#########7| 48690/50000 [32:35:13<41:24,  1.90s/it]
Outer training loop:  97%|#########7| 48691/50000 [32:35:14<40:21,  1.85s/it]
Outer training loop:  97%|#########7| 48692/50000 [32:35:16<40:16,  1.85s/it]
Outer training loop:  97%|#########7| 48693/50000 [32:35:18<41:02,  1.88s/it]
Outer training loop:  97%|#########7| 48694/50000 [32:35:20<41:21,  1.90s/it]
Outer training loop:  97%|#########7| 48695/50000 [32:35:22<41:06,  1.89s/it]
Outer training loop:  97%|#########7| 48696/50000 [32:35:24<41:26,  1.91s/it]
Outer training loop:  97%|#########7| 48697/50000 [32:35:26<41:06,  1.89s/it]
Outer training loop:  97%|#########7| 48698/50000 [32:35:28<40:39,  1.87s/it]
Outer training loop:  97%|#########7| 48699/50000 [32:35:30<40:19,  1.86s/it]
Outer training loop:  97%|#########7| 48700/50000 [32:35:32<42:16,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.85s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  97%|#########7| 48701/50000 [32:35:39<1:18:12,  3.61s/it]
Outer training loop:  97%|#########7| 48702/50000 [32:35:41<1:06:37,  3.08s/it]
Outer training loop:  97%|#########7| 48703/50000 [32:35:43<58:50,  2.72s/it]  
Outer training loop:  97%|#########7| 48704/50000 [32:35:45<54:24,  2.52s/it]
Outer training loop:  97%|#########7| 48705/50000 [32:35:47<49:42,  2.30s/it]
Outer training loop:  97%|#########7| 48706/50000 [32:35:49<46:57,  2.18s/it]
Outer training loop:  97%|#########7| 48707/50000 [32:35:50<44:28,  2.06s/it]
Outer training loop:  97%|#########7| 48708/50000 [32:35:52<44:02,  2.05s/it]
Outer training loop:  97%|#########7| 48709/50000 [32:35:54<42:38,  1.98s/it]
Outer training loop:  97%|#########7| 48710/50000 [32:35:56<41:15,  1.92s/it]
Outer training loop:  97%|#########7| 48711/50000 [32:35:58<42:09,  1.96s/it]
Outer training loop:  97%|#########7| 48712/50000 [32:36:00<41:08,  1.92s/it]
Outer training loop:  97%|#########7| 48713/50000 [32:36:02<40:48,  1.90s/it]
Outer training loop:  97%|#########7| 48714/50000 [32:36:04<41:32,  1.94s/it]
Outer training loop:  97%|#########7| 48715/50000 [32:36:06<41:19,  1.93s/it]
Outer training loop:  97%|#########7| 48716/50000 [32:36:08<40:35,  1.90s/it]
Outer training loop:  97%|#########7| 48717/50000 [32:36:09<40:59,  1.92s/it]
Outer training loop:  97%|#########7| 48718/50000 [32:36:11<40:39,  1.90s/it]
Outer training loop:  97%|#########7| 48719/50000 [32:36:13<39:31,  1.85s/it]
Outer training loop:  97%|#########7| 48720/50000 [32:36:15<40:43,  1.91s/it]
Outer training loop:  97%|#########7| 48721/50000 [32:36:17<39:47,  1.87s/it]
Outer training loop:  97%|#########7| 48722/50000 [32:36:19<38:57,  1.83s/it]
Outer training loop:  97%|#########7| 48723/50000 [32:36:21<39:15,  1.84s/it]
Outer training loop:  97%|#########7| 48724/50000 [32:36:23<40:13,  1.89s/it]
Outer training loop:  97%|#########7| 48725/50000 [32:36:24<39:22,  1.85s/it]
Outer training loop:  97%|#########7| 48726/50000 [32:36:26<39:54,  1.88s/it]
Outer training loop:  97%|#########7| 48727/50000 [32:36:28<40:03,  1.89s/it]
Outer training loop:  97%|#########7| 48728/50000 [32:36:30<40:25,  1.91s/it]
Outer training loop:  97%|#########7| 48729/50000 [32:36:32<40:06,  1.89s/it]
Outer training loop:  97%|#########7| 48730/50000 [32:36:34<39:56,  1.89s/it]
Outer training loop:  97%|#########7| 48731/50000 [32:36:36<41:36,  1.97s/it]
Outer training loop:  97%|#########7| 48732/50000 [32:36:38<40:38,  1.92s/it]
Outer training loop:  97%|#########7| 48733/50000 [32:36:40<40:38,  1.92s/it]
Outer training loop:  97%|#########7| 48734/50000 [32:36:42<40:05,  1.90s/it]
Outer training loop:  97%|#########7| 48735/50000 [32:36:44<42:20,  2.01s/it]
Outer training loop:  97%|#########7| 48736/50000 [32:36:46<41:21,  1.96s/it]
Outer training loop:  97%|#########7| 48737/50000 [32:36:47<40:21,  1.92s/it]
Outer training loop:  97%|#########7| 48738/50000 [32:36:50<41:21,  1.97s/it]
Outer training loop:  97%|#########7| 48739/50000 [32:36:52<41:09,  1.96s/it]
Outer training loop:  97%|#########7| 48740/50000 [32:36:53<41:13,  1.96s/it]
Outer training loop:  97%|#########7| 48741/50000 [32:36:56<41:36,  1.98s/it]
Outer training loop:  97%|#########7| 48742/50000 [32:36:57<40:13,  1.92s/it]
Outer training loop:  97%|#########7| 48743/50000 [32:36:59<39:22,  1.88s/it]
Outer training loop:  97%|#########7| 48744/50000 [32:37:01<38:36,  1.84s/it]
Outer training loop:  97%|#########7| 48745/50000 [32:37:03<40:08,  1.92s/it]
Outer training loop:  97%|#########7| 48746/50000 [32:37:05<39:58,  1.91s/it]
Outer training loop:  97%|#########7| 48747/50000 [32:37:07<39:13,  1.88s/it]
Outer training loop:  97%|#########7| 48748/50000 [32:37:08<38:29,  1.84s/it]
Outer training loop:  97%|#########7| 48749/50000 [32:37:10<39:06,  1.88s/it]
Outer training loop:  98%|#########7| 48750/50000 [32:37:12<38:30,  1.85s/it]
Outer training loop:  98%|#########7| 48751/50000 [32:37:14<39:33,  1.90s/it]
Outer training loop:  98%|#########7| 48752/50000 [32:37:16<40:10,  1.93s/it]
Outer training loop:  98%|#########7| 48753/50000 [32:37:18<39:46,  1.91s/it]
Outer training loop:  98%|#########7| 48754/50000 [32:37:20<38:39,  1.86s/it]
Outer training loop:  98%|#########7| 48755/50000 [32:37:22<39:26,  1.90s/it]
Outer training loop:  98%|#########7| 48756/50000 [32:37:24<38:41,  1.87s/it]
Outer training loop:  98%|#########7| 48757/50000 [32:37:25<38:37,  1.86s/it]
Outer training loop:  98%|#########7| 48758/50000 [32:37:27<39:29,  1.91s/it]
Outer training loop:  98%|#########7| 48759/50000 [32:37:29<39:22,  1.90s/it]
Outer training loop:  98%|#########7| 48760/50000 [32:37:31<38:38,  1.87s/it]
Outer training loop:  98%|#########7| 48761/50000 [32:37:33<38:20,  1.86s/it]
Outer training loop:  98%|#########7| 48762/50000 [32:37:35<39:20,  1.91s/it]
Outer training loop:  98%|#########7| 48763/50000 [32:37:37<38:48,  1.88s/it]
Outer training loop:  98%|#########7| 48764/50000 [32:37:39<38:24,  1.86s/it]
Outer training loop:  98%|#########7| 48765/50000 [32:37:41<39:44,  1.93s/it]
Outer training loop:  98%|#########7| 48766/50000 [32:37:43<39:09,  1.90s/it]
Outer training loop:  98%|#########7| 48767/50000 [32:37:44<38:57,  1.90s/it]
Outer training loop:  98%|#########7| 48768/50000 [32:37:46<38:06,  1.86s/it]
Outer training loop:  98%|#########7| 48769/50000 [32:37:48<39:30,  1.93s/it]
Outer training loop:  98%|#########7| 48770/50000 [32:37:50<39:11,  1.91s/it]
Outer training loop:  98%|#########7| 48771/50000 [32:37:52<38:36,  1.88s/it]
Outer training loop:  98%|#########7| 48772/50000 [32:37:54<38:57,  1.90s/it]
Outer training loop:  98%|#########7| 48773/50000 [32:37:56<38:51,  1.90s/it]
Outer training loop:  98%|#########7| 48774/50000 [32:37:58<38:22,  1.88s/it]
Outer training loop:  98%|#########7| 48775/50000 [32:37:59<38:01,  1.86s/it]
Outer training loop:  98%|#########7| 48776/50000 [32:38:01<39:07,  1.92s/it]
Outer training loop:  98%|#########7| 48777/50000 [32:38:03<38:11,  1.87s/it]
Outer training loop:  98%|#########7| 48778/50000 [32:38:05<38:35,  1.89s/it]
Outer training loop:  98%|#########7| 48779/50000 [32:38:07<38:29,  1.89s/it]
Outer training loop:  98%|#########7| 48780/50000 [32:38:09<38:42,  1.90s/it]
Outer training loop:  98%|#########7| 48781/50000 [32:38:11<38:55,  1.92s/it]
Outer training loop:  98%|#########7| 48782/50000 [32:38:13<37:55,  1.87s/it]
Outer training loop:  98%|#########7| 48783/50000 [32:38:15<37:49,  1.86s/it]
Outer training loop:  98%|#########7| 48784/50000 [32:38:16<37:12,  1.84s/it]
Outer training loop:  98%|#########7| 48785/50000 [32:38:18<37:53,  1.87s/it]
Outer training loop:  98%|#########7| 48786/50000 [32:38:20<37:59,  1.88s/it]
Outer training loop:  98%|#########7| 48787/50000 [32:38:22<38:27,  1.90s/it]
Outer training loop:  98%|#########7| 48788/50000 [32:38:24<37:44,  1.87s/it]
Outer training loop:  98%|#########7| 48789/50000 [32:38:26<37:00,  1.83s/it]
Outer training loop:  98%|#########7| 48790/50000 [32:38:28<38:11,  1.89s/it]
Outer training loop:  98%|#########7| 48791/50000 [32:38:30<37:55,  1.88s/it]
Outer training loop:  98%|#########7| 48792/50000 [32:38:31<37:23,  1.86s/it]
Outer training loop:  98%|#########7| 48793/50000 [32:38:33<37:51,  1.88s/it]
Outer training loop:  98%|#########7| 48794/50000 [32:38:35<39:00,  1.94s/it]
Outer training loop:  98%|#########7| 48795/50000 [32:38:37<38:35,  1.92s/it]
Outer training loop:  98%|#########7| 48796/50000 [32:38:39<37:36,  1.87s/it]
Outer training loop:  98%|#########7| 48797/50000 [32:38:41<39:04,  1.95s/it]
Outer training loop:  98%|#########7| 48798/50000 [32:38:43<38:50,  1.94s/it]
Outer training loop:  98%|#########7| 48799/50000 [32:38:45<38:21,  1.92s/it]
Outer training loop:  98%|#########7| 48800/50000 [32:38:47<37:33,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.89s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.87s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]

Outer training loop:  98%|#########7| 48801/50000 [32:38:54<1:11:46,  3.59s/it]
Outer training loop:  98%|#########7| 48802/50000 [32:38:56<1:01:27,  3.08s/it]
Outer training loop:  98%|#########7| 48803/50000 [32:38:58<54:15,  2.72s/it]  
Outer training loop:  98%|#########7| 48804/50000 [32:39:00<49:04,  2.46s/it]
Outer training loop:  98%|#########7| 48805/50000 [32:39:02<46:33,  2.34s/it]
Outer training loop:  98%|#########7| 48806/50000 [32:39:04<44:06,  2.22s/it]
Outer training loop:  98%|#########7| 48807/50000 [32:39:06<42:11,  2.12s/it]
Outer training loop:  98%|#########7| 48808/50000 [32:39:08<41:40,  2.10s/it]
Outer training loop:  98%|#########7| 48809/50000 [32:39:10<40:16,  2.03s/it]
Outer training loop:  98%|#########7| 48810/50000 [32:39:12<38:59,  1.97s/it]
Outer training loop:  98%|#########7| 48811/50000 [32:39:13<38:27,  1.94s/it]
Outer training loop:  98%|#########7| 48812/50000 [32:39:15<38:38,  1.95s/it]
Outer training loop:  98%|#########7| 48813/50000 [32:39:17<37:27,  1.89s/it]
Outer training loop:  98%|#########7| 48814/50000 [32:39:19<37:26,  1.89s/it]
Outer training loop:  98%|#########7| 48815/50000 [32:39:21<38:25,  1.95s/it]
Outer training loop:  98%|#########7| 48816/50000 [32:39:23<37:28,  1.90s/it]
Outer training loop:  98%|#########7| 48817/50000 [32:39:25<37:36,  1.91s/it]
Outer training loop:  98%|#########7| 48818/50000 [32:39:27<37:09,  1.89s/it]
Outer training loop:  98%|#########7| 48819/50000 [32:39:29<38:16,  1.94s/it]
Outer training loop:  98%|#########7| 48820/50000 [32:39:31<37:29,  1.91s/it]
Outer training loop:  98%|#########7| 48821/50000 [32:39:32<37:16,  1.90s/it]
Outer training loop:  98%|#########7| 48822/50000 [32:39:34<37:36,  1.92s/it]
Outer training loop:  98%|#########7| 48823/50000 [32:39:36<37:18,  1.90s/it]
Outer training loop:  98%|#########7| 48824/50000 [32:39:38<36:47,  1.88s/it]
Outer training loop:  98%|#########7| 48825/50000 [32:39:40<36:52,  1.88s/it]
Outer training loop:  98%|#########7| 48826/50000 [32:39:42<37:07,  1.90s/it]
Outer training loop:  98%|#########7| 48827/50000 [32:39:44<36:59,  1.89s/it]
Outer training loop:  98%|#########7| 48828/50000 [32:39:46<36:38,  1.88s/it]
Outer training loop:  98%|#########7| 48829/50000 [32:39:48<38:23,  1.97s/it]
Outer training loop:  98%|#########7| 48830/50000 [32:39:50<38:25,  1.97s/it]
Outer training loop:  98%|#########7| 48831/50000 [32:39:52<37:43,  1.94s/it]
Outer training loop:  98%|#########7| 48832/50000 [32:39:54<38:29,  1.98s/it]
Outer training loop:  98%|#########7| 48833/50000 [32:39:56<38:44,  1.99s/it]
Outer training loop:  98%|#########7| 48834/50000 [32:39:58<38:04,  1.96s/it]
Outer training loop:  98%|#########7| 48835/50000 [32:39:59<36:47,  1.90s/it]
Outer training loop:  98%|#########7| 48836/50000 [32:40:02<38:09,  1.97s/it]
Outer training loop:  98%|#########7| 48837/50000 [32:40:03<38:07,  1.97s/it]
Outer training loop:  98%|#########7| 48838/50000 [32:40:05<38:08,  1.97s/it]
Outer training loop:  98%|#########7| 48839/50000 [32:40:08<39:01,  2.02s/it]
Outer training loop:  98%|#########7| 48840/50000 [32:40:09<37:24,  1.94s/it]
Outer training loop:  98%|#########7| 48841/50000 [32:40:11<37:02,  1.92s/it]
Outer training loop:  98%|#########7| 48842/50000 [32:40:13<37:42,  1.95s/it]
Outer training loop:  98%|#########7| 48843/50000 [32:40:15<37:03,  1.92s/it]
Outer training loop:  98%|#########7| 48844/50000 [32:40:17<36:27,  1.89s/it]
Outer training loop:  98%|#########7| 48845/50000 [32:40:19<37:06,  1.93s/it]
Outer training loop:  98%|#########7| 48846/50000 [32:40:21<37:17,  1.94s/it]
Outer training loop:  98%|#########7| 48847/50000 [32:40:23<36:28,  1.90s/it]
Outer training loop:  98%|#########7| 48848/50000 [32:40:25<37:38,  1.96s/it]
Outer training loop:  98%|#########7| 48849/50000 [32:40:27<36:28,  1.90s/it]
Outer training loop:  98%|#########7| 48850/50000 [32:40:28<36:28,  1.90s/it]
Outer training loop:  98%|#########7| 48851/50000 [32:40:30<36:50,  1.92s/it]
Outer training loop:  98%|#########7| 48852/50000 [32:40:32<36:01,  1.88s/it]
Outer training loop:  98%|#########7| 48853/50000 [32:40:34<35:52,  1.88s/it]
Outer training loop:  98%|#########7| 48854/50000 [32:40:36<37:08,  1.94s/it]
Outer training loop:  98%|#########7| 48855/50000 [32:40:38<36:42,  1.92s/it]
Outer training loop:  98%|#########7| 48856/50000 [32:40:40<36:04,  1.89s/it]
Outer training loop:  98%|#########7| 48857/50000 [32:40:42<35:36,  1.87s/it]
Outer training loop:  98%|#########7| 48858/50000 [32:40:44<36:41,  1.93s/it]
Outer training loop:  98%|#########7| 48859/50000 [32:40:45<35:24,  1.86s/it]
Outer training loop:  98%|#########7| 48860/50000 [32:40:47<35:19,  1.86s/it]
Outer training loop:  98%|#########7| 48861/50000 [32:40:49<35:10,  1.85s/it]
Outer training loop:  98%|#########7| 48862/50000 [32:40:51<36:13,  1.91s/it]
Outer training loop:  98%|#########7| 48863/50000 [32:40:53<35:31,  1.88s/it]
Outer training loop:  98%|#########7| 48864/50000 [32:40:55<34:47,  1.84s/it]
Outer training loop:  98%|#########7| 48865/50000 [32:40:57<36:12,  1.91s/it]
Outer training loop:  98%|#########7| 48866/50000 [32:40:59<35:18,  1.87s/it]
Outer training loop:  98%|#########7| 48867/50000 [32:41:00<34:47,  1.84s/it]
Outer training loop:  98%|#########7| 48868/50000 [32:41:02<35:54,  1.90s/it]
Outer training loop:  98%|#########7| 48869/50000 [32:41:04<35:47,  1.90s/it]
Outer training loop:  98%|#########7| 48870/50000 [32:41:06<35:16,  1.87s/it]
Outer training loop:  98%|#########7| 48871/50000 [32:41:08<34:56,  1.86s/it]
Outer training loop:  98%|#########7| 48872/50000 [32:41:10<36:11,  1.92s/it]
Outer training loop:  98%|#########7| 48873/50000 [32:41:12<35:43,  1.90s/it]
Outer training loop:  98%|#########7| 48874/50000 [32:41:14<35:26,  1.89s/it]
Outer training loop:  98%|#########7| 48875/50000 [32:41:16<36:04,  1.92s/it]
Outer training loop:  98%|#########7| 48876/50000 [32:41:18<35:13,  1.88s/it]
Outer training loop:  98%|#########7| 48877/50000 [32:41:19<34:29,  1.84s/it]
Outer training loop:  98%|#########7| 48878/50000 [32:41:21<34:53,  1.87s/it]
Outer training loop:  98%|#########7| 48879/50000 [32:41:23<34:16,  1.83s/it]
Outer training loop:  98%|#########7| 48880/50000 [32:41:25<34:23,  1.84s/it]
Outer training loop:  98%|#########7| 48881/50000 [32:41:27<34:40,  1.86s/it]
Outer training loop:  98%|#########7| 48882/50000 [32:41:29<36:37,  1.97s/it]
Outer training loop:  98%|#########7| 48883/50000 [32:41:31<36:01,  1.94s/it]
Outer training loop:  98%|#########7| 48884/50000 [32:41:33<35:38,  1.92s/it]
Outer training loop:  98%|#########7| 48885/50000 [32:41:34<34:37,  1.86s/it]
Outer training loop:  98%|#########7| 48886/50000 [32:41:36<35:39,  1.92s/it]
Outer training loop:  98%|#########7| 48887/50000 [32:41:38<35:32,  1.92s/it]
Outer training loop:  98%|#########7| 48888/50000 [32:41:40<35:07,  1.90s/it]
Outer training loop:  98%|#########7| 48889/50000 [32:41:42<36:32,  1.97s/it]
Outer training loop:  98%|#########7| 48890/50000 [32:41:44<35:56,  1.94s/it]
Outer training loop:  98%|#########7| 48891/50000 [32:41:46<35:07,  1.90s/it]
Outer training loop:  98%|#########7| 48892/50000 [32:41:48<35:20,  1.91s/it]
Outer training loop:  98%|#########7| 48893/50000 [32:41:50<34:29,  1.87s/it]
Outer training loop:  98%|#########7| 48894/50000 [32:41:52<34:36,  1.88s/it]
Outer training loop:  98%|#########7| 48895/50000 [32:41:54<35:12,  1.91s/it]
Outer training loop:  98%|#########7| 48896/50000 [32:41:56<34:56,  1.90s/it]
Outer training loop:  98%|#########7| 48897/50000 [32:41:57<34:31,  1.88s/it]
Outer training loop:  98%|#########7| 48898/50000 [32:41:59<33:56,  1.85s/it]
Outer training loop:  98%|#########7| 48899/50000 [32:42:01<35:11,  1.92s/it]
Outer training loop:  98%|#########7| 48900/50000 [32:42:03<34:35,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.72s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.79s/it]

Outer training loop:  98%|#########7| 48901/50000 [32:42:10<1:04:23,  3.52s/it]
Outer training loop:  98%|#########7| 48902/50000 [32:42:12<54:48,  2.99s/it]  
Outer training loop:  98%|#########7| 48903/50000 [32:42:14<49:26,  2.70s/it]
Outer training loop:  98%|#########7| 48904/50000 [32:42:16<43:47,  2.40s/it]
Outer training loop:  98%|#########7| 48905/50000 [32:42:18<40:06,  2.20s/it]
Outer training loop:  98%|#########7| 48906/50000 [32:42:19<38:01,  2.09s/it]
Outer training loop:  98%|#########7| 48907/50000 [32:42:21<37:37,  2.07s/it]
Outer training loop:  98%|#########7| 48908/50000 [32:42:23<35:25,  1.95s/it]
Outer training loop:  98%|#########7| 48909/50000 [32:42:25<34:03,  1.87s/it]
Outer training loop:  98%|#########7| 48910/50000 [32:42:27<34:56,  1.92s/it]
Outer training loop:  98%|#########7| 48911/50000 [32:42:29<34:25,  1.90s/it]
Outer training loop:  98%|#########7| 48912/50000 [32:42:30<33:40,  1.86s/it]
Outer training loop:  98%|#########7| 48913/50000 [32:42:32<33:11,  1.83s/it]
Outer training loop:  98%|#########7| 48914/50000 [32:42:34<34:28,  1.90s/it]
Outer training loop:  98%|#########7| 48915/50000 [32:42:36<34:56,  1.93s/it]
Outer training loop:  98%|#########7| 48916/50000 [32:42:38<34:29,  1.91s/it]
Outer training loop:  98%|#########7| 48917/50000 [32:42:40<34:42,  1.92s/it]
Outer training loop:  98%|#########7| 48918/50000 [32:42:42<34:09,  1.89s/it]
Outer training loop:  98%|#########7| 48919/50000 [32:42:44<33:44,  1.87s/it]
Outer training loop:  98%|#########7| 48920/50000 [32:42:46<34:22,  1.91s/it]
Outer training loop:  98%|#########7| 48921/50000 [32:42:47<33:05,  1.84s/it]
Outer training loop:  98%|#########7| 48922/50000 [32:42:49<33:06,  1.84s/it]
Outer training loop:  98%|#########7| 48923/50000 [32:42:51<33:29,  1.87s/it]
Outer training loop:  98%|#########7| 48924/50000 [32:42:53<33:56,  1.89s/it]
Outer training loop:  98%|#########7| 48925/50000 [32:42:55<34:14,  1.91s/it]
Outer training loop:  98%|#########7| 48926/50000 [32:42:57<34:02,  1.90s/it]
Outer training loop:  98%|#########7| 48927/50000 [32:42:59<35:02,  1.96s/it]
Outer training loop:  98%|#########7| 48928/50000 [32:43:01<34:17,  1.92s/it]
Outer training loop:  98%|#########7| 48929/50000 [32:43:03<35:11,  1.97s/it]
Outer training loop:  98%|#########7| 48930/50000 [32:43:05<35:15,  1.98s/it]
Outer training loop:  98%|#########7| 48931/50000 [32:43:07<35:52,  2.01s/it]
Outer training loop:  98%|#########7| 48932/50000 [32:43:09<34:42,  1.95s/it]
Outer training loop:  98%|#########7| 48933/50000 [32:43:11<34:20,  1.93s/it]
Outer training loop:  98%|#########7| 48934/50000 [32:43:13<34:45,  1.96s/it]
Outer training loop:  98%|#########7| 48935/50000 [32:43:15<34:31,  1.95s/it]
Outer training loop:  98%|#########7| 48936/50000 [32:43:16<33:23,  1.88s/it]
Outer training loop:  98%|#########7| 48937/50000 [32:43:18<33:30,  1.89s/it]
Outer training loop:  98%|#########7| 48938/50000 [32:43:20<33:42,  1.90s/it]
Outer training loop:  98%|#########7| 48939/50000 [32:43:22<33:18,  1.88s/it]
Outer training loop:  98%|#########7| 48940/50000 [32:43:24<32:35,  1.84s/it]
Outer training loop:  98%|#########7| 48941/50000 [32:43:26<34:37,  1.96s/it]
Outer training loop:  98%|#########7| 48942/50000 [32:43:28<34:17,  1.95s/it]
Outer training loop:  98%|#########7| 48943/50000 [32:43:30<33:45,  1.92s/it]
Outer training loop:  98%|#########7| 48944/50000 [32:43:32<34:06,  1.94s/it]
Outer training loop:  98%|#########7| 48945/50000 [32:43:34<33:39,  1.91s/it]
Outer training loop:  98%|#########7| 48946/50000 [32:43:36<33:33,  1.91s/it]
Outer training loop:  98%|#########7| 48947/50000 [32:43:37<33:13,  1.89s/it]
Outer training loop:  98%|#########7| 48948/50000 [32:43:39<33:40,  1.92s/it]
Outer training loop:  98%|#########7| 48949/50000 [32:43:41<32:55,  1.88s/it]
Outer training loop:  98%|#########7| 48950/50000 [32:43:43<32:37,  1.86s/it]
Outer training loop:  98%|#########7| 48951/50000 [32:43:45<33:26,  1.91s/it]
Outer training loop:  98%|#########7| 48952/50000 [32:43:47<32:48,  1.88s/it]
Outer training loop:  98%|#########7| 48953/50000 [32:43:49<32:44,  1.88s/it]
Outer training loop:  98%|#########7| 48954/50000 [32:43:51<32:34,  1.87s/it]
Outer training loop:  98%|#########7| 48955/50000 [32:43:53<32:55,  1.89s/it]
Outer training loop:  98%|#########7| 48956/50000 [32:43:54<31:55,  1.84s/it]
Outer training loop:  98%|#########7| 48957/50000 [32:43:56<32:14,  1.85s/it]
Outer training loop:  98%|#########7| 48958/50000 [32:43:58<31:48,  1.83s/it]
Outer training loop:  98%|#########7| 48959/50000 [32:44:00<32:33,  1.88s/it]
Outer training loop:  98%|#########7| 48960/50000 [32:44:02<32:26,  1.87s/it]
Outer training loop:  98%|#########7| 48961/50000 [32:44:03<31:34,  1.82s/it]
Outer training loop:  98%|#########7| 48962/50000 [32:44:05<32:24,  1.87s/it]
Outer training loop:  98%|#########7| 48963/50000 [32:44:07<31:54,  1.85s/it]
Outer training loop:  98%|#########7| 48964/50000 [32:44:09<31:13,  1.81s/it]
Outer training loop:  98%|#########7| 48965/50000 [32:44:11<32:34,  1.89s/it]
Outer training loop:  98%|#########7| 48966/50000 [32:44:13<32:33,  1.89s/it]
Outer training loop:  98%|#########7| 48967/50000 [32:44:15<32:07,  1.87s/it]
Outer training loop:  98%|#########7| 48968/50000 [32:44:17<31:50,  1.85s/it]
Outer training loop:  98%|#########7| 48969/50000 [32:44:19<33:05,  1.93s/it]
Outer training loop:  98%|#########7| 48970/50000 [32:44:20<32:25,  1.89s/it]
Outer training loop:  98%|#########7| 48971/50000 [32:44:22<32:30,  1.90s/it]
Outer training loop:  98%|#########7| 48972/50000 [32:44:24<33:04,  1.93s/it]
Outer training loop:  98%|#########7| 48973/50000 [32:44:26<33:16,  1.94s/it]
Outer training loop:  98%|#########7| 48974/50000 [32:44:28<32:33,  1.90s/it]
Outer training loop:  98%|#########7| 48975/50000 [32:44:30<33:28,  1.96s/it]
Outer training loop:  98%|#########7| 48976/50000 [32:44:32<32:39,  1.91s/it]
Outer training loop:  98%|#########7| 48977/50000 [32:44:34<32:24,  1.90s/it]
Outer training loop:  98%|#########7| 48978/50000 [32:44:36<33:08,  1.95s/it]
Outer training loop:  98%|#########7| 48979/50000 [32:44:38<33:11,  1.95s/it]
Outer training loop:  98%|#########7| 48980/50000 [32:44:40<32:51,  1.93s/it]
Outer training loop:  98%|#########7| 48981/50000 [32:44:42<32:10,  1.89s/it]
Outer training loop:  98%|#########7| 48982/50000 [32:44:44<32:31,  1.92s/it]
Outer training loop:  98%|#########7| 48983/50000 [32:44:45<31:55,  1.88s/it]
Outer training loop:  98%|#########7| 48984/50000 [32:44:47<31:42,  1.87s/it]
Outer training loop:  98%|#########7| 48985/50000 [32:44:49<33:04,  1.96s/it]
Outer training loop:  98%|#########7| 48986/50000 [32:44:51<32:10,  1.90s/it]
Outer training loop:  98%|#########7| 48987/50000 [32:44:53<31:46,  1.88s/it]
Outer training loop:  98%|#########7| 48988/50000 [32:44:55<32:21,  1.92s/it]
Outer training loop:  98%|#########7| 48989/50000 [32:44:57<31:53,  1.89s/it]
Outer training loop:  98%|#########7| 48990/50000 [32:44:59<31:08,  1.85s/it]
Outer training loop:  98%|#########7| 48991/50000 [32:45:00<30:42,  1.83s/it]
Outer training loop:  98%|#########7| 48992/50000 [32:45:02<31:27,  1.87s/it]
Outer training loop:  98%|#########7| 48993/50000 [32:45:04<31:26,  1.87s/it]
Outer training loop:  98%|#########7| 48994/50000 [32:45:06<30:55,  1.84s/it]
Outer training loop:  98%|#########7| 48995/50000 [32:45:08<31:54,  1.91s/it]
Outer training loop:  98%|#########7| 48996/50000 [32:45:10<31:46,  1.90s/it]
Outer training loop:  98%|#########7| 48997/50000 [32:45:12<31:35,  1.89s/it]
Outer training loop:  98%|#########7| 48998/50000 [32:45:14<31:28,  1.88s/it]
Outer training loop:  98%|#########7| 48999/50000 [32:45:16<32:20,  1.94s/it]
Outer training loop:  98%|#########8| 49000/50000 [32:45:18<31:47,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:02<00:04,  2.00s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.97s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.87s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:27,  1.80s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:29,  1.85s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:28,  1.85s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:25,  1.84s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:23,  1.84s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:26,  1.91s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:22,  1.87s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:20,  1.87s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:24,  1.95s/it]

Evaluating policy:  12%|#2        | 10/83 [00:18<02:19,  1.91s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:17,  1.91s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:13,  1.89s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:17,  1.96s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:12,  1.91s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:04,  1.86s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.85s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:59,  1.87s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.88s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:53,  1.84s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:57,  1.93s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:54,  1.90s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:50,  1.88s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:49,  1.92s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:47,  1.91s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:47,  1.96s/it]

Evaluating policy:  35%|###4      | 29/83 [00:55<01:43,  1.92s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:40,  1.89s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:37,  1.88s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:35,  1.88s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:38,  1.97s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:33,  1.92s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:29,  1.85s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:28,  1.88s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.87s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:14<01:25,  1.95s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:20,  1.88s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.87s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:17,  1.88s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:14,  1.87s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:10,  1.81s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:08,  1.81s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:11,  1.92s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:07,  1.88s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.84s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<00:59,  1.80s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:57,  1.81s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<00:55,  1.80s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:57,  1.93s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:54,  1.87s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.82s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:49,  1.82s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.84s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:43,  1.82s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:52<00:41,  1.82s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:54<00:43,  1.98s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:40,  1.93s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:35,  1.85s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:30,  1.77s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:28,  1.76s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:26,  1.79s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:24,  1.77s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:25,  1.96s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.88s/it]

Evaluating policy:  87%|########6 | 72/83 [02:14<00:20,  1.86s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:18,  1.82s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.83s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:15,  1.92s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:13,  1.89s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.87s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.88s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:08,  2.07s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:06,  2.03s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.97s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.87s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:22,  1.76s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:21,  1.77s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:21,  1.79s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:33,  1.96s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:29,  1.94s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:26,  1.93s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:21,  1.88s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.93s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:20,  1.92s/it]

Evaluating policy:  13%|#3        | 11/83 [00:20<02:15,  1.89s/it]

Evaluating policy:  14%|#4        | 12/83 [00:22<02:12,  1.86s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.88s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:05,  1.88s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:02,  1.86s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:06,  1.94s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<02:01,  1.90s/it]

Evaluating policy:  24%|##4       | 20/83 [00:37<01:58,  1.87s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:54,  1.85s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:57,  1.92s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:55,  1.92s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:51,  1.88s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:48,  1.86s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:47,  1.88s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:49,  1.95s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:45,  1.92s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:41,  1.88s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:39,  1.87s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:36,  1.86s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:33,  1.83s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:36,  1.94s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:33,  1.91s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:31,  1.91s/it]

Evaluating policy:  43%|####3     | 36/83 [01:08<01:29,  1.91s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:26,  1.88s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:23,  1.85s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:23,  1.91s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:21,  1.89s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.87s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:19,  1.95s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:15,  1.90s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:12,  1.86s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:16,  2.01s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:12,  1.97s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:29<01:09,  1.93s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:31<01:05,  1.88s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:03,  1.87s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:01,  1.87s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<00:59,  1.85s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<01:00,  1.96s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:57,  1.92s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:54,  1.88s/it]

Evaluating policy:  66%|######6   | 55/83 [01:44<00:52,  1.87s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:48,  1.85s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:46,  1.86s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:43,  1.83s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:45,  1.97s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:41,  1.90s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.89s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.87s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:35,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.82s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:04<00:31,  1.83s/it]

Evaluating policy:  81%|########  | 67/83 [02:06<00:29,  1.84s/it]

Evaluating policy:  82%|########1 | 68/83 [02:08<00:29,  1.98s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:27,  1.96s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.92s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.90s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.85s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:19<00:16,  1.82s/it]

Evaluating policy:  90%|######### | 75/83 [02:21<00:14,  1.83s/it]

Evaluating policy:  92%|#########1| 76/83 [02:23<00:12,  1.82s/it]

Evaluating policy:  93%|#########2| 77/83 [02:25<00:10,  1.81s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.95s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.90s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.89s/it]

Evaluating policy:  98%|#########7| 81/83 [02:32<00:03,  1.86s/it]

Evaluating policy:  99%|#########8| 82/83 [02:34<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:36<00:00,  1.85s/it]

Outer training loop:  98%|#########8| 49001/50000 [32:50:38<27:01:54, 97.41s/it]
Outer training loop:  98%|#########8| 49002/50000 [32:50:40<19:02:57, 68.72s/it]
Outer training loop:  98%|#########8| 49003/50000 [32:50:41<13:28:35, 48.66s/it]
Outer training loop:  98%|#########8| 49004/50000 [32:50:43<9:34:16, 34.59s/it] 
Outer training loop:  98%|#########8| 49005/50000 [32:50:45<6:50:24, 24.75s/it]
Outer training loop:  98%|#########8| 49006/50000 [32:50:47<4:58:24, 18.01s/it]
Outer training loop:  98%|#########8| 49007/50000 [32:50:49<3:37:32, 13.14s/it]
Outer training loop:  98%|#########8| 49008/50000 [32:50:51<2:41:30,  9.77s/it]
Outer training loop:  98%|#########8| 49009/50000 [32:50:53<2:01:59,  7.39s/it]
Outer training loop:  98%|#########8| 49010/50000 [32:50:55<1:36:20,  5.84s/it]
Outer training loop:  98%|#########8| 49011/50000 [32:50:57<1:15:42,  4.59s/it]
Outer training loop:  98%|#########8| 49012/50000 [32:50:58<1:01:38,  3.74s/it]
Outer training loop:  98%|#########8| 49013/50000 [32:51:00<52:47,  3.21s/it]  
Outer training loop:  98%|#########8| 49014/50000 [32:51:02<45:42,  2.78s/it]
Outer training loop:  98%|#########8| 49015/50000 [32:51:04<41:54,  2.55s/it]
Outer training loop:  98%|#########8| 49016/50000 [32:51:06<39:49,  2.43s/it]
Outer training loop:  98%|#########8| 49017/50000 [32:51:08<36:54,  2.25s/it]
Outer training loop:  98%|#########8| 49018/50000 [32:51:10<34:36,  2.11s/it]
Outer training loop:  98%|#########8| 49019/50000 [32:51:12<32:59,  2.02s/it]
Outer training loop:  98%|#########8| 49020/50000 [32:51:14<32:51,  2.01s/it]
Outer training loop:  98%|#########8| 49021/50000 [32:51:16<31:33,  1.93s/it]
Outer training loop:  98%|#########8| 49022/50000 [32:51:17<30:35,  1.88s/it]
Outer training loop:  98%|#########8| 49023/50000 [32:51:19<31:28,  1.93s/it]
Outer training loop:  98%|#########8| 49024/50000 [32:51:21<30:30,  1.88s/it]
Outer training loop:  98%|#########8| 49025/50000 [32:51:23<30:12,  1.86s/it]
Outer training loop:  98%|#########8| 49026/50000 [32:51:25<30:19,  1.87s/it]
Outer training loop:  98%|#########8| 49027/50000 [32:51:27<30:11,  1.86s/it]
Outer training loop:  98%|#########8| 49028/50000 [32:51:29<30:17,  1.87s/it]
Outer training loop:  98%|#########8| 49029/50000 [32:51:30<30:17,  1.87s/it]
Outer training loop:  98%|#########8| 49030/50000 [32:51:33<31:15,  1.93s/it]
Outer training loop:  98%|#########8| 49031/50000 [32:51:34<30:44,  1.90s/it]
Outer training loop:  98%|#########8| 49032/50000 [32:51:36<30:38,  1.90s/it]
Outer training loop:  98%|#########8| 49033/50000 [32:51:38<31:11,  1.94s/it]
Outer training loop:  98%|#########8| 49034/50000 [32:51:40<30:51,  1.92s/it]
Outer training loop:  98%|#########8| 49035/50000 [32:51:42<29:47,  1.85s/it]
Outer training loop:  98%|#########8| 49036/50000 [32:51:44<30:24,  1.89s/it]
Outer training loop:  98%|#########8| 49037/50000 [32:51:46<30:02,  1.87s/it]
Outer training loop:  98%|#########8| 49038/50000 [32:51:47<29:36,  1.85s/it]
Outer training loop:  98%|#########8| 49039/50000 [32:51:50<31:06,  1.94s/it]
Outer training loop:  98%|#########8| 49040/50000 [32:51:51<30:25,  1.90s/it]
Outer training loop:  98%|#########8| 49041/50000 [32:51:53<30:15,  1.89s/it]
Outer training loop:  98%|#########8| 49042/50000 [32:51:55<30:23,  1.90s/it]
Outer training loop:  98%|#########8| 49043/50000 [32:51:57<30:23,  1.91s/it]
Outer training loop:  98%|#########8| 49044/50000 [32:51:59<29:19,  1.84s/it]
Outer training loop:  98%|#########8| 49045/50000 [32:52:01<28:50,  1.81s/it]
Outer training loop:  98%|#########8| 49046/50000 [32:52:03<29:45,  1.87s/it]
Outer training loop:  98%|#########8| 49047/50000 [32:52:04<29:28,  1.86s/it]
Outer training loop:  98%|#########8| 49048/50000 [32:52:06<29:33,  1.86s/it]
Outer training loop:  98%|#########8| 49049/50000 [32:52:08<29:22,  1.85s/it]
Outer training loop:  98%|#########8| 49050/50000 [32:52:10<30:25,  1.92s/it]
Outer training loop:  98%|#########8| 49051/50000 [32:52:12<29:35,  1.87s/it]
Outer training loop:  98%|#########8| 49052/50000 [32:52:14<30:16,  1.92s/it]
Outer training loop:  98%|#########8| 49053/50000 [32:52:16<29:11,  1.85s/it]
Outer training loop:  98%|#########8| 49054/50000 [32:52:18<29:28,  1.87s/it]
Outer training loop:  98%|#########8| 49055/50000 [32:52:19<29:16,  1.86s/it]
Outer training loop:  98%|#########8| 49056/50000 [32:52:21<30:09,  1.92s/it]
Outer training loop:  98%|#########8| 49057/50000 [32:52:23<29:14,  1.86s/it]
Outer training loop:  98%|#########8| 49058/50000 [32:52:25<29:19,  1.87s/it]
Outer training loop:  98%|#########8| 49059/50000 [32:52:27<30:06,  1.92s/it]
Outer training loop:  98%|#########8| 49060/50000 [32:52:29<30:54,  1.97s/it]
Outer training loop:  98%|#########8| 49061/50000 [32:52:31<30:01,  1.92s/it]
Outer training loop:  98%|#########8| 49062/50000 [32:52:33<29:46,  1.90s/it]
Outer training loop:  98%|#########8| 49063/50000 [32:52:35<29:25,  1.88s/it]
Outer training loop:  98%|#########8| 49064/50000 [32:52:37<29:36,  1.90s/it]
Outer training loop:  98%|#########8| 49065/50000 [32:52:38<29:21,  1.88s/it]
Outer training loop:  98%|#########8| 49066/50000 [32:52:41<30:01,  1.93s/it]
Outer training loop:  98%|#########8| 49067/50000 [32:52:42<29:33,  1.90s/it]
Outer training loop:  98%|#########8| 49068/50000 [32:52:44<29:30,  1.90s/it]
Outer training loop:  98%|#########8| 49069/50000 [32:52:46<30:01,  1.93s/it]
Outer training loop:  98%|#########8| 49070/50000 [32:52:48<29:56,  1.93s/it]
Outer training loop:  98%|#########8| 49071/50000 [32:52:50<29:26,  1.90s/it]
Outer training loop:  98%|#########8| 49072/50000 [32:52:52<30:03,  1.94s/it]
Outer training loop:  98%|#########8| 49073/50000 [32:52:54<29:38,  1.92s/it]
Outer training loop:  98%|#########8| 49074/50000 [32:52:56<29:41,  1.92s/it]
Outer training loop:  98%|#########8| 49075/50000 [32:52:58<29:29,  1.91s/it]
Outer training loop:  98%|#########8| 49076/50000 [32:53:00<29:41,  1.93s/it]
Outer training loop:  98%|#########8| 49077/50000 [32:53:02<29:05,  1.89s/it]
Outer training loop:  98%|#########8| 49078/50000 [32:53:03<28:52,  1.88s/it]
Outer training loop:  98%|#########8| 49079/50000 [32:53:05<29:49,  1.94s/it]
Outer training loop:  98%|#########8| 49080/50000 [32:53:07<28:55,  1.89s/it]
Outer training loop:  98%|#########8| 49081/50000 [32:53:09<28:42,  1.87s/it]
Outer training loop:  98%|#########8| 49082/50000 [32:53:11<29:11,  1.91s/it]
Outer training loop:  98%|#########8| 49083/50000 [32:53:13<29:37,  1.94s/it]
Outer training loop:  98%|#########8| 49084/50000 [32:53:15<28:47,  1.89s/it]
Outer training loop:  98%|#########8| 49085/50000 [32:53:17<29:29,  1.93s/it]
Outer training loop:  98%|#########8| 49086/50000 [32:53:19<28:45,  1.89s/it]
Outer training loop:  98%|#########8| 49087/50000 [32:53:21<28:49,  1.89s/it]
Outer training loop:  98%|#########8| 49088/50000 [32:53:22<28:04,  1.85s/it]
Outer training loop:  98%|#########8| 49089/50000 [32:53:24<28:22,  1.87s/it]
Outer training loop:  98%|#########8| 49090/50000 [32:53:26<27:47,  1.83s/it]
Outer training loop:  98%|#########8| 49091/50000 [32:53:28<27:53,  1.84s/it]
Outer training loop:  98%|#########8| 49092/50000 [32:53:30<28:46,  1.90s/it]
Outer training loop:  98%|#########8| 49093/50000 [32:53:32<28:11,  1.86s/it]
Outer training loop:  98%|#########8| 49094/50000 [32:53:33<28:07,  1.86s/it]
Outer training loop:  98%|#########8| 49095/50000 [32:53:36<28:51,  1.91s/it]
Outer training loop:  98%|#########8| 49096/50000 [32:53:37<28:32,  1.89s/it]
Outer training loop:  98%|#########8| 49097/50000 [32:53:39<27:34,  1.83s/it]
Outer training loop:  98%|#########8| 49098/50000 [32:53:41<27:29,  1.83s/it]
Outer training loop:  98%|#########8| 49099/50000 [32:53:43<28:21,  1.89s/it]
Outer training loop:  98%|#########8| 49100/50000 [32:53:45<28:09,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.90s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  98%|#########8| 49101/50000 [32:53:52<52:22,  3.50s/it]
Outer training loop:  98%|#########8| 49102/50000 [32:53:54<44:26,  2.97s/it]
Outer training loop:  98%|#########8| 49103/50000 [32:53:56<40:04,  2.68s/it]
Outer training loop:  98%|#########8| 49104/50000 [32:53:58<36:12,  2.42s/it]
Outer training loop:  98%|#########8| 49105/50000 [32:53:59<33:06,  2.22s/it]
Outer training loop:  98%|#########8| 49106/50000 [32:54:01<32:04,  2.15s/it]
Outer training loop:  98%|#########8| 49107/50000 [32:54:03<31:27,  2.11s/it]
Outer training loop:  98%|#########8| 49108/50000 [32:54:05<30:02,  2.02s/it]
Outer training loop:  98%|#########8| 49109/50000 [32:54:07<29:48,  2.01s/it]
Outer training loop:  98%|#########8| 49110/50000 [32:54:09<28:59,  1.95s/it]
Outer training loop:  98%|#########8| 49111/50000 [32:54:11<28:54,  1.95s/it]
Outer training loop:  98%|#########8| 49112/50000 [32:54:13<28:13,  1.91s/it]
Outer training loop:  98%|#########8| 49113/50000 [32:54:15<28:03,  1.90s/it]
Outer training loop:  98%|#########8| 49114/50000 [32:54:16<27:39,  1.87s/it]
Outer training loop:  98%|#########8| 49115/50000 [32:54:18<27:22,  1.86s/it]
Outer training loop:  98%|#########8| 49116/50000 [32:54:20<28:28,  1.93s/it]
Outer training loop:  98%|#########8| 49117/50000 [32:54:22<27:47,  1.89s/it]
Outer training loop:  98%|#########8| 49118/50000 [32:54:24<27:47,  1.89s/it]
Outer training loop:  98%|#########8| 49119/50000 [32:54:26<29:09,  1.99s/it]
Outer training loop:  98%|#########8| 49120/50000 [32:54:28<28:47,  1.96s/it]
Outer training loop:  98%|#########8| 49121/50000 [32:54:30<28:06,  1.92s/it]
Outer training loop:  98%|#########8| 49122/50000 [32:54:32<28:48,  1.97s/it]
Outer training loop:  98%|#########8| 49123/50000 [32:54:34<28:01,  1.92s/it]
Outer training loop:  98%|#########8| 49124/50000 [32:54:36<27:32,  1.89s/it]
Outer training loop:  98%|#########8| 49125/50000 [32:54:38<28:09,  1.93s/it]
Outer training loop:  98%|#########8| 49126/50000 [32:54:40<27:35,  1.89s/it]
Outer training loop:  98%|#########8| 49127/50000 [32:54:41<27:41,  1.90s/it]
Outer training loop:  98%|#########8| 49128/50000 [32:54:43<28:21,  1.95s/it]
Outer training loop:  98%|#########8| 49129/50000 [32:54:45<27:40,  1.91s/it]
Outer training loop:  98%|#########8| 49130/50000 [32:54:47<27:24,  1.89s/it]
Outer training loop:  98%|#########8| 49131/50000 [32:54:49<26:27,  1.83s/it]
Outer training loop:  98%|#########8| 49132/50000 [32:54:51<27:17,  1.89s/it]
Outer training loop:  98%|#########8| 49133/50000 [32:54:53<26:53,  1.86s/it]
Outer training loop:  98%|#########8| 49134/50000 [32:54:55<27:13,  1.89s/it]
Outer training loop:  98%|#########8| 49135/50000 [32:54:57<27:20,  1.90s/it]
Outer training loop:  98%|#########8| 49136/50000 [32:54:58<27:21,  1.90s/it]
Outer training loop:  98%|#########8| 49137/50000 [32:55:00<27:11,  1.89s/it]
Outer training loop:  98%|#########8| 49138/50000 [32:55:02<27:53,  1.94s/it]
Outer training loop:  98%|#########8| 49139/50000 [32:55:04<27:23,  1.91s/it]
Outer training loop:  98%|#########8| 49140/50000 [32:55:06<27:05,  1.89s/it]
Outer training loop:  98%|#########8| 49141/50000 [32:55:08<26:35,  1.86s/it]
Outer training loop:  98%|#########8| 49142/50000 [32:55:10<27:42,  1.94s/it]
Outer training loop:  98%|#########8| 49143/50000 [32:55:12<26:58,  1.89s/it]
Outer training loop:  98%|#########8| 49144/50000 [32:55:13<26:11,  1.84s/it]
Outer training loop:  98%|#########8| 49145/50000 [32:55:15<25:55,  1.82s/it]
Outer training loop:  98%|#########8| 49146/50000 [32:55:17<26:33,  1.87s/it]
Outer training loop:  98%|#########8| 49147/50000 [32:55:19<25:45,  1.81s/it]
Outer training loop:  98%|#########8| 49148/50000 [32:55:21<25:46,  1.81s/it]
Outer training loop:  98%|#########8| 49149/50000 [32:55:23<26:28,  1.87s/it]
Outer training loop:  98%|#########8| 49150/50000 [32:55:24<26:08,  1.85s/it]
Outer training loop:  98%|#########8| 49151/50000 [32:55:26<26:02,  1.84s/it]
Outer training loop:  98%|#########8| 49152/50000 [32:55:28<26:38,  1.88s/it]
Outer training loop:  98%|#########8| 49153/50000 [32:55:30<26:03,  1.85s/it]
Outer training loop:  98%|#########8| 49154/50000 [32:55:32<26:14,  1.86s/it]
Outer training loop:  98%|#########8| 49155/50000 [32:55:34<26:05,  1.85s/it]
Outer training loop:  98%|#########8| 49156/50000 [32:55:36<26:40,  1.90s/it]
Outer training loop:  98%|#########8| 49157/50000 [32:55:38<27:28,  1.96s/it]
Outer training loop:  98%|#########8| 49158/50000 [32:55:40<26:50,  1.91s/it]
Outer training loop:  98%|#########8| 49159/50000 [32:55:42<26:57,  1.92s/it]
Outer training loop:  98%|#########8| 49160/50000 [32:55:43<26:25,  1.89s/it]
Outer training loop:  98%|#########8| 49161/50000 [32:55:45<26:35,  1.90s/it]
Outer training loop:  98%|#########8| 49162/50000 [32:55:47<25:48,  1.85s/it]
Outer training loop:  98%|#########8| 49163/50000 [32:55:49<26:26,  1.90s/it]
Outer training loop:  98%|#########8| 49164/50000 [32:55:51<26:11,  1.88s/it]
Outer training loop:  98%|#########8| 49165/50000 [32:55:53<25:29,  1.83s/it]
Outer training loop:  98%|#########8| 49166/50000 [32:55:55<26:22,  1.90s/it]
Outer training loop:  98%|#########8| 49167/50000 [32:55:56<25:37,  1.85s/it]
Outer training loop:  98%|#########8| 49168/50000 [32:55:58<25:29,  1.84s/it]
Outer training loop:  98%|#########8| 49169/50000 [32:56:00<24:59,  1.80s/it]
Outer training loop:  98%|#########8| 49170/50000 [32:56:02<25:43,  1.86s/it]
Outer training loop:  98%|#########8| 49171/50000 [32:56:04<25:20,  1.83s/it]
Outer training loop:  98%|#########8| 49172/50000 [32:56:06<25:30,  1.85s/it]
Outer training loop:  98%|#########8| 49173/50000 [32:56:08<26:46,  1.94s/it]
Outer training loop:  98%|#########8| 49174/50000 [32:56:10<26:09,  1.90s/it]
Outer training loop:  98%|#########8| 49175/50000 [32:56:11<25:51,  1.88s/it]
Outer training loop:  98%|#########8| 49176/50000 [32:56:13<26:23,  1.92s/it]
Outer training loop:  98%|#########8| 49177/50000 [32:56:15<25:53,  1.89s/it]
Outer training loop:  98%|#########8| 49178/50000 [32:56:17<25:25,  1.86s/it]
Outer training loop:  98%|#########8| 49179/50000 [32:56:19<25:45,  1.88s/it]
Outer training loop:  98%|#########8| 49180/50000 [32:56:21<26:49,  1.96s/it]
Outer training loop:  98%|#########8| 49181/50000 [32:56:23<25:46,  1.89s/it]
Outer training loop:  98%|#########8| 49182/50000 [32:56:25<25:34,  1.88s/it]
Outer training loop:  98%|#########8| 49183/50000 [32:56:27<26:29,  1.95s/it]
Outer training loop:  98%|#########8| 49184/50000 [32:56:29<26:17,  1.93s/it]
Outer training loop:  98%|#########8| 49185/50000 [32:56:31<25:54,  1.91s/it]
Outer training loop:  98%|#########8| 49186/50000 [32:56:32<25:31,  1.88s/it]
Outer training loop:  98%|#########8| 49187/50000 [32:56:34<25:59,  1.92s/it]
Outer training loop:  98%|#########8| 49188/50000 [32:56:36<25:21,  1.87s/it]
Outer training loop:  98%|#########8| 49189/50000 [32:56:38<25:12,  1.87s/it]
Outer training loop:  98%|#########8| 49190/50000 [32:56:40<26:19,  1.95s/it]
Outer training loop:  98%|#########8| 49191/50000 [32:56:42<25:40,  1.90s/it]
Outer training loop:  98%|#########8| 49192/50000 [32:56:44<25:47,  1.92s/it]
Outer training loop:  98%|#########8| 49193/50000 [32:56:46<25:10,  1.87s/it]
Outer training loop:  98%|#########8| 49194/50000 [32:56:48<25:41,  1.91s/it]
Outer training loop:  98%|#########8| 49195/50000 [32:56:50<25:33,  1.90s/it]
Outer training loop:  98%|#########8| 49196/50000 [32:56:51<25:35,  1.91s/it]
Outer training loop:  98%|#########8| 49197/50000 [32:56:54<26:05,  1.95s/it]
Outer training loop:  98%|#########8| 49198/50000 [32:56:55<25:59,  1.94s/it]
Outer training loop:  98%|#########8| 49199/50000 [32:56:57<25:29,  1.91s/it]
Outer training loop:  98%|#########8| 49200/50000 [32:56:59<25:24,  1.91s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.73s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.80s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.82s/it]

Outer training loop:  98%|#########8| 49201/50000 [32:57:07<47:39,  3.58s/it]
Outer training loop:  98%|#########8| 49202/50000 [32:57:09<40:57,  3.08s/it]
Outer training loop:  98%|#########8| 49203/50000 [32:57:10<36:14,  2.73s/it]
Outer training loop:  98%|#########8| 49204/50000 [32:57:12<33:05,  2.49s/it]
Outer training loop:  98%|#########8| 49205/50000 [32:57:15<31:28,  2.38s/it]
Outer training loop:  98%|#########8| 49206/50000 [32:57:16<28:59,  2.19s/it]
Outer training loop:  98%|#########8| 49207/50000 [32:57:18<27:37,  2.09s/it]
Outer training loop:  98%|#########8| 49208/50000 [32:57:20<27:37,  2.09s/it]
Outer training loop:  98%|#########8| 49209/50000 [32:57:22<26:04,  1.98s/it]
Outer training loop:  98%|#########8| 49210/50000 [32:57:24<25:20,  1.92s/it]
Outer training loop:  98%|#########8| 49211/50000 [32:57:26<24:59,  1.90s/it]
Outer training loop:  98%|#########8| 49212/50000 [32:57:28<25:36,  1.95s/it]
Outer training loop:  98%|#########8| 49213/50000 [32:57:29<24:55,  1.90s/it]
Outer training loop:  98%|#########8| 49214/50000 [32:57:31<24:47,  1.89s/it]
Outer training loop:  98%|#########8| 49215/50000 [32:57:33<25:22,  1.94s/it]
Outer training loop:  98%|#########8| 49216/50000 [32:57:35<24:59,  1.91s/it]
Outer training loop:  98%|#########8| 49217/50000 [32:57:37<24:16,  1.86s/it]
Outer training loop:  98%|#########8| 49218/50000 [32:57:39<24:45,  1.90s/it]
Outer training loop:  98%|#########8| 49219/50000 [32:57:41<24:39,  1.89s/it]
Outer training loop:  98%|#########8| 49220/50000 [32:57:43<24:31,  1.89s/it]
Outer training loop:  98%|#########8| 49221/50000 [32:57:44<24:05,  1.86s/it]
Outer training loop:  98%|#########8| 49222/50000 [32:57:46<24:44,  1.91s/it]
Outer training loop:  98%|#########8| 49223/50000 [32:57:48<24:26,  1.89s/it]
Outer training loop:  98%|#########8| 49224/50000 [32:57:50<24:10,  1.87s/it]
Outer training loop:  98%|#########8| 49225/50000 [32:57:52<25:01,  1.94s/it]
Outer training loop:  98%|#########8| 49226/50000 [32:57:54<24:38,  1.91s/it]
Outer training loop:  98%|#########8| 49227/50000 [32:57:56<24:06,  1.87s/it]
Outer training loop:  98%|#########8| 49228/50000 [32:57:58<24:21,  1.89s/it]
Outer training loop:  98%|#########8| 49229/50000 [32:58:00<24:44,  1.93s/it]
Outer training loop:  98%|#########8| 49230/50000 [32:58:02<24:18,  1.89s/it]
Outer training loop:  98%|#########8| 49231/50000 [32:58:04<24:11,  1.89s/it]
Outer training loop:  98%|#########8| 49232/50000 [32:58:06<24:53,  1.95s/it]
Outer training loop:  98%|#########8| 49233/50000 [32:58:07<24:23,  1.91s/it]
Outer training loop:  98%|#########8| 49234/50000 [32:58:09<23:46,  1.86s/it]
Outer training loop:  98%|#########8| 49235/50000 [32:58:11<23:33,  1.85s/it]
Outer training loop:  98%|#########8| 49236/50000 [32:58:13<24:10,  1.90s/it]
Outer training loop:  98%|#########8| 49237/50000 [32:58:15<23:58,  1.89s/it]
Outer training loop:  98%|#########8| 49238/50000 [32:58:17<24:01,  1.89s/it]
Outer training loop:  98%|#########8| 49239/50000 [32:58:19<25:02,  1.97s/it]
Outer training loop:  98%|#########8| 49240/50000 [32:58:21<24:55,  1.97s/it]
Outer training loop:  98%|#########8| 49241/50000 [32:58:23<24:27,  1.93s/it]
Outer training loop:  98%|#########8| 49242/50000 [32:58:25<24:42,  1.96s/it]
Outer training loop:  98%|#########8| 49243/50000 [32:58:27<24:26,  1.94s/it]
Outer training loop:  98%|#########8| 49244/50000 [32:58:28<23:53,  1.90s/it]
Outer training loop:  98%|#########8| 49245/50000 [32:58:30<23:30,  1.87s/it]
Outer training loop:  98%|#########8| 49246/50000 [32:58:32<24:12,  1.93s/it]
Outer training loop:  98%|#########8| 49247/50000 [32:58:34<23:57,  1.91s/it]
Outer training loop:  98%|#########8| 49248/50000 [32:58:36<23:27,  1.87s/it]
Outer training loop:  98%|#########8| 49249/50000 [32:58:38<23:34,  1.88s/it]
Outer training loop:  98%|#########8| 49250/50000 [32:58:40<23:44,  1.90s/it]
Outer training loop:  99%|#########8| 49251/50000 [32:58:42<23:10,  1.86s/it]
Outer training loop:  99%|#########8| 49252/50000 [32:58:44<23:41,  1.90s/it]
Outer training loop:  99%|#########8| 49253/50000 [32:58:45<23:26,  1.88s/it]
Outer training loop:  99%|#########8| 49254/50000 [32:58:47<23:41,  1.91s/it]
Outer training loop:  99%|#########8| 49255/50000 [32:58:49<23:27,  1.89s/it]
Outer training loop:  99%|#########8| 49256/50000 [32:58:51<23:49,  1.92s/it]
Outer training loop:  99%|#########8| 49257/50000 [32:58:53<23:32,  1.90s/it]
Outer training loop:  99%|#########8| 49258/50000 [32:58:55<23:23,  1.89s/it]
Outer training loop:  99%|#########8| 49259/50000 [32:58:57<23:49,  1.93s/it]
Outer training loop:  99%|#########8| 49260/50000 [32:58:59<24:02,  1.95s/it]
Outer training loop:  99%|#########8| 49261/50000 [32:59:01<23:16,  1.89s/it]
Outer training loop:  99%|#########8| 49262/50000 [32:59:03<23:54,  1.94s/it]
Outer training loop:  99%|#########8| 49263/50000 [32:59:05<23:55,  1.95s/it]
Outer training loop:  99%|#########8| 49264/50000 [32:59:06<23:11,  1.89s/it]
Outer training loop:  99%|#########8| 49265/50000 [32:59:08<22:39,  1.85s/it]
Outer training loop:  99%|#########8| 49266/50000 [32:59:10<23:17,  1.90s/it]
Outer training loop:  99%|#########8| 49267/50000 [32:59:12<22:56,  1.88s/it]
Outer training loop:  99%|#########8| 49268/50000 [32:59:14<22:56,  1.88s/it]
Outer training loop:  99%|#########8| 49269/50000 [32:59:16<23:28,  1.93s/it]
Outer training loop:  99%|#########8| 49270/50000 [32:59:18<23:23,  1.92s/it]
Outer training loop:  99%|#########8| 49271/50000 [32:59:20<23:22,  1.92s/it]
Outer training loop:  99%|#########8| 49272/50000 [32:59:22<23:14,  1.92s/it]
Outer training loop:  99%|#########8| 49273/50000 [32:59:24<23:31,  1.94s/it]
Outer training loop:  99%|#########8| 49274/50000 [32:59:25<22:43,  1.88s/it]
Outer training loop:  99%|#########8| 49275/50000 [32:59:27<22:11,  1.84s/it]
Outer training loop:  99%|#########8| 49276/50000 [32:59:29<22:32,  1.87s/it]
Outer training loop:  99%|#########8| 49277/50000 [32:59:31<22:13,  1.84s/it]
Outer training loop:  99%|#########8| 49278/50000 [32:59:33<21:59,  1.83s/it]
Outer training loop:  99%|#########8| 49279/50000 [32:59:34<21:39,  1.80s/it]
Outer training loop:  99%|#########8| 49280/50000 [32:59:36<22:19,  1.86s/it]
Outer training loop:  99%|#########8| 49281/50000 [32:59:38<22:11,  1.85s/it]
Outer training loop:  99%|#########8| 49282/50000 [32:59:40<22:50,  1.91s/it]
Outer training loop:  99%|#########8| 49283/50000 [32:59:42<22:14,  1.86s/it]
Outer training loop:  99%|#########8| 49284/50000 [32:59:44<22:27,  1.88s/it]
Outer training loop:  99%|#########8| 49285/50000 [32:59:46<22:17,  1.87s/it]
Outer training loop:  99%|#########8| 49286/50000 [32:59:48<21:53,  1.84s/it]
Outer training loop:  99%|#########8| 49287/50000 [32:59:50<22:42,  1.91s/it]
Outer training loop:  99%|#########8| 49288/50000 [32:59:52<22:19,  1.88s/it]
Outer training loop:  99%|#########8| 49289/50000 [32:59:53<21:54,  1.85s/it]
Outer training loop:  99%|#########8| 49290/50000 [32:59:55<22:07,  1.87s/it]
Outer training loop:  99%|#########8| 49291/50000 [32:59:57<22:37,  1.92s/it]
Outer training loop:  99%|#########8| 49292/50000 [32:59:59<22:01,  1.87s/it]
Outer training loop:  99%|#########8| 49293/50000 [33:00:01<21:21,  1.81s/it]
Outer training loop:  99%|#########8| 49294/50000 [33:00:03<21:58,  1.87s/it]
Outer training loop:  99%|#########8| 49295/50000 [33:00:04<21:41,  1.85s/it]
Outer training loop:  99%|#########8| 49296/50000 [33:00:06<21:37,  1.84s/it]
Outer training loop:  99%|#########8| 49297/50000 [33:00:08<21:12,  1.81s/it]
Outer training loop:  99%|#########8| 49298/50000 [33:00:10<21:55,  1.87s/it]
Outer training loop:  99%|#########8| 49299/50000 [33:00:12<21:12,  1.82s/it]
Outer training loop:  99%|#########8| 49300/50000 [33:00:14<20:57,  1.80s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.70s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.74s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.84s/it]

Outer training loop:  99%|#########8| 49301/50000 [33:00:21<41:11,  3.54s/it]
Outer training loop:  99%|#########8| 49302/50000 [33:00:23<34:57,  3.00s/it]
Outer training loop:  99%|#########8| 49303/50000 [33:00:25<30:39,  2.64s/it]
Outer training loop:  99%|#########8| 49304/50000 [33:00:27<28:00,  2.41s/it]
Outer training loop:  99%|#########8| 49305/50000 [33:00:29<26:45,  2.31s/it]
Outer training loop:  99%|#########8| 49306/50000 [33:00:30<25:08,  2.17s/it]
Outer training loop:  99%|#########8| 49307/50000 [33:00:32<23:48,  2.06s/it]
Outer training loop:  99%|#########8| 49308/50000 [33:00:34<23:14,  2.01s/it]
Outer training loop:  99%|#########8| 49309/50000 [33:00:36<23:21,  2.03s/it]
Outer training loop:  99%|#########8| 49310/50000 [33:00:38<22:36,  1.97s/it]
Outer training loop:  99%|#########8| 49311/50000 [33:00:40<22:16,  1.94s/it]
Outer training loop:  99%|#########8| 49312/50000 [33:00:42<22:08,  1.93s/it]
Outer training loop:  99%|#########8| 49313/50000 [33:00:44<21:51,  1.91s/it]
Outer training loop:  99%|#########8| 49314/50000 [33:00:46<21:53,  1.91s/it]
Outer training loop:  99%|#########8| 49315/50000 [33:00:47<21:38,  1.90s/it]
Outer training loop:  99%|#########8| 49316/50000 [33:00:50<22:05,  1.94s/it]
Outer training loop:  99%|#########8| 49317/50000 [33:00:51<21:31,  1.89s/it]
Outer training loop:  99%|#########8| 49318/50000 [33:00:53<21:14,  1.87s/it]
Outer training loop:  99%|#########8| 49319/50000 [33:00:55<21:49,  1.92s/it]
Outer training loop:  99%|#########8| 49320/50000 [33:00:57<21:05,  1.86s/it]
Outer training loop:  99%|#########8| 49321/50000 [33:00:59<21:13,  1.88s/it]
Outer training loop:  99%|#########8| 49322/50000 [33:01:01<21:33,  1.91s/it]
Outer training loop:  99%|#########8| 49323/50000 [33:01:03<21:08,  1.87s/it]
Outer training loop:  99%|#########8| 49324/50000 [33:01:04<21:03,  1.87s/it]
Outer training loop:  99%|#########8| 49325/50000 [33:01:06<20:59,  1.87s/it]
Outer training loop:  99%|#########8| 49326/50000 [33:01:08<21:15,  1.89s/it]
Outer training loop:  99%|#########8| 49327/50000 [33:01:10<21:23,  1.91s/it]
Outer training loop:  99%|#########8| 49328/50000 [33:01:12<21:22,  1.91s/it]
Outer training loop:  99%|#########8| 49329/50000 [33:01:14<21:47,  1.95s/it]
Outer training loop:  99%|#########8| 49330/50000 [33:01:16<21:03,  1.89s/it]
Outer training loop:  99%|#########8| 49331/50000 [33:01:18<20:45,  1.86s/it]
Outer training loop:  99%|#########8| 49332/50000 [33:01:19<20:27,  1.84s/it]
Outer training loop:  99%|#########8| 49333/50000 [33:01:21<21:02,  1.89s/it]
Outer training loop:  99%|#########8| 49334/50000 [33:01:23<20:37,  1.86s/it]
Outer training loop:  99%|#########8| 49335/50000 [33:01:25<20:19,  1.83s/it]
Outer training loop:  99%|#########8| 49336/50000 [33:01:27<20:25,  1.85s/it]
Outer training loop:  99%|#########8| 49337/50000 [33:01:29<21:10,  1.92s/it]
Outer training loop:  99%|#########8| 49338/50000 [33:01:31<20:35,  1.87s/it]
Outer training loop:  99%|#########8| 49339/50000 [33:01:33<20:16,  1.84s/it]
Outer training loop:  99%|#########8| 49340/50000 [33:01:35<20:46,  1.89s/it]
Outer training loop:  99%|#########8| 49341/50000 [33:01:36<20:44,  1.89s/it]
Outer training loop:  99%|#########8| 49342/50000 [33:01:38<20:32,  1.87s/it]
Outer training loop:  99%|#########8| 49343/50000 [33:01:40<20:57,  1.91s/it]
Outer training loop:  99%|#########8| 49344/50000 [33:01:42<20:36,  1.88s/it]
Outer training loop:  99%|#########8| 49345/50000 [33:01:44<20:24,  1.87s/it]
Outer training loop:  99%|#########8| 49346/50000 [33:01:46<20:02,  1.84s/it]
Outer training loop:  99%|#########8| 49347/50000 [33:01:48<20:38,  1.90s/it]
Outer training loop:  99%|#########8| 49348/50000 [33:01:49<20:11,  1.86s/it]
Outer training loop:  99%|#########8| 49349/50000 [33:01:51<20:03,  1.85s/it]
Outer training loop:  99%|#########8| 49350/50000 [33:01:53<20:44,  1.92s/it]
Outer training loop:  99%|#########8| 49351/50000 [33:01:55<21:18,  1.97s/it]
Outer training loop:  99%|#########8| 49352/50000 [33:01:57<20:43,  1.92s/it]
Outer training loop:  99%|#########8| 49353/50000 [33:01:59<21:25,  1.99s/it]
Outer training loop:  99%|#########8| 49354/50000 [33:02:01<20:54,  1.94s/it]
Outer training loop:  99%|#########8| 49355/50000 [33:02:03<20:05,  1.87s/it]
Outer training loop:  99%|#########8| 49356/50000 [33:02:05<19:52,  1.85s/it]
Outer training loop:  99%|#########8| 49357/50000 [33:02:07<20:28,  1.91s/it]
Outer training loop:  99%|#########8| 49358/50000 [33:02:09<20:11,  1.89s/it]
Outer training loop:  99%|#########8| 49359/50000 [33:02:10<20:05,  1.88s/it]
Outer training loop:  99%|#########8| 49360/50000 [33:02:12<20:24,  1.91s/it]
Outer training loop:  99%|#########8| 49361/50000 [33:02:14<20:35,  1.93s/it]
Outer training loop:  99%|#########8| 49362/50000 [33:02:16<20:24,  1.92s/it]
Outer training loop:  99%|#########8| 49363/50000 [33:02:18<20:52,  1.97s/it]
Outer training loop:  99%|#########8| 49364/50000 [33:02:20<20:27,  1.93s/it]
Outer training loop:  99%|#########8| 49365/50000 [33:02:22<19:49,  1.87s/it]
Outer training loop:  99%|#########8| 49366/50000 [33:02:24<19:53,  1.88s/it]
Outer training loop:  99%|#########8| 49367/50000 [33:02:26<20:28,  1.94s/it]
Outer training loop:  99%|#########8| 49368/50000 [33:02:28<20:20,  1.93s/it]
Outer training loop:  99%|#########8| 49369/50000 [33:02:30<20:02,  1.91s/it]
Outer training loop:  99%|#########8| 49370/50000 [33:02:32<20:22,  1.94s/it]
Outer training loop:  99%|#########8| 49371/50000 [33:02:34<20:05,  1.92s/it]
Outer training loop:  99%|#########8| 49372/50000 [33:02:36<20:08,  1.92s/it]
Outer training loop:  99%|#########8| 49373/50000 [33:02:37<19:56,  1.91s/it]
Outer training loop:  99%|#########8| 49374/50000 [33:02:39<19:41,  1.89s/it]
Outer training loop:  99%|#########8| 49375/50000 [33:02:41<19:35,  1.88s/it]
Outer training loop:  99%|#########8| 49376/50000 [33:02:43<19:17,  1.85s/it]
Outer training loop:  99%|#########8| 49377/50000 [33:02:45<19:28,  1.88s/it]
Outer training loop:  99%|#########8| 49378/50000 [33:02:47<19:25,  1.87s/it]
Outer training loop:  99%|#########8| 49379/50000 [33:02:49<19:11,  1.85s/it]
Outer training loop:  99%|#########8| 49380/50000 [33:02:51<19:40,  1.90s/it]
Outer training loop:  99%|#########8| 49381/50000 [33:02:52<18:59,  1.84s/it]
Outer training loop:  99%|#########8| 49382/50000 [33:02:54<18:46,  1.82s/it]
Outer training loop:  99%|#########8| 49383/50000 [33:02:56<18:37,  1.81s/it]
Outer training loop:  99%|#########8| 49384/50000 [33:02:58<19:33,  1.91s/it]
Outer training loop:  99%|#########8| 49385/50000 [33:03:00<19:04,  1.86s/it]
Outer training loop:  99%|#########8| 49386/50000 [33:03:01<18:47,  1.84s/it]
Outer training loop:  99%|#########8| 49387/50000 [33:03:03<18:59,  1.86s/it]
Outer training loop:  99%|#########8| 49388/50000 [33:03:05<18:48,  1.84s/it]
Outer training loop:  99%|#########8| 49389/50000 [33:03:07<18:31,  1.82s/it]
Outer training loop:  99%|#########8| 49390/50000 [33:03:09<18:20,  1.80s/it]
Outer training loop:  99%|#########8| 49391/50000 [33:03:11<18:51,  1.86s/it]
Outer training loop:  99%|#########8| 49392/50000 [33:03:12<18:29,  1.82s/it]
Outer training loop:  99%|#########8| 49393/50000 [33:03:14<18:26,  1.82s/it]
Outer training loop:  99%|#########8| 49394/50000 [33:03:16<18:43,  1.85s/it]
Outer training loop:  99%|#########8| 49395/50000 [33:03:18<18:37,  1.85s/it]
Outer training loop:  99%|#########8| 49396/50000 [33:03:20<19:03,  1.89s/it]
Outer training loop:  99%|#########8| 49397/50000 [33:03:22<19:26,  1.93s/it]
Outer training loop:  99%|#########8| 49398/50000 [33:03:24<19:17,  1.92s/it]
Outer training loop:  99%|#########8| 49399/50000 [33:03:26<19:28,  1.94s/it]
Outer training loop:  99%|#########8| 49400/50000 [33:03:28<19:28,  1.95s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.84s/it]

Evaluating policy:  67%|######6   | 2/3 [00:04<00:01,  1.94s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop:  99%|#########8| 49401/50000 [33:03:36<37:17,  3.74s/it]
Outer training loop:  99%|#########8| 49402/50000 [33:03:38<31:53,  3.20s/it]
Outer training loop:  99%|#########8| 49403/50000 [33:03:40<29:10,  2.93s/it]
Outer training loop:  99%|#########8| 49404/50000 [33:03:42<25:59,  2.62s/it]
Outer training loop:  99%|#########8| 49405/50000 [33:03:44<23:32,  2.37s/it]
Outer training loop:  99%|#########8| 49406/50000 [33:03:46<21:56,  2.22s/it]
Outer training loop:  99%|#########8| 49407/50000 [33:03:48<21:29,  2.17s/it]
Outer training loop:  99%|#########8| 49408/50000 [33:03:50<20:29,  2.08s/it]
Outer training loop:  99%|#########8| 49409/50000 [33:03:51<19:53,  2.02s/it]
Outer training loop:  99%|#########8| 49410/50000 [33:03:53<19:21,  1.97s/it]
Outer training loop:  99%|#########8| 49411/50000 [33:03:55<19:52,  2.03s/it]
Outer training loop:  99%|#########8| 49412/50000 [33:03:57<19:35,  2.00s/it]
Outer training loop:  99%|#########8| 49413/50000 [33:03:59<19:13,  1.97s/it]
Outer training loop:  99%|#########8| 49414/50000 [33:04:01<19:29,  2.00s/it]
Outer training loop:  99%|#########8| 49415/50000 [33:04:03<18:57,  1.94s/it]
Outer training loop:  99%|#########8| 49416/50000 [33:04:05<18:40,  1.92s/it]
Outer training loop:  99%|#########8| 49417/50000 [33:04:07<19:19,  1.99s/it]
Outer training loop:  99%|#########8| 49418/50000 [33:04:09<19:00,  1.96s/it]
Outer training loop:  99%|#########8| 49419/50000 [33:04:11<18:47,  1.94s/it]
Outer training loop:  99%|#########8| 49420/50000 [33:04:13<18:47,  1.94s/it]
Outer training loop:  99%|#########8| 49421/50000 [33:04:15<19:16,  2.00s/it]
Outer training loop:  99%|#########8| 49422/50000 [33:04:17<18:42,  1.94s/it]
Outer training loop:  99%|#########8| 49423/50000 [33:04:19<18:13,  1.89s/it]
Outer training loop:  99%|#########8| 49424/50000 [33:04:21<18:56,  1.97s/it]
Outer training loop:  99%|#########8| 49425/50000 [33:04:23<18:22,  1.92s/it]
Outer training loop:  99%|#########8| 49426/50000 [33:04:24<17:54,  1.87s/it]
Outer training loop:  99%|#########8| 49427/50000 [33:04:26<17:56,  1.88s/it]
Outer training loop:  99%|#########8| 49428/50000 [33:04:28<18:56,  1.99s/it]
Outer training loop:  99%|#########8| 49429/50000 [33:04:30<18:09,  1.91s/it]
Outer training loop:  99%|#########8| 49430/50000 [33:04:32<17:56,  1.89s/it]
Outer training loop:  99%|#########8| 49431/50000 [33:04:34<18:31,  1.95s/it]
Outer training loop:  99%|#########8| 49432/50000 [33:04:36<18:13,  1.93s/it]
Outer training loop:  99%|#########8| 49433/50000 [33:04:38<17:42,  1.87s/it]
Outer training loop:  99%|#########8| 49434/50000 [33:04:40<18:32,  1.96s/it]
Outer training loop:  99%|#########8| 49435/50000 [33:04:42<17:59,  1.91s/it]
Outer training loop:  99%|#########8| 49436/50000 [33:04:44<18:01,  1.92s/it]
Outer training loop:  99%|#########8| 49437/50000 [33:04:45<17:41,  1.89s/it]
Outer training loop:  99%|#########8| 49438/50000 [33:04:48<18:11,  1.94s/it]
Outer training loop:  99%|#########8| 49439/50000 [33:04:49<17:58,  1.92s/it]
Outer training loop:  99%|#########8| 49440/50000 [33:04:51<17:52,  1.92s/it]
Outer training loop:  99%|#########8| 49441/50000 [33:04:53<18:04,  1.94s/it]
Outer training loop:  99%|#########8| 49442/50000 [33:04:55<18:02,  1.94s/it]
Outer training loop:  99%|#########8| 49443/50000 [33:04:57<18:12,  1.96s/it]
Outer training loop:  99%|#########8| 49444/50000 [33:04:59<18:34,  2.00s/it]
Outer training loop:  99%|#########8| 49445/50000 [33:05:01<18:32,  2.00s/it]
Outer training loop:  99%|#########8| 49446/50000 [33:05:03<17:47,  1.93s/it]
Outer training loop:  99%|#########8| 49447/50000 [33:05:05<18:39,  2.03s/it]
Outer training loop:  99%|#########8| 49448/50000 [33:05:07<18:31,  2.01s/it]
Outer training loop:  99%|#########8| 49449/50000 [33:05:09<18:23,  2.00s/it]
Outer training loop:  99%|#########8| 49450/50000 [33:05:11<18:32,  2.02s/it]
Outer training loop:  99%|#########8| 49451/50000 [33:05:13<18:12,  1.99s/it]
Outer training loop:  99%|#########8| 49452/50000 [33:05:15<17:29,  1.92s/it]
Outer training loop:  99%|#########8| 49453/50000 [33:05:17<17:40,  1.94s/it]
Outer training loop:  99%|#########8| 49454/50000 [33:05:19<18:01,  1.98s/it]
Outer training loop:  99%|#########8| 49455/50000 [33:05:21<17:38,  1.94s/it]
Outer training loop:  99%|#########8| 49456/50000 [33:05:23<17:23,  1.92s/it]
Outer training loop:  99%|#########8| 49457/50000 [33:05:25<18:02,  1.99s/it]
Outer training loop:  99%|#########8| 49458/50000 [33:05:27<17:37,  1.95s/it]
Outer training loop:  99%|#########8| 49459/50000 [33:05:29<17:20,  1.92s/it]
Outer training loop:  99%|#########8| 49460/50000 [33:05:31<17:28,  1.94s/it]
Outer training loop:  99%|#########8| 49461/50000 [33:05:33<18:03,  2.01s/it]
Outer training loop:  99%|#########8| 49462/50000 [33:05:35<17:42,  1.98s/it]
Outer training loop:  99%|#########8| 49463/50000 [33:05:37<17:53,  2.00s/it]
Outer training loop:  99%|#########8| 49464/50000 [33:05:39<18:01,  2.02s/it]
Outer training loop:  99%|#########8| 49465/50000 [33:05:41<17:45,  1.99s/it]
Outer training loop:  99%|#########8| 49466/50000 [33:05:43<17:34,  1.97s/it]
Outer training loop:  99%|#########8| 49467/50000 [33:05:45<17:16,  1.94s/it]
Outer training loop:  99%|#########8| 49468/50000 [33:05:47<17:26,  1.97s/it]
Outer training loop:  99%|#########8| 49469/50000 [33:05:48<16:57,  1.92s/it]
Outer training loop:  99%|#########8| 49470/50000 [33:05:50<17:10,  1.94s/it]
Outer training loop:  99%|#########8| 49471/50000 [33:05:52<17:15,  1.96s/it]
Outer training loop:  99%|#########8| 49472/50000 [33:05:54<16:42,  1.90s/it]
Outer training loop:  99%|#########8| 49473/50000 [33:05:56<16:38,  1.89s/it]
Outer training loop:  99%|#########8| 49474/50000 [33:05:58<17:18,  1.97s/it]
Outer training loop:  99%|#########8| 49475/50000 [33:06:00<17:22,  1.99s/it]
Outer training loop:  99%|#########8| 49476/50000 [33:06:02<17:04,  1.96s/it]
Outer training loop:  99%|#########8| 49477/50000 [33:06:04<17:29,  2.01s/it]
Outer training loop:  99%|#########8| 49478/50000 [33:06:06<17:18,  1.99s/it]
Outer training loop:  99%|#########8| 49479/50000 [33:06:08<16:54,  1.95s/it]
Outer training loop:  99%|#########8| 49480/50000 [33:06:10<16:50,  1.94s/it]
Outer training loop:  99%|#########8| 49481/50000 [33:06:12<17:13,  1.99s/it]
Outer training loop:  99%|#########8| 49482/50000 [33:06:14<16:53,  1.96s/it]
Outer training loop:  99%|#########8| 49483/50000 [33:06:16<16:26,  1.91s/it]
Outer training loop:  99%|#########8| 49484/50000 [33:06:18<16:46,  1.95s/it]
Outer training loop:  99%|#########8| 49485/50000 [33:06:20<16:26,  1.92s/it]
Outer training loop:  99%|#########8| 49486/50000 [33:06:22<16:17,  1.90s/it]
Outer training loop:  99%|#########8| 49487/50000 [33:06:23<16:10,  1.89s/it]
Outer training loop:  99%|#########8| 49488/50000 [33:06:25<15:58,  1.87s/it]
Outer training loop:  99%|#########8| 49489/50000 [33:06:27<16:03,  1.88s/it]
Outer training loop:  99%|#########8| 49490/50000 [33:06:29<16:25,  1.93s/it]
Outer training loop:  99%|#########8| 49491/50000 [33:06:31<16:15,  1.92s/it]
Outer training loop:  99%|#########8| 49492/50000 [33:06:33<16:06,  1.90s/it]
Outer training loop:  99%|#########8| 49493/50000 [33:06:35<16:17,  1.93s/it]
Outer training loop:  99%|#########8| 49494/50000 [33:06:37<17:06,  2.03s/it]
Outer training loop:  99%|#########8| 49495/50000 [33:06:39<16:35,  1.97s/it]
Outer training loop:  99%|#########8| 49496/50000 [33:06:41<16:09,  1.92s/it]
Outer training loop:  99%|#########8| 49497/50000 [33:06:43<16:38,  1.99s/it]
Outer training loop:  99%|#########8| 49498/50000 [33:06:45<16:47,  2.01s/it]
Outer training loop:  99%|#########8| 49499/50000 [33:06:47<16:11,  1.94s/it]
Outer training loop:  99%|#########9| 49500/50000 [33:06:49<15:39,  1.88s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.84s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.85s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:30,  1.83s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:28,  1.83s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:26,  1.83s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:29,  1.89s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:28,  1.91s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:25,  1.89s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:27,  1.94s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:22,  1.91s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:23,  1.94s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:18,  1.90s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:21,  1.96s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:18,  1.95s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:13,  1.91s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:09,  1.87s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:06,  1.86s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:06,  1.90s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:04,  1.89s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:00,  1.86s/it]

Evaluating policy:  23%|##2       | 19/83 [00:35<01:57,  1.84s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<02:02,  1.95s/it]

Evaluating policy:  25%|##5       | 21/83 [00:39<01:58,  1.91s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.85s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:52,  1.87s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.85s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.94s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.90s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:45,  1.89s/it]

Evaluating policy:  34%|###3      | 28/83 [00:52<01:42,  1.85s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:38,  1.83s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:37,  1.83s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:41,  1.96s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:37,  1.91s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:35,  1.91s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.86s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:27,  1.83s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:26,  1.84s/it]

Evaluating policy:  45%|####4     | 37/83 [01:09<01:23,  1.81s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:24,  1.88s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:22,  1.87s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:18,  1.83s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:16,  1.83s/it]

Evaluating policy:  51%|#####     | 42/83 [01:18<01:15,  1.85s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:20<01:13,  1.83s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:22<01:13,  1.88s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:24<01:14,  1.96s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:26<01:09,  1.88s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.85s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.85s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:32<01:02,  1.83s/it]

Evaluating policy:  60%|######    | 50/83 [01:33<01:00,  1.83s/it]

Evaluating policy:  61%|######1   | 51/83 [01:35<00:58,  1.83s/it]

Evaluating policy:  63%|######2   | 52/83 [01:37<01:00,  1.97s/it]

Evaluating policy:  64%|######3   | 53/83 [01:39<00:58,  1.94s/it]

Evaluating policy:  65%|######5   | 54/83 [01:41<00:54,  1.89s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:52,  1.88s/it]

Evaluating policy:  67%|######7   | 56/83 [01:45<00:50,  1.87s/it]

Evaluating policy:  69%|######8   | 57/83 [01:47<00:49,  1.88s/it]

Evaluating policy:  70%|######9   | 58/83 [01:49<00:47,  1.89s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:50<00:44,  1.84s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:45,  1.99s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:42,  1.95s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:56<00:39,  1.89s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:58<00:37,  1.86s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:00<00:34,  1.84s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:02<00:32,  1.81s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:03<00:30,  1.78s/it]

Evaluating policy:  81%|########  | 67/83 [02:05<00:28,  1.77s/it]

Evaluating policy:  82%|########1 | 68/83 [02:07<00:29,  1.95s/it]

Evaluating policy:  83%|########3 | 69/83 [02:09<00:26,  1.89s/it]

Evaluating policy:  84%|########4 | 70/83 [02:11<00:24,  1.88s/it]

Evaluating policy:  86%|########5 | 71/83 [02:13<00:22,  1.84s/it]

Evaluating policy:  87%|########6 | 72/83 [02:15<00:19,  1.81s/it]

Evaluating policy:  88%|########7 | 73/83 [02:16<00:17,  1.79s/it]

Evaluating policy:  89%|########9 | 74/83 [02:18<00:16,  1.81s/it]

Evaluating policy:  90%|######### | 75/83 [02:20<00:14,  1.82s/it]

Evaluating policy:  92%|#########1| 76/83 [02:22<00:12,  1.81s/it]

Evaluating policy:  93%|#########2| 77/83 [02:24<00:11,  1.99s/it]

Evaluating policy:  94%|#########3| 78/83 [02:26<00:09,  1.91s/it]

Evaluating policy:  95%|#########5| 79/83 [02:28<00:07,  1.93s/it]

Evaluating policy:  96%|#########6| 80/83 [02:30<00:05,  1.90s/it]

Evaluating policy:  98%|#########7| 81/83 [02:31<00:03,  1.85s/it]

Evaluating policy:  99%|#########8| 82/83 [02:33<00:01,  1.86s/it]

Evaluating policy: 100%|##########| 83/83 [02:35<00:00,  1.79s/it]


Evaluating policy:   0%|          | 0/83 [00:00<?, ?it/s]

Evaluating policy:   1%|1         | 1/83 [00:01<02:22,  1.74s/it]

Evaluating policy:   2%|2         | 2/83 [00:03<02:24,  1.79s/it]

Evaluating policy:   4%|3         | 3/83 [00:05<02:20,  1.76s/it]

Evaluating policy:   5%|4         | 4/83 [00:07<02:36,  1.98s/it]

Evaluating policy:   6%|6         | 5/83 [00:09<02:30,  1.93s/it]

Evaluating policy:   7%|7         | 6/83 [00:11<02:28,  1.92s/it]

Evaluating policy:   8%|8         | 7/83 [00:13<02:24,  1.90s/it]

Evaluating policy:  10%|9         | 8/83 [00:15<02:24,  1.93s/it]

Evaluating policy:  11%|#         | 9/83 [00:17<02:21,  1.91s/it]

Evaluating policy:  12%|#2        | 10/83 [00:19<02:17,  1.88s/it]

Evaluating policy:  13%|#3        | 11/83 [00:21<02:18,  1.92s/it]

Evaluating policy:  14%|#4        | 12/83 [00:23<02:19,  1.96s/it]

Evaluating policy:  16%|#5        | 13/83 [00:24<02:12,  1.89s/it]

Evaluating policy:  17%|#6        | 14/83 [00:26<02:10,  1.89s/it]

Evaluating policy:  18%|#8        | 15/83 [00:28<02:08,  1.89s/it]

Evaluating policy:  19%|#9        | 16/83 [00:30<02:10,  1.95s/it]

Evaluating policy:  20%|##        | 17/83 [00:32<02:07,  1.93s/it]

Evaluating policy:  22%|##1       | 18/83 [00:34<02:04,  1.92s/it]

Evaluating policy:  23%|##2       | 19/83 [00:36<02:00,  1.88s/it]

Evaluating policy:  24%|##4       | 20/83 [00:38<01:57,  1.86s/it]

Evaluating policy:  25%|##5       | 21/83 [00:40<01:59,  1.92s/it]

Evaluating policy:  27%|##6       | 22/83 [00:41<01:53,  1.86s/it]

Evaluating policy:  28%|##7       | 23/83 [00:43<01:51,  1.86s/it]

Evaluating policy:  29%|##8       | 24/83 [00:45<01:49,  1.86s/it]

Evaluating policy:  30%|###       | 25/83 [00:47<01:52,  1.95s/it]

Evaluating policy:  31%|###1      | 26/83 [00:49<01:48,  1.91s/it]

Evaluating policy:  33%|###2      | 27/83 [00:51<01:43,  1.84s/it]

Evaluating policy:  34%|###3      | 28/83 [00:53<01:41,  1.84s/it]

Evaluating policy:  35%|###4      | 29/83 [00:54<01:39,  1.84s/it]

Evaluating policy:  36%|###6      | 30/83 [00:56<01:38,  1.85s/it]

Evaluating policy:  37%|###7      | 31/83 [00:58<01:40,  1.94s/it]

Evaluating policy:  39%|###8      | 32/83 [01:00<01:36,  1.89s/it]

Evaluating policy:  40%|###9      | 33/83 [01:02<01:33,  1.87s/it]

Evaluating policy:  41%|####      | 34/83 [01:04<01:30,  1.84s/it]

Evaluating policy:  42%|####2     | 35/83 [01:06<01:27,  1.82s/it]

Evaluating policy:  43%|####3     | 36/83 [01:07<01:26,  1.84s/it]

Evaluating policy:  45%|####4     | 37/83 [01:10<01:29,  1.94s/it]

Evaluating policy:  46%|####5     | 38/83 [01:11<01:25,  1.89s/it]

Evaluating policy:  47%|####6     | 39/83 [01:13<01:21,  1.86s/it]

Evaluating policy:  48%|####8     | 40/83 [01:15<01:20,  1.86s/it]

Evaluating policy:  49%|####9     | 41/83 [01:17<01:18,  1.86s/it]

Evaluating policy:  51%|#####     | 42/83 [01:19<01:22,  2.00s/it]

Evaluating policy:  52%|#####1    | 43/83 [01:21<01:17,  1.94s/it]

Evaluating policy:  53%|#####3    | 44/83 [01:23<01:14,  1.92s/it]

Evaluating policy:  54%|#####4    | 45/83 [01:25<01:12,  1.92s/it]

Evaluating policy:  55%|#####5    | 46/83 [01:27<01:09,  1.89s/it]

Evaluating policy:  57%|#####6    | 47/83 [01:28<01:06,  1.86s/it]

Evaluating policy:  58%|#####7    | 48/83 [01:30<01:04,  1.84s/it]

Evaluating policy:  59%|#####9    | 49/83 [01:33<01:07,  1.98s/it]

Evaluating policy:  60%|######    | 50/83 [01:34<01:02,  1.91s/it]

Evaluating policy:  61%|######1   | 51/83 [01:36<01:00,  1.89s/it]

Evaluating policy:  63%|######2   | 52/83 [01:38<00:57,  1.86s/it]

Evaluating policy:  64%|######3   | 53/83 [01:40<00:55,  1.86s/it]

Evaluating policy:  65%|######5   | 54/83 [01:42<00:53,  1.83s/it]

Evaluating policy:  66%|######6   | 55/83 [01:43<00:51,  1.83s/it]

Evaluating policy:  67%|######7   | 56/83 [01:46<00:53,  1.99s/it]

Evaluating policy:  69%|######8   | 57/83 [01:48<00:50,  1.96s/it]

Evaluating policy:  70%|######9   | 58/83 [01:50<00:48,  1.93s/it]

Evaluating policy:  71%|#######1  | 59/83 [01:51<00:45,  1.88s/it]

Evaluating policy:  72%|#######2  | 60/83 [01:53<00:43,  1.88s/it]

Evaluating policy:  73%|#######3  | 61/83 [01:55<00:42,  1.91s/it]

Evaluating policy:  75%|#######4  | 62/83 [01:57<00:39,  1.88s/it]

Evaluating policy:  76%|#######5  | 63/83 [01:59<00:37,  1.85s/it]

Evaluating policy:  77%|#######7  | 64/83 [02:01<00:34,  1.83s/it]

Evaluating policy:  78%|#######8  | 65/83 [02:03<00:35,  1.98s/it]

Evaluating policy:  80%|#######9  | 66/83 [02:05<00:33,  1.96s/it]

Evaluating policy:  81%|########  | 67/83 [02:07<00:30,  1.94s/it]

Evaluating policy:  82%|########1 | 68/83 [02:09<00:28,  1.92s/it]

Evaluating policy:  83%|########3 | 69/83 [02:10<00:26,  1.90s/it]

Evaluating policy:  84%|########4 | 70/83 [02:12<00:24,  1.86s/it]

Evaluating policy:  86%|########5 | 71/83 [02:14<00:22,  1.86s/it]

Evaluating policy:  87%|########6 | 72/83 [02:16<00:20,  1.83s/it]

Evaluating policy:  88%|########7 | 73/83 [02:18<00:18,  1.85s/it]

Evaluating policy:  89%|########9 | 74/83 [02:20<00:18,  2.01s/it]

Evaluating policy:  90%|######### | 75/83 [02:22<00:15,  1.96s/it]

Evaluating policy:  92%|#########1| 76/83 [02:24<00:13,  1.94s/it]

Evaluating policy:  93%|#########2| 77/83 [02:26<00:11,  1.90s/it]

Evaluating policy:  94%|#########3| 78/83 [02:27<00:09,  1.85s/it]

Evaluating policy:  95%|#########5| 79/83 [02:29<00:07,  1.86s/it]

Evaluating policy:  96%|#########6| 80/83 [02:31<00:05,  1.82s/it]

Evaluating policy:  98%|#########7| 81/83 [02:33<00:03,  1.83s/it]

Evaluating policy:  99%|#########8| 82/83 [02:35<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 83/83 [02:37<00:00,  2.02s/it]

Outer training loop:  99%|#########9| 49501/50000 [33:12:09<13:31:32, 97.58s/it]
Outer training loop:  99%|#########9| 49502/50000 [33:12:11<9:31:25, 68.85s/it] 
Outer training loop:  99%|#########9| 49503/50000 [33:12:13<6:43:47, 48.75s/it]
Outer training loop:  99%|#########9| 49504/50000 [33:12:15<4:46:24, 34.65s/it]
Outer training loop:  99%|#########9| 49505/50000 [33:12:17<3:24:36, 24.80s/it]
Outer training loop:  99%|#########9| 49506/50000 [33:12:18<2:27:21, 17.90s/it]
Outer training loop:  99%|#########9| 49507/50000 [33:12:20<1:47:25, 13.07s/it]
Outer training loop:  99%|#########9| 49508/50000 [33:12:22<1:19:36,  9.71s/it]
Outer training loop:  99%|#########9| 49509/50000 [33:12:24<1:00:01,  7.33s/it]
Outer training loop:  99%|#########9| 49510/50000 [33:12:26<46:30,  5.70s/it]  
Outer training loop:  99%|#########9| 49511/50000 [33:12:28<36:57,  4.54s/it]
Outer training loop:  99%|#########9| 49512/50000 [33:12:30<31:41,  3.90s/it]
Outer training loop:  99%|#########9| 49513/50000 [33:12:32<26:25,  3.26s/it]
Outer training loop:  99%|#########9| 49514/50000 [33:12:34<22:59,  2.84s/it]
Outer training loop:  99%|#########9| 49515/50000 [33:12:36<20:49,  2.58s/it]
Outer training loop:  99%|#########9| 49516/50000 [33:12:37<18:57,  2.35s/it]
Outer training loop:  99%|#########9| 49517/50000 [33:12:39<17:29,  2.17s/it]
Outer training loop:  99%|#########9| 49518/50000 [33:12:41<16:33,  2.06s/it]
Outer training loop:  99%|#########9| 49519/50000 [33:12:43<16:22,  2.04s/it]
Outer training loop:  99%|#########9| 49520/50000 [33:12:45<15:58,  2.00s/it]
Outer training loop:  99%|#########9| 49521/50000 [33:12:47<15:31,  1.95s/it]
Outer training loop:  99%|#########9| 49522/50000 [33:12:49<15:37,  1.96s/it]
Outer training loop:  99%|#########9| 49523/50000 [33:12:51<15:18,  1.93s/it]
Outer training loop:  99%|#########9| 49524/50000 [33:12:52<14:56,  1.88s/it]
Outer training loop:  99%|#########9| 49525/50000 [33:12:54<14:36,  1.85s/it]
Outer training loop:  99%|#########9| 49526/50000 [33:12:56<15:01,  1.90s/it]
Outer training loop:  99%|#########9| 49527/50000 [33:12:58<14:54,  1.89s/it]
Outer training loop:  99%|#########9| 49528/50000 [33:13:00<14:53,  1.89s/it]
Outer training loop:  99%|#########9| 49529/50000 [33:13:02<15:19,  1.95s/it]
Outer training loop:  99%|#########9| 49530/50000 [33:13:04<14:59,  1.91s/it]
Outer training loop:  99%|#########9| 49531/50000 [33:13:06<15:04,  1.93s/it]
Outer training loop:  99%|#########9| 49532/50000 [33:13:08<15:20,  1.97s/it]
Outer training loop:  99%|#########9| 49533/50000 [33:13:10<15:11,  1.95s/it]
Outer training loop:  99%|#########9| 49534/50000 [33:13:12<14:52,  1.91s/it]
Outer training loop:  99%|#########9| 49535/50000 [33:13:13<14:32,  1.88s/it]
Outer training loop:  99%|#########9| 49536/50000 [33:13:15<14:46,  1.91s/it]
Outer training loop:  99%|#########9| 49537/50000 [33:13:17<14:37,  1.90s/it]
Outer training loop:  99%|#########9| 49538/50000 [33:13:19<14:49,  1.92s/it]
Outer training loop:  99%|#########9| 49539/50000 [33:13:21<15:24,  2.00s/it]
Outer training loop:  99%|#########9| 49540/50000 [33:13:23<15:18,  2.00s/it]
Outer training loop:  99%|#########9| 49541/50000 [33:13:25<14:56,  1.95s/it]
Outer training loop:  99%|#########9| 49542/50000 [33:13:27<14:59,  1.96s/it]
Outer training loop:  99%|#########9| 49543/50000 [33:13:29<15:09,  1.99s/it]
Outer training loop:  99%|#########9| 49544/50000 [33:13:31<14:58,  1.97s/it]
Outer training loop:  99%|#########9| 49545/50000 [33:13:33<14:35,  1.92s/it]
Outer training loop:  99%|#########9| 49546/50000 [33:13:35<14:48,  1.96s/it]
Outer training loop:  99%|#########9| 49547/50000 [33:13:37<14:26,  1.91s/it]
Outer training loop:  99%|#########9| 49548/50000 [33:13:39<14:17,  1.90s/it]
Outer training loop:  99%|#########9| 49549/50000 [33:13:41<14:52,  1.98s/it]
Outer training loop:  99%|#########9| 49550/50000 [33:13:43<14:43,  1.96s/it]
Outer training loop:  99%|#########9| 49551/50000 [33:13:45<14:21,  1.92s/it]
Outer training loop:  99%|#########9| 49552/50000 [33:13:46<14:08,  1.89s/it]
Outer training loop:  99%|#########9| 49553/50000 [33:13:48<14:10,  1.90s/it]
Outer training loop:  99%|#########9| 49554/50000 [33:13:50<14:01,  1.89s/it]
Outer training loop:  99%|#########9| 49555/50000 [33:13:52<13:59,  1.89s/it]
Outer training loop:  99%|#########9| 49556/50000 [33:13:54<13:42,  1.85s/it]
Outer training loop:  99%|#########9| 49557/50000 [33:13:56<14:02,  1.90s/it]
Outer training loop:  99%|#########9| 49558/50000 [33:13:58<14:04,  1.91s/it]
Outer training loop:  99%|#########9| 49559/50000 [33:14:00<13:46,  1.88s/it]
Outer training loop:  99%|#########9| 49560/50000 [33:14:02<14:00,  1.91s/it]
Outer training loop:  99%|#########9| 49561/50000 [33:14:04<14:00,  1.91s/it]
Outer training loop:  99%|#########9| 49562/50000 [33:14:05<13:46,  1.89s/it]
Outer training loop:  99%|#########9| 49563/50000 [33:14:07<13:49,  1.90s/it]
Outer training loop:  99%|#########9| 49564/50000 [33:14:09<14:09,  1.95s/it]
Outer training loop:  99%|#########9| 49565/50000 [33:14:11<13:50,  1.91s/it]
Outer training loop:  99%|#########9| 49566/50000 [33:14:13<13:46,  1.91s/it]
Outer training loop:  99%|#########9| 49567/50000 [33:14:15<13:43,  1.90s/it]
Outer training loop:  99%|#########9| 49568/50000 [33:14:17<13:27,  1.87s/it]
Outer training loop:  99%|#########9| 49569/50000 [33:14:19<13:27,  1.87s/it]
Outer training loop:  99%|#########9| 49570/50000 [33:14:21<13:36,  1.90s/it]
Outer training loop:  99%|#########9| 49571/50000 [33:14:23<14:10,  1.98s/it]
Outer training loop:  99%|#########9| 49572/50000 [33:14:24<13:35,  1.91s/it]
Outer training loop:  99%|#########9| 49573/50000 [33:14:26<13:33,  1.90s/it]
Outer training loop:  99%|#########9| 49574/50000 [33:14:29<14:10,  2.00s/it]
Outer training loop:  99%|#########9| 49575/50000 [33:14:30<13:52,  1.96s/it]
Outer training loop:  99%|#########9| 49576/50000 [33:14:32<13:39,  1.93s/it]
Outer training loop:  99%|#########9| 49577/50000 [33:14:34<14:03,  1.99s/it]
Outer training loop:  99%|#########9| 49578/50000 [33:14:36<13:58,  1.99s/it]
Outer training loop:  99%|#########9| 49579/50000 [33:14:38<13:54,  1.98s/it]
Outer training loop:  99%|#########9| 49580/50000 [33:14:40<13:27,  1.92s/it]
Outer training loop:  99%|#########9| 49581/50000 [33:14:42<13:33,  1.94s/it]
Outer training loop:  99%|#########9| 49582/50000 [33:14:44<13:18,  1.91s/it]
Outer training loop:  99%|#########9| 49583/50000 [33:14:46<12:54,  1.86s/it]
Outer training loop:  99%|#########9| 49584/50000 [33:14:48<13:04,  1.89s/it]
Outer training loop:  99%|#########9| 49585/50000 [33:14:50<13:21,  1.93s/it]
Outer training loop:  99%|#########9| 49586/50000 [33:14:52<13:01,  1.89s/it]
Outer training loop:  99%|#########9| 49587/50000 [33:14:53<12:43,  1.85s/it]
Outer training loop:  99%|#########9| 49588/50000 [33:14:55<12:51,  1.87s/it]
Outer training loop:  99%|#########9| 49589/50000 [33:14:57<12:37,  1.84s/it]
Outer training loop:  99%|#########9| 49590/50000 [33:14:59<12:38,  1.85s/it]
Outer training loop:  99%|#########9| 49591/50000 [33:15:01<13:01,  1.91s/it]
Outer training loop:  99%|#########9| 49592/50000 [33:15:03<12:55,  1.90s/it]
Outer training loop:  99%|#########9| 49593/50000 [33:15:05<12:51,  1.90s/it]
Outer training loop:  99%|#########9| 49594/50000 [33:15:07<13:28,  1.99s/it]
Outer training loop:  99%|#########9| 49595/50000 [33:15:09<13:11,  1.95s/it]
Outer training loop:  99%|#########9| 49596/50000 [33:15:11<12:49,  1.90s/it]
Outer training loop:  99%|#########9| 49597/50000 [33:15:12<12:29,  1.86s/it]
Outer training loop:  99%|#########9| 49598/50000 [33:15:14<12:46,  1.91s/it]
Outer training loop:  99%|#########9| 49599/50000 [33:15:16<12:26,  1.86s/it]
Outer training loop:  99%|#########9| 49600/50000 [33:15:18<12:21,  1.85s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.77s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.81s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.78s/it]

Outer training loop:  99%|#########9| 49601/50000 [33:15:25<23:34,  3.55s/it]
Outer training loop:  99%|#########9| 49602/50000 [33:15:27<20:25,  3.08s/it]
Outer training loop:  99%|#########9| 49603/50000 [33:15:29<18:06,  2.74s/it]
Outer training loop:  99%|#########9| 49604/50000 [33:15:31<16:14,  2.46s/it]
Outer training loop:  99%|#########9| 49605/50000 [33:15:33<15:26,  2.35s/it]
Outer training loop:  99%|#########9| 49606/50000 [33:15:35<14:35,  2.22s/it]
Outer training loop:  99%|#########9| 49607/50000 [33:15:37<13:46,  2.10s/it]
Outer training loop:  99%|#########9| 49608/50000 [33:15:39<13:09,  2.01s/it]
Outer training loop:  99%|#########9| 49609/50000 [33:15:41<13:13,  2.03s/it]
Outer training loop:  99%|#########9| 49610/50000 [33:15:43<12:47,  1.97s/it]
Outer training loop:  99%|#########9| 49611/50000 [33:15:45<12:37,  1.95s/it]
Outer training loop:  99%|#########9| 49612/50000 [33:15:47<12:43,  1.97s/it]
Outer training loop:  99%|#########9| 49613/50000 [33:15:48<12:15,  1.90s/it]
Outer training loop:  99%|#########9| 49614/50000 [33:15:50<12:13,  1.90s/it]
Outer training loop:  99%|#########9| 49615/50000 [33:15:52<12:12,  1.90s/it]
Outer training loop:  99%|#########9| 49616/50000 [33:15:54<11:58,  1.87s/it]
Outer training loop:  99%|#########9| 49617/50000 [33:15:56<11:52,  1.86s/it]
Outer training loop:  99%|#########9| 49618/50000 [33:15:58<11:48,  1.85s/it]
Outer training loop:  99%|#########9| 49619/50000 [33:16:00<12:17,  1.93s/it]
Outer training loop:  99%|#########9| 49620/50000 [33:16:02<12:07,  1.91s/it]
Outer training loop:  99%|#########9| 49621/50000 [33:16:03<11:54,  1.88s/it]
Outer training loop:  99%|#########9| 49622/50000 [33:16:05<12:05,  1.92s/it]
Outer training loop:  99%|#########9| 49623/50000 [33:16:07<12:07,  1.93s/it]
Outer training loop:  99%|#########9| 49624/50000 [33:16:09<11:58,  1.91s/it]
Outer training loop:  99%|#########9| 49625/50000 [33:16:11<12:03,  1.93s/it]
Outer training loop:  99%|#########9| 49626/50000 [33:16:13<12:02,  1.93s/it]
Outer training loop:  99%|#########9| 49627/50000 [33:16:15<11:38,  1.87s/it]
Outer training loop:  99%|#########9| 49628/50000 [33:16:17<11:21,  1.83s/it]
Outer training loop:  99%|#########9| 49629/50000 [33:16:19<11:48,  1.91s/it]
Outer training loop:  99%|#########9| 49630/50000 [33:16:21<11:43,  1.90s/it]
Outer training loop:  99%|#########9| 49631/50000 [33:16:22<11:38,  1.89s/it]
Outer training loop:  99%|#########9| 49632/50000 [33:16:24<11:29,  1.87s/it]
Outer training loop:  99%|#########9| 49633/50000 [33:16:26<11:50,  1.94s/it]
Outer training loop:  99%|#########9| 49634/50000 [33:16:28<11:39,  1.91s/it]
Outer training loop:  99%|#########9| 49635/50000 [33:16:30<11:16,  1.85s/it]
Outer training loop:  99%|#########9| 49636/50000 [33:16:32<11:29,  1.89s/it]
Outer training loop:  99%|#########9| 49637/50000 [33:16:34<11:33,  1.91s/it]
Outer training loop:  99%|#########9| 49638/50000 [33:16:36<11:17,  1.87s/it]
Outer training loop:  99%|#########9| 49639/50000 [33:16:38<11:13,  1.86s/it]
Outer training loop:  99%|#########9| 49640/50000 [33:16:39<11:15,  1.88s/it]
Outer training loop:  99%|#########9| 49641/50000 [33:16:41<11:08,  1.86s/it]
Outer training loop:  99%|#########9| 49642/50000 [33:16:43<10:59,  1.84s/it]
Outer training loop:  99%|#########9| 49643/50000 [33:16:45<11:10,  1.88s/it]
Outer training loop:  99%|#########9| 49644/50000 [33:16:47<10:54,  1.84s/it]
Outer training loop:  99%|#########9| 49645/50000 [33:16:48<10:37,  1.80s/it]
Outer training loop:  99%|#########9| 49646/50000 [33:16:50<10:45,  1.82s/it]
Outer training loop:  99%|#########9| 49647/50000 [33:16:52<10:58,  1.87s/it]
Outer training loop:  99%|#########9| 49648/50000 [33:16:54<11:02,  1.88s/it]
Outer training loop:  99%|#########9| 49649/50000 [33:16:56<10:50,  1.85s/it]
Outer training loop:  99%|#########9| 49650/50000 [33:16:58<11:15,  1.93s/it]
Outer training loop:  99%|#########9| 49651/50000 [33:17:00<10:57,  1.88s/it]
Outer training loop:  99%|#########9| 49652/50000 [33:17:02<10:45,  1.85s/it]
Outer training loop:  99%|#########9| 49653/50000 [33:17:04<10:57,  1.90s/it]
Outer training loop:  99%|#########9| 49654/50000 [33:17:05<10:44,  1.86s/it]
Outer training loop:  99%|#########9| 49655/50000 [33:17:07<10:41,  1.86s/it]
Outer training loop:  99%|#########9| 49656/50000 [33:17:09<10:33,  1.84s/it]
Outer training loop:  99%|#########9| 49657/50000 [33:17:11<10:45,  1.88s/it]
Outer training loop:  99%|#########9| 49658/50000 [33:17:13<10:40,  1.87s/it]
Outer training loop:  99%|#########9| 49659/50000 [33:17:15<10:24,  1.83s/it]
Outer training loop:  99%|#########9| 49660/50000 [33:17:16<10:08,  1.79s/it]
Outer training loop:  99%|#########9| 49661/50000 [33:17:18<10:14,  1.81s/it]
Outer training loop:  99%|#########9| 49662/50000 [33:17:20<10:07,  1.80s/it]
Outer training loop:  99%|#########9| 49663/50000 [33:17:22<10:03,  1.79s/it]
Outer training loop:  99%|#########9| 49664/50000 [33:17:24<10:06,  1.81s/it]
Outer training loop:  99%|#########9| 49665/50000 [33:17:26<10:16,  1.84s/it]
Outer training loop:  99%|#########9| 49666/50000 [33:17:27<10:15,  1.84s/it]
Outer training loop:  99%|#########9| 49667/50000 [33:17:29<10:22,  1.87s/it]
Outer training loop:  99%|#########9| 49668/50000 [33:17:31<10:09,  1.84s/it]
Outer training loop:  99%|#########9| 49669/50000 [33:17:33<10:13,  1.85s/it]
Outer training loop:  99%|#########9| 49670/50000 [33:17:35<10:29,  1.91s/it]
Outer training loop:  99%|#########9| 49671/50000 [33:17:37<10:28,  1.91s/it]
Outer training loop:  99%|#########9| 49672/50000 [33:17:39<10:16,  1.88s/it]
Outer training loop:  99%|#########9| 49673/50000 [33:17:41<10:11,  1.87s/it]
Outer training loop:  99%|#########9| 49674/50000 [33:17:43<10:16,  1.89s/it]
Outer training loop:  99%|#########9| 49675/50000 [33:17:44<10:10,  1.88s/it]
Outer training loop:  99%|#########9| 49676/50000 [33:17:46<10:18,  1.91s/it]
Outer training loop:  99%|#########9| 49677/50000 [33:17:48<10:32,  1.96s/it]
Outer training loop:  99%|#########9| 49678/50000 [33:17:50<10:06,  1.88s/it]
Outer training loop:  99%|#########9| 49679/50000 [33:17:52<09:57,  1.86s/it]
Outer training loop:  99%|#########9| 49680/50000 [33:17:54<10:05,  1.89s/it]
Outer training loop:  99%|#########9| 49681/50000 [33:17:56<10:08,  1.91s/it]
Outer training loop:  99%|#########9| 49682/50000 [33:17:58<09:51,  1.86s/it]
Outer training loop:  99%|#########9| 49683/50000 [33:17:59<09:43,  1.84s/it]
Outer training loop:  99%|#########9| 49684/50000 [33:18:01<09:55,  1.89s/it]
Outer training loop:  99%|#########9| 49685/50000 [33:18:03<09:42,  1.85s/it]
Outer training loop:  99%|#########9| 49686/50000 [33:18:05<09:40,  1.85s/it]
Outer training loop:  99%|#########9| 49687/50000 [33:18:07<09:51,  1.89s/it]
Outer training loop:  99%|#########9| 49688/50000 [33:18:09<09:32,  1.83s/it]
Outer training loop:  99%|#########9| 49689/50000 [33:18:11<09:37,  1.86s/it]
Outer training loop:  99%|#########9| 49690/50000 [33:18:12<09:32,  1.85s/it]
Outer training loop:  99%|#########9| 49691/50000 [33:18:14<09:51,  1.91s/it]
Outer training loop:  99%|#########9| 49692/50000 [33:18:16<09:38,  1.88s/it]
Outer training loop:  99%|#########9| 49693/50000 [33:18:18<09:23,  1.84s/it]
Outer training loop:  99%|#########9| 49694/50000 [33:18:20<09:33,  1.87s/it]
Outer training loop:  99%|#########9| 49695/50000 [33:18:22<09:23,  1.85s/it]
Outer training loop:  99%|#########9| 49696/50000 [33:18:24<09:15,  1.83s/it]
Outer training loop:  99%|#########9| 49697/50000 [33:18:25<09:19,  1.85s/it]
Outer training loop:  99%|#########9| 49698/50000 [33:18:28<09:41,  1.93s/it]
Outer training loop:  99%|#########9| 49699/50000 [33:18:29<09:27,  1.89s/it]
Outer training loop:  99%|#########9| 49700/50000 [33:18:31<09:26,  1.89s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.81s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.83s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.83s/it]

Outer training loop:  99%|#########9| 49701/50000 [33:18:39<17:49,  3.58s/it]
Outer training loop:  99%|#########9| 49702/50000 [33:18:41<15:04,  3.04s/it]
Outer training loop:  99%|#########9| 49703/50000 [33:18:42<13:18,  2.69s/it]
Outer training loop:  99%|#########9| 49704/50000 [33:18:44<11:58,  2.43s/it]
Outer training loop:  99%|#########9| 49705/50000 [33:18:46<11:13,  2.28s/it]
Outer training loop:  99%|#########9| 49706/50000 [33:18:48<10:34,  2.16s/it]
Outer training loop:  99%|#########9| 49707/50000 [33:18:50<10:06,  2.07s/it]
Outer training loop:  99%|#########9| 49708/50000 [33:18:52<10:00,  2.05s/it]
Outer training loop:  99%|#########9| 49709/50000 [33:18:54<09:27,  1.95s/it]
Outer training loop:  99%|#########9| 49710/50000 [33:18:55<09:15,  1.92s/it]
Outer training loop:  99%|#########9| 49711/50000 [33:18:57<09:11,  1.91s/it]
Outer training loop:  99%|#########9| 49712/50000 [33:18:59<09:14,  1.92s/it]
Outer training loop:  99%|#########9| 49713/50000 [33:19:01<09:01,  1.89s/it]
Outer training loop:  99%|#########9| 49714/50000 [33:19:03<08:51,  1.86s/it]
Outer training loop:  99%|#########9| 49715/50000 [33:19:05<09:03,  1.91s/it]
Outer training loop:  99%|#########9| 49716/50000 [33:19:07<08:54,  1.88s/it]
Outer training loop:  99%|#########9| 49717/50000 [33:19:09<08:47,  1.86s/it]
Outer training loop:  99%|#########9| 49718/50000 [33:19:10<08:37,  1.83s/it]
Outer training loop:  99%|#########9| 49719/50000 [33:19:12<08:51,  1.89s/it]
Outer training loop:  99%|#########9| 49720/50000 [33:19:14<08:38,  1.85s/it]
Outer training loop:  99%|#########9| 49721/50000 [33:19:16<08:29,  1.83s/it]
Outer training loop:  99%|#########9| 49722/50000 [33:19:18<08:27,  1.83s/it]
Outer training loop:  99%|#########9| 49723/50000 [33:19:20<09:05,  1.97s/it]
Outer training loop:  99%|#########9| 49724/50000 [33:19:22<08:52,  1.93s/it]
Outer training loop:  99%|#########9| 49725/50000 [33:19:24<08:37,  1.88s/it]
Outer training loop:  99%|#########9| 49726/50000 [33:19:26<08:42,  1.91s/it]
Outer training loop:  99%|#########9| 49727/50000 [33:19:27<08:30,  1.87s/it]
Outer training loop:  99%|#########9| 49728/50000 [33:19:29<08:17,  1.83s/it]
Outer training loop:  99%|#########9| 49729/50000 [33:19:31<08:14,  1.82s/it]
Outer training loop:  99%|#########9| 49730/50000 [33:19:33<08:20,  1.86s/it]
Outer training loop:  99%|#########9| 49731/50000 [33:19:35<08:23,  1.87s/it]
Outer training loop:  99%|#########9| 49732/50000 [33:19:37<08:17,  1.86s/it]
Outer training loop:  99%|#########9| 49733/50000 [33:19:39<08:23,  1.88s/it]
Outer training loop:  99%|#########9| 49734/50000 [33:19:40<08:17,  1.87s/it]
Outer training loop:  99%|#########9| 49735/50000 [33:19:42<08:03,  1.82s/it]
Outer training loop:  99%|#########9| 49736/50000 [33:19:44<08:20,  1.90s/it]
Outer training loop:  99%|#########9| 49737/50000 [33:19:46<08:09,  1.86s/it]
Outer training loop:  99%|#########9| 49738/50000 [33:19:48<08:19,  1.91s/it]
Outer training loop:  99%|#########9| 49739/50000 [33:19:50<08:18,  1.91s/it]
Outer training loop:  99%|#########9| 49740/50000 [33:19:52<08:06,  1.87s/it]
Outer training loop:  99%|#########9| 49741/50000 [33:19:53<07:57,  1.84s/it]
Outer training loop:  99%|#########9| 49742/50000 [33:19:55<07:44,  1.80s/it]
Outer training loop:  99%|#########9| 49743/50000 [33:19:57<07:53,  1.84s/it]
Outer training loop:  99%|#########9| 49744/50000 [33:19:59<07:52,  1.85s/it]
Outer training loop:  99%|#########9| 49745/50000 [33:20:01<07:56,  1.87s/it]
Outer training loop:  99%|#########9| 49746/50000 [33:20:03<08:00,  1.89s/it]
Outer training loop:  99%|#########9| 49747/50000 [33:20:05<07:53,  1.87s/it]
Outer training loop:  99%|#########9| 49748/50000 [33:20:06<07:46,  1.85s/it]
Outer training loop:  99%|#########9| 49749/50000 [33:20:08<07:56,  1.90s/it]
Outer training loop: 100%|#########9| 49750/50000 [33:20:10<07:54,  1.90s/it]
Outer training loop: 100%|#########9| 49751/50000 [33:20:12<08:05,  1.95s/it]
Outer training loop: 100%|#########9| 49752/50000 [33:20:14<08:11,  1.98s/it]
Outer training loop: 100%|#########9| 49753/50000 [33:20:16<07:58,  1.94s/it]
Outer training loop: 100%|#########9| 49754/50000 [33:20:18<07:42,  1.88s/it]
Outer training loop: 100%|#########9| 49755/50000 [33:20:20<07:31,  1.84s/it]
Outer training loop: 100%|#########9| 49756/50000 [33:20:22<07:38,  1.88s/it]
Outer training loop: 100%|#########9| 49757/50000 [33:20:24<07:34,  1.87s/it]
Outer training loop: 100%|#########9| 49758/50000 [33:20:26<07:37,  1.89s/it]
Outer training loop: 100%|#########9| 49759/50000 [33:20:27<07:38,  1.90s/it]
Outer training loop: 100%|#########9| 49760/50000 [33:20:29<07:31,  1.88s/it]
Outer training loop: 100%|#########9| 49761/50000 [33:20:31<07:24,  1.86s/it]
Outer training loop: 100%|#########9| 49762/50000 [33:20:33<07:14,  1.83s/it]
Outer training loop: 100%|#########9| 49763/50000 [33:20:35<07:31,  1.90s/it]
Outer training loop: 100%|#########9| 49764/50000 [33:20:37<07:20,  1.87s/it]
Outer training loop: 100%|#########9| 49765/50000 [33:20:39<07:20,  1.88s/it]
Outer training loop: 100%|#########9| 49766/50000 [33:20:41<07:29,  1.92s/it]
Outer training loop: 100%|#########9| 49767/50000 [33:20:42<07:11,  1.85s/it]
Outer training loop: 100%|#########9| 49768/50000 [33:20:44<07:05,  1.83s/it]
Outer training loop: 100%|#########9| 49769/50000 [33:20:46<06:58,  1.81s/it]
Outer training loop: 100%|#########9| 49770/50000 [33:20:48<07:15,  1.89s/it]
Outer training loop: 100%|#########9| 49771/50000 [33:20:50<07:13,  1.89s/it]
Outer training loop: 100%|#########9| 49772/50000 [33:20:52<07:04,  1.86s/it]
Outer training loop: 100%|#########9| 49773/50000 [33:20:54<07:21,  1.95s/it]
Outer training loop: 100%|#########9| 49774/50000 [33:20:56<07:22,  1.96s/it]
Outer training loop: 100%|#########9| 49775/50000 [33:20:58<07:10,  1.91s/it]
Outer training loop: 100%|#########9| 49776/50000 [33:20:59<07:08,  1.91s/it]
Outer training loop: 100%|#########9| 49777/50000 [33:21:02<07:18,  1.97s/it]
Outer training loop: 100%|#########9| 49778/50000 [33:21:03<07:09,  1.93s/it]
Outer training loop: 100%|#########9| 49779/50000 [33:21:05<06:58,  1.89s/it]
Outer training loop: 100%|#########9| 49780/50000 [33:21:07<07:04,  1.93s/it]
Outer training loop: 100%|#########9| 49781/50000 [33:21:09<07:02,  1.93s/it]
Outer training loop: 100%|#########9| 49782/50000 [33:21:11<06:50,  1.88s/it]
Outer training loop: 100%|#########9| 49783/50000 [33:21:13<06:40,  1.84s/it]
Outer training loop: 100%|#########9| 49784/50000 [33:21:15<06:51,  1.91s/it]
Outer training loop: 100%|#########9| 49785/50000 [33:21:17<06:44,  1.88s/it]
Outer training loop: 100%|#########9| 49786/50000 [33:21:18<06:40,  1.87s/it]
Outer training loop: 100%|#########9| 49787/50000 [33:21:21<06:51,  1.93s/it]
Outer training loop: 100%|#########9| 49788/50000 [33:21:22<06:42,  1.90s/it]
Outer training loop: 100%|#########9| 49789/50000 [33:21:24<06:37,  1.88s/it]
Outer training loop: 100%|#########9| 49790/50000 [33:21:26<06:26,  1.84s/it]
Outer training loop: 100%|#########9| 49791/50000 [33:21:28<06:38,  1.91s/it]
Outer training loop: 100%|#########9| 49792/50000 [33:21:30<06:31,  1.88s/it]
Outer training loop: 100%|#########9| 49793/50000 [33:21:32<06:22,  1.85s/it]
Outer training loop: 100%|#########9| 49794/50000 [33:21:34<06:32,  1.91s/it]
Outer training loop: 100%|#########9| 49795/50000 [33:21:36<06:30,  1.90s/it]
Outer training loop: 100%|#########9| 49796/50000 [33:21:37<06:24,  1.88s/it]
Outer training loop: 100%|#########9| 49797/50000 [33:21:40<06:40,  1.97s/it]
Outer training loop: 100%|#########9| 49798/50000 [33:21:41<06:24,  1.90s/it]
Outer training loop: 100%|#########9| 49799/50000 [33:21:43<06:13,  1.86s/it]
Outer training loop: 100%|#########9| 49800/50000 [33:21:45<06:07,  1.84s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.86s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.90s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop: 100%|#########9| 49801/50000 [33:21:53<11:57,  3.61s/it]
Outer training loop: 100%|#########9| 49802/50000 [33:21:54<10:11,  3.09s/it]
Outer training loop: 100%|#########9| 49803/50000 [33:21:56<08:55,  2.72s/it]
Outer training loop: 100%|#########9| 49804/50000 [33:21:58<08:03,  2.46s/it]
Outer training loop: 100%|#########9| 49805/50000 [33:22:00<07:26,  2.29s/it]
Outer training loop: 100%|#########9| 49806/50000 [33:22:02<06:56,  2.14s/it]
Outer training loop: 100%|#########9| 49807/50000 [33:22:04<06:35,  2.05s/it]
Outer training loop: 100%|#########9| 49808/50000 [33:22:06<06:26,  2.01s/it]
Outer training loop: 100%|#########9| 49809/50000 [33:22:08<06:24,  2.01s/it]
Outer training loop: 100%|#########9| 49810/50000 [33:22:09<06:08,  1.94s/it]
Outer training loop: 100%|#########9| 49811/50000 [33:22:11<06:00,  1.91s/it]
Outer training loop: 100%|#########9| 49812/50000 [33:22:13<06:07,  1.95s/it]
Outer training loop: 100%|#########9| 49813/50000 [33:22:15<05:56,  1.91s/it]
Outer training loop: 100%|#########9| 49814/50000 [33:22:17<05:46,  1.86s/it]
Outer training loop: 100%|#########9| 49815/50000 [33:22:19<05:38,  1.83s/it]
Outer training loop: 100%|#########9| 49816/50000 [33:22:21<05:49,  1.90s/it]
Outer training loop: 100%|#########9| 49817/50000 [33:22:23<05:47,  1.90s/it]
Outer training loop: 100%|#########9| 49818/50000 [33:22:24<05:38,  1.86s/it]
Outer training loop: 100%|#########9| 49819/50000 [33:22:26<05:32,  1.83s/it]
Outer training loop: 100%|#########9| 49820/50000 [33:22:28<05:46,  1.92s/it]
Outer training loop: 100%|#########9| 49821/50000 [33:22:30<05:38,  1.89s/it]
Outer training loop: 100%|#########9| 49822/50000 [33:22:32<05:30,  1.86s/it]
Outer training loop: 100%|#########9| 49823/50000 [33:22:34<05:38,  1.91s/it]
Outer training loop: 100%|#########9| 49824/50000 [33:22:36<05:35,  1.91s/it]
Outer training loop: 100%|#########9| 49825/50000 [33:22:38<05:26,  1.86s/it]
Outer training loop: 100%|#########9| 49826/50000 [33:22:40<05:32,  1.91s/it]
Outer training loop: 100%|#########9| 49827/50000 [33:22:41<05:26,  1.89s/it]
Outer training loop: 100%|#########9| 49828/50000 [33:22:43<05:17,  1.85s/it]
Outer training loop: 100%|#########9| 49829/50000 [33:22:45<05:14,  1.84s/it]
Outer training loop: 100%|#########9| 49830/50000 [33:22:47<05:23,  1.90s/it]
Outer training loop: 100%|#########9| 49831/50000 [33:22:49<05:13,  1.86s/it]
Outer training loop: 100%|#########9| 49832/50000 [33:22:51<05:11,  1.85s/it]
Outer training loop: 100%|#########9| 49833/50000 [33:22:53<05:16,  1.89s/it]
Outer training loop: 100%|#########9| 49834/50000 [33:22:54<05:07,  1.85s/it]
Outer training loop: 100%|#########9| 49835/50000 [33:22:56<05:05,  1.85s/it]
Outer training loop: 100%|#########9| 49836/50000 [33:22:58<05:12,  1.91s/it]
Outer training loop: 100%|#########9| 49837/50000 [33:23:00<05:06,  1.88s/it]
Outer training loop: 100%|#########9| 49838/50000 [33:23:02<05:02,  1.87s/it]
Outer training loop: 100%|#########9| 49839/50000 [33:23:04<04:57,  1.85s/it]
Outer training loop: 100%|#########9| 49840/50000 [33:23:06<05:01,  1.88s/it]
Outer training loop: 100%|#########9| 49841/50000 [33:23:08<05:00,  1.89s/it]
Outer training loop: 100%|#########9| 49842/50000 [33:23:09<04:52,  1.85s/it]
Outer training loop: 100%|#########9| 49843/50000 [33:23:11<05:03,  1.93s/it]
Outer training loop: 100%|#########9| 49844/50000 [33:23:13<04:51,  1.87s/it]
Outer training loop: 100%|#########9| 49845/50000 [33:23:15<04:44,  1.83s/it]
Outer training loop: 100%|#########9| 49846/50000 [33:23:17<04:40,  1.82s/it]
Outer training loop: 100%|#########9| 49847/50000 [33:23:19<04:51,  1.90s/it]
Outer training loop: 100%|#########9| 49848/50000 [33:23:21<04:43,  1.86s/it]
Outer training loop: 100%|#########9| 49849/50000 [33:23:22<04:39,  1.85s/it]
Outer training loop: 100%|#########9| 49850/50000 [33:23:24<04:43,  1.89s/it]
Outer training loop: 100%|#########9| 49851/50000 [33:23:26<04:43,  1.91s/it]
Outer training loop: 100%|#########9| 49852/50000 [33:23:28<04:39,  1.89s/it]
Outer training loop: 100%|#########9| 49853/50000 [33:23:30<04:34,  1.87s/it]
Outer training loop: 100%|#########9| 49854/50000 [33:23:32<04:35,  1.89s/it]
Outer training loop: 100%|#########9| 49855/50000 [33:23:34<04:33,  1.89s/it]
Outer training loop: 100%|#########9| 49856/50000 [33:23:36<04:27,  1.86s/it]
Outer training loop: 100%|#########9| 49857/50000 [33:23:38<04:31,  1.90s/it]
Outer training loop: 100%|#########9| 49858/50000 [33:23:39<04:30,  1.91s/it]
Outer training loop: 100%|#########9| 49859/50000 [33:23:41<04:24,  1.88s/it]
Outer training loop: 100%|#########9| 49860/50000 [33:23:43<04:29,  1.93s/it]
Outer training loop: 100%|#########9| 49861/50000 [33:23:45<04:25,  1.91s/it]
Outer training loop: 100%|#########9| 49862/50000 [33:23:47<04:17,  1.86s/it]
Outer training loop: 100%|#########9| 49863/50000 [33:23:49<04:08,  1.82s/it]
Outer training loop: 100%|#########9| 49864/50000 [33:23:51<04:19,  1.91s/it]
Outer training loop: 100%|#########9| 49865/50000 [33:23:53<04:11,  1.86s/it]
Outer training loop: 100%|#########9| 49866/50000 [33:23:54<04:08,  1.85s/it]
Outer training loop: 100%|#########9| 49867/50000 [33:23:56<04:11,  1.89s/it]
Outer training loop: 100%|#########9| 49868/50000 [33:23:58<04:16,  1.94s/it]
Outer training loop: 100%|#########9| 49869/50000 [33:24:00<04:09,  1.90s/it]
Outer training loop: 100%|#########9| 49870/50000 [33:24:02<04:06,  1.90s/it]
Outer training loop: 100%|#########9| 49871/50000 [33:24:04<04:14,  1.98s/it]
Outer training loop: 100%|#########9| 49872/50000 [33:24:06<04:07,  1.93s/it]
Outer training loop: 100%|#########9| 49873/50000 [33:24:08<03:59,  1.89s/it]
Outer training loop: 100%|#########9| 49874/50000 [33:24:10<03:59,  1.90s/it]
Outer training loop: 100%|#########9| 49875/50000 [33:24:12<04:01,  1.93s/it]
Outer training loop: 100%|#########9| 49876/50000 [33:24:14<03:59,  1.93s/it]
Outer training loop: 100%|#########9| 49877/50000 [33:24:16<03:51,  1.88s/it]
Outer training loop: 100%|#########9| 49878/50000 [33:24:18<03:56,  1.94s/it]
Outer training loop: 100%|#########9| 49879/50000 [33:24:19<03:47,  1.88s/it]
Outer training loop: 100%|#########9| 49880/50000 [33:24:21<03:42,  1.85s/it]
Outer training loop: 100%|#########9| 49881/50000 [33:24:23<03:43,  1.88s/it]
Outer training loop: 100%|#########9| 49882/50000 [33:24:25<03:39,  1.86s/it]
Outer training loop: 100%|#########9| 49883/50000 [33:24:27<03:39,  1.87s/it]
Outer training loop: 100%|#########9| 49884/50000 [33:24:29<03:38,  1.88s/it]
Outer training loop: 100%|#########9| 49885/50000 [33:24:31<03:44,  1.95s/it]
Outer training loop: 100%|#########9| 49886/50000 [33:24:33<03:36,  1.90s/it]
Outer training loop: 100%|#########9| 49887/50000 [33:24:34<03:34,  1.90s/it]
Outer training loop: 100%|#########9| 49888/50000 [33:24:37<03:40,  1.97s/it]
Outer training loop: 100%|#########9| 49889/50000 [33:24:38<03:31,  1.91s/it]
Outer training loop: 100%|#########9| 49890/50000 [33:24:40<03:26,  1.88s/it]
Outer training loop: 100%|#########9| 49891/50000 [33:24:42<03:27,  1.90s/it]
Outer training loop: 100%|#########9| 49892/50000 [33:24:44<03:24,  1.89s/it]
Outer training loop: 100%|#########9| 49893/50000 [33:24:46<03:16,  1.84s/it]
Outer training loop: 100%|#########9| 49894/50000 [33:24:47<03:12,  1.82s/it]
Outer training loop: 100%|#########9| 49895/50000 [33:24:50<03:17,  1.88s/it]
Outer training loop: 100%|#########9| 49896/50000 [33:24:51<03:12,  1.86s/it]
Outer training loop: 100%|#########9| 49897/50000 [33:24:53<03:10,  1.85s/it]
Outer training loop: 100%|#########9| 49898/50000 [33:24:55<03:17,  1.93s/it]
Outer training loop: 100%|#########9| 49899/50000 [33:24:57<03:11,  1.90s/it]
Outer training loop: 100%|#########9| 49900/50000 [33:24:59<03:11,  1.92s/it]

Evaluating policy:   0%|          | 0/3 [00:00<?, ?it/s]

Evaluating policy:  33%|###3      | 1/3 [00:01<00:03,  1.93s/it]

Evaluating policy:  67%|######6   | 2/3 [00:03<00:01,  1.96s/it]

Evaluating policy: 100%|##########| 3/3 [00:05<00:00,  1.88s/it]

Outer training loop: 100%|#########9| 49901/50000 [33:25:06<05:52,  3.56s/it]
Outer training loop: 100%|#########9| 49902/50000 [33:25:08<04:59,  3.06s/it]
Outer training loop: 100%|#########9| 49903/50000 [33:25:10<04:19,  2.68s/it]
Outer training loop: 100%|#########9| 49904/50000 [33:25:12<03:51,  2.41s/it]
Outer training loop: 100%|#########9| 49905/50000 [33:25:14<03:38,  2.30s/it]
Outer training loop: 100%|#########9| 49906/50000 [33:25:16<03:23,  2.16s/it]
Outer training loop: 100%|#########9| 49907/50000 [33:25:18<03:10,  2.04s/it]
Outer training loop: 100%|#########9| 49908/50000 [33:25:19<03:03,  1.99s/it]
Outer training loop: 100%|#########9| 49909/50000 [33:25:22<03:03,  2.02s/it]
Outer training loop: 100%|#########9| 49910/50000 [33:25:23<02:57,  1.97s/it]
Outer training loop: 100%|#########9| 49911/50000 [33:25:25<02:52,  1.94s/it]
Outer training loop: 100%|#########9| 49912/50000 [33:25:27<02:51,  1.95s/it]
Outer training loop: 100%|#########9| 49913/50000 [33:25:29<02:46,  1.92s/it]
Outer training loop: 100%|#########9| 49914/50000 [33:25:31<02:40,  1.87s/it]
Outer training loop: 100%|#########9| 49915/50000 [33:25:33<02:41,  1.90s/it]
Outer training loop: 100%|#########9| 49916/50000 [33:25:35<02:35,  1.85s/it]
Outer training loop: 100%|#########9| 49917/50000 [33:25:37<02:39,  1.92s/it]
Outer training loop: 100%|#########9| 49918/50000 [33:25:39<02:40,  1.96s/it]
Outer training loop: 100%|#########9| 49919/50000 [33:25:41<02:36,  1.94s/it]
Outer training loop: 100%|#########9| 49920/50000 [33:25:43<02:35,  1.94s/it]
Outer training loop: 100%|#########9| 49921/50000 [33:25:44<02:31,  1.91s/it]
Outer training loop: 100%|#########9| 49922/50000 [33:25:46<02:32,  1.95s/it]
Outer training loop: 100%|#########9| 49923/50000 [33:25:48<02:28,  1.93s/it]
Outer training loop: 100%|#########9| 49924/50000 [33:25:50<02:23,  1.89s/it]
Outer training loop: 100%|#########9| 49925/50000 [33:25:52<02:24,  1.93s/it]
Outer training loop: 100%|#########9| 49926/50000 [33:25:54<02:19,  1.88s/it]
Outer training loop: 100%|#########9| 49927/50000 [33:25:56<02:15,  1.85s/it]
Outer training loop: 100%|#########9| 49928/50000 [33:25:58<02:17,  1.91s/it]
Outer training loop: 100%|#########9| 49929/50000 [33:25:59<02:12,  1.86s/it]
Outer training loop: 100%|#########9| 49930/50000 [33:26:01<02:11,  1.88s/it]
Outer training loop: 100%|#########9| 49931/50000 [33:26:03<02:07,  1.84s/it]
Outer training loop: 100%|#########9| 49932/50000 [33:26:05<02:12,  1.94s/it]
Outer training loop: 100%|#########9| 49933/50000 [33:26:07<02:06,  1.89s/it]
Outer training loop: 100%|#########9| 49934/50000 [33:26:09<02:01,  1.84s/it]
Outer training loop: 100%|#########9| 49935/50000 [33:26:11<02:01,  1.87s/it]
Outer training loop: 100%|#########9| 49936/50000 [33:26:13<02:00,  1.88s/it]
Outer training loop: 100%|#########9| 49937/50000 [33:26:14<01:54,  1.82s/it]
Outer training loop: 100%|#########9| 49938/50000 [33:26:16<01:51,  1.81s/it]
Outer training loop: 100%|#########9| 49939/50000 [33:26:18<01:52,  1.85s/it]
Outer training loop: 100%|#########9| 49940/50000 [33:26:20<01:52,  1.88s/it]
Outer training loop: 100%|#########9| 49941/50000 [33:26:22<01:50,  1.87s/it]
Outer training loop: 100%|#########9| 49942/50000 [33:26:24<01:48,  1.88s/it]
Outer training loop: 100%|#########9| 49943/50000 [33:26:26<01:45,  1.85s/it]
Outer training loop: 100%|#########9| 49944/50000 [33:26:27<01:44,  1.86s/it]
Outer training loop: 100%|#########9| 49945/50000 [33:26:29<01:40,  1.83s/it]
Outer training loop: 100%|#########9| 49946/50000 [33:26:31<01:40,  1.85s/it]
Outer training loop: 100%|#########9| 49947/50000 [33:26:33<01:36,  1.82s/it]
Outer training loop: 100%|#########9| 49948/50000 [33:26:35<01:34,  1.82s/it]
Outer training loop: 100%|#########9| 49949/50000 [33:26:37<01:35,  1.87s/it]
Outer training loop: 100%|#########9| 49950/50000 [33:26:38<01:33,  1.86s/it]
Outer training loop: 100%|#########9| 49951/50000 [33:26:40<01:30,  1.86s/it]
Outer training loop: 100%|#########9| 49952/50000 [33:26:42<01:30,  1.88s/it]
Outer training loop: 100%|#########9| 49953/50000 [33:26:44<01:28,  1.89s/it]
Outer training loop: 100%|#########9| 49954/50000 [33:26:46<01:26,  1.87s/it]
Outer training loop: 100%|#########9| 49955/50000 [33:26:48<01:23,  1.85s/it]
Outer training loop: 100%|#########9| 49956/50000 [33:26:50<01:25,  1.93s/it]
Outer training loop: 100%|#########9| 49957/50000 [33:26:52<01:21,  1.90s/it]
Outer training loop: 100%|#########9| 49958/50000 [33:26:54<01:18,  1.86s/it]
Outer training loop: 100%|#########9| 49959/50000 [33:26:55<01:16,  1.88s/it]
Outer training loop: 100%|#########9| 49960/50000 [33:26:57<01:13,  1.84s/it]
Outer training loop: 100%|#########9| 49961/50000 [33:26:59<01:12,  1.85s/it]
Outer training loop: 100%|#########9| 49962/50000 [33:27:01<01:11,  1.88s/it]
Outer training loop: 100%|#########9| 49963/50000 [33:27:03<01:09,  1.87s/it]
Outer training loop: 100%|#########9| 49964/50000 [33:27:05<01:05,  1.82s/it]
Outer training loop: 100%|#########9| 49965/50000 [33:27:06<01:04,  1.83s/it]
Outer training loop: 100%|#########9| 49966/50000 [33:27:08<01:03,  1.85s/it]
Outer training loop: 100%|#########9| 49967/50000 [33:27:10<01:02,  1.90s/it]
Outer training loop: 100%|#########9| 49968/50000 [33:27:12<00:58,  1.84s/it]
Outer training loop: 100%|#########9| 49969/50000 [33:27:14<00:58,  1.89s/it]
Outer training loop: 100%|#########9| 49970/50000 [33:27:16<00:55,  1.86s/it]
Outer training loop: 100%|#########9| 49971/50000 [33:27:18<00:53,  1.83s/it]
Outer training loop: 100%|#########9| 49972/50000 [33:27:19<00:51,  1.82s/it]
Outer training loop: 100%|#########9| 49973/50000 [33:27:21<00:50,  1.88s/it]
Outer training loop: 100%|#########9| 49974/50000 [33:27:23<00:48,  1.86s/it]
Outer training loop: 100%|#########9| 49975/50000 [33:27:25<00:47,  1.90s/it]
Outer training loop: 100%|#########9| 49976/50000 [33:27:27<00:46,  1.94s/it]
Outer training loop: 100%|#########9| 49977/50000 [33:27:29<00:43,  1.90s/it]
Outer training loop: 100%|#########9| 49978/50000 [33:27:31<00:40,  1.86s/it]
Outer training loop: 100%|#########9| 49979/50000 [33:27:33<00:38,  1.85s/it]
Outer training loop: 100%|#########9| 49980/50000 [33:27:35<00:38,  1.93s/it]
Outer training loop: 100%|#########9| 49981/50000 [33:27:36<00:35,  1.87s/it]
Outer training loop: 100%|#########9| 49982/50000 [33:27:38<00:32,  1.81s/it]
Outer training loop: 100%|#########9| 49983/50000 [33:27:40<00:31,  1.88s/it]
Outer training loop: 100%|#########9| 49984/50000 [33:27:42<00:30,  1.88s/it]
Outer training loop: 100%|#########9| 49985/50000 [33:27:44<00:28,  1.87s/it]
Outer training loop: 100%|#########9| 49986/50000 [33:27:46<00:26,  1.92s/it]
Outer training loop: 100%|#########9| 49987/50000 [33:27:48<00:24,  1.87s/it]
Outer training loop: 100%|#########9| 49988/50000 [33:27:50<00:22,  1.86s/it]
Outer training loop: 100%|#########9| 49989/50000 [33:27:51<00:20,  1.88s/it]
Outer training loop: 100%|#########9| 49990/50000 [33:27:54<00:19,  1.94s/it]
Outer training loop: 100%|#########9| 49991/50000 [33:27:55<00:16,  1.87s/it]
Outer training loop: 100%|#########9| 49992/50000 [33:27:57<00:14,  1.86s/it]
Outer training loop: 100%|#########9| 49993/50000 [33:27:59<00:13,  1.92s/it]
Outer training loop: 100%|#########9| 49994/50000 [33:28:01<00:11,  1.88s/it]
Outer training loop: 100%|#########9| 49995/50000 [33:28:03<00:09,  1.89s/it]
Outer training loop: 100%|#########9| 49996/50000 [33:28:05<00:07,  1.97s/it]
Outer training loop: 100%|#########9| 49997/50000 [33:28:07<00:05,  1.95s/it]
Outer training loop: 100%|#########9| 49998/50000 [33:28:09<00:03,  1.89s/it]
Outer training loop: 100%|#########9| 49999/50000 [33:28:11<00:01,  1.94s/it]
Outer training loop: 100%|##########| 50000/50000 [33:28:13<00:00,  1.93s/it]