mxnet wavenet training log

log.txt

[tts] python train.py                                                                      master  ✭ ✱                                                                                                [78/1825]
2016-10-19 14:34:39,821 Start training with [gpu(3)]
2016-10-19 14:35:02,437 Epoch[0] Batch [10]     Speed: 8.75 samples/sec Train-mymae=57.056759
2016-10-19 14:35:22,736 Epoch[0] Batch [20]     Speed: 7.88 samples/sec Train-mymae=30.858146
2016-10-19 14:35:42,603 Epoch[0] Batch [30]     Speed: 8.05 samples/sec Train-mymae=29.581398
2016-10-19 14:36:03,529 Epoch[0] Batch [40]     Speed: 7.65 samples/sec Train-mymae=28.388074
2016-10-19 14:36:23,578 Epoch[0] Batch [50]     Speed: 7.98 samples/sec Train-mymae=29.426588
2016-10-19 14:36:44,323 Epoch[0] Batch [60]     Speed: 7.71 samples/sec Train-mymae=29.216556
2016-10-19 14:37:06,312 Epoch[0] Batch [70]     Speed: 7.28 samples/sec Train-mymae=27.873466
2016-10-19 14:37:26,722 Epoch[0] Batch [80]     Speed: 7.84 samples/sec Train-mymae=29.810743
2016-10-19 14:37:47,514 Epoch[0] Batch [90]     Speed: 7.70 samples/sec Train-mymae=30.334974
2016-10-19 14:38:07,281 Epoch[0] Batch [100]    Speed: 8.09 samples/sec Train-mymae=29.147760
2016-10-19 14:38:27,795 Epoch[0] Batch [110]    Speed: 7.80 samples/sec Train-mymae=30.240936
2016-10-19 14:38:48,626 Epoch[0] Batch [120]    Speed: 7.68 samples/sec Train-mymae=30.538801
2016-10-19 14:39:08,312 Epoch[0] Batch [130]    Speed: 8.13 samples/sec Train-mymae=31.033553
2016-10-19 14:39:28,702 Epoch[0] Batch [140]    Speed: 7.85 samples/sec Train-mymae=29.461625
2016-10-19 14:39:48,493 Epoch[0] Batch [150]    Speed: 8.08 samples/sec Train-mymae=27.210755
2016-10-19 14:40:08,384 Epoch[0] Batch [160]    Speed: 8.04 samples/sec Train-mymae=28.530111
2016-10-19 14:40:28,698 Epoch[0] Batch [170]    Speed: 7.88 samples/sec Train-mymae=32.708387
2016-10-19 14:40:48,659 Epoch[0] Batch [180]    Speed: 8.02 samples/sec Train-mymae=29.993312
2016-10-19 14:41:08,229 Epoch[0] Batch [190]    Speed: 8.18 samples/sec Train-mymae=29.386732
2016-10-19 14:41:27,830 Epoch[0] Batch [200]    Speed: 8.16 samples/sec Train-mymae=27.920722
2016-10-19 14:41:47,327 Epoch[0] Batch [210]    Speed: 8.21 samples/sec Train-mymae=29.762688
2016-10-19 14:42:06,790 Epoch[0] Batch [220]    Speed: 8.22 samples/sec Train-mymae=24.196779
2016-10-19 14:42:26,094 Epoch[0] Batch [230]    Speed: 8.29 samples/sec Train-mymae=19.973991
2016-10-19 14:42:45,630 Epoch[0] Batch [240]    Speed: 8.19 samples/sec Train-mymae=17.930430
2016-10-19 14:43:05,062 Epoch[0] Batch [250]    Speed: 8.23 samples/sec Train-mymae=17.034664
2016-10-19 14:43:24,710 Epoch[0] Batch [260]    Speed: 8.14 samples/sec Train-mymae=19.263878
2016-10-19 14:43:44,114 Epoch[0] Batch [270]    Speed: 8.25 samples/sec Train-mymae=17.071230
2016-10-19 14:44:03,492 Epoch[0] Batch [280]    Speed: 8.26 samples/sec Train-mymae=16.884779
2016-10-19 14:44:23,525 Epoch[0] Batch [290]    Speed: 7.99 samples/sec Train-mymae=17.828298
2016-10-19 14:44:43,161 Epoch[0] Batch [300]    Speed: 8.15 samples/sec Train-mymae=16.734015
2016-10-19 14:45:02,732 Epoch[0] Batch [310]    Speed: 8.18 samples/sec Train-mymae=16.495318
2016-10-19 14:45:22,231 Epoch[0] Batch [320]    Speed: 8.21 samples/sec Train-mymae=16.142198
2016-10-19 14:45:41,853 Epoch[0] Batch [330]    Speed: 8.15 samples/sec Train-mymae=15.791646
2016-10-19 14:46:01,757 Epoch[0] Batch [340]    Speed: 8.04 samples/sec Train-mymae=15.934451
2016-10-19 14:46:21,722 Epoch[0] Batch [350]    Speed: 8.01 samples/sec Train-mymae=16.374101
2016-10-19 14:46:41,309 Epoch[0] Batch [360]    Speed: 8.17 samples/sec Train-mymae=16.627341
2016-10-19 14:47:00,913 Epoch[0] Batch [370]    Speed: 8.16 samples/sec Train-mymae=16.769457
2016-10-19 14:47:20,420 Epoch[0] Batch [380]    Speed: 8.20 samples/sec Train-mymae=16.545075
2016-10-19 14:47:40,171 Epoch[0] Batch [390]    Speed: 8.10 samples/sec Train-mymae=15.617439
2016-10-19 14:47:59,710 Epoch[0] Batch [400]    Speed: 8.19 samples/sec Train-mymae=15.528881
2016-10-19 14:48:19,461 Epoch[0] Batch [410]    Speed: 8.10 samples/sec Train-mymae=15.567142
2016-10-19 14:48:39,110 Epoch[0] Batch [420]    Speed: 8.14 samples/sec Train-mymae=15.105454
2016-10-19 14:48:59,296 Epoch[0] Batch [430]    Speed: 7.93 samples/sec Train-mymae=14.919322
2016-10-19 14:49:19,079 Epoch[0] Batch [440]    Speed: 8.09 samples/sec Train-mymae=14.517386
2016-10-19 14:49:38,570 Epoch[0] Batch [450]    Speed: 8.21 samples/sec Train-mymae=14.355115
2016-10-19 14:49:58,781 Epoch[0] Batch [460]    Speed: 7.92 samples/sec Train-mymae=14.551800
2016-10-19 14:50:18,777 Epoch[0] Batch [470]    Speed: 8.00 samples/sec Train-mymae=14.487068
2016-10-19 14:50:38,304 Epoch[0] Batch [480]    Speed: 8.19 samples/sec Train-mymae=13.970604
2016-10-19 14:50:58,197 Epoch[0] Batch [490]    Speed: 8.04 samples/sec Train-mymae=13.941695
2016-10-19 14:51:17,648 Epoch[0] Batch [500]    Speed: 8.23 samples/sec Train-mymae=14.480248
2016-10-19 14:51:37,143 Epoch[0] Batch [510]    Speed: 8.21 samples/sec Train-mymae=14.292431
2016-10-19 14:51:57,029 Epoch[0] Batch [520]    Speed: 8.05 samples/sec Train-mymae=14.342782
2016-10-19 14:52:16,584 Epoch[0] Batch [530]    Speed: 8.18 samples/sec Train-mymae=14.155475
2016-10-19 14:52:36,534 Epoch[0] Batch [540]    Speed: 8.02 samples/sec Train-mymae=13.642085