# Assignment 1. Music Century Classification

**Assignment Responsible**: Natalie Lang.

In this assignment, we will build models to predict which
**century** a piece of music was released.  We will be using the "YearPredictionMSD Data Set"
based on the Million Song Dataset. The data is available to download from the UCI
Machine Learning Repository. Here are some links about the data:

- https://archive.ics.uci.edu/ml/datasets/yearpredictionmsd
- http://millionsongdataset.com/pages/tasks-demos/#yearrecognition

Note that you are note allowed to import additional packages **(especially not PyTorch)**. One of the objectives is to understand how the training procedure actually operates, before working with PyTorch's autograd engine which does it all for us.

## Question 1. Data (21%)

Start by setting up a Google Colab notebook in which to do your work.
Since you are working with a partner, you might find this link helpful:

- https://colab.research.google.com/github/googlecolab/colabtools/blob/master/notebooks/colab-github-demo.ipynb

The recommended way to work together is pair coding, where you and your partner are sitting together and writing code together.

To process and read the data, we use the popular `pandas` package for data analysis.

```
import pandas
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(1) # To reproduce our experiment
```

Now that your notebook is set up, we can load the data into the notebook. The code below provides
two ways of loading the data: directly from the internet, or through mounting Google Drive.
The first method is easier but slower, and the second method is a bit involved at first, but
can save you time later on. You will need to mount Google Drive for later assignments, so we recommend
figuring how to do that now.

Here are some resources to help you get started:

- http.://colab.research.google.com/notebooks/io.ipynb

```
load_from_drive = False

if not load_from_drive:
  csv_path = "http://archive.ics.uci.edu/ml/machine-learning-databases/00203/YearPredictionMSD.txt.zip"
else:
  from google.colab import drive
  drive.mount('/content/gdrive')
  csv_path = '/content/gdrive/My Drive/YearPredictionMSD.txt.zip' # TODO - UPDATE ME WITH THE TRUE PATH!

t_label = ["year"]
x_labels = ["var%d" % i for i in range(1, 91)]
df = pandas.read_csv(csv_path, names=t_label + x_labels)
```

Now that the data is loaded to your Colab notebook, you should be able to display the Pandas
DataFrame `df` as a table:

```
df
```

      year
      var1
      var2
      var3
      var4
      var5
      var6
      var7
      var8
      var9
      var10
      var11
      var12
      var13
      var14
      var15
      var16
      var17
      var18
      var19
      var20
      var21
      var22
      var23
      var24
      var25
      var26
      var27
      var28
      var29
      var30
      var31
      var32
      var33
      var34
      var35
      var36
      var37
      var38
      var39
      ...
      var51
      var52
      var53
      var54
      var55
      var56
      var57
      var58
      var59
      var60
      var61
      var62
      var63
      var64
      var65
      var66
      var67
      var68
      var69
      var70
      var71
      var72
      var73
      var74
      var75
      var76
      var77
      var78
      var79
      var80
      var81
      var82
      var83
      var84
      var85
      var86
      var87
      var88
      var89
      var90

      0
      2001
      49.94357
      21.47114
      73.07750
      8.74861
      -17.40628
      -13.09905
      -25.01202
      -12.23257
      7.83089
      -2.46783
      3.32136
      -2.31521
      10.20556
      611.10913
      951.08960
      698.11428
      408.98485
      383.70912
      326.51512
      238.11327
      251.42414
      187.17351
      100.42652
      179.19498
      -8.41558
      -317.87038
      95.86266
      48.10259
      -95.66303
      -18.06215
      1.96984
      34.42438
      11.72670
      1.36790
      7.79444
      -0.36994
      -133.67852
      -83.26165
      -37.29765
      ...
      -25.38187
      -3.90772
      13.29258
      41.55060
      -7.26272
      -21.00863
      105.50848
      64.29856
      26.08481
      -44.59110
      -8.30657
      7.93706
      -10.73660
      -95.44766
      -82.03307
      -35.59194
      4.69525
      70.95626
      28.09139
      6.02015
      -37.13767
      -41.12450
      -8.40816
      7.19877
      -8.60176
      -5.90857
      -12.32437
      14.68734
      -54.32125
      40.14786
      13.01620
      -54.40548
      58.99367
      15.37344
      1.11144
      -23.08793
      68.40795
      -1.82223
      -27.46348
      2.26327

      1
      2001
      48.73215
      18.42930
      70.32679
      12.94636
      -10.32437
      -24.83777
      8.76630
      -0.92019
      18.76548
      4.59210
      2.21920
      0.34006
      44.38997
      2056.93836
      605.40696
      457.41175
      777.15347
      415.64880
      746.47775
      366.45320
      317.82946
      273.07917
      141.75921
      317.35269
      19.48271
      -65.25496
      162.75145
      135.00765
      -96.28436
      -86.87955
      17.38087
      45.90742
      32.49908
      -32.85429
      45.10830
      26.84939
      -302.57328
      -41.71932
      -138.85034
      ...
      28.55107
      1.52298
      70.99515
      -43.63073
      -42.55014
      129.82848
      79.95420
      -87.14554
      -45.75446
      -65.82100
      -43.90031
      -19.45705
      12.59163
      -407.64130
      42.91189
      12.15850
      -88.37882
      42.25246
      46.49209
      -30.17747
      45.98495
      130.47892
      13.88281
      -4.00055
      17.85965
      -18.32138
      -87.99109
      14.37524
      -22.70119
      -58.81266
      5.66812
      -19.68073
      33.04964
      42.87836
      -9.90378
      -32.22788
      70.49388
      12.04941
      58.43453
      26.92061

      2
      2001
      50.95714
      31.85602
      55.81851
      13.41693
      -6.57898
      -18.54940
      -3.27872
      -2.35035
      16.07017
      1.39518
      2.73553
      0.82804
      7.46586
      699.54544
      1016.00954
      594.06748
      355.73663
      507.39931
      387.69910
      287.15347
      112.37152
      161.68928
      144.14353
      199.29693
      -4.24359
      -297.00587
      -148.36392
      -7.94726
      -18.71630
      12.77542
      -25.37725
      9.71410
      0.13843
      26.79723
      6.30760
      28.70107
      -74.89005
      -289.19553
      -166.26089
      ...
      18.50939
      16.97216
      24.26629
      -10.50788
      -8.68412
      54.75759
      194.74034
      7.95966
      -18.22685
      0.06463
      -2.63069
      26.02561
      1.75729
      -262.36917
      -233.60089
      -2.50502
      -12.14279
      81.37617
      2.07554
      -1.82381
      183.65292
      22.64797
      -39.98887
      43.37381
      -31.56737
      -4.88840
      -36.53213
      -23.94662
      -84.19275
      66.00518
      3.03800
      26.05866
      -50.92779
      10.93792
      -0.07568
      43.20130
      -115.00698
      -0.05859
      39.67068
      -0.66345

      3
      2001
      48.24750
      -1.89837
      36.29772
      2.58776
      0.97170
      -26.21683
      5.05097
      -10.34124
      3.55005
      -6.36304
      6.63016
      -3.35142
      37.64085
      2174.08189
      697.43346
      459.24587
      742.78961
      229.30783
      387.89697
      249.06662
      245.89870
      176.20527
      98.82222
      150.97286
      78.49057
      -62.00282
      43.49659
      -96.42719
      -108.96608
      14.22854
      14.54178
      -23.55608
      -39.36953
      -43.59209
      20.83714
      35.63919
      -181.34947
      -93.66614
      -90.55616
      ...
      4.56917
      -37.32280
      4.15159
      12.24315
      35.02697
      -178.89573
      82.46573
      -20.49425
      101.78577
      -19.77808
      -21.52657
      3.36303
      -11.63176
      51.55411
      -50.57576
      -28.14755
      -83.15795
      -7.35260
      -22.11505
      1.18279
      -122.70467
      150.57360
      24.37468
      41.19821
      -37.04318
      -28.72986
      162.19614
      22.18309
      -8.63509
      85.23416
      34.57337
      -171.70734
      -16.96705
      -46.67617
      -12.51516
      82.58061
      -72.08993
      9.90558
      199.62971
      18.85382

      4
      2001
      50.97020
      42.20998
      67.09964
      8.46791
      -15.85279
      -16.81409
      -12.48207
      -9.37636
      12.63699
      0.93609
      1.60923
      2.19223
      47.32082
      894.28471
      809.86615
      318.78559
      435.04497
      341.61467
      334.30734
      322.99589
      190.61921
      235.84715
      96.89517
      210.58870
      5.60463
      -199.63958
      204.85812
      -77.17695
      -65.79741
      -6.95097
      -12.15262
      -3.85410
      20.68990
      -20.30480
      37.15045
      11.20673
      -124.09519
      -295.98542
      -33.31169
      ...
      45.25506
      10.42226
      27.88782
      -17.12676
      -31.54772
      -76.86293
      41.17343
      -138.32535
      -53.96905
      -21.30266
      -24.87362
      -2.46595
      -4.05003
      -56.51161
      -34.56445
      -5.07092
      -47.75605
      64.81513
      -97.42948
      -12.59418
      55.23699
      28.85657
      54.53513
      -31.97077
      20.03279
      -8.07892
      -55.12617
      26.58961
      -10.27183
      -30.64232
      9.92661
      -55.95724
      64.92712
      -17.72522
      -1.49237
      -7.50035
      51.76631
      7.88713
      55.66926
      28.74903

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      515340
      2006
      51.28467
      45.88068
      22.19582
      -5.53319
      -3.61835
      -16.36914
      2.12652
      5.18160
      -8.66890
      2.67217
      0.45234
      2.51380
      18.79583
      592.17931
      619.01842
      681.30323
      415.21939
      639.90327
      287.20710
      375.31963
      212.76265
      246.26651
      143.48234
      217.45556
      9.90577
      -62.51153
      -76.96635
      -60.62065
      67.81811
      -9.20742
      -30.73303
      21.58525
      -31.21664
      -36.39659
      28.18814
      39.46981
      -77.13200
      -43.39948
      -57.69462
      ...
      -74.40960
      78.78128
      -14.74786
      18.02148
      -19.61304
      -50.34714
      87.06521
      43.77874
      -5.00339
      101.08108
      -13.34314
      -59.17573
      -46.22182
      -27.10155
      -7.07840
      23.04732
      29.32027
      2.10740
      -5.77951
      2.68326
      -13.78081
      6.33542
      -37.38191
      -14.90918
      26.87263
      7.07232
      -127.04955
      86.78200
      -68.14511
      67.44416
      4.81440
      -3.75991
      -30.92584
      26.33968
      -5.03390
      21.86037
      -142.29410
      3.42901
      -41.14721
      -15.46052

      515341
      2006
      49.87870
      37.93125
      18.65987
      -3.63581
      -27.75665
      -18.52988
      7.76108
      3.56109
      -2.50351
      2.20175
      -0.58487
      -9.78657
      35.81410
      1047.28364
      1451.87226
      633.17982
      448.46796
      826.14418
      277.55902
      202.20787
      241.85866
      199.31274
      180.60934
      168.49980
      89.28058
      237.30605
      -72.22211
      -10.02772
      -41.24980
      -7.59473
      -5.23307
      24.88978
      39.42813
      -40.17760
      26.51372
      79.84191
      -15.49724
      46.37942
      -209.97900
      ...
      -61.06002
      50.86072
      -3.54799
      36.50303
      20.94570
      -79.43478
      -15.49133
      17.79165
      95.84510
      -37.68620
      8.51302
      13.72492
      -71.83419
      -191.37407
      -34.71662
      28.34789
      45.25187
      17.07862
      31.46894
      -13.44802
      38.68815
      109.03046
      -42.45525
      18.67531
      -50.86612
      11.26242
      59.30165
      178.15846
      -29.04997
      70.22336
      32.38589
      -32.75535
      -61.05473
      56.65182
      15.29965
      95.88193
      -10.63242
      12.96552
      92.11633
      10.88815

      515342
      2006
      45.12852
      12.65758
      -38.72018
      8.80882
      -29.29985
      -2.28706
      -18.40424
      -22.28726
      -4.52429
      -11.46411
      3.28514
      1.99943
      27.77109
      1693.72442
      3825.48305
      2714.53243
      1036.34216
      1171.81248
      468.44308
      1042.15436
      278.94429
      497.83085
      423.82729
      239.91028
      -61.01287
      -1383.48696
      -1828.43740
      -131.54731
      138.81510
      51.36991
      -45.25035
      138.31791
      -107.60348
      -17.01878
      -36.53276
      226.67213
      716.76768
      -267.06525
      -362.27860
      ...
      191.56779
      72.49396
      -38.96949
      61.22195
      24.49062
      182.62433
      510.41684
      -379.38804
      226.54992
      -201.28237
      6.89971
      86.07237
      -42.85773
      -215.01900
      88.60866
      14.51385
      -28.33832
      255.17385
      14.17125
      25.06417
      218.85618
      -222.53173
      35.58546
      30.88622
      -24.91594
      -2.65009
      -69.53483
      333.67598
      -28.24399
      202.51566
      -18.73598
      -71.15954
      -123.98443
      121.26989
      10.89629
      34.62409
      -248.61020
      -6.07171
      53.96319
      -8.09364

      515343
      2006
      44.16614
      32.38368
      -3.34971
      -2.49165
      -19.59278
      -18.67098
      8.78428
      4.02039
      -12.01230
      -0.74075
      -1.26523
      -4.41983
      140.44937
      2850.23336
      1875.28895
      1362.98053
      784.39737
      908.09838
      367.12005
      692.58547
      286.72625
      395.46735
      221.19089
      211.62098
      141.17304
      647.52054
      -451.67671
      -170.33993
      -106.30851
      129.80285
      -118.54997
      116.14019
      -18.36186
      -29.42843
      13.59803
      296.86552
      -332.24640
      219.84847
      -180.27193
      ...
      14.33401
      -10.61959
      -37.44137
      32.72492
      -16.62357
      -343.07974
      148.00075
      -64.73672
      59.16029
      -129.60142
      24.47146
      -90.78617
      -34.58624
      -285.37506
      -8.78066
      63.91160
      58.86067
      43.28537
      22.69472
      -0.93940
      417.76862
      78.26912
      -173.95232
      -35.42845
      10.59859
      -16.51518
      157.75671
      294.31838
      -37.30155
      80.00327
      67.16763
      282.77624
      -4.63677
      144.00125
      21.62652
      -29.72432
      71.47198
      20.32240
      14.83107
      39.74909

      515344
      2005
      51.85726
      59.11655
      26.39436
      -5.46030
      -20.69012
      -19.95528
      -6.72771
      2.29590
      10.31018
      6.26597
      -1.78800
      -6.19786
      20.16600
      598.45275
      1140.69539
      721.49244
      272.84841
      564.06690
      199.41547
      189.04637
      217.32042
      137.13390
      150.34608
      98.21589
      48.12644
      -601.59295
      10.58466
      -83.35368
      96.86756
      69.40708
      8.06033
      -26.01693
      -2.93173
      26.18398
      -12.24660
      -14.52391
      -121.61676
      119.15632
      -229.55722
      ...
      1.78072
      64.75548
      24.55866
      -1.12509
      -13.58287
      -99.66038
      -124.73875
      67.02630
      33.05618
      60.25818
      28.00288
      10.62425
      -8.86772
      78.13543
      -181.10013
      74.69489
      57.45083
      114.08816
      -9.91322
      7.53612
      97.06395
      233.17754
      -100.68441
      27.67012
      -37.33008
      -0.34676
      -207.78766
      116.75005
      -91.82912
      8.35020
      -11.50511
      -69.18291
      60.58456
      28.64599
      -4.39620
      -64.56491
      -45.61012
      -5.51512
      32.35602
      12.17352

515345 rows × 91 columns

To set up our data for classification, we'll use the "year" field to represent
whether a song was released in the 20-th century. In our case `df["year"]` will be 1 if
the year was released after 2000, and 0 otherwise.

```
df["year"] = df["year"].map(lambda x: int(x > 2000))
```

```
df.head(20)
```

      year
      var1
      var2
      var3
      var4
      var5
      var6
      var7
      var8
      var9
      var10
      var11
      var12
      var13
      var14
      var15
      var16
      var17
      var18
      var19
      var20
      var21
      var22
      var23
      var24
      var25
      var26
      var27
      var28
      var29
      var30
      var31
      var32
      var33
      var34
      var35
      var36
      var37
      var38
      var39
      ...
      var51
      var52
      var53
      var54
      var55
      var56
      var57
      var58
      var59
      var60
      var61
      var62
      var63
      var64
      var65
      var66
      var67
      var68
      var69
      var70
      var71
      var72
      var73
      var74
      var75
      var76
      var77
      var78
      var79
      var80
      var81
      var82
      var83
      var84
      var85
      var86
      var87
      var88
      var89
      var90

      0
      1
      49.94357
      21.47114
      73.07750
      8.74861
      -17.40628
      -13.09905
      -25.01202
      -12.23257
      7.83089
      -2.46783
      3.32136
      -2.31521
      10.20556
      611.10913
      951.08960
      698.11428
      408.98485
      383.70912
      326.51512
      238.11327
      251.42414
      187.17351
      100.42652
      179.19498
      -8.41558
      -317.87038
      95.86266
      48.10259
      -95.66303
      -18.06215
      1.96984
      34.42438
      11.72670
      1.36790
      7.79444
      -0.36994
      -133.67852
      -83.26165
      -37.29765
      ...
      -25.38187
      -3.90772
      13.29258
      41.55060
      -7.26272
      -21.00863
      105.50848
      64.29856
      26.08481
      -44.59110
      -8.30657
      7.93706
      -10.73660
      -95.44766
      -82.03307
      -35.59194
      4.69525
      70.95626
      28.09139
      6.02015
      -37.13767
      -41.12450
      -8.40816
      7.19877
      -8.60176
      -5.90857
      -12.32437
      14.68734
      -54.32125
      40.14786
      13.01620
      -54.40548
      58.99367
      15.37344
      1.11144
      -23.08793
      68.40795
      -1.82223
      -27.46348
      2.26327

      1
      1
      48.73215
      18.42930
      70.32679
      12.94636
      -10.32437
      -24.83777
      8.76630
      -0.92019
      18.76548
      4.59210
      2.21920
      0.34006
      44.38997
      2056.93836
      605.40696
      457.41175
      777.15347
      415.64880
      746.47775
      366.45320
      317.82946
      273.07917
      141.75921
      317.35269
      19.48271
      -65.25496
      162.75145
      135.00765
      -96.28436
      -86.87955
      17.38087
      45.90742
      32.49908
      -32.85429
      45.10830
      26.84939
      -302.57328
      -41.71932
      -138.85034
      ...
      28.55107
      1.52298
      70.99515
      -43.63073
      -42.55014
      129.82848
      79.95420
      -87.14554
      -45.75446
      -65.82100
      -43.90031
      -19.45705
      12.59163
      -407.64130
      42.91189
      12.15850
      -88.37882
      42.25246
      46.49209
      -30.17747
      45.98495
      130.47892
      13.88281
      -4.00055
      17.85965
      -18.32138
      -87.99109
      14.37524
      -22.70119
      -58.81266
      5.66812
      -19.68073
      33.04964
      42.87836
      -9.90378
      -32.22788
      70.49388
      12.04941
      58.43453
      26.92061

      2
      1
      50.95714
      31.85602
      55.81851
      13.41693
      -6.57898
      -18.54940
      -3.27872
      -2.35035
      16.07017
      1.39518
      2.73553
      0.82804
      7.46586
      699.54544
      1016.00954
      594.06748
      355.73663
      507.39931
      387.69910
      287.15347
      112.37152
      161.68928
      144.14353
      199.29693
      -4.24359
      -297.00587
      -148.36392
      -7.94726
      -18.71630
      12.77542
      -25.37725
      9.71410
      0.13843
      26.79723
      6.30760
      28.70107
      -74.89005
      -289.19553
      -166.26089
      ...
      18.50939
      16.97216
      24.26629
      -10.50788
      -8.68412
      54.75759
      194.74034
      7.95966
      -18.22685
      0.06463
      -2.63069
      26.02561
      1.75729
      -262.36917
      -233.60089
      -2.50502
      -12.14279
      81.37617
      2.07554
      -1.82381
      183.65292
      22.64797
      -39.98887
      43.37381
      -31.56737
      -4.88840
      -36.53213
      -23.94662
      -84.19275
      66.00518
      3.03800
      26.05866
      -50.92779
      10.93792
      -0.07568
      43.20130
      -115.00698
      -0.05859
      39.67068
      -0.66345

      3
      1
      48.24750
      -1.89837
      36.29772
      2.58776
      0.97170
      -26.21683
      5.05097
      -10.34124
      3.55005
      -6.36304
      6.63016
      -3.35142
      37.64085
      2174.08189
      697.43346
      459.24587
      742.78961
      229.30783
      387.89697
      249.06662
      245.89870
      176.20527
      98.82222
      150.97286
      78.49057
      -62.00282
      43.49659
      -96.42719
      -108.96608
      14.22854
      14.54178
      -23.55608
      -39.36953
      -43.59209
      20.83714
      35.63919
      -181.34947
      -93.66614
      -90.55616
      ...
      4.56917
      -37.32280
      4.15159
      12.24315
      35.02697
      -178.89573
      82.46573
      -20.49425
      101.78577
      -19.77808
      -21.52657
      3.36303
      -11.63176
      51.55411
      -50.57576
      -28.14755
      -83.15795
      -7.35260
      -22.11505
      1.18279
      -122.70467
      150.57360
      24.37468
      41.19821
      -37.04318
      -28.72986
      162.19614
      22.18309
      -8.63509
      85.23416
      34.57337
      -171.70734
      -16.96705
      -46.67617
      -12.51516
      82.58061
      -72.08993
      9.90558
      199.62971
      18.85382

      4
      1
      50.97020
      42.20998
      67.09964
      8.46791
      -15.85279
      -16.81409
      -12.48207
      -9.37636
      12.63699
      0.93609
      1.60923
      2.19223
      47.32082
      894.28471
      809.86615
      318.78559
      435.04497
      341.61467
      334.30734
      322.99589
      190.61921
      235.84715
      96.89517
      210.58870
      5.60463
      -199.63958
      204.85812
      -77.17695
      -65.79741
      -6.95097
      -12.15262
      -3.85410
      20.68990
      -20.30480
      37.15045
      11.20673
      -124.09519
      -295.98542
      -33.31169
      ...
      45.25506
      10.42226
      27.88782
      -17.12676
      -31.54772
      -76.86293
      41.17343
      -138.32535
      -53.96905
      -21.30266
      -24.87362
      -2.46595
      -4.05003
      -56.51161
      -34.56445
      -5.07092
      -47.75605
      64.81513
      -97.42948
      -12.59418
      55.23699
      28.85657
      54.53513
      -31.97077
      20.03279
      -8.07892
      -55.12617
      26.58961
      -10.27183
      -30.64232
      9.92661
      -55.95724
      64.92712
      -17.72522
      -1.49237
      -7.50035
      51.76631
      7.88713
      55.66926
      28.74903

      5
      1
      50.54767
      0.31568
      92.35066
      22.38696
      -25.51870
      -19.04928
      20.67345
      -5.19943
      3.63566
      -4.69088
      2.49578
      -3.02468
      7.69273
      1004.95615
      785.06709
      591.99232
      495.75332
      291.38165
      434.08355
      291.55265
      303.58860
      216.12189
      126.10703
      147.28090
      6.17712
      -133.67424
      8.69259
      -13.76138
      -52.72072
      -24.00961
      -10.82297
      11.42039
      32.10442
      -26.26130
      -4.98856
      20.84154
      -129.69145
      -157.91059
      -43.29252
      ...
      45.07251
      -6.20682
      17.53456
      43.66313
      -4.80024
      -33.62226
      -15.01731
      -33.74416
      2.12072
      -46.83225
      19.99596
      26.31683
      0.51657
      -285.36622
      -52.26245
      -13.00341
      -9.41185
      69.69108
      33.54907
      -11.39665
      93.61103
      163.39892
      5.00362
      15.23807
      -13.48394
      -6.37431
      -37.31287
      92.47370
      -90.00149
      47.25143
      6.59753
      -50.69577
      26.02574
      18.94430
      -0.33730
      6.09352
      35.18381
      5.00283
      -11.02257
      0.02263

      6
      1
      50.57546
      33.17843
      50.53517
      11.55217
      -27.24764
      -8.78206
      -12.04282
      -9.53930
      28.61811
      8.25435
      -0.43743
      5.66265
      11.07787
      1080.98902
      1230.78393
      1301.63542
      952.84686
      783.02498
      560.79536
      696.19620
      253.36266
      316.92697
      151.75689
      144.07059
      -3.02894
      -111.65251
      -56.64580
      464.86598
      150.52166
      84.69609
      -91.71196
      89.31272
      16.49867
      -4.47074
      -2.02539
      13.27637
      -153.73456
      199.01552
      -278.79072
      ...
      0.05909
      -92.07551
      7.80480
      -46.15966
      -39.03309
      32.52065
      164.15989
      -247.22025
      -100.28773
      -55.58712
      8.38343
      -4.57294
      -20.08525
      -357.00069
      -232.78978
      -112.81679
      -66.16128
      43.25003
      18.48417
      -2.50274
      3.25927
      94.57509
      -24.31254
      62.97582
      -19.41809
      10.35282
      -91.89392
      10.51922
      -74.98521
      12.29948
      11.63681
      25.44182
      134.62382
      21.51982
      8.17570
      35.46251
      11.57736
      4.50056
      -4.62739
      1.40192

      7
      1
      48.26892
      8.97526
      75.23158
      24.04945
      -16.02105
      -14.09491
      8.11871
      -1.87566
      7.46701
      1.18189
      1.46625
      -6.34226
      8.40470
      1619.74629
      1762.43083
      714.83843
      1115.90792
      525.85703
      576.66300
      350.41961
      315.44672
      267.94363
      198.28158
      201.97524
      17.14701
      -517.47978
      -11.33396
      59.52820
      -25.10400
      23.48782
      12.63888
      40.85580
      -3.72243
      -16.97392
      0.04284
      59.45956
      -191.92216
      445.36223
      -120.62667
      ...
      63.71442
      3.70732
      -9.36662
      -25.75461
      35.16677
      -33.17382
      13.80469
      -107.14403
      140.04250
      -124.16153
      -18.33032
      -9.99397
      8.96011
      -411.27991
      -99.75061
      -75.51735
      -88.57128
      22.90222
      4.14618
      -16.83238
      142.14168
      326.91932
      -3.49405
      7.58991
      -32.60725
      -4.44469
      -56.48952
      -31.19491
      -32.75384
      -52.97111
      18.03989
      -58.46192
      -65.56438
      46.99856
      -4.09602
      56.37650
      -18.29975
      -0.30633
      3.98364
      -3.72556

      8
      1
      49.75468
      33.99581
      56.73846
      2.89581
      -2.92429
      -26.44413
      1.71392
      -0.55644
      22.08594
      7.43847
      -0.03578
      1.66534
      46.02060
      789.58109
      607.78514
      248.54603
      288.44004
      330.16459
      356.34405
      194.21792
      194.68297
      158.43568
      104.05610
      183.10695
      -39.54223
      -225.39832
      78.57149
      20.62209
      -9.44189
      15.94754
      51.24174
      -4.36054
      17.60516
      9.03638
      16.69044
      42.70944
      -77.62818
      46.71905
      -109.71028
      ...
      36.17475
      3.44547
      46.40983
      -21.52684
      -10.50872
      -8.62787
      55.23069
      -41.74452
      -20.24522
      21.78558
      -8.69348
      -1.85679
      3.60117
      -53.94252
      -54.66970
      -15.14957
      -46.61675
      48.45009
      2.77737
      -7.80854
      38.29390
      120.35575
      23.84978
      14.87696
      -41.76961
      -21.05519
      -45.79645
      9.16222
      -21.48052
      -9.70822
      18.70812
      5.20391
      -27.75192
      17.22100
      -0.85210
      -15.67150
      -26.36257
      5.48708
      -9.13495
      6.08680

      9
      1
      45.17809
      46.34234
      -40.65357
      -2.47909
      1.21253
      -0.65302
      -6.95536
      -12.20040
      17.02512
      2.00002
      -1.87785
      9.85499
      25.59837
      1905.18577
      3676.09074
      1976.85531
      913.11216
      1957.52415
      955.98525
      942.72667
      439.85991
      591.66138
      493.40770
      496.38516
      33.94285
      -255.90134
      -762.28079
      -66.10935
      -128.02217
      198.12908
      -34.44957
      176.00397
      -140.80069
      -22.56380
      12.77945
      193.30164
      314.20949
      576.29519
      -429.58643
      ...
      -17.48938
      75.58779
      93.29243
      85.83507
      47.13972
      312.85482
      135.50478
      -32.47886
      49.67063
      -214.73180
      -77.83503
      -47.26902
      7.58366
      -352.56581
      -36.15655
      -53.39933
      -98.60417
      -82.37799
      45.81588
      -16.91676
      18.35888
      -315.68965
      -3.14554
      125.45269
      -130.18808
      -3.06337
      42.26602
      -9.04929
      26.41570
      23.36165
      -4.36742
      -87.55285
      -70.79677
      76.57355
      -7.71727
      3.26926
      -298.49845
      11.49326
      -89.21804
      -15.09719

      10
      1
      39.13076
      -23.01763
      -36.20583
      1.67519
      -4.27101
      13.01158
      8.05718
      -8.41088
      6.27370
      -7.81564
      -12.29472
      -12.26186
      45.05025
      2022.36720
      3860.79421
      3186.91395
      849.65196
      2835.16969
      847.01876
      1001.27685
      347.33791
      500.59470
      536.16777
      318.73821
      50.13600
      451.14001
      -589.91962
      -95.42003
      -99.51192
      -28.43500
      22.34786
      132.68163
      -111.84550
      106.94141
      -9.94676
      269.57001
      416.04429
      3.08594
      -312.96128
      ...
      364.86126
      4.89096
      -110.07105
      19.43169
      -13.69554
      464.30985
      203.55456
      -450.84974
      9.54696
      -433.98556
      40.33083
      60.29120
      66.28236
      -398.01020
      582.88308
      -93.75781
      -31.61288
      -311.99559
      106.91372
      -8.56906
      -42.26640
      269.83186
      10.39815
      -14.56199
      36.15556
      51.77120
      23.70949
      -15.79585
      216.13271
      100.04021
      32.86051
      -26.08461
      -186.82429
      113.58176
      9.28727
      44.60282
      158.00425
      -2.59543
      109.19723
      23.36143

      11
      1
      37.66498
      -34.05910
      -17.36060
      -26.77781
      -39.95119
      -20.75000
      -0.10231
      -0.89972
      -1.30205
      -0.93041
      -3.30157
      -2.37522
      35.42988
      1166.91594
      1373.01959
      409.04286
      609.84543
      271.33121
      392.43089
      180.98525
      215.37174
      137.27208
      69.71585
      187.80590
      -12.67805
      -347.02236
      271.33219
      40.08570
      42.13962
      10.70729
      36.75105
      6.28685
      3.24624
      -3.85517
      10.67336
      -9.47190
      -62.45325
      212.55872
      -33.45944
      ...
      14.26269
      12.13975
      17.85889
      16.82677
      -73.67298
      30.33815
      -101.70001
      -46.13208
      14.26704
      4.31515
      -25.67264
      8.70269
      11.69122
      -59.19425
      -43.59348
      -81.06458
      -8.27943
      -16.76813
      0.83299
      47.83962
      -50.41094
      -146.83861
      37.05135
      9.58587
      11.61329
      -11.13671
      86.66011
      -21.11909
      -4.36777
      44.53390
      11.18909
      45.20614
      53.83925
      2.59467
      -4.00958
      -47.74886
      -170.92864
      -5.19009
      8.83617
      -7.16056

      12
      1
      26.51957
      -148.15762
      -13.30095
      -7.25851
      17.22029
      -21.99439
      5.51947
      3.48418
      2.61738
      2.51194
      -0.53980
      4.94747
      55.79019
      5492.46406
      1704.58482
      953.13106
      959.38535
      788.92858
      365.45762
      270.37477
      524.61443
      194.81728
      139.44778
      375.02904
      -434.18365
      -627.01684
      642.48227
      -440.32465
      -319.81435
      10.13236
      38.64956
      -30.14092
      75.22352
      -48.15860
      80.88233
      183.14023
      -178.08073
      -594.06429
      185.11781
      ...
      137.58951
      38.38606
      -3.31309
      5.48498
      -128.84776
      -303.86613
      188.82297
      -108.38722
      -69.68996
      84.70685
      -54.19731
      -112.09276
      85.47803
      38.20818
      -277.12066
      132.27890
      91.74732
      -11.84666
      -28.23002
      -21.30369
      205.87882
      -22.13147
      -52.71243
      -122.98466
      -0.77068
      -56.79478
      91.28164
      -2.31187
      49.34151
      -243.95844
      23.80442
      251.76360
      18.81642
      157.09656
      -27.79449
      -137.72740
      115.28414
      23.00230
      -164.02536
      51.54138

      13
      1
      37.68491
      -26.84185
      -27.10566
      -14.95883
      -5.87200
      -21.68979
      4.87374
      -18.01800
      1.52141
      -6.81668
      6.80117
      21.17530
      50.35368
      6431.63159
      3755.88724
      850.19745
      2540.53952
      400.64901
      614.21009
      324.55934
      395.25844
      241.66513
      99.66852
      489.65963
      -466.60181
      4015.57689
      1042.51072
      787.79078
      -240.53556
      -82.21473
      92.62933
      25.79866
      47.82510
      -24.31114
      39.59818
      -318.96984
      1174.17699
      2574.00913
      -116.86438
      ...
      -10.44655
      -86.37776
      -2.66989
      -150.73054
      -309.01020
      -263.88605
      -511.63129
      -331.45866
      524.10757
      -103.81583
      -11.11616
      146.64710
      47.03819
      -555.49233
      -312.62275
      17.44093
      114.07544
      30.33500
      233.32721
      36.70535
      -449.22124
      507.86020
      179.49985
      -62.12860
      20.81610
      37.22555
      821.44873
      179.59958
      -37.21579
      -632.61945
      -67.57637
      234.27192
      -72.34557
      -362.25101
      -25.55019
      -89.08971
      -891.58937
      14.11648
      -1030.99180
      99.28967

      14
      0
      39.11695
      -8.29767
      -51.37966
      -4.42668
      -30.06506
      -11.95916
      -0.85322
      -8.86179
      11.36680
      3.78199
      1.54568
      0.58662
      28.49070
      7965.81458
      1411.06042
      1218.84885
      645.93064
      1138.49365
      552.35786
      652.69173
      281.26048
      396.15845
      231.01221
      215.80169
      345.50690
      -1163.24259
      -117.88194
      162.42114
      339.56477
      32.54661
      -78.76608
      9.22956
      -7.94960
      -15.24124
      29.36270
      -19.91118
      530.29303
      284.81573
      2.50910
      ...
      50.89571
      18.59393
      53.83297
      -89.69260
      -31.45442
      188.56576
      70.85815
      -259.86825
      65.62430
      -208.96960
      -50.55450
      1.06368
      0.31427
      396.93348
      105.80580
      72.06757
      -160.48858
      -97.33616
      -69.36746
      37.07706
      -113.27781
      -95.78902
      155.13377
      -22.10761
      -1.08949
      13.36146
      833.00787
      -88.78803
      30.70543
      -63.16836
      42.22923
      478.26580
      -10.33823
      -103.76858
      39.19511
      -98.76636
      -122.81061
      -2.14942
      -211.48202
      -12.81569

      15
      1
      35.05129
      -67.97714
      -14.20239
      -6.68696
      -0.61230
      -18.70341
      -1.31928
      -9.46370
      5.53492
      2.79989
      3.34150
      18.01445
      48.14418
      1938.39116
      755.44238
      1350.57299
      793.09536
      793.43819
      585.72478
      280.00734
      260.63960
      168.98085
      148.49783
      539.22389
      -99.68273
      198.55553
      332.84941
      -139.15915
      -166.48695
      -186.65004
      143.44962
      -22.08680
      37.34160
      -30.72572
      64.79269
      57.94387
      332.16642
      77.96230
      312.90695
      ...
      52.35627
      13.93583
      -2.03342
      -107.12775
      -6.42999
      215.24319
      122.33200
      -183.75334
      89.84912
      3.34699
      -33.62856
      38.22842
      77.13977
      150.36511
      -56.21045
      34.03026
      -34.86760
      60.25658
      2.52359
      -56.10479
      -21.25331
      77.13640
      134.26121
      26.21428
      -152.23191
      -52.85864
      -101.18125
      55.52082
      -64.97701
      227.31886
      10.25585
      94.90539
      15.95689
      -98.15732
      -9.64859
      -93.52834
      -95.82981
      20.73063
      -562.07671
      43.44696

      16
      1
      33.63129
      -96.14912
      -89.38216
      -12.11699
      13.77252
      -6.69377
      -33.36843
      -24.81437
      21.22757
      0.26310
      0.42982
      -6.59226
      27.25596
      2407.95516
      1257.41135
      795.17898
      645.83254
      1014.07214
      474.67866
      697.26135
      334.85298
      358.30744
      324.45833
      165.58489
      -8.08451
      101.52431
      39.60959
      7.06185
      -78.21043
      88.00470
      129.65276
      50.90538
      -21.26991
      32.76996
      -29.96947
      48.47300
      441.40686
      454.01182
      95.66968
      ...
      55.84917
      92.63264
      30.35791
      -79.20043
      -7.46621
      602.37381
      188.96249
      -98.71438
      176.37440
      -248.00977
      44.62138
      24.37125
      30.40357
      79.40089
      230.05627
      -2.13554
      30.01898
      -41.89611
      -1.77960
      46.71421
      149.76135
      177.92434
      -8.49394
      47.67341
      85.99877
      6.63650
      -213.70857
      90.57614
      -15.13552
      -99.22940
      49.93249
      -14.47489
      40.70590
      58.63692
      8.81522
      27.28474
      5.78046
      3.44539
      259.10825
      10.28525

      17
      0
      41.38639
      -20.78665
      51.80155
      17.21415
      -36.44189
      -11.53169
      11.75252
      -7.62428
      -3.65488
      -5.08109
      4.37624
      -1.39493
      18.28400
      3157.88796
      1116.07117
      913.19211
      638.12946
      535.61444
      440.68379
      435.75048
      545.13836
      286.93019
      154.24744
      278.14609
      168.02465
      -392.26915
      -109.59025
      -243.33016
      57.76966
      7.51054
      84.23430
      -67.00385
      80.72070
      -0.00321
      2.12590
      41.51370
      -389.17834
      -421.90682
      -294.54854
      ...
      127.18184
      -106.70486
      -18.10968
      19.19753
      -2.79115
      31.93573
      10.40156
      -226.50662
      -2.38144
      -157.93284
      0.46577
      27.78523
      6.41002
      -173.66782
      29.73562
      -44.29195
      -81.53220
      50.49506
      65.00208
      -17.76363
      -37.10280
      264.10009
      100.74257
      -57.74383
      -22.25492
      -17.44555
      335.82434
      80.63635
      -37.20103
      7.30588
      50.37614
      -40.48205
      48.07805
      -7.62399
      6.51934
      -30.46090
      -53.87264
      4.44627
      58.16913
      -0.02409

      18
      0
      37.45034
      11.42615
      56.28982
      19.58426
      -16.43530
      2.22457
      1.02668
      -7.34736
      -0.01184
      1.24013
      2.57660
      2.79598
      30.02393
      1249.33039
      3438.98500
      962.85885
      1654.99981
      1066.80390
      606.24732
      278.63327
      273.36684
      250.97765
      257.04632
      294.90282
      41.74146
      -455.47604
      473.68747
      -72.16130
      409.83317
      111.23583
      98.74929
      19.15393
      -38.69791
      -20.10176
      57.44465
      -127.58760
      -189.37100
      1126.01208
      -98.48301
      ...
      55.86703
      -15.58939
      -40.62299
      -45.18282
      -85.17021
      180.09138
      491.26688
      6.42139
      263.30124
      -152.86190
      -16.74493
      -33.13223
      -10.64964
      -367.08218
      49.57986
      17.60102
      -112.39761
      79.62965
      -92.92839
      -49.77861
      -53.65950
      184.29601
      -37.85258
      -218.50833
      -37.55140
      -17.84213
      -59.35940
      -168.74190
      -48.89748
      -99.69609
      -22.46207
      -25.77228
      -322.42841
      -146.57408
      13.61588
      92.22918
      -439.80259
      25.73235
      157.22967
      38.70617

      19
      0
      39.71092
      -4.92800
      12.88590
      -11.87773
      2.48031
      -16.11028
      -16.40421
      -8.29657
      9.86817
      -0.17431
      0.98765
      7.37903
      13.84987
      1256.34345
      800.61127
      715.08117
      632.19829
      357.08486
      218.48245
      278.90104
      229.11644
      228.51202
      87.35304
      144.36923
      34.96307
      -123.85672
      -10.33970
      -171.19156
      -89.00293
      39.72671
      -17.78437
      4.56124
      17.07687
      -20.75794
      16.49367
      63.67926
      -113.39747
      12.27230
      -188.87545
      ...
      -15.69508
      1.68706
      17.69997
      -3.35888
      4.21699
      25.45729
      147.84361
      -69.50808
      81.47487
      -91.89773
      9.59723
      2.52910
      -9.48278
      -81.54172
      -30.46621
      8.10179
      9.51480
      25.88937
      28.29212
      5.73829
      158.30039
      96.20428
      55.04618
      18.25457
      -18.37851
      1.06499
      -76.40237
      111.81246
      -57.71450
      29.55411
      11.92816
      -73.72412
      16.19039
      9.79606
      9.71693
      -9.90907
      -20.65851
      2.34002
      -31.57015
      1.58400

20 rows × 91 columns

### Part (a) -- 7%

The data set description text asks us to respect the below train/test split to
avoid the "producer effect". That is, we want to make sure that no song from a single artist
ends up in both the training and test set.

Explain why it would be problematic to have
some songs from an artist in the training set, and other songs from the same artist in the
test set. (Hint: Remember that we want our test accuracy to predict how well the model
will perform in practice on a song it hasn't learned about.)

```
df_train = df[:463715]
df_test = df[463715:]

# convert to numpy
train_xs = df_train[x_labels].to_numpy()
train_ts = df_train[t_label].to_numpy()
test_xs = df_test[x_labels].to_numpy()
test_ts = df_test[t_label].to_numpy()

# Write your explanation here
'''
To evaluate our model properly, we want to know how well the model will perform over a new songs that it hasn't encountered before.
Using songs from the same artist for both the training set and the test set can lead to a situation in which the model learns to extract features based on the singer,
e.g., the singer's voice, and will decide based on those features.
In this scenario, the model might perform well on samples of songs in the test where their corresponding singers also appeared in the train set.
But this doesn't guarantee that the model will generalize for songs in the test set that their singers didn't appear in the train set.
'''

```

    "\nTo evaluate our model properly, we want to know how well the model will perform over a new songs that it hasn't encountered before.\nUsing songs from the same artist for both the training set and the test set can lead to a situation in which the model learns to extract features based on the singer,\ne.g., the singer's voice, and will decide based on those features.\nIn this scenario, the model might perform well on samples of songs in the test where their corresponding singers also appeared in the train set.\nBut this doesn't guarantee that the model will generalize for songs in the test set that their singers didn't appear in the train set.\n"

### Part (b) -- 7%

It can be beneficial to **normalize** the columns, so that each column (feature)
has the *same* mean and standard deviation.

```
feature_means = df_train.mean()[1:].to_numpy() # the [1:] removes the mean of the "year" field
feature_stds  = df_train.std()[1:].to_numpy()

train_norm_xs = (train_xs - feature_means) / feature_stds
test_norm_xs = (test_xs - feature_means) / feature_stds
```

Notice how in our code, we normalized the test set using the *training data means and standard deviations*.
This is *not* a bug.

Explain why it would be improper to compute and use test set means
and standard deviations. (Hint: Remember what we want to use the test accuracy to measure.)

```
# Write your explanation here
"""
We normalize the data with respect to the training set to make the input data used for training "more identical distributed" during training.
We are using test data to evaluate our model performance over new unseen data.
If we are using the average and standard deviation of this test set to normalize it, we enforce dependency between different samples in the test set.
This harms the idea of model evaluation over new unseen data, e.g., if we evaluate our model for two different test sets, both contain some identical samples.
Following this approach will lead to different outcomes for the same input sample (because the average and the standard deviation will be different), which demonstrates the problem with this approach.
"""
```

    '\nWe normalize the data with respect to the training set to make the input data used for training "more identical distributed" during training.\nWe are using test data to evaluate our model performance over new unseen data.\nIf we are using the average and standard deviation of this test set to normalize it, we enforce dependency between different samples in the test set. \nThis harms the idea of model evaluation over new unseen data, e.g., if we evaluate our model for two different test sets, both contain some identical samples.\nFollowing this approach will lead to different outcomes for the same input sample (because the average and the standard deviation will be different), which demonstrates the problem with this approach.\n'

### Part (c) -- 7%

Finally, we'll move some of the data in our training set into a validation set.

Explain why we should limit how many times we use the test set, and that we should use the validation
set during the model building process.

```
# shuffle the training set
reindex = np.random.permutation(len(train_xs))
train_xs = train_xs[reindex]
train_norm_xs = train_norm_xs[reindex]
train_ts = train_ts[reindex]

# use the first 50000 elements of `train_xs` as the validation set
train_xs, val_xs           = train_xs[50000:], train_xs[:50000]
train_norm_xs, val_norm_xs = train_norm_xs[50000:], train_norm_xs[:50000]
train_ts, val_ts           = train_ts[50000:], train_ts[:50000]

# Write your explanation here
'''
If we will not limit the times that we are using the test set, we may encounter overfitting phenomena over the test data,
which in turn will lead to good preformance over the test set, this phenomena may occur becuase we are tuning our model to achieve
good preformance over a specific set of test samples.
This is a problem because the main reason that we are using a test set is to evaluate our model preformance over an unseen data,
i.e., we test how well the training procedure will be able to generalize for unseen samples.
We can avoid this issue by examing our model preformance (during the bulding procedure) via
another set which we call "validation" set. The validation dataset is different from the test dataset,
both datasets are held back from the training of the model,but the validation set used to give an unbiased preformance estimation
of the final tuned model when comparing between different final models."
'''
```

    '\nIf we will not limit the times that we are using the test set, we may encounter overfitting phenomena over the test data, \nwhich in turn will lead to good preformance over the test set, this phenomena may occur becuase we are tuning our model to achieve\ngood preformance over a specific set of test samples. \nThis is a problem because the main reason that we are using a test set is to evaluate our model preformance over an unseen data,\ni.e., we test how well the training procedure will be able to generalize for unseen samples. \nWe can avoid this issue by examing our model preformance (during the bulding procedure) via\nanother set which we call "validation" set. The validation dataset is different from the test dataset, \nboth datasets are held back from the training of the model,but the validation set used to give an unbiased preformance estimation\nof the final tuned model when comparing between different final models."\n'

## Part 2. Classification (79%)

We will first build a *classification* model to perform decade classification.
These helper functions are written for you. All other code that you write in this section should be vectorized whenever possible (i.e., avoid unnecessary loops).

```
def sigmoid(z):
  return 1 / (1 + np.exp(-z))

def cross_entropy(t, y):
  epsilon=1e-15 #to avoid log(0)
  return -t * np.log(y+epsilon) - (1 - t) * np.log(1 - y+epsilon)

def cost(y, t):
  return np.mean(cross_entropy(t, y))

def get_accuracy(y, t):
  acc = 0
  N = 0
  for i in range(len(y)):
    N += 1
    if (y[i] >= 0.5 and t[i] == 1) or (y[i] < 0.5 and t[i] == 0):
      acc += 1
  return acc / N
```

### Part (a) -- 7%

Write a function `pred` that computes the prediction `y` based on logistic regression, i.e., a single layer with weights `w` and bias `b`. The output is given by:
\begin{equation}
y = \sigma({\bf w}^T {\bf x} + b),
\end{equation}
where the value of $y$ is an estimate of the probability that the song is released in the current century, namely ${\rm year} =1$.

```
def pred(w, b, X):
  """
  Returns the prediction `y` of the target based on the weights `w` and scalar
  bias `b`.

  Preconditions: np.shape(w) == (90,)
                 type(b) == float
                 np.shape(X) = (N, 90) for some N

  >>> pred(np.zeros(90), 1, np.ones([2, 90]))
  array([0.73105858, 0.73105858]) # It's okay if your output differs in the last
  decimals
  """
  return sigmoid(np.dot(X,w)+b)

```

### Part (b) -- 7%

Write a function `derivative_cost` that computes and returns the gradients
$\frac{\partial\mathcal{L}}{\partial {\bf w}}$ and
$\frac{\partial\mathcal{L}}{\partial b}$. Here, `X` is the input, `y` is the prediction, and `t` is the true label.

```
def derivative_cost(X, y, t):
  """
  Returns a tuple containing the gradients dLdw and dLdb.

  Precondition: np.shape(X) == (N, 90) for some N
                np.shape(y) == (N,)
                np.shape(t) == (N,)

  Postcondition: np.shape(dLdw) = (90,)
           type(dLdb) = float
  """
  N = np.shape(y)[0]
  dLdb = np.mean(y - t)
  dLdw = 1/N * np.dot(y - t,X)
  return (dLdw, dLdb)
```

# **Explenation on Gradients**

First we define our loss function as followed:

$L(w,b)=-\frac{1}{N}\sum_{n=1}^{N}t_n\log(y_n)+(1-t_n)\log(1-y_n)$.

Let $z_n = \mathbf{w}^T\mathbf{x}_n+b$

where we denote bold letter as a vector.

By the chain-rule:

$\frac{\partial L}{\partial w}= \sum_{n=1}^{N}\frac{\partial L}{\partial y_n}\cdot \frac{\partial y_n}{\partial z_n} \cdot \frac{\partial z_n}{\partial w}$

$\frac{\partial L}{\partial b}= \sum_{n=1}^{N} \frac{\partial L}{\partial y_n}\cdot \frac{\partial y_n}{\partial z_n} \cdot \frac{\partial z_n}{\partial b}$

where:

$\frac{\partial L}{\partial y_n}=-\frac{1}{N}(\frac{t_n}{y_n}-\frac{1-t_n}{1-y_n} )=\frac{y_n-t_n}{N \cdot y_n \cdot (1-y_n)}$,

$\frac{\partial y_n}{\partial z_n} = \sigma'(z_n) = \sigma(z_n)(1-\sigma(z_n))=y_n(1-y_n)$

$\frac{\partial z_n}{\partial b} =1$,

$\frac{\partial z_n}{\partial w}=\mathbf{x}_n$

These boils down to:

$\frac{\partial L}{\partial b} = \sum_{n=1}^{N} \frac{y_n-t_n}{N \cdot y_n \cdot (1-y_n)} \cdot y_n(1-y_n) \cdot 1 = \frac{1}{N}\sum_{n=1}^{N}y_n-t_n$

$\frac{\partial L}{\partial w} = \sum_{n=1}^{N} \frac{y_n-t_n}{N \cdot y_n \cdot (1-y_n)} \cdot y_n(1-y_n) \cdot \mathbf{x}_b = \frac{1}{N}\sum_{n=1}^{N}(y_n-t_n)\mathbf{x}_n$

### Part (c) -- 7%

We can check that our derivative is implemented correctly using the finite difference rule. In 1D, the
finite difference rule tells us that for small $h$, we should have

$$\frac{f(x+h) - f(x)}{h} \approx f'(x)$$

Show that $\frac{\partial\mathcal{L}}{\partial b}$  is implement correctly
by comparing the result from `derivative_cost` with the empirical cost derivative computed using the above numerical approximation.

```
# Your code goes here
def compute_analytical_deriviate(w,b,t,X,h=1e-05):
  '''
  First we calculate the derivative with respect to the bias term
  '''
  # Calculate the predications
  y = pred(w, b, X)
  # Calculate the loss of those predications
  L = cost(y,t)
  y_new = pred(w,b+h,X)
  L_new = cost(y_new,t)

  # Calculate the analytic derivative
  deriviate_b =(L_new-L)/h

  '''
  Now we calculate the derivative with respect to the weights
  '''
  L_new = np.zeros(len(w))
  deriviate_w = np.zeros(len(w))
  for weight_idx in range(len(w)):
    # Compute cost of w[weight_idx] + h
    w_new = np.copy(w)
    w_new[weight_idx] = w_new[weight_idx] + h
    # Calculate the predications
    y_new = pred(w_new,b,X)
    # Calculate the loss of those predications
    L_new[weight_idx]=cost(y_new,t)
    deriviate_w[weight_idx] = (L_new[weight_idx]-L)/h

  return deriviate_w, deriviate_b

```

Now we are going to check our derivative using an example

```
w=np.zeros(90)
h=1e-05
b=1
t=np.array([1,1])
X=np.ones([2, 90])
y = pred(w, b, X)
dw,db = derivative_cost(X,y,t)
dw_analytic, db_analytic = compute_analytical_deriviate(w,b,t,X,h)
print("The bias analytical derivative results is -", db_analytic)
print("The bias algorithm derivative results is - ", db)
print(f"The error is {np.abs(db-db_analytic)}")
```

    The bias analytical derivative results is - -0.26894043830827385
    The bias algorithm derivative results is -  -0.2689414213699951
    The error is 9.830617212491788e-07

### Part (d) -- 7%

Show that $\frac{\partial\mathcal{L}}{\partial {\bf w}}$  is implement correctly.

```
print("The weights analytical derivative results is -", dw_analytic)
print("The weights algorithm derivative results is - ", dw)
print(f"The maximal error is {np.max(np.abs(dw-dw_analytic))}")
```

    The weights analytical derivative results is - [-0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044
     -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044 -0.26894044]
    The weights algorithm derivative results is -  [-0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142
     -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142 -0.26894142]
    The maximal error is 9.830617212491788e-07

Now we will check over couple of real examples from the training set

```
h = 1e-9;
# First we initialize the weights by random numbers
w=np.random.rand(90)
b=np.random.rand(1)

# Now we extract the actual samples from the dataset
t = (train_ts[:1]).squeeze()
X = train_norm_xs[:1]
y = pred(w, b, X)

dw,db = derivative_cost(X,y,t)
dw_analytic, db_analytic = compute_analytical_deriviate(w,b,t,X,h)
print("The analytical derivative bias results is -", db_analytic)
print("The algorithm derivative bias results is - ", db)
print(f"Bias Error is {np.abs(db-db_analytic)}")

print("The analytical derivative weights results is -", dw_analytic)
print("The algorithm derivative weights results is - ", dw)
print(f"Maximal weights Error is {np.max(np.abs(dw-dw_analytic))}")
```

    The analytical derivative bias results is - -0.9672760370449395
    The algorithm derivative bias results is -  -0.9672759967800327
    Bias Error is 4.0264906742137896e-08
    The analytical derivative weights results is - [ 0.97880459  1.33712197  1.5150321   1.17287824 -0.1407332  -0.28086733
      0.99645137  1.66069425  0.34143222  2.53612864 -0.60000049  0.2660081
     -0.16358959  0.27339908 -0.18315438 -0.02620393  0.57133454  0.29290792
     -0.22191848 -0.26613023  0.32624214 -0.12356471  0.2247269  -0.1315108
      0.79538109  0.54556981  0.37574432  0.19421131  0.50898086  0.31249137
     -0.95565378 -0.54595706  0.15626922  0.11944934 -0.86183682 -0.96469277
     -0.30012703 -0.25347191 -0.2640399  -0.20129987  0.12283019  0.02585665
     -0.75847728  0.77280982  0.88499963 -0.2634799   0.25427704  0.86388496
     -0.74310602 -0.05215162  0.02074074 -0.18614754  0.64476735  0.68632078
     -0.79160367 -0.5105103  -0.51600768  0.62609251 -0.36517989  1.05336495
      0.91098018  0.16509771  0.39282799  0.36208059 -1.05262732  0.20108226
      0.01534106 -0.28936853  0.35263614 -0.85813934  0.51580074 -0.01067191
     -0.41914161  0.52804738 -0.2147611  -0.83647089  0.07340262 -1.70637771
     -0.25002356  0.29500091  0.20058133 -0.08705836  0.93793728  0.12571144
     -1.7977615  -0.31657921  0.64228134  0.73033668 -0.51113247  0.58821525]
    The algorithm derivative weights results is -  [ 0.97880513  1.33712198  1.51503252  1.17287858 -0.14073285 -0.28086701
      0.99645136  1.66069465  0.34143218  2.53612882 -0.60000029  0.26600803
     -0.16358953  0.27339904 -0.18315388 -0.02620335  0.57133477  0.29290862
     -0.22191785 -0.26613003  0.3262425  -0.12356409  0.22472752 -0.13151021
      0.79538094  0.54556964  0.37574461  0.19421122  0.5089815   0.31249165
     -0.95565349 -0.54595737  0.15626943  0.11944995 -0.86183685 -0.96469248
     -0.30012698 -0.25347199 -0.26404002 -0.20129932  0.12282997  0.02585716
     -0.75847651  0.77280993  0.88499994 -0.26347972  0.25427681  0.86388554
     -0.74310507 -0.05215134  0.02074135 -0.18614762  0.64476718  0.68632109
     -0.79160365 -0.5105102  -0.51600786  0.62609255 -0.36517956  1.05336573
      0.91098042  0.16509771  0.39282848  0.3620809  -1.05262721  0.20108315
      0.01534155 -0.28936764  0.35263587 -0.8581389   0.51580074 -0.01067161
     -0.41914165  0.52804727 -0.21476141 -0.83647112  0.07340269 -1.70637775
     -0.25002302  0.29500102  0.20058141 -0.08705784  0.93793721  0.12571158
     -1.79776144 -0.31657967  0.64228073  0.73033717 -0.51113169  0.58821569]
    Maximal weights Error is 9.504835127849276e-07

### Part (e) -- 7%

Now that you have a gradient function that works, we can actually run gradient descent.
Complete the following code that will run stochastic: gradient descent training:

```
def run_gradient_descent(w0, b0, mu=0.1, batch_size=100, max_iters=100):
  """Return the values of (w, b) after running gradient descent for max_iters.
  We use:
    - train_norm_xs and train_ts as the training set
    - val_norm_xs and val_ts as the test set
    - mu as the learning rate
    - (w0, b0) as the initial values of (w, b)

  Precondition: np.shape(w0) == (90,)
                type(b0) == float

  Postcondition: np.shape(w) == (90,)
                 type(b) == float
  """
  w = w0
  b = b0
  iter = 0
  val_loss = []

  while iter < max_iters:
    # shuffle the training set (there is code above for how to do this)
    new_indices = np.random.permutation(len(train_norm_xs))
    train_norm_xs_temp = train_norm_xs[new_indices]
    train_ts_temp = train_ts[new_indices]

    for i in range(0, len(train_norm_xs), batch_size): # iterate over each minibatch
      # minibatch that we are working with:
      X = train_norm_xs_temp[i:(i + batch_size)]
      t = train_ts_temp[i:(i + batch_size), 0]

      # since len(train_norm_xs) does not divide batch_size evenly, we will skip over
      # the "last" minibatch
      if np.shape(X)[0] != batch_size:
        continue

      # compute the prediction
      y = pred(w, b, X)
      # update w and b
      dw,db = derivative_cost(X,y,t)
      w -= mu*dw
      b -= mu*db

      # increment the iteration count
      iter += 1
      # compute and print the *validation* loss and accuracy
      if (iter % 10 == 0):
        # Calculate the predications over the validation data
        y_val = pred(w,b,val_norm_xs)
        # Calculate the cost over the predications
        val_cost = cost(y_val,val_ts.squeeze())
        val_acc = get_accuracy(y_val, val_ts)
        print("Iter %d. [Val Acc %.0f%%, Loss %f]" % (iter, val_acc * 100, val_cost))
        val_loss.append(val_cost)
      if iter >= max_iters:
        break

      # Think what parameters you should return for further use
      # We choose to return in addition to w and b the validation loss in order to evaluate our model training procedure over unseen data.
  return w, b, val_loss

```

### Part (f) -- 7%

Call `run_gradient_descent` with the weights and biases all initialized to zero.
Show that if the learning rate $\mu$ is too small, then convergence is slow.
Also, show that if $\mu$ is too large, then the optimization algorirthm does not converge. The demonstration should be made using plots showing these effects.

```
w0 = np.zeros(90)
b0 = 0
Iterations = 1000
mus=[0.005,0.1,5]
for mu in mus:
  print("mu=",mu,":")
  w,b,losses=run_gradient_descent(w0,b0,mu, max_iters=Iterations)
  print("-------------------------")
  plt.semilogy(range(0,Iterations,10),losses)

plt.legend(mus,loc='best')
plt.xlabel("Iteration")
plt.ylabel("Loss")
plt.grid()
plt.title("Learning rate convergance properties")

```

We conclude from the figure above that for a small learning rate, the algorithm may be capable to converge but very slowly due to the small changes in the parameters space. On the other hand, we see that by using too large learning rate, the algorithm is incapable to converge due to the large changes in the parameters space. Thus, we conclude from the empirical results that choosing a learning rate somewhere in between those learning rates leads to the best performance.
We explain this phenomenon by the fact that the algorithm can change its parameters efficiently towards a good optimum. Meaning that we have found a good trade-off between a small enough step size that allows convergence but not too small such that it leads to faster convergence rate to a good enough optimum.

### Part (g) -- 7%

Find the optimial value of ${\bf w}$ and $b$ using your code. Explain how you chose
the learning rate $\mu$ and the batch size. Show plots demostrating good and bad behaviours.

```
'''
Hyperparameters search
'''

mus = [0.01, 0.1, 0.5, 1]
batch_sizes = [1, 64, 1024, 50000]
Iterations = 1000
loss_results = np.zeros((len(mus),len(batch_sizes)))
w_history = np.zeros((len(mus),len(batch_sizes), 90))
b_history = np.zeros((len(mus),len(batch_sizes), 1))

w0 = np.random.randn(90)
b0 = np.random.randn(1)[0]

for mu_idx,mu in enumerate(mus):
  for minibatch_idx, minibatch_size in enumerate(batch_sizes):
    print("mu=",mu, 'batch_size=', minibatch_size, ':')
    w,b,losses=run_gradient_descent(w0,b0,mu, max_iters=Iterations)
    print("-------------------------")
    loss_results[mu_idx][minibatch_idx] = losses[-1]
    w_history[mu_idx][minibatch_idx] = w
    b_history[mu_idx][minibatch_idx] = b

best_mu_idx, best_batch_size_idx = np.unravel_index(np.argmin(loss_results, axis=None), loss_results.shape)
print("optimal mu=",mus[best_mu_idx], 'optimal batch_size=', batch_sizes[best_batch_size_idx])
'''
Save the optimal weight and bias
'''
opt_w = w_history[best_mu_idx][best_batch_size_idx]
opt_b = b_history[best_mu_idx][best_batch_size_idx]
```

    mu= 0.01 batch_size= 1 :
    Iter 10. [Val Acc 47%, Loss 2.747708]
    Iter 20. [Val Acc 47%, Loss 2.714000]
    Iter 30. [Val Acc 47%, Loss 2.686253]
    Iter 40. [Val Acc 47%, Loss 2.659955]
    Iter 50. [Val Acc 47%, Loss 2.633081]
    Iter 60. [Val Acc 47%, Loss 2.603889]
    Iter 70. [Val Acc 48%, Loss 2.580692]
    Iter 80. [Val Acc 48%, Loss 2.557785]
    Iter 90. [Val Acc 48%, Loss 2.530571]
    Iter 100. [Val Acc 48%, Loss 2.503728]
    Iter 110. [Val Acc 48%, Loss 2.479285]
    Iter 120. [Val Acc 48%, Loss 2.455511]
    Iter 130. [Val Acc 48%, Loss 2.436268]
    Iter 140. [Val Acc 48%, Loss 2.415122]
    Iter 150. [Val Acc 48%, Loss 2.394333]
    Iter 160. [Val Acc 48%, Loss 2.377478]
    Iter 170. [Val Acc 48%, Loss 2.357796]
    Iter 180. [Val Acc 48%, Loss 2.338882]
    Iter 190. [Val Acc 48%, Loss 2.319985]
    Iter 200. [Val Acc 48%, Loss 2.298555]
    Iter 210. [Val Acc 48%, Loss 2.279912]
    Iter 220. [Val Acc 49%, Loss 2.262655]
    Iter 230. [Val Acc 49%, Loss 2.244858]
    Iter 240. [Val Acc 49%, Loss 2.228405]
    Iter 250. [Val Acc 49%, Loss 2.213221]
    Iter 260. [Val Acc 49%, Loss 2.195344]
    Iter 270. [Val Acc 49%, Loss 2.180319]
    Iter 280. [Val Acc 49%, Loss 2.164565]
    Iter 290. [Val Acc 49%, Loss 2.145138]
    Iter 300. [Val Acc 49%, Loss 2.128512]
    Iter 310. [Val Acc 49%, Loss 2.111651]
    Iter 320. [Val Acc 49%, Loss 2.095608]
    Iter 330. [Val Acc 49%, Loss 2.079349]
    Iter 340. [Val Acc 50%, Loss 2.063636]
    Iter 350. [Val Acc 50%, Loss 2.049249]
    Iter 360. [Val Acc 50%, Loss 2.034857]
    Iter 370. [Val Acc 50%, Loss 2.018879]
    Iter 380. [Val Acc 50%, Loss 2.003994]
    Iter 390. [Val Acc 50%, Loss 1.988755]
    Iter 400. [Val Acc 50%, Loss 1.975895]
    Iter 410. [Val Acc 50%, Loss 1.960965]
    Iter 420. [Val Acc 50%, Loss 1.946616]
    Iter 430. [Val Acc 50%, Loss 1.931607]
    Iter 440. [Val Acc 50%, Loss 1.917479]
    Iter 450. [Val Acc 50%, Loss 1.902136]
    Iter 460. [Val Acc 50%, Loss 1.888529]
    Iter 470. [Val Acc 51%, Loss 1.877311]
    Iter 480. [Val Acc 51%, Loss 1.865819]
    Iter 490. [Val Acc 51%, Loss 1.851953]
    Iter 500. [Val Acc 51%, Loss 1.839929]
    Iter 510. [Val Acc 51%, Loss 1.826131]
    Iter 520. [Val Acc 51%, Loss 1.814031]
    Iter 530. [Val Acc 51%, Loss 1.801829]
    Iter 540. [Val Acc 51%, Loss 1.788531]
    Iter 550. [Val Acc 51%, Loss 1.775909]
    Iter 560. [Val Acc 51%, Loss 1.763521]
    Iter 570. [Val Acc 51%, Loss 1.751194]
    Iter 580. [Val Acc 51%, Loss 1.739099]
    Iter 590. [Val Acc 52%, Loss 1.727596]
    Iter 600. [Val Acc 52%, Loss 1.716846]
    Iter 610. [Val Acc 52%, Loss 1.705644]
    Iter 620. [Val Acc 52%, Loss 1.694009]
    Iter 630. [Val Acc 52%, Loss 1.683048]
    Iter 640. [Val Acc 52%, Loss 1.672278]
    Iter 650. [Val Acc 52%, Loss 1.660642]
    Iter 660. [Val Acc 52%, Loss 1.649790]
    Iter 670. [Val Acc 52%, Loss 1.639040]
    Iter 680. [Val Acc 53%, Loss 1.626574]
    Iter 690. [Val Acc 53%, Loss 1.616777]
    Iter 700. [Val Acc 53%, Loss 1.605414]
    Iter 710. [Val Acc 53%, Loss 1.594303]
    Iter 720. [Val Acc 53%, Loss 1.585565]
    Iter 730. [Val Acc 53%, Loss 1.576315]
    Iter 740. [Val Acc 53%, Loss 1.566632]
    Iter 750. [Val Acc 53%, Loss 1.557272]
    Iter 760. [Val Acc 53%, Loss 1.547641]
    Iter 770. [Val Acc 53%, Loss 1.539211]
    Iter 780. [Val Acc 53%, Loss 1.530813]
    Iter 790. [Val Acc 54%, Loss 1.521222]
    Iter 800. [Val Acc 54%, Loss 1.512906]
    Iter 810. [Val Acc 54%, Loss 1.503275]
    Iter 820. [Val Acc 54%, Loss 1.494597]
    Iter 830. [Val Acc 54%, Loss 1.485597]
    Iter 840. [Val Acc 54%, Loss 1.476089]
    Iter 850. [Val Acc 54%, Loss 1.468392]
    Iter 860. [Val Acc 54%, Loss 1.460788]
    Iter 870. [Val Acc 54%, Loss 1.452696]
    Iter 880. [Val Acc 54%, Loss 1.443291]
    Iter 890. [Val Acc 54%, Loss 1.436108]
    Iter 900. [Val Acc 54%, Loss 1.426818]
    Iter 910. [Val Acc 54%, Loss 1.418355]
    Iter 920. [Val Acc 55%, Loss 1.410911]
    Iter 930. [Val Acc 55%, Loss 1.402065]
    Iter 940. [Val Acc 55%, Loss 1.394395]
    Iter 950. [Val Acc 55%, Loss 1.386117]
    Iter 960. [Val Acc 55%, Loss 1.377559]
    Iter 970. [Val Acc 55%, Loss 1.370999]
    Iter 980. [Val Acc 55%, Loss 1.363807]
    Iter 990. [Val Acc 55%, Loss 1.356797]
    Iter 1000. [Val Acc 55%, Loss 1.349479]
    -------------------------
    mu= 0.01 batch_size= 64 :
    Iter 10. [Val Acc 55%, Loss 1.345617]
    Iter 20. [Val Acc 56%, Loss 1.338529]
    Iter 30. [Val Acc 56%, Loss 1.331585]
    Iter 40. [Val Acc 56%, Loss 1.325503]
    Iter 50. [Val Acc 56%, Loss 1.318606]
    Iter 60. [Val Acc 56%, Loss 1.310718]
    Iter 70. [Val Acc 56%, Loss 1.304447]
    Iter 80. [Val Acc 56%, Loss 1.297898]
    Iter 90. [Val Acc 56%, Loss 1.290197]
    Iter 100. [Val Acc 56%, Loss 1.283876]
    Iter 110. [Val Acc 56%, Loss 1.277738]
    Iter 120. [Val Acc 56%, Loss 1.271026]
    Iter 130. [Val Acc 56%, Loss 1.263781]
    Iter 140. [Val Acc 56%, Loss 1.257526]
    Iter 150. [Val Acc 57%, Loss 1.251464]
    Iter 160. [Val Acc 57%, Loss 1.245631]
    Iter 170. [Val Acc 57%, Loss 1.239308]
    Iter 180. [Val Acc 57%, Loss 1.232742]
    Iter 190. [Val Acc 57%, Loss 1.227313]
    Iter 200. [Val Acc 57%, Loss 1.221064]
    Iter 210. [Val Acc 57%, Loss 1.214861]
    Iter 220. [Val Acc 57%, Loss 1.208799]
    Iter 230. [Val Acc 57%, Loss 1.202713]
    Iter 240. [Val Acc 57%, Loss 1.197273]
    Iter 250. [Val Acc 57%, Loss 1.191149]
    Iter 260. [Val Acc 57%, Loss 1.185029]
    Iter 270. [Val Acc 57%, Loss 1.179632]
    Iter 280. [Val Acc 57%, Loss 1.174148]
    Iter 290. [Val Acc 57%, Loss 1.168859]
    Iter 300. [Val Acc 57%, Loss 1.163904]
    Iter 310. [Val Acc 58%, Loss 1.158641]
    Iter 320. [Val Acc 58%, Loss 1.153226]
    Iter 330. [Val Acc 58%, Loss 1.148096]
    Iter 340. [Val Acc 58%, Loss 1.142611]
    Iter 350. [Val Acc 58%, Loss 1.137139]
    Iter 360. [Val Acc 58%, Loss 1.132120]
    Iter 370. [Val Acc 58%, Loss 1.126368]
    Iter 380. [Val Acc 58%, Loss 1.121926]
    Iter 390. [Val Acc 58%, Loss 1.116211]
    Iter 400. [Val Acc 58%, Loss 1.111316]
    Iter 410. [Val Acc 58%, Loss 1.107111]
    Iter 420. [Val Acc 58%, Loss 1.102484]
    Iter 430. [Val Acc 58%, Loss 1.098378]
    Iter 440. [Val Acc 58%, Loss 1.094737]
    Iter 450. [Val Acc 58%, Loss 1.090213]
    Iter 460. [Val Acc 59%, Loss 1.085753]
    Iter 470. [Val Acc 59%, Loss 1.081230]
    Iter 480. [Val Acc 59%, Loss 1.076412]
    Iter 490. [Val Acc 59%, Loss 1.072144]
    Iter 500. [Val Acc 59%, Loss 1.067320]
    Iter 510. [Val Acc 59%, Loss 1.061990]
    Iter 520. [Val Acc 59%, Loss 1.057249]
    Iter 530. [Val Acc 59%, Loss 1.053846]
    Iter 540. [Val Acc 59%, Loss 1.049935]
    Iter 550. [Val Acc 59%, Loss 1.046121]
    Iter 560. [Val Acc 59%, Loss 1.041967]
    Iter 570. [Val Acc 59%, Loss 1.038177]
    Iter 580. [Val Acc 59%, Loss 1.033472]
    Iter 590. [Val Acc 59%, Loss 1.029379]
    Iter 600. [Val Acc 59%, Loss 1.024871]
    Iter 610. [Val Acc 59%, Loss 1.020544]
    Iter 620. [Val Acc 59%, Loss 1.017094]
    Iter 630. [Val Acc 60%, Loss 1.013561]
    Iter 640. [Val Acc 60%, Loss 1.009934]
    Iter 650. [Val Acc 60%, Loss 1.005945]
    Iter 660. [Val Acc 60%, Loss 1.002079]
    Iter 670. [Val Acc 60%, Loss 0.998660]
    Iter 680. [Val Acc 60%, Loss 0.994886]
    Iter 690. [Val Acc 60%, Loss 0.991315]
    Iter 700. [Val Acc 60%, Loss 0.987764]
    Iter 710. [Val Acc 60%, Loss 0.984638]
    Iter 720. [Val Acc 60%, Loss 0.981326]
    Iter 730. [Val Acc 60%, Loss 0.977929]
    Iter 740. [Val Acc 60%, Loss 0.974272]
    Iter 750. [Val Acc 60%, Loss 0.971192]
    Iter 760. [Val Acc 60%, Loss 0.968324]
    Iter 770. [Val Acc 60%, Loss 0.965537]
    Iter 780. [Val Acc 60%, Loss 0.962614]
    Iter 790. [Val Acc 61%, Loss 0.959136]
    Iter 800. [Val Acc 61%, Loss 0.956208]
    Iter 810. [Val Acc 61%, Loss 0.953530]
    Iter 820. [Val Acc 61%, Loss 0.950446]
    Iter 830. [Val Acc 61%, Loss 0.946679]
    Iter 840. [Val Acc 61%, Loss 0.943529]
    Iter 850. [Val Acc 61%, Loss 0.940725]
    Iter 860. [Val Acc 61%, Loss 0.937726]
    Iter 870. [Val Acc 61%, Loss 0.934797]
    Iter 880. [Val Acc 61%, Loss 0.931442]
    Iter 890. [Val Acc 61%, Loss 0.928389]
    Iter 900. [Val Acc 61%, Loss 0.925764]
    Iter 910. [Val Acc 61%, Loss 0.922628]
    Iter 920. [Val Acc 61%, Loss 0.919816]
    Iter 930. [Val Acc 61%, Loss 0.917270]
    Iter 940. [Val Acc 62%, Loss 0.914550]
    Iter 950. [Val Acc 62%, Loss 0.911570]
    Iter 960. [Val Acc 62%, Loss 0.909385]
    Iter 970. [Val Acc 62%, Loss 0.906585]
    Iter 980. [Val Acc 62%, Loss 0.904106]
    Iter 990. [Val Acc 62%, Loss 0.901269]
    Iter 1000. [Val Acc 62%, Loss 0.898525]
    -------------------------
    mu= 0.01 batch_size= 1024 :
    Iter 10. [Val Acc 62%, Loss 0.906328]
    Iter 20. [Val Acc 62%, Loss 0.903532]
    Iter 30. [Val Acc 62%, Loss 0.901044]
    Iter 40. [Val Acc 62%, Loss 0.898690]
    Iter 50. [Val Acc 62%, Loss 0.896345]
    Iter 60. [Val Acc 62%, Loss 0.893715]
    Iter 70. [Val Acc 62%, Loss 0.891497]
    Iter 80. [Val Acc 62%, Loss 0.888453]
    Iter 90. [Val Acc 62%, Loss 0.886079]
    Iter 100. [Val Acc 62%, Loss 0.883580]
    Iter 110. [Val Acc 62%, Loss 0.880946]
    Iter 120. [Val Acc 62%, Loss 0.878768]
    Iter 130. [Val Acc 62%, Loss 0.876273]
    Iter 140. [Val Acc 62%, Loss 0.874109]
    Iter 150. [Val Acc 63%, Loss 0.871913]
    Iter 160. [Val Acc 63%, Loss 0.869762]
    Iter 170. [Val Acc 63%, Loss 0.867458]
    Iter 180. [Val Acc 63%, Loss 0.865235]
    Iter 190. [Val Acc 63%, Loss 0.863043]
    Iter 200. [Val Acc 63%, Loss 0.860969]
    Iter 210. [Val Acc 63%, Loss 0.859043]
    Iter 220. [Val Acc 63%, Loss 0.856785]
    Iter 230. [Val Acc 63%, Loss 0.854760]
    Iter 240. [Val Acc 63%, Loss 0.852633]
    Iter 250. [Val Acc 63%, Loss 0.850425]
    Iter 260. [Val Acc 63%, Loss 0.848362]
    Iter 270. [Val Acc 63%, Loss 0.846445]
    Iter 280. [Val Acc 63%, Loss 0.844571]
    Iter 290. [Val Acc 63%, Loss 0.842505]
    Iter 300. [Val Acc 63%, Loss 0.840632]
    Iter 310. [Val Acc 63%, Loss 0.838513]
    Iter 320. [Val Acc 63%, Loss 0.836432]
    Iter 330. [Val Acc 63%, Loss 0.834393]
    Iter 340. [Val Acc 63%, Loss 0.832571]
    Iter 350. [Val Acc 64%, Loss 0.830582]
    Iter 360. [Val Acc 64%, Loss 0.828517]
    Iter 370. [Val Acc 64%, Loss 0.826750]
    Iter 380. [Val Acc 64%, Loss 0.824939]
    Iter 390. [Val Acc 64%, Loss 0.823122]
    Iter 400. [Val Acc 64%, Loss 0.821391]
    Iter 410. [Val Acc 64%, Loss 0.819657]
    Iter 420. [Val Acc 64%, Loss 0.818024]
    Iter 430. [Val Acc 64%, Loss 0.815976]
    Iter 440. [Val Acc 64%, Loss 0.814484]
    Iter 450. [Val Acc 64%, Loss 0.813169]
    Iter 460. [Val Acc 64%, Loss 0.811369]
    Iter 470. [Val Acc 64%, Loss 0.809472]
    Iter 480. [Val Acc 64%, Loss 0.807631]
    Iter 490. [Val Acc 64%, Loss 0.805938]
    Iter 500. [Val Acc 64%, Loss 0.804531]
    Iter 510. [Val Acc 64%, Loss 0.803300]
    Iter 520. [Val Acc 64%, Loss 0.801580]
    Iter 530. [Val Acc 64%, Loss 0.800029]
    Iter 540. [Val Acc 64%, Loss 0.798580]
    Iter 550. [Val Acc 64%, Loss 0.797229]
    Iter 560. [Val Acc 64%, Loss 0.795180]
    Iter 570. [Val Acc 64%, Loss 0.793616]
    Iter 580. [Val Acc 64%, Loss 0.792142]
    Iter 590. [Val Acc 64%, Loss 0.790722]
    Iter 600. [Val Acc 64%, Loss 0.789357]
    Iter 610. [Val Acc 64%, Loss 0.787887]
    Iter 620. [Val Acc 64%, Loss 0.786438]
    Iter 630. [Val Acc 65%, Loss 0.784737]
    Iter 640. [Val Acc 65%, Loss 0.783578]
    Iter 650. [Val Acc 65%, Loss 0.782101]
    Iter 660. [Val Acc 65%, Loss 0.780594]
    Iter 670. [Val Acc 65%, Loss 0.779144]
    Iter 680. [Val Acc 65%, Loss 0.777746]
    Iter 690. [Val Acc 65%, Loss 0.776177]
    Iter 700. [Val Acc 65%, Loss 0.774936]
    Iter 710. [Val Acc 65%, Loss 0.773153]
    Iter 720. [Val Acc 65%, Loss 0.771496]
    Iter 730. [Val Acc 65%, Loss 0.770301]
    Iter 740. [Val Acc 65%, Loss 0.768918]
    Iter 750. [Val Acc 65%, Loss 0.767392]
    Iter 760. [Val Acc 65%, Loss 0.766347]
    Iter 770. [Val Acc 65%, Loss 0.765385]
    Iter 780. [Val Acc 65%, Loss 0.764265]
    Iter 790. [Val Acc 65%, Loss 0.762733]
    Iter 800. [Val Acc 65%, Loss 0.761390]
    Iter 810. [Val Acc 65%, Loss 0.760108]
    Iter 820. [Val Acc 65%, Loss 0.758848]
    Iter 830. [Val Acc 65%, Loss 0.757540]
    Iter 840. [Val Acc 65%, Loss 0.756335]
    Iter 850. [Val Acc 65%, Loss 0.754976]
    Iter 860. [Val Acc 65%, Loss 0.754003]
    Iter 870. [Val Acc 65%, Loss 0.752844]
    Iter 880. [Val Acc 65%, Loss 0.751843]
    Iter 890. [Val Acc 65%, Loss 0.750819]
    Iter 900. [Val Acc 65%, Loss 0.749556]
    Iter 910. [Val Acc 65%, Loss 0.748219]
    Iter 920. [Val Acc 65%, Loss 0.747076]
    Iter 930. [Val Acc 66%, Loss 0.745959]
    Iter 940. [Val Acc 66%, Loss 0.744811]
    Iter 950. [Val Acc 66%, Loss 0.743666]
    Iter 960. [Val Acc 66%, Loss 0.742686]
    Iter 970. [Val Acc 66%, Loss 0.741863]
    Iter 980. [Val Acc 66%, Loss 0.740832]
    Iter 990. [Val Acc 66%, Loss 0.739786]
    Iter 1000. [Val Acc 66%, Loss 0.738588]
    -------------------------
    mu= 0.01 batch_size= 50000 :
    Iter 10. [Val Acc 65%, Loss 0.755230]
    Iter 20. [Val Acc 65%, Loss 0.753799]
    Iter 30. [Val Acc 65%, Loss 0.752042]
    Iter 40. [Val Acc 66%, Loss 0.750682]
    Iter 50. [Val Acc 66%, Loss 0.748914]
    Iter 60. [Val Acc 66%, Loss 0.747344]
    Iter 70. [Val Acc 66%, Loss 0.745985]
    Iter 80. [Val Acc 66%, Loss 0.744480]
    Iter 90. [Val Acc 66%, Loss 0.743040]
    Iter 100. [Val Acc 66%, Loss 0.741452]
    Iter 110. [Val Acc 66%, Loss 0.740027]
    Iter 120. [Val Acc 66%, Loss 0.738746]
    Iter 130. [Val Acc 66%, Loss 0.737431]
    Iter 140. [Val Acc 66%, Loss 0.736303]
    Iter 150. [Val Acc 66%, Loss 0.734928]
    Iter 160. [Val Acc 66%, Loss 0.733650]
    Iter 170. [Val Acc 66%, Loss 0.732754]
    Iter 180. [Val Acc 66%, Loss 0.731638]
    Iter 190. [Val Acc 66%, Loss 0.730632]
    Iter 200. [Val Acc 66%, Loss 0.729331]
    Iter 210. [Val Acc 66%, Loss 0.728069]
    Iter 220. [Val Acc 66%, Loss 0.726973]
    Iter 230. [Val Acc 66%, Loss 0.725977]
    Iter 240. [Val Acc 66%, Loss 0.724940]
    Iter 250. [Val Acc 67%, Loss 0.723628]
    Iter 260. [Val Acc 67%, Loss 0.722392]
    Iter 270. [Val Acc 67%, Loss 0.721598]
    Iter 280. [Val Acc 67%, Loss 0.720821]
    Iter 290. [Val Acc 67%, Loss 0.719714]
    Iter 300. [Val Acc 67%, Loss 0.718522]
    Iter 310. [Val Acc 67%, Loss 0.717457]
    Iter 320. [Val Acc 67%, Loss 0.716587]
    Iter 330. [Val Acc 67%, Loss 0.715639]
    Iter 340. [Val Acc 67%, Loss 0.714676]
    Iter 350. [Val Acc 67%, Loss 0.713573]
    Iter 360. [Val Acc 67%, Loss 0.712725]
    Iter 370. [Val Acc 67%, Loss 0.711915]
    Iter 380. [Val Acc 67%, Loss 0.710702]
    Iter 390. [Val Acc 67%, Loss 0.709541]
    Iter 400. [Val Acc 67%, Loss 0.708734]
    Iter 410. [Val Acc 67%, Loss 0.707741]
    Iter 420. [Val Acc 67%, Loss 0.706738]
    Iter 430. [Val Acc 67%, Loss 0.705661]
    Iter 440. [Val Acc 67%, Loss 0.704696]
    Iter 450. [Val Acc 67%, Loss 0.703845]
    Iter 460. [Val Acc 67%, Loss 0.703125]
    Iter 470. [Val Acc 67%, Loss 0.702134]
    Iter 480. [Val Acc 67%, Loss 0.701258]
    Iter 490. [Val Acc 67%, Loss 0.700424]
    Iter 500. [Val Acc 67%, Loss 0.699536]
    Iter 510. [Val Acc 67%, Loss 0.698573]
    Iter 520. [Val Acc 67%, Loss 0.697681]
    Iter 530. [Val Acc 67%, Loss 0.696743]
    Iter 540. [Val Acc 67%, Loss 0.695802]
    Iter 550. [Val Acc 67%, Loss 0.694991]
    Iter 560. [Val Acc 67%, Loss 0.694154]
    Iter 570. [Val Acc 67%, Loss 0.693397]
    Iter 580. [Val Acc 67%, Loss 0.692732]
    Iter 590. [Val Acc 67%, Loss 0.691846]
    Iter 600. [Val Acc 67%, Loss 0.691009]
    Iter 610. [Val Acc 67%, Loss 0.690373]
    Iter 620. [Val Acc 67%, Loss 0.689547]
    Iter 630. [Val Acc 67%, Loss 0.688896]
    Iter 640. [Val Acc 68%, Loss 0.688155]
    Iter 650. [Val Acc 68%, Loss 0.687193]
    Iter 660. [Val Acc 68%, Loss 0.686396]
    Iter 670. [Val Acc 68%, Loss 0.685603]
    Iter 680. [Val Acc 68%, Loss 0.684812]
    Iter 690. [Val Acc 68%, Loss 0.684158]
    Iter 700. [Val Acc 68%, Loss 0.683343]
    Iter 710. [Val Acc 68%, Loss 0.682598]
    Iter 720. [Val Acc 68%, Loss 0.682021]
    Iter 730. [Val Acc 68%, Loss 0.681436]
    Iter 740. [Val Acc 68%, Loss 0.680796]
    Iter 750. [Val Acc 68%, Loss 0.679902]
    Iter 760. [Val Acc 68%, Loss 0.679117]
    Iter 770. [Val Acc 68%, Loss 0.678494]
    Iter 780. [Val Acc 68%, Loss 0.677840]
    Iter 790. [Val Acc 68%, Loss 0.677069]
    Iter 800. [Val Acc 68%, Loss 0.676521]
    Iter 810. [Val Acc 68%, Loss 0.675886]
    Iter 820. [Val Acc 68%, Loss 0.675161]
    Iter 830. [Val Acc 68%, Loss 0.674611]
    Iter 840. [Val Acc 68%, Loss 0.674039]
    Iter 850. [Val Acc 68%, Loss 0.673409]
    Iter 860. [Val Acc 68%, Loss 0.672753]
    Iter 870. [Val Acc 68%, Loss 0.672043]
    Iter 880. [Val Acc 68%, Loss 0.671548]
    Iter 890. [Val Acc 68%, Loss 0.671124]
    Iter 900. [Val Acc 68%, Loss 0.670509]
    Iter 910. [Val Acc 68%, Loss 0.669867]
    Iter 920. [Val Acc 68%, Loss 0.669327]
    Iter 930. [Val Acc 68%, Loss 0.668823]
    Iter 940. [Val Acc 68%, Loss 0.668254]
    Iter 950. [Val Acc 68%, Loss 0.667773]
    Iter 960. [Val Acc 68%, Loss 0.667096]
    Iter 970. [Val Acc 68%, Loss 0.666527]
    Iter 980. [Val Acc 68%, Loss 0.666055]
    Iter 990. [Val Acc 68%, Loss 0.665470]
    Iter 1000. [Val Acc 68%, Loss 0.664856]
    -------------------------
    mu= 0.1 batch_size= 1 :
    Iter 10. [Val Acc 68%, Loss 0.679409]
    Iter 20. [Val Acc 68%, Loss 0.671601]
    Iter 30. [Val Acc 68%, Loss 0.665270]
    Iter 40. [Val Acc 69%, Loss 0.660963]
    Iter 50. [Val Acc 69%, Loss 0.652989]
    Iter 60. [Val Acc 69%, Loss 0.647111]
    Iter 70. [Val Acc 69%, Loss 0.645079]
    Iter 80. [Val Acc 70%, Loss 0.639254]
    Iter 90. [Val Acc 70%, Loss 0.635633]
    Iter 100. [Val Acc 70%, Loss 0.632096]
    Iter 110. [Val Acc 70%, Loss 0.628129]
    Iter 120. [Val Acc 70%, Loss 0.625318]
    Iter 130. [Val Acc 70%, Loss 0.623155]
    Iter 140. [Val Acc 71%, Loss 0.621235]
    Iter 150. [Val Acc 71%, Loss 0.616948]
    Iter 160. [Val Acc 71%, Loss 0.615482]
    Iter 170. [Val Acc 71%, Loss 0.612431]
    Iter 180. [Val Acc 70%, Loss 0.612584]
    Iter 190. [Val Acc 71%, Loss 0.610688]
    Iter 200. [Val Acc 71%, Loss 0.608137]
    Iter 210. [Val Acc 71%, Loss 0.603869]
    Iter 220. [Val Acc 71%, Loss 0.603650]
    Iter 230. [Val Acc 71%, Loss 0.602367]
    Iter 240. [Val Acc 71%, Loss 0.599944]
    Iter 250. [Val Acc 71%, Loss 0.597481]
    Iter 260. [Val Acc 71%, Loss 0.596554]
    Iter 270. [Val Acc 71%, Loss 0.596659]
    Iter 280. [Val Acc 71%, Loss 0.595849]
    Iter 290. [Val Acc 71%, Loss 0.597362]
    Iter 300. [Val Acc 72%, Loss 0.592679]
    Iter 310. [Val Acc 72%, Loss 0.591685]
    Iter 320. [Val Acc 72%, Loss 0.592053]
    Iter 330. [Val Acc 72%, Loss 0.591132]
    Iter 340. [Val Acc 72%, Loss 0.589139]
    Iter 350. [Val Acc 72%, Loss 0.586347]
    Iter 360. [Val Acc 72%, Loss 0.584569]
    Iter 370. [Val Acc 72%, Loss 0.586159]
    Iter 380. [Val Acc 72%, Loss 0.585876]
    Iter 390. [Val Acc 72%, Loss 0.584457]
    Iter 400. [Val Acc 72%, Loss 0.583038]
    Iter 410. [Val Acc 72%, Loss 0.583724]
    Iter 420. [Val Acc 72%, Loss 0.583272]
    Iter 430. [Val Acc 72%, Loss 0.580109]
    Iter 440. [Val Acc 72%, Loss 0.580403]
    Iter 450. [Val Acc 72%, Loss 0.580265]
    Iter 460. [Val Acc 72%, Loss 0.581923]
    Iter 470. [Val Acc 73%, Loss 0.578253]
    Iter 480. [Val Acc 73%, Loss 0.579597]
    Iter 490. [Val Acc 73%, Loss 0.577945]
    Iter 500. [Val Acc 73%, Loss 0.576872]
    Iter 510. [Val Acc 72%, Loss 0.580532]
    Iter 520. [Val Acc 73%, Loss 0.577793]
    Iter 530. [Val Acc 73%, Loss 0.575839]
    Iter 540. [Val Acc 73%, Loss 0.573407]
    Iter 550. [Val Acc 73%, Loss 0.572914]
    Iter 560. [Val Acc 73%, Loss 0.571298]
    Iter 570. [Val Acc 73%, Loss 0.573441]
    Iter 580. [Val Acc 73%, Loss 0.573641]
    Iter 590. [Val Acc 73%, Loss 0.569347]
    Iter 600. [Val Acc 73%, Loss 0.570235]
    Iter 610. [Val Acc 73%, Loss 0.570195]
    Iter 620. [Val Acc 73%, Loss 0.570531]
    Iter 630. [Val Acc 73%, Loss 0.569130]
    Iter 640. [Val Acc 73%, Loss 0.570484]
    Iter 650. [Val Acc 73%, Loss 0.570825]
    Iter 660. [Val Acc 73%, Loss 0.570190]
    Iter 670. [Val Acc 73%, Loss 0.568838]
    Iter 680. [Val Acc 73%, Loss 0.568787]
    Iter 690. [Val Acc 73%, Loss 0.573686]
    Iter 700. [Val Acc 73%, Loss 0.569949]
    Iter 710. [Val Acc 73%, Loss 0.569296]
    Iter 720. [Val Acc 73%, Loss 0.567528]
    Iter 730. [Val Acc 73%, Loss 0.567788]
    Iter 740. [Val Acc 73%, Loss 0.565928]
    Iter 750. [Val Acc 73%, Loss 0.567437]
    Iter 760. [Val Acc 73%, Loss 0.566625]
    Iter 770. [Val Acc 73%, Loss 0.565927]
    Iter 780. [Val Acc 73%, Loss 0.565327]
    Iter 790. [Val Acc 73%, Loss 0.564760]
    Iter 800. [Val Acc 73%, Loss 0.565287]
    Iter 810. [Val Acc 73%, Loss 0.564235]
    Iter 820. [Val Acc 73%, Loss 0.564770]
    Iter 830. [Val Acc 73%, Loss 0.565330]
    Iter 840. [Val Acc 73%, Loss 0.570959]
    Iter 850. [Val Acc 73%, Loss 0.567317]
    Iter 860. [Val Acc 73%, Loss 0.566348]
    Iter 870. [Val Acc 73%, Loss 0.570058]
    Iter 880. [Val Acc 73%, Loss 0.565754]
    Iter 890. [Val Acc 73%, Loss 0.567168]
    Iter 900. [Val Acc 73%, Loss 0.567377]
    Iter 910. [Val Acc 73%, Loss 0.570124]
    Iter 920. [Val Acc 73%, Loss 0.569124]
    Iter 930. [Val Acc 73%, Loss 0.563923]
    Iter 940. [Val Acc 73%, Loss 0.567770]
    Iter 950. [Val Acc 73%, Loss 0.565879]
    Iter 960. [Val Acc 73%, Loss 0.564209]
    Iter 970. [Val Acc 73%, Loss 0.565112]
    Iter 980. [Val Acc 73%, Loss 0.563434]
    Iter 990. [Val Acc 73%, Loss 0.563550]
    Iter 1000. [Val Acc 73%, Loss 0.563739]
    -------------------------
    mu= 0.1 batch_size= 64 :
    Iter 10. [Val Acc 70%, Loss 0.592290]
    Iter 20. [Val Acc 70%, Loss 0.585235]
    Iter 30. [Val Acc 71%, Loss 0.579702]
    Iter 40. [Val Acc 72%, Loss 0.572464]
    Iter 50. [Val Acc 72%, Loss 0.571763]
    Iter 60. [Val Acc 73%, Loss 0.571309]
    Iter 70. [Val Acc 73%, Loss 0.571737]
    Iter 80. [Val Acc 73%, Loss 0.567088]
    Iter 90. [Val Acc 73%, Loss 0.565484]
    Iter 100. [Val Acc 73%, Loss 0.565041]
    Iter 110. [Val Acc 73%, Loss 0.563843]
    Iter 120. [Val Acc 73%, Loss 0.567036]
    Iter 130. [Val Acc 73%, Loss 0.563507]
    Iter 140. [Val Acc 73%, Loss 0.563263]
    Iter 150. [Val Acc 73%, Loss 0.562121]
    Iter 160. [Val Acc 73%, Loss 0.562797]
    Iter 170. [Val Acc 73%, Loss 0.564114]
    Iter 180. [Val Acc 73%, Loss 0.563105]
    Iter 190. [Val Acc 73%, Loss 0.562744]
    Iter 200. [Val Acc 73%, Loss 0.562479]
    Iter 210. [Val Acc 73%, Loss 0.562616]
    Iter 220. [Val Acc 73%, Loss 0.566107]
    Iter 230. [Val Acc 73%, Loss 0.564263]
    Iter 240. [Val Acc 73%, Loss 0.562913]
    Iter 250. [Val Acc 73%, Loss 0.562721]
    Iter 260. [Val Acc 73%, Loss 0.561437]
    Iter 270. [Val Acc 73%, Loss 0.562036]
    Iter 280. [Val Acc 73%, Loss 0.561121]
    Iter 290. [Val Acc 73%, Loss 0.562473]
    Iter 300. [Val Acc 73%, Loss 0.562500]
    Iter 310. [Val Acc 73%, Loss 0.563313]
    Iter 320. [Val Acc 73%, Loss 0.562950]
    Iter 330. [Val Acc 73%, Loss 0.561976]
    Iter 340. [Val Acc 73%, Loss 0.561059]
    Iter 350. [Val Acc 73%, Loss 0.560430]
    Iter 360. [Val Acc 73%, Loss 0.561136]
    Iter 370. [Val Acc 73%, Loss 0.560419]
    Iter 380. [Val Acc 73%, Loss 0.560722]
    Iter 390. [Val Acc 73%, Loss 0.560222]
    Iter 400. [Val Acc 73%, Loss 0.561431]
    Iter 410. [Val Acc 73%, Loss 0.560787]
    Iter 420. [Val Acc 73%, Loss 0.560813]
    Iter 430. [Val Acc 73%, Loss 0.563041]
    Iter 440. [Val Acc 73%, Loss 0.562902]
    Iter 450. [Val Acc 73%, Loss 0.562116]
    Iter 460. [Val Acc 73%, Loss 0.562575]
    Iter 470. [Val Acc 73%, Loss 0.562176]
    Iter 480. [Val Acc 73%, Loss 0.561807]
    Iter 490. [Val Acc 73%, Loss 0.561357]
    Iter 500. [Val Acc 73%, Loss 0.561262]
    Iter 510. [Val Acc 73%, Loss 0.564092]
    Iter 520. [Val Acc 73%, Loss 0.561313]
    Iter 530. [Val Acc 73%, Loss 0.560520]
    Iter 540. [Val Acc 73%, Loss 0.561145]
    Iter 550. [Val Acc 73%, Loss 0.563100]
    Iter 560. [Val Acc 73%, Loss 0.562265]
    Iter 570. [Val Acc 73%, Loss 0.562030]
    Iter 580. [Val Acc 73%, Loss 0.559475]
    Iter 590. [Val Acc 73%, Loss 0.560141]
    Iter 600. [Val Acc 73%, Loss 0.561045]
    Iter 610. [Val Acc 73%, Loss 0.561885]
    Iter 620. [Val Acc 73%, Loss 0.560077]
    Iter 630. [Val Acc 74%, Loss 0.559479]
    Iter 640. [Val Acc 74%, Loss 0.559959]
    Iter 650. [Val Acc 73%, Loss 0.560518]
    Iter 660. [Val Acc 73%, Loss 0.561056]
    Iter 670. [Val Acc 73%, Loss 0.559532]
    Iter 680. [Val Acc 74%, Loss 0.559924]
    Iter 690. [Val Acc 73%, Loss 0.561509]
    Iter 700. [Val Acc 73%, Loss 0.560791]
    Iter 710. [Val Acc 73%, Loss 0.560777]
    Iter 720. [Val Acc 74%, Loss 0.559533]
    Iter 730. [Val Acc 74%, Loss 0.559723]
    Iter 740. [Val Acc 74%, Loss 0.560264]
    Iter 750. [Val Acc 74%, Loss 0.560015]
    Iter 760. [Val Acc 73%, Loss 0.562035]
    Iter 770. [Val Acc 73%, Loss 0.561162]
    Iter 780. [Val Acc 73%, Loss 0.561993]
    Iter 790. [Val Acc 73%, Loss 0.562308]
    Iter 800. [Val Acc 74%, Loss 0.560469]
    Iter 810. [Val Acc 73%, Loss 0.560603]
    Iter 820. [Val Acc 73%, Loss 0.562177]
    Iter 830. [Val Acc 73%, Loss 0.559160]
    Iter 840. [Val Acc 73%, Loss 0.560149]
    Iter 850. [Val Acc 74%, Loss 0.560336]
    Iter 860. [Val Acc 73%, Loss 0.560631]
    Iter 870. [Val Acc 73%, Loss 0.560824]
    Iter 880. [Val Acc 73%, Loss 0.561900]
    Iter 890. [Val Acc 74%, Loss 0.560320]
    Iter 900. [Val Acc 73%, Loss 0.561141]
    Iter 910. [Val Acc 73%, Loss 0.561781]
    Iter 920. [Val Acc 73%, Loss 0.562192]
    Iter 930. [Val Acc 73%, Loss 0.561363]
    Iter 940. [Val Acc 74%, Loss 0.560408]
    Iter 950. [Val Acc 73%, Loss 0.560634]
    Iter 960. [Val Acc 73%, Loss 0.561237]
    Iter 970. [Val Acc 73%, Loss 0.560777]
    Iter 980. [Val Acc 73%, Loss 0.560328]
    Iter 990. [Val Acc 73%, Loss 0.561638]
    Iter 1000. [Val Acc 73%, Loss 0.561736]
    -------------------------
    mu= 0.1 batch_size= 1024 :
    Iter 10. [Val Acc 70%, Loss 0.588655]
    Iter 20. [Val Acc 71%, Loss 0.579326]
    Iter 30. [Val Acc 72%, Loss 0.572884]
    Iter 40. [Val Acc 72%, Loss 0.568879]
    Iter 50. [Val Acc 73%, Loss 0.566292]
    Iter 60. [Val Acc 73%, Loss 0.564891]
    Iter 70. [Val Acc 73%, Loss 0.561464]
    Iter 80. [Val Acc 73%, Loss 0.561306]
    Iter 90. [Val Acc 73%, Loss 0.561099]
    Iter 100. [Val Acc 73%, Loss 0.562469]
    Iter 110. [Val Acc 73%, Loss 0.563372]
    Iter 120. [Val Acc 73%, Loss 0.561993]
    Iter 130. [Val Acc 73%, Loss 0.561547]
    Iter 140. [Val Acc 73%, Loss 0.561684]
    Iter 150. [Val Acc 73%, Loss 0.558833]
    Iter 160. [Val Acc 73%, Loss 0.559328]
    Iter 170. [Val Acc 73%, Loss 0.559064]
    Iter 180. [Val Acc 73%, Loss 0.562386]
    Iter 190. [Val Acc 73%, Loss 0.562403]
    Iter 200. [Val Acc 73%, Loss 0.561534]
    Iter 210. [Val Acc 73%, Loss 0.561327]
    Iter 220. [Val Acc 73%, Loss 0.560810]
    Iter 230. [Val Acc 74%, Loss 0.560575]
    Iter 240. [Val Acc 74%, Loss 0.559611]
    Iter 250. [Val Acc 74%, Loss 0.559275]
    Iter 260. [Val Acc 73%, Loss 0.559620]
    Iter 270. [Val Acc 73%, Loss 0.562252]
    Iter 280. [Val Acc 73%, Loss 0.560616]
    Iter 290. [Val Acc 74%, Loss 0.562359]
    Iter 300. [Val Acc 73%, Loss 0.560249]
    Iter 310. [Val Acc 73%, Loss 0.561329]
    Iter 320. [Val Acc 73%, Loss 0.564209]
    Iter 330. [Val Acc 73%, Loss 0.561896]
    Iter 340. [Val Acc 73%, Loss 0.560753]
    Iter 350. [Val Acc 73%, Loss 0.560915]
    Iter 360. [Val Acc 73%, Loss 0.560551]
    Iter 370. [Val Acc 73%, Loss 0.560028]
    Iter 380. [Val Acc 73%, Loss 0.559377]
    Iter 390. [Val Acc 74%, Loss 0.559433]
    Iter 400. [Val Acc 73%, Loss 0.560036]
    Iter 410. [Val Acc 73%, Loss 0.559280]
    Iter 420. [Val Acc 73%, Loss 0.559554]
    Iter 430. [Val Acc 73%, Loss 0.559622]
    Iter 440. [Val Acc 73%, Loss 0.562370]
    Iter 450. [Val Acc 73%, Loss 0.560087]
    Iter 460. [Val Acc 73%, Loss 0.561199]
    Iter 470. [Val Acc 73%, Loss 0.560547]
    Iter 480. [Val Acc 73%, Loss 0.560935]
    Iter 490. [Val Acc 73%, Loss 0.560348]
    Iter 500. [Val Acc 73%, Loss 0.560621]
    Iter 510. [Val Acc 73%, Loss 0.560382]
    Iter 520. [Val Acc 73%, Loss 0.561751]
    Iter 530. [Val Acc 73%, Loss 0.563205]
    Iter 540. [Val Acc 73%, Loss 0.564479]
    Iter 550. [Val Acc 73%, Loss 0.565028]
    Iter 560. [Val Acc 73%, Loss 0.561909]
    Iter 570. [Val Acc 73%, Loss 0.561870]
    Iter 580. [Val Acc 73%, Loss 0.563272]
    Iter 590. [Val Acc 73%, Loss 0.561354]
    Iter 600. [Val Acc 73%, Loss 0.561165]
    Iter 610. [Val Acc 73%, Loss 0.559553]
    Iter 620. [Val Acc 73%, Loss 0.562268]
    Iter 630. [Val Acc 73%, Loss 0.559616]
    Iter 640. [Val Acc 73%, Loss 0.560678]
    Iter 650. [Val Acc 73%, Loss 0.560354]
    Iter 660. [Val Acc 73%, Loss 0.563479]
    Iter 670. [Val Acc 74%, Loss 0.560747]
    Iter 680. [Val Acc 73%, Loss 0.559797]
    Iter 690. [Val Acc 73%, Loss 0.561270]
    Iter 700. [Val Acc 74%, Loss 0.559975]
    Iter 710. [Val Acc 73%, Loss 0.565731]
    Iter 720. [Val Acc 74%, Loss 0.562695]
    Iter 730. [Val Acc 73%, Loss 0.560558]
    Iter 740. [Val Acc 73%, Loss 0.561093]
    Iter 750. [Val Acc 74%, Loss 0.561564]
    Iter 760. [Val Acc 74%, Loss 0.560283]
    Iter 770. [Val Acc 73%, Loss 0.560645]
    Iter 780. [Val Acc 73%, Loss 0.561482]
    Iter 790. [Val Acc 74%, Loss 0.559948]
    Iter 800. [Val Acc 73%, Loss 0.560110]
    Iter 810. [Val Acc 73%, Loss 0.561499]
    Iter 820. [Val Acc 73%, Loss 0.560626]
    Iter 830. [Val Acc 73%, Loss 0.561546]
    Iter 840. [Val Acc 73%, Loss 0.562858]
    Iter 850. [Val Acc 73%, Loss 0.560825]
    Iter 860. [Val Acc 73%, Loss 0.561020]
    Iter 870. [Val Acc 73%, Loss 0.560428]
    Iter 880. [Val Acc 73%, Loss 0.558847]
    Iter 890. [Val Acc 73%, Loss 0.560968]
    Iter 900. [Val Acc 73%, Loss 0.560754]
    Iter 910. [Val Acc 73%, Loss 0.559668]
    Iter 920. [Val Acc 73%, Loss 0.559342]
    Iter 930. [Val Acc 73%, Loss 0.560140]
    Iter 940. [Val Acc 73%, Loss 0.558062]
    Iter 950. [Val Acc 73%, Loss 0.559619]
    Iter 960. [Val Acc 73%, Loss 0.561443]
    Iter 970. [Val Acc 73%, Loss 0.559474]
    Iter 980. [Val Acc 73%, Loss 0.560724]
    Iter 990. [Val Acc 73%, Loss 0.563589]
    Iter 1000. [Val Acc 73%, Loss 0.559996]
    -------------------------
    mu= 0.1 batch_size= 50000 :
    Iter 10. [Val Acc 70%, Loss 0.587093]
    Iter 20. [Val Acc 71%, Loss 0.581235]
    Iter 30. [Val Acc 72%, Loss 0.574132]
    Iter 40. [Val Acc 72%, Loss 0.571098]
    Iter 50. [Val Acc 72%, Loss 0.568353]
    Iter 60. [Val Acc 73%, Loss 0.564279]
    Iter 70. [Val Acc 73%, Loss 0.564496]
    Iter 80. [Val Acc 73%, Loss 0.562809]
    Iter 90. [Val Acc 73%, Loss 0.561855]
    Iter 100. [Val Acc 73%, Loss 0.560910]
    Iter 110. [Val Acc 73%, Loss 0.560603]
    Iter 120. [Val Acc 73%, Loss 0.561356]
    Iter 130. [Val Acc 73%, Loss 0.562828]
    Iter 140. [Val Acc 73%, Loss 0.561949]
    Iter 150. [Val Acc 73%, Loss 0.560387]
    Iter 160. [Val Acc 73%, Loss 0.561294]
    Iter 170. [Val Acc 73%, Loss 0.560468]
    Iter 180. [Val Acc 73%, Loss 0.564345]
    Iter 190. [Val Acc 73%, Loss 0.559498]
    Iter 200. [Val Acc 74%, Loss 0.562014]
    Iter 210. [Val Acc 73%, Loss 0.559234]
    Iter 220. [Val Acc 73%, Loss 0.559281]
    Iter 230. [Val Acc 73%, Loss 0.559418]
    Iter 240. [Val Acc 73%, Loss 0.561295]
    Iter 250. [Val Acc 73%, Loss 0.559458]
    Iter 260. [Val Acc 74%, Loss 0.559566]
    Iter 270. [Val Acc 73%, Loss 0.560627]
    Iter 280. [Val Acc 73%, Loss 0.559881]
    Iter 290. [Val Acc 73%, Loss 0.558768]
    Iter 300. [Val Acc 73%, Loss 0.559939]
    Iter 310. [Val Acc 73%, Loss 0.560570]
    Iter 320. [Val Acc 73%, Loss 0.561866]
    Iter 330. [Val Acc 73%, Loss 0.559524]
    Iter 340. [Val Acc 73%, Loss 0.559664]
    Iter 350. [Val Acc 73%, Loss 0.559450]
    Iter 360. [Val Acc 73%, Loss 0.559615]
    Iter 370. [Val Acc 73%, Loss 0.559145]
    Iter 380. [Val Acc 74%, Loss 0.559157]
    Iter 390. [Val Acc 73%, Loss 0.559673]
    Iter 400. [Val Acc 74%, Loss 0.559804]
    Iter 410. [Val Acc 73%, Loss 0.562760]
    Iter 420. [Val Acc 73%, Loss 0.560370]
    Iter 430. [Val Acc 73%, Loss 0.561171]
    Iter 440. [Val Acc 73%, Loss 0.564657]
    Iter 450. [Val Acc 73%, Loss 0.561261]
    Iter 460. [Val Acc 73%, Loss 0.563374]
    Iter 470. [Val Acc 73%, Loss 0.559754]
    Iter 480. [Val Acc 73%, Loss 0.560187]
    Iter 490. [Val Acc 73%, Loss 0.559167]
    Iter 500. [Val Acc 73%, Loss 0.561825]
    Iter 510. [Val Acc 73%, Loss 0.558672]
    Iter 520. [Val Acc 73%, Loss 0.560738]
    Iter 530. [Val Acc 73%, Loss 0.560045]
    Iter 540. [Val Acc 73%, Loss 0.559471]
    Iter 550. [Val Acc 73%, Loss 0.560768]
    Iter 560. [Val Acc 73%, Loss 0.560978]
    Iter 570. [Val Acc 74%, Loss 0.559859]
    Iter 580. [Val Acc 73%, Loss 0.560666]
    Iter 590. [Val Acc 74%, Loss 0.559373]
    Iter 600. [Val Acc 74%, Loss 0.558546]
    Iter 610. [Val Acc 74%, Loss 0.558270]
    Iter 620. [Val Acc 74%, Loss 0.559078]
    Iter 630. [Val Acc 73%, Loss 0.559734]
    Iter 640. [Val Acc 74%, Loss 0.559508]
    Iter 650. [Val Acc 73%, Loss 0.564200]
    Iter 660. [Val Acc 73%, Loss 0.560632]
    Iter 670. [Val Acc 73%, Loss 0.559314]
    Iter 680. [Val Acc 73%, Loss 0.561126]
    Iter 690. [Val Acc 74%, Loss 0.559224]
    Iter 700. [Val Acc 73%, Loss 0.560274]
    Iter 710. [Val Acc 73%, Loss 0.560439]
    Iter 720. [Val Acc 74%, Loss 0.558450]
    Iter 730. [Val Acc 74%, Loss 0.558533]
    Iter 740. [Val Acc 74%, Loss 0.558783]
    Iter 750. [Val Acc 73%, Loss 0.560155]
    Iter 760. [Val Acc 74%, Loss 0.558911]
    Iter 770. [Val Acc 73%, Loss 0.564054]
    Iter 780. [Val Acc 73%, Loss 0.560115]
    Iter 790. [Val Acc 73%, Loss 0.559701]
    Iter 800. [Val Acc 73%, Loss 0.560570]
    Iter 810. [Val Acc 73%, Loss 0.559442]
    Iter 820. [Val Acc 73%, Loss 0.560083]
    Iter 830. [Val Acc 73%, Loss 0.560481]
    Iter 840. [Val Acc 73%, Loss 0.559980]
    Iter 850. [Val Acc 73%, Loss 0.559379]
    Iter 860. [Val Acc 73%, Loss 0.559846]
    Iter 870. [Val Acc 73%, Loss 0.561364]
    Iter 880. [Val Acc 73%, Loss 0.562525]
    Iter 890. [Val Acc 73%, Loss 0.561186]
    Iter 900. [Val Acc 73%, Loss 0.559891]
    Iter 910. [Val Acc 73%, Loss 0.560325]
    Iter 920. [Val Acc 73%, Loss 0.561209]
    Iter 930. [Val Acc 73%, Loss 0.559837]
    Iter 940. [Val Acc 73%, Loss 0.558987]
    Iter 950. [Val Acc 73%, Loss 0.561734]
    Iter 960. [Val Acc 73%, Loss 0.559366]
    Iter 970. [Val Acc 73%, Loss 0.560490]
    Iter 980. [Val Acc 73%, Loss 0.559810]
    Iter 990. [Val Acc 74%, Loss 0.558847]
    Iter 1000. [Val Acc 74%, Loss 0.558762]
    -------------------------
    mu= 0.5 batch_size= 1 :
    Iter 10. [Val Acc 71%, Loss 0.581826]
    Iter 20. [Val Acc 71%, Loss 0.587028]
    Iter 30. [Val Acc 72%, Loss 0.583999]
    Iter 40. [Val Acc 73%, Loss 0.574338]
    Iter 50. [Val Acc 72%, Loss 0.581830]
    Iter 60. [Val Acc 71%, Loss 0.592591]
    Iter 70. [Val Acc 71%, Loss 0.584839]
    Iter 80. [Val Acc 72%, Loss 0.582638]
    Iter 90. [Val Acc 72%, Loss 0.580157]
    Iter 100. [Val Acc 72%, Loss 0.582450]
    Iter 110. [Val Acc 72%, Loss 0.582991]
    Iter 120. [Val Acc 71%, Loss 0.584508]
    Iter 130. [Val Acc 71%, Loss 0.580782]
    Iter 140. [Val Acc 73%, Loss 0.578765]
    Iter 150. [Val Acc 72%, Loss 0.577924]
    Iter 160. [Val Acc 72%, Loss 0.589741]
    Iter 170. [Val Acc 71%, Loss 0.595559]
    Iter 180. [Val Acc 71%, Loss 0.589188]
    Iter 190. [Val Acc 72%, Loss 0.581944]
    Iter 200. [Val Acc 70%, Loss 0.629927]
    Iter 210. [Val Acc 72%, Loss 0.579404]
    Iter 220. [Val Acc 71%, Loss 0.592928]
    Iter 230. [Val Acc 72%, Loss 0.583686]
    Iter 240. [Val Acc 71%, Loss 0.603126]
    Iter 250. [Val Acc 72%, Loss 0.580994]
    Iter 260. [Val Acc 72%, Loss 0.588131]
    Iter 270. [Val Acc 72%, Loss 0.586901]
    Iter 280. [Val Acc 72%, Loss 0.584389]
    Iter 290. [Val Acc 73%, Loss 0.583983]
    Iter 300. [Val Acc 71%, Loss 0.592173]
    Iter 310. [Val Acc 72%, Loss 0.581581]
    Iter 320. [Val Acc 72%, Loss 0.589940]
    Iter 330. [Val Acc 72%, Loss 0.585270]
    Iter 340. [Val Acc 71%, Loss 0.599005]
    Iter 350. [Val Acc 72%, Loss 0.584067]
    Iter 360. [Val Acc 72%, Loss 0.575166]
    Iter 370. [Val Acc 72%, Loss 0.587395]
    Iter 380. [Val Acc 71%, Loss 0.588687]
    Iter 390. [Val Acc 72%, Loss 0.588800]
    Iter 400. [Val Acc 72%, Loss 0.586720]
    Iter 410. [Val Acc 72%, Loss 0.582128]
    Iter 420. [Val Acc 71%, Loss 0.598927]
    Iter 430. [Val Acc 71%, Loss 0.602086]
    Iter 440. [Val Acc 73%, Loss 0.572028]
    Iter 450. [Val Acc 72%, Loss 0.581195]
    Iter 460. [Val Acc 72%, Loss 0.584306]
    Iter 470. [Val Acc 72%, Loss 0.575389]
    Iter 480. [Val Acc 71%, Loss 0.592165]
    Iter 490. [Val Acc 70%, Loss 0.604073]
    Iter 500. [Val Acc 72%, Loss 0.587972]
    Iter 510. [Val Acc 70%, Loss 0.607682]
    Iter 520. [Val Acc 72%, Loss 0.580047]
    Iter 530. [Val Acc 72%, Loss 0.578526]
    Iter 540. [Val Acc 72%, Loss 0.588844]
    Iter 550. [Val Acc 73%, Loss 0.579851]
    Iter 560. [Val Acc 72%, Loss 0.589152]
    Iter 570. [Val Acc 72%, Loss 0.585632]
    Iter 580. [Val Acc 72%, Loss 0.580526]
    Iter 590. [Val Acc 72%, Loss 0.580519]
    Iter 600. [Val Acc 72%, Loss 0.576636]
    Iter 610. [Val Acc 72%, Loss 0.574260]
    Iter 620. [Val Acc 73%, Loss 0.573735]
    Iter 630. [Val Acc 72%, Loss 0.577194]
    Iter 640. [Val Acc 73%, Loss 0.578603]
    Iter 650. [Val Acc 72%, Loss 0.595883]
    Iter 660. [Val Acc 72%, Loss 0.582291]
    Iter 670. [Val Acc 73%, Loss 0.578250]
    Iter 680. [Val Acc 70%, Loss 0.602881]
    Iter 690. [Val Acc 72%, Loss 0.584581]
    Iter 700. [Val Acc 72%, Loss 0.577787]
    Iter 710. [Val Acc 72%, Loss 0.580681]
    Iter 720. [Val Acc 71%, Loss 0.597347]
    Iter 730. [Val Acc 72%, Loss 0.578597]
    Iter 740. [Val Acc 72%, Loss 0.578396]
    Iter 750. [Val Acc 72%, Loss 0.590653]
    Iter 760. [Val Acc 71%, Loss 0.591098]
    Iter 770. [Val Acc 72%, Loss 0.579618]
    Iter 780. [Val Acc 72%, Loss 0.581651]
    Iter 790. [Val Acc 72%, Loss 0.584673]
    Iter 800. [Val Acc 72%, Loss 0.590660]
    Iter 810. [Val Acc 71%, Loss 0.594152]
    Iter 820. [Val Acc 70%, Loss 0.595766]
    Iter 830. [Val Acc 71%, Loss 0.585496]
    Iter 840. [Val Acc 72%, Loss 0.580777]
    Iter 850. [Val Acc 70%, Loss 0.592646]
    Iter 860. [Val Acc 72%, Loss 0.581417]
    Iter 870. [Val Acc 72%, Loss 0.580703]
    Iter 880. [Val Acc 72%, Loss 0.585556]
    Iter 890. [Val Acc 72%, Loss 0.575730]
    Iter 900. [Val Acc 71%, Loss 0.603648]
    Iter 910. [Val Acc 72%, Loss 0.587088]
    Iter 920. [Val Acc 72%, Loss 0.579448]
    Iter 930. [Val Acc 71%, Loss 0.597108]
    Iter 940. [Val Acc 71%, Loss 0.594602]
    Iter 950. [Val Acc 70%, Loss 0.611177]
    Iter 960. [Val Acc 70%, Loss 0.606051]
    Iter 970. [Val Acc 72%, Loss 0.580752]
    Iter 980. [Val Acc 72%, Loss 0.580853]
    Iter 990. [Val Acc 72%, Loss 0.576678]
    Iter 1000. [Val Acc 72%, Loss 0.585513]
    -------------------------
    mu= 0.5 batch_size= 64 :
    Iter 10. [Val Acc 70%, Loss 0.606173]
    Iter 20. [Val Acc 72%, Loss 0.577952]
    Iter 30. [Val Acc 72%, Loss 0.588444]
    Iter 40. [Val Acc 72%, Loss 0.594655]
    Iter 50. [Val Acc 72%, Loss 0.588764]
    Iter 60. [Val Acc 72%, Loss 0.584706]
    Iter 70. [Val Acc 73%, Loss 0.575156]
    Iter 80. [Val Acc 72%, Loss 0.583978]
    Iter 90. [Val Acc 72%, Loss 0.584858]
    Iter 100. [Val Acc 72%, Loss 0.576421]
    Iter 110. [Val Acc 72%, Loss 0.585372]
    Iter 120. [Val Acc 72%, Loss 0.584066]
    Iter 130. [Val Acc 72%, Loss 0.587507]
    Iter 140. [Val Acc 71%, Loss 0.595414]
    Iter 150. [Val Acc 72%, Loss 0.579779]
    Iter 160. [Val Acc 73%, Loss 0.575165]
    Iter 170. [Val Acc 73%, Loss 0.575980]
    Iter 180. [Val Acc 72%, Loss 0.573750]
    Iter 190. [Val Acc 73%, Loss 0.576006]
    Iter 200. [Val Acc 71%, Loss 0.588216]
    Iter 210. [Val Acc 72%, Loss 0.572224]
    Iter 220. [Val Acc 72%, Loss 0.591985]
    Iter 230. [Val Acc 72%, Loss 0.583819]
    Iter 240. [Val Acc 70%, Loss 0.606819]
    Iter 250. [Val Acc 72%, Loss 0.577677]
    Iter 260. [Val Acc 73%, Loss 0.573635]
    Iter 270. [Val Acc 72%, Loss 0.588160]
    Iter 280. [Val Acc 72%, Loss 0.581591]
    Iter 290. [Val Acc 73%, Loss 0.577572]
    Iter 300. [Val Acc 73%, Loss 0.571340]
    Iter 310. [Val Acc 72%, Loss 0.583058]
    Iter 320. [Val Acc 71%, Loss 0.598982]
    Iter 330. [Val Acc 72%, Loss 0.582725]
    Iter 340. [Val Acc 72%, Loss 0.586177]
    Iter 350. [Val Acc 71%, Loss 0.595499]
    Iter 360. [Val Acc 71%, Loss 0.599800]
    Iter 370. [Val Acc 70%, Loss 0.636335]
    Iter 380. [Val Acc 71%, Loss 0.590918]
    Iter 390. [Val Acc 72%, Loss 0.575994]
    Iter 400. [Val Acc 72%, Loss 0.578398]
    Iter 410. [Val Acc 71%, Loss 0.589907]
    Iter 420. [Val Acc 72%, Loss 0.591513]
    Iter 430. [Val Acc 72%, Loss 0.581741]
    Iter 440. [Val Acc 69%, Loss 0.629751]
    Iter 450. [Val Acc 73%, Loss 0.572014]
    Iter 460. [Val Acc 71%, Loss 0.582875]
    Iter 470. [Val Acc 71%, Loss 0.586935]
    Iter 480. [Val Acc 72%, Loss 0.575920]
    Iter 490. [Val Acc 72%, Loss 0.586978]
    Iter 500. [Val Acc 73%, Loss 0.573966]
    Iter 510. [Val Acc 73%, Loss 0.576928]
    Iter 520. [Val Acc 72%, Loss 0.579794]
    Iter 530. [Val Acc 70%, Loss 0.594891]
    Iter 540. [Val Acc 73%, Loss 0.567873]
    Iter 550. [Val Acc 71%, Loss 0.597763]
    Iter 560. [Val Acc 72%, Loss 0.586892]
    Iter 570. [Val Acc 69%, Loss 0.660220]
    Iter 580. [Val Acc 72%, Loss 0.591923]
    Iter 590. [Val Acc 72%, Loss 0.584776]
    Iter 600. [Val Acc 72%, Loss 0.583607]
    Iter 610. [Val Acc 72%, Loss 0.586966]
    Iter 620. [Val Acc 72%, Loss 0.581606]
    Iter 630. [Val Acc 72%, Loss 0.589886]
    Iter 640. [Val Acc 73%, Loss 0.572823]
    Iter 650. [Val Acc 71%, Loss 0.597025]
    Iter 660. [Val Acc 72%, Loss 0.584365]
    Iter 670. [Val Acc 72%, Loss 0.591095]
    Iter 680. [Val Acc 72%, Loss 0.601383]
    Iter 690. [Val Acc 72%, Loss 0.581686]
    Iter 700. [Val Acc 73%, Loss 0.579637]
    Iter 710. [Val Acc 72%, Loss 0.585211]
    Iter 720. [Val Acc 73%, Loss 0.578331]
    Iter 730. [Val Acc 73%, Loss 0.574596]
    Iter 740. [Val Acc 72%, Loss 0.582471]
    Iter 750. [Val Acc 71%, Loss 0.600619]
    Iter 760. [Val Acc 72%, Loss 0.575043]
    Iter 770. [Val Acc 73%, Loss 0.582886]
    Iter 780. [Val Acc 73%, Loss 0.589236]
    Iter 790. [Val Acc 73%, Loss 0.583253]
    Iter 800. [Val Acc 71%, Loss 0.603794]
    Iter 810. [Val Acc 73%, Loss 0.591337]
    Iter 820. [Val Acc 71%, Loss 0.588915]
    Iter 830. [Val Acc 71%, Loss 0.607162]
    Iter 840. [Val Acc 72%, Loss 0.579243]
    Iter 850. [Val Acc 72%, Loss 0.579465]
    Iter 860. [Val Acc 71%, Loss 0.593722]
    Iter 870. [Val Acc 72%, Loss 0.590360]
    Iter 880. [Val Acc 70%, Loss 0.619208]
    Iter 890. [Val Acc 72%, Loss 0.579833]
    Iter 900. [Val Acc 72%, Loss 0.581384]
    Iter 910. [Val Acc 72%, Loss 0.581583]
    Iter 920. [Val Acc 72%, Loss 0.586999]
    Iter 930. [Val Acc 72%, Loss 0.587639]
    Iter 940. [Val Acc 71%, Loss 0.590603]
    Iter 950. [Val Acc 72%, Loss 0.578699]
    Iter 960. [Val Acc 72%, Loss 0.599932]
    Iter 970. [Val Acc 73%, Loss 0.573691]
    Iter 980. [Val Acc 72%, Loss 0.588930]
    Iter 990. [Val Acc 72%, Loss 0.589555]
    Iter 1000. [Val Acc 71%, Loss 0.591204]
    -------------------------
    mu= 0.5 batch_size= 1024 :
    Iter 10. [Val Acc 71%, Loss 0.589210]
    Iter 20. [Val Acc 72%, Loss 0.603029]
    Iter 30. [Val Acc 65%, Loss 0.744261]
    Iter 40. [Val Acc 71%, Loss 0.602223]
    Iter 50. [Val Acc 72%, Loss 0.590086]
    Iter 60. [Val Acc 73%, Loss 0.583704]
    Iter 70. [Val Acc 72%, Loss 0.587972]
    Iter 80. [Val Acc 73%, Loss 0.577430]
    Iter 90. [Val Acc 73%, Loss 0.569298]
    Iter 100. [Val Acc 71%, Loss 0.598593]
    Iter 110. [Val Acc 73%, Loss 0.581486]
    Iter 120. [Val Acc 72%, Loss 0.599007]
    Iter 130. [Val Acc 72%, Loss 0.575619]
    Iter 140. [Val Acc 71%, Loss 0.591104]
    Iter 150. [Val Acc 72%, Loss 0.580432]
    Iter 160. [Val Acc 72%, Loss 0.580477]
    Iter 170. [Val Acc 72%, Loss 0.581285]
    Iter 180. [Val Acc 72%, Loss 0.591494]
    Iter 190. [Val Acc 71%, Loss 0.582207]
    Iter 200. [Val Acc 72%, Loss 0.586299]
    Iter 210. [Val Acc 72%, Loss 0.599889]
    Iter 220. [Val Acc 72%, Loss 0.580448]
    Iter 230. [Val Acc 72%, Loss 0.580767]
    Iter 240. [Val Acc 73%, Loss 0.577602]
    Iter 250. [Val Acc 71%, Loss 0.600360]
    Iter 260. [Val Acc 72%, Loss 0.590913]
    Iter 270. [Val Acc 72%, Loss 0.589140]
    Iter 280. [Val Acc 71%, Loss 0.593471]
    Iter 290. [Val Acc 72%, Loss 0.578113]
    Iter 300. [Val Acc 72%, Loss 0.572821]
    Iter 310. [Val Acc 72%, Loss 0.591579]
    Iter 320. [Val Acc 73%, Loss 0.582045]
    Iter 330. [Val Acc 73%, Loss 0.576280]
    Iter 340. [Val Acc 72%, Loss 0.579601]
    Iter 350. [Val Acc 71%, Loss 0.605440]
    Iter 360. [Val Acc 72%, Loss 0.584608]
    Iter 370. [Val Acc 72%, Loss 0.582736]
    Iter 380. [Val Acc 72%, Loss 0.575237]
    Iter 390. [Val Acc 72%, Loss 0.575676]
    Iter 400. [Val Acc 71%, Loss 0.608766]
    Iter 410. [Val Acc 73%, Loss 0.576581]
    Iter 420. [Val Acc 72%, Loss 0.581078]
    Iter 430. [Val Acc 71%, Loss 0.594714]
    Iter 440. [Val Acc 72%, Loss 0.583900]
    Iter 450. [Val Acc 69%, Loss 0.656565]
    Iter 460. [Val Acc 71%, Loss 0.589886]
    Iter 470. [Val Acc 71%, Loss 0.600099]
    Iter 480. [Val Acc 71%, Loss 0.606505]
    Iter 490. [Val Acc 71%, Loss 0.593084]
    Iter 500. [Val Acc 72%, Loss 0.580861]
    Iter 510. [Val Acc 72%, Loss 0.577896]
    Iter 520. [Val Acc 72%, Loss 0.586358]
    Iter 530. [Val Acc 73%, Loss 0.575788]
    Iter 540. [Val Acc 72%, Loss 0.587966]
    Iter 550. [Val Acc 72%, Loss 0.589470]
    Iter 560. [Val Acc 72%, Loss 0.587236]
    Iter 570. [Val Acc 72%, Loss 0.592914]
    Iter 580. [Val Acc 72%, Loss 0.586646]
    Iter 590. [Val Acc 72%, Loss 0.583243]
    Iter 600. [Val Acc 72%, Loss 0.583611]
    Iter 610. [Val Acc 72%, Loss 0.587218]
    Iter 620. [Val Acc 70%, Loss 0.601483]
    Iter 630. [Val Acc 71%, Loss 0.592276]
    Iter 640. [Val Acc 72%, Loss 0.586130]
    Iter 650. [Val Acc 71%, Loss 0.585050]
    Iter 660. [Val Acc 71%, Loss 0.600727]
    Iter 670. [Val Acc 72%, Loss 0.592090]
    Iter 680. [Val Acc 71%, Loss 0.590011]
    Iter 690. [Val Acc 73%, Loss 0.575244]
    Iter 700. [Val Acc 72%, Loss 0.585503]
    Iter 710. [Val Acc 72%, Loss 0.593004]
    Iter 720. [Val Acc 71%, Loss 0.586811]
    Iter 730. [Val Acc 70%, Loss 0.604519]
    Iter 740. [Val Acc 72%, Loss 0.579787]
    Iter 750. [Val Acc 72%, Loss 0.578156]
    Iter 760. [Val Acc 71%, Loss 0.584004]
    Iter 770. [Val Acc 72%, Loss 0.575973]
    Iter 780. [Val Acc 72%, Loss 0.580794]
    Iter 790. [Val Acc 71%, Loss 0.594087]
    Iter 800. [Val Acc 72%, Loss 0.585583]
    Iter 810. [Val Acc 72%, Loss 0.591155]
    Iter 820. [Val Acc 72%, Loss 0.582567]
    Iter 830. [Val Acc 71%, Loss 0.603483]
    Iter 840. [Val Acc 72%, Loss 0.575608]
    Iter 850. [Val Acc 72%, Loss 0.581330]
    Iter 860. [Val Acc 73%, Loss 0.575886]
    Iter 870. [Val Acc 72%, Loss 0.584772]
    Iter 880. [Val Acc 72%, Loss 0.583165]
    Iter 890. [Val Acc 72%, Loss 0.582396]
    Iter 900. [Val Acc 72%, Loss 0.582213]
    Iter 910. [Val Acc 72%, Loss 0.586524]
    Iter 920. [Val Acc 72%, Loss 0.584792]
    Iter 930. [Val Acc 71%, Loss 0.593124]
    Iter 940. [Val Acc 71%, Loss 0.587475]
    Iter 950. [Val Acc 72%, Loss 0.577246]
    Iter 960. [Val Acc 72%, Loss 0.584198]
    Iter 970. [Val Acc 72%, Loss 0.591193]
    Iter 980. [Val Acc 73%, Loss 0.588141]
    Iter 990. [Val Acc 72%, Loss 0.578465]
    Iter 1000. [Val Acc 71%, Loss 0.600692]
    -------------------------
    mu= 0.5 batch_size= 50000 :
    Iter 10. [Val Acc 72%, Loss 0.592683]
    Iter 20. [Val Acc 72%, Loss 0.579416]
    Iter 30. [Val Acc 72%, Loss 0.583694]
    Iter 40. [Val Acc 73%, Loss 0.578678]
    Iter 50. [Val Acc 73%, Loss 0.581302]
    Iter 60. [Val Acc 72%, Loss 0.579771]
    Iter 70. [Val Acc 71%, Loss 0.593267]
    Iter 80. [Val Acc 72%, Loss 0.578987]
    Iter 90. [Val Acc 72%, Loss 0.589618]
    Iter 100. [Val Acc 71%, Loss 0.601089]
    Iter 110. [Val Acc 69%, Loss 0.639295]
    Iter 120. [Val Acc 72%, Loss 0.587517]
    Iter 130. [Val Acc 72%, Loss 0.583127]
    Iter 140. [Val Acc 72%, Loss 0.580135]
    Iter 150. [Val Acc 73%, Loss 0.573530]
    Iter 160. [Val Acc 71%, Loss 0.588180]
    Iter 170. [Val Acc 72%, Loss 0.585047]
    Iter 180. [Val Acc 72%, Loss 0.581876]
    Iter 190. [Val Acc 72%, Loss 0.583647]
    Iter 200. [Val Acc 72%, Loss 0.583407]
    Iter 210. [Val Acc 72%, Loss 0.586119]
    Iter 220. [Val Acc 72%, Loss 0.593284]
    Iter 230. [Val Acc 73%, Loss 0.572868]
    Iter 240. [Val Acc 72%, Loss 0.589600]
    Iter 250. [Val Acc 72%, Loss 0.583290]
    Iter 260. [Val Acc 72%, Loss 0.587017]
    Iter 270. [Val Acc 73%, Loss 0.577588]
    Iter 280. [Val Acc 72%, Loss 0.591181]
    Iter 290. [Val Acc 73%, Loss 0.576520]
    Iter 300. [Val Acc 72%, Loss 0.583725]
    Iter 310. [Val Acc 73%, Loss 0.573857]
    Iter 320. [Val Acc 72%, Loss 0.587531]
    Iter 330. [Val Acc 73%, Loss 0.586787]
    Iter 340. [Val Acc 70%, Loss 0.620427]
    Iter 350. [Val Acc 70%, Loss 0.611747]
    Iter 360. [Val Acc 73%, Loss 0.576199]
    Iter 370. [Val Acc 73%, Loss 0.579737]
    Iter 380. [Val Acc 73%, Loss 0.577388]
    Iter 390. [Val Acc 73%, Loss 0.572786]
    Iter 400. [Val Acc 71%, Loss 0.596543]
    Iter 410. [Val Acc 72%, Loss 0.577185]
    Iter 420. [Val Acc 71%, Loss 0.598082]
    Iter 430. [Val Acc 73%, Loss 0.575929]
    Iter 440. [Val Acc 72%, Loss 0.578314]
    Iter 450. [Val Acc 72%, Loss 0.589949]
    Iter 460. [Val Acc 72%, Loss 0.596015]
    Iter 470. [Val Acc 72%, Loss 0.589569]
    Iter 480. [Val Acc 72%, Loss 0.587915]
    Iter 490. [Val Acc 71%, Loss 0.597030]
    Iter 500. [Val Acc 72%, Loss 0.579541]
    Iter 510. [Val Acc 72%, Loss 0.584864]
    Iter 520. [Val Acc 72%, Loss 0.582931]
    Iter 530. [Val Acc 72%, Loss 0.590454]
    Iter 540. [Val Acc 71%, Loss 0.585117]
    Iter 550. [Val Acc 72%, Loss 0.592542]
    Iter 560. [Val Acc 72%, Loss 0.587635]
    Iter 570. [Val Acc 71%, Loss 0.603857]
    Iter 580. [Val Acc 72%, Loss 0.584066]
    Iter 590. [Val Acc 73%, Loss 0.578845]
    Iter 600. [Val Acc 72%, Loss 0.594258]
    Iter 610. [Val Acc 72%, Loss 0.590250]
    Iter 620. [Val Acc 72%, Loss 0.592393]
    Iter 630. [Val Acc 72%, Loss 0.582076]
    Iter 640. [Val Acc 71%, Loss 0.588331]
    Iter 650. [Val Acc 71%, Loss 0.601237]
    Iter 660. [Val Acc 71%, Loss 0.601165]
    Iter 670. [Val Acc 71%, Loss 0.593265]
    Iter 680. [Val Acc 71%, Loss 0.610716]
    Iter 690. [Val Acc 72%, Loss 0.588713]
    Iter 700. [Val Acc 73%, Loss 0.575681]
    Iter 710. [Val Acc 72%, Loss 0.579087]
    Iter 720. [Val Acc 73%, Loss 0.575245]
    Iter 730. [Val Acc 72%, Loss 0.584133]
    Iter 740. [Val Acc 72%, Loss 0.580486]
    Iter 750. [Val Acc 73%, Loss 0.576796]
    Iter 760. [Val Acc 71%, Loss 0.589501]
    Iter 770. [Val Acc 72%, Loss 0.579829]
    Iter 780. [Val Acc 72%, Loss 0.585343]
    Iter 790. [Val Acc 72%, Loss 0.580408]
    Iter 800. [Val Acc 73%, Loss 0.571175]
    Iter 810. [Val Acc 72%, Loss 0.580470]
    Iter 820. [Val Acc 72%, Loss 0.583257]
    Iter 830. [Val Acc 72%, Loss 0.578053]
    Iter 840. [Val Acc 73%, Loss 0.576516]
    Iter 850. [Val Acc 72%, Loss 0.572452]
    Iter 860. [Val Acc 72%, Loss 0.577578]
    Iter 870. [Val Acc 72%, Loss 0.582210]
    Iter 880. [Val Acc 72%, Loss 0.592086]
    Iter 890. [Val Acc 73%, Loss 0.576531]
    Iter 900. [Val Acc 73%, Loss 0.574552]
    Iter 910. [Val Acc 72%, Loss 0.585882]
    Iter 920. [Val Acc 72%, Loss 0.583272]
    Iter 930. [Val Acc 72%, Loss 0.575140]
    Iter 940. [Val Acc 72%, Loss 0.585038]
    Iter 950. [Val Acc 73%, Loss 0.580774]
    Iter 960. [Val Acc 71%, Loss 0.593172]
    Iter 970. [Val Acc 71%, Loss 0.590174]
    Iter 980. [Val Acc 72%, Loss 0.588020]
    Iter 990. [Val Acc 72%, Loss 0.573353]
    Iter 1000. [Val Acc 72%, Loss 0.583535]
    -------------------------
    mu= 1 batch_size= 1 :
    Iter 10. [Val Acc 65%, Loss 0.835307]
    Iter 20. [Val Acc 70%, Loss 0.624538]
    Iter 30. [Val Acc 70%, Loss 0.648395]
    Iter 40. [Val Acc 71%, Loss 0.611388]
    Iter 50. [Val Acc 71%, Loss 0.606219]
    Iter 60. [Val Acc 71%, Loss 0.622821]
    Iter 70. [Val Acc 71%, Loss 0.606585]
    Iter 80. [Val Acc 70%, Loss 0.636981]
    Iter 90. [Val Acc 71%, Loss 0.607346]
    Iter 100. [Val Acc 68%, Loss 0.674119]
    Iter 110. [Val Acc 71%, Loss 0.631252]
    Iter 120. [Val Acc 71%, Loss 0.606393]
    Iter 130. [Val Acc 69%, Loss 0.632669]
    Iter 140. [Val Acc 70%, Loss 0.626606]
    Iter 150. [Val Acc 70%, Loss 0.633659]
    Iter 160. [Val Acc 66%, Loss 0.704981]
    Iter 170. [Val Acc 69%, Loss 0.640457]
    Iter 180. [Val Acc 70%, Loss 0.638301]
    Iter 190. [Val Acc 68%, Loss 0.678143]
    Iter 200. [Val Acc 70%, Loss 0.635142]
    Iter 210. [Val Acc 68%, Loss 0.664631]
    Iter 220. [Val Acc 70%, Loss 0.632760]
    Iter 230. [Val Acc 71%, Loss 0.621955]
    Iter 240. [Val Acc 71%, Loss 0.628745]
    Iter 250. [Val Acc 71%, Loss 0.618109]
    Iter 260. [Val Acc 70%, Loss 0.646655]
    Iter 270. [Val Acc 72%, Loss 0.611295]
    Iter 280. [Val Acc 70%, Loss 0.642134]
    Iter 290. [Val Acc 72%, Loss 0.603068]
    Iter 300. [Val Acc 71%, Loss 0.612133]
    Iter 310. [Val Acc 71%, Loss 0.634346]
    Iter 320. [Val Acc 71%, Loss 0.627186]
    Iter 330. [Val Acc 72%, Loss 0.599014]
    Iter 340. [Val Acc 72%, Loss 0.599767]
    Iter 350. [Val Acc 71%, Loss 0.620333]
    Iter 360. [Val Acc 70%, Loss 0.640612]
    Iter 370. [Val Acc 70%, Loss 0.632140]
    Iter 380. [Val Acc 71%, Loss 0.605680]
    Iter 390. [Val Acc 66%, Loss 0.692406]
    Iter 400. [Val Acc 71%, Loss 0.618953]
    Iter 410. [Val Acc 70%, Loss 0.631884]
    Iter 420. [Val Acc 71%, Loss 0.610257]
    Iter 430. [Val Acc 71%, Loss 0.615260]
    Iter 440. [Val Acc 68%, Loss 0.674961]
    Iter 450. [Val Acc 70%, Loss 0.636760]
    Iter 460. [Val Acc 71%, Loss 0.622234]
    Iter 470. [Val Acc 72%, Loss 0.614751]
    Iter 480. [Val Acc 68%, Loss 0.679807]
    Iter 490. [Val Acc 67%, Loss 0.676813]
    Iter 500. [Val Acc 70%, Loss 0.633809]
    Iter 510. [Val Acc 70%, Loss 0.634957]
    Iter 520. [Val Acc 70%, Loss 0.677149]
    Iter 530. [Val Acc 70%, Loss 0.632326]
    Iter 540. [Val Acc 72%, Loss 0.598494]
    Iter 550. [Val Acc 68%, Loss 0.649885]
    Iter 560. [Val Acc 65%, Loss 0.747819]
    Iter 570. [Val Acc 71%, Loss 0.632694]
    Iter 580. [Val Acc 70%, Loss 0.623372]
    Iter 590. [Val Acc 70%, Loss 0.664098]
    Iter 600. [Val Acc 67%, Loss 0.690233]
    Iter 610. [Val Acc 70%, Loss 0.650998]
    Iter 620. [Val Acc 71%, Loss 0.624001]
    Iter 630. [Val Acc 71%, Loss 0.615079]
    Iter 640. [Val Acc 70%, Loss 0.623445]
    Iter 650. [Val Acc 68%, Loss 0.666189]
    Iter 660. [Val Acc 69%, Loss 0.665159]
    Iter 670. [Val Acc 71%, Loss 0.609871]
    Iter 680. [Val Acc 71%, Loss 0.627761]
    Iter 690. [Val Acc 68%, Loss 0.768204]
    Iter 700. [Val Acc 67%, Loss 0.682128]
    Iter 710. [Val Acc 67%, Loss 0.711350]
    Iter 720. [Val Acc 71%, Loss 0.615439]
    Iter 730. [Val Acc 70%, Loss 0.642120]
    Iter 740. [Val Acc 69%, Loss 0.666598]
    Iter 750. [Val Acc 71%, Loss 0.626535]
    Iter 760. [Val Acc 71%, Loss 0.621660]
    Iter 770. [Val Acc 70%, Loss 0.640999]
    Iter 780. [Val Acc 72%, Loss 0.608338]
    Iter 790. [Val Acc 71%, Loss 0.633275]
    Iter 800. [Val Acc 70%, Loss 0.626994]
    Iter 810. [Val Acc 70%, Loss 0.619370]
    Iter 820. [Val Acc 70%, Loss 0.632602]
    Iter 830. [Val Acc 70%, Loss 0.632094]
    Iter 840. [Val Acc 69%, Loss 0.631691]
    Iter 850. [Val Acc 68%, Loss 0.653616]
    Iter 860. [Val Acc 71%, Loss 0.644223]
    Iter 870. [Val Acc 69%, Loss 0.627117]
    Iter 880. [Val Acc 68%, Loss 0.663094]
    Iter 890. [Val Acc 70%, Loss 0.634297]
    Iter 900. [Val Acc 71%, Loss 0.625664]
    Iter 910. [Val Acc 71%, Loss 0.637111]
    Iter 920. [Val Acc 71%, Loss 0.627618]
    Iter 930. [Val Acc 72%, Loss 0.614587]
    Iter 940. [Val Acc 69%, Loss 0.648853]
    Iter 950. [Val Acc 71%, Loss 0.610537]
    Iter 960. [Val Acc 70%, Loss 0.645824]
    Iter 970. [Val Acc 69%, Loss 0.663227]
    Iter 980. [Val Acc 71%, Loss 0.609769]
    Iter 990. [Val Acc 71%, Loss 0.609153]
    Iter 1000. [Val Acc 71%, Loss 0.616064]
    -------------------------
    mu= 1 batch_size= 64 :
    Iter 10. [Val Acc 70%, Loss 0.629198]
    Iter 20. [Val Acc 66%, Loss 0.695369]
    Iter 30. [Val Acc 71%, Loss 0.607919]
    Iter 40. [Val Acc 69%, Loss 0.638659]
    Iter 50. [Val Acc 67%, Loss 0.688053]
    Iter 60. [Val Acc 68%, Loss 0.693892]
    Iter 70. [Val Acc 71%, Loss 0.615990]
    Iter 80. [Val Acc 72%, Loss 0.599219]
    Iter 90. [Val Acc 71%, Loss 0.629675]
    Iter 100. [Val Acc 71%, Loss 0.622717]
    Iter 110. [Val Acc 69%, Loss 0.641184]
    Iter 120. [Val Acc 71%, Loss 0.611580]
    Iter 130. [Val Acc 69%, Loss 0.649739]
    Iter 140. [Val Acc 70%, Loss 0.634307]
    Iter 150. [Val Acc 69%, Loss 0.632484]
    Iter 160. [Val Acc 71%, Loss 0.613134]
    Iter 170. [Val Acc 69%, Loss 0.652090]
    Iter 180. [Val Acc 67%, Loss 0.734856]
    Iter 190. [Val Acc 71%, Loss 0.612898]
    Iter 200. [Val Acc 70%, Loss 0.626283]
    Iter 210. [Val Acc 70%, Loss 0.632082]
    Iter 220. [Val Acc 72%, Loss 0.616310]
    Iter 230. [Val Acc 71%, Loss 0.612020]
    Iter 240. [Val Acc 68%, Loss 0.669219]
    Iter 250. [Val Acc 71%, Loss 0.617777]
    Iter 260. [Val Acc 64%, Loss 0.754732]
    Iter 270. [Val Acc 70%, Loss 0.636857]
    Iter 280. [Val Acc 71%, Loss 0.623842]
    Iter 290. [Val Acc 72%, Loss 0.612712]
    Iter 300. [Val Acc 67%, Loss 0.708748]
    Iter 310. [Val Acc 71%, Loss 0.608174]
    Iter 320. [Val Acc 65%, Loss 0.778001]
    Iter 330. [Val Acc 71%, Loss 0.620782]
    Iter 340. [Val Acc 71%, Loss 0.632840]
    Iter 350. [Val Acc 71%, Loss 0.619756]
    Iter 360. [Val Acc 71%, Loss 0.621645]
    Iter 370. [Val Acc 72%, Loss 0.609539]
    Iter 380. [Val Acc 70%, Loss 0.625794]
    Iter 390. [Val Acc 71%, Loss 0.624004]
    Iter 400. [Val Acc 71%, Loss 0.606671]
    Iter 410. [Val Acc 70%, Loss 0.625599]
    Iter 420. [Val Acc 70%, Loss 0.618382]
    Iter 430. [Val Acc 71%, Loss 0.609106]
    Iter 440. [Val Acc 70%, Loss 0.623340]
    Iter 450. [Val Acc 71%, Loss 0.597850]
    Iter 460. [Val Acc 70%, Loss 0.640124]
    Iter 470. [Val Acc 70%, Loss 0.632079]
    Iter 480. [Val Acc 70%, Loss 0.630415]
    Iter 490. [Val Acc 70%, Loss 0.635406]
    Iter 500. [Val Acc 69%, Loss 0.649818]
    Iter 510. [Val Acc 71%, Loss 0.609786]
    Iter 520. [Val Acc 71%, Loss 0.612525]
    Iter 530. [Val Acc 70%, Loss 0.637019]
    Iter 540. [Val Acc 71%, Loss 0.611348]
    Iter 550. [Val Acc 70%, Loss 0.609801]
    Iter 560. [Val Acc 66%, Loss 0.740150]
    Iter 570. [Val Acc 71%, Loss 0.618078]
    Iter 580. [Val Acc 71%, Loss 0.629268]
    Iter 590. [Val Acc 71%, Loss 0.600015]
    Iter 600. [Val Acc 69%, Loss 0.683220]
    Iter 610. [Val Acc 68%, Loss 0.681337]
    Iter 620. [Val Acc 70%, Loss 0.619363]
    Iter 630. [Val Acc 70%, Loss 0.626569]
    Iter 640. [Val Acc 70%, Loss 0.627416]
    Iter 650. [Val Acc 70%, Loss 0.630546]
    Iter 660. [Val Acc 66%, Loss 0.734672]
    Iter 670. [Val Acc 66%, Loss 0.792481]
    Iter 680. [Val Acc 71%, Loss 0.614539]
    Iter 690. [Val Acc 68%, Loss 0.660943]
    Iter 700. [Val Acc 71%, Loss 0.625004]
    Iter 710. [Val Acc 71%, Loss 0.633742]
    Iter 720. [Val Acc 71%, Loss 0.611698]
    Iter 730. [Val Acc 70%, Loss 0.623131]
    Iter 740. [Val Acc 71%, Loss 0.616610]
    Iter 750. [Val Acc 70%, Loss 0.624855]
    Iter 760. [Val Acc 66%, Loss 0.702875]
    Iter 770. [Val Acc 68%, Loss 0.676050]
    Iter 780. [Val Acc 71%, Loss 0.624335]
    Iter 790. [Val Acc 70%, Loss 0.641198]
    Iter 800. [Val Acc 71%, Loss 0.619213]
    Iter 810. [Val Acc 67%, Loss 0.690588]
    Iter 820. [Val Acc 72%, Loss 0.600636]
    Iter 830. [Val Acc 72%, Loss 0.604822]
    Iter 840. [Val Acc 70%, Loss 0.630850]
    Iter 850. [Val Acc 70%, Loss 0.642288]
    Iter 860. [Val Acc 68%, Loss 0.695363]
    Iter 870. [Val Acc 69%, Loss 0.651445]
    Iter 880. [Val Acc 71%, Loss 0.618260]
    Iter 890. [Val Acc 67%, Loss 0.701989]
    Iter 900. [Val Acc 72%, Loss 0.610612]
    Iter 910. [Val Acc 72%, Loss 0.616255]
    Iter 920. [Val Acc 70%, Loss 0.635534]
    Iter 930. [Val Acc 70%, Loss 0.633450]
    Iter 940. [Val Acc 68%, Loss 0.706726]
    Iter 950. [Val Acc 70%, Loss 0.636933]
    Iter 960. [Val Acc 62%, Loss 0.797707]
    Iter 970. [Val Acc 72%, Loss 0.625017]
    Iter 980. [Val Acc 70%, Loss 0.638366]
    Iter 990. [Val Acc 70%, Loss 0.629172]
    Iter 1000. [Val Acc 71%, Loss 0.604121]
    -------------------------
    mu= 1 batch_size= 1024 :
    Iter 10. [Val Acc 68%, Loss 0.649252]
    Iter 20. [Val Acc 71%, Loss 0.611786]
    Iter 30. [Val Acc 71%, Loss 0.610930]
    Iter 40. [Val Acc 70%, Loss 0.627276]
    Iter 50. [Val Acc 65%, Loss 0.758272]
    Iter 60. [Val Acc 71%, Loss 0.612275]
    Iter 70. [Val Acc 70%, Loss 0.626646]
    Iter 80. [Val Acc 70%, Loss 0.612988]
    Iter 90. [Val Acc 71%, Loss 0.604877]
    Iter 100. [Val Acc 69%, Loss 0.645123]
    Iter 110. [Val Acc 70%, Loss 0.613425]
    Iter 120. [Val Acc 71%, Loss 0.613258]
    Iter 130. [Val Acc 70%, Loss 0.612942]
    Iter 140. [Val Acc 67%, Loss 0.662790]
    Iter 150. [Val Acc 68%, Loss 0.672127]
    Iter 160. [Val Acc 70%, Loss 0.629385]
    Iter 170. [Val Acc 71%, Loss 0.606426]
    Iter 180. [Val Acc 70%, Loss 0.630142]
    Iter 190. [Val Acc 71%, Loss 0.621830]
    Iter 200. [Val Acc 71%, Loss 0.617339]
    Iter 210. [Val Acc 69%, Loss 0.643586]
    Iter 220. [Val Acc 72%, Loss 0.598228]
    Iter 230. [Val Acc 69%, Loss 0.648760]
    Iter 240. [Val Acc 72%, Loss 0.609973]
    Iter 250. [Val Acc 68%, Loss 0.663813]
    Iter 260. [Val Acc 71%, Loss 0.627936]
    Iter 270. [Val Acc 71%, Loss 0.603384]
    Iter 280. [Val Acc 70%, Loss 0.624479]
    Iter 290. [Val Acc 69%, Loss 0.659565]
    Iter 300. [Val Acc 70%, Loss 0.623267]
    Iter 310. [Val Acc 69%, Loss 0.682700]
    Iter 320. [Val Acc 71%, Loss 0.612237]
    Iter 330. [Val Acc 63%, Loss 0.782602]
    Iter 340. [Val Acc 70%, Loss 0.622531]
    Iter 350. [Val Acc 66%, Loss 0.769405]
    Iter 360. [Val Acc 70%, Loss 0.641148]
    Iter 370. [Val Acc 71%, Loss 0.626583]
    Iter 380. [Val Acc 71%, Loss 0.612345]
    Iter 390. [Val Acc 70%, Loss 0.630485]
    Iter 400. [Val Acc 72%, Loss 0.603227]
    Iter 410. [Val Acc 70%, Loss 0.643332]
    Iter 420. [Val Acc 69%, Loss 0.701056]
    Iter 430. [Val Acc 70%, Loss 0.639313]
    Iter 440. [Val Acc 70%, Loss 0.675662]
    Iter 450. [Val Acc 71%, Loss 0.611969]
    Iter 460. [Val Acc 71%, Loss 0.630756]
    Iter 470. [Val Acc 68%, Loss 0.702735]
    Iter 480. [Val Acc 71%, Loss 0.608782]
    Iter 490. [Val Acc 72%, Loss 0.613070]
    Iter 500. [Val Acc 69%, Loss 0.652675]
    Iter 510. [Val Acc 70%, Loss 0.621619]
    Iter 520. [Val Acc 70%, Loss 0.645772]
    Iter 530. [Val Acc 70%, Loss 0.617525]
    Iter 540. [Val Acc 70%, Loss 0.636999]
    Iter 550. [Val Acc 67%, Loss 0.705442]
    Iter 560. [Val Acc 71%, Loss 0.609372]
    Iter 570. [Val Acc 71%, Loss 0.624787]
    Iter 580. [Val Acc 63%, Loss 0.909281]
    Iter 590. [Val Acc 69%, Loss 0.660922]
    Iter 600. [Val Acc 70%, Loss 0.629545]
    Iter 610. [Val Acc 70%, Loss 0.633079]
    Iter 620. [Val Acc 71%, Loss 0.622469]
    Iter 630. [Val Acc 70%, Loss 0.621392]
    Iter 640. [Val Acc 70%, Loss 0.630855]
    Iter 650. [Val Acc 71%, Loss 0.605529]
    Iter 660. [Val Acc 71%, Loss 0.611042]
    Iter 670. [Val Acc 71%, Loss 0.610900]
    Iter 680. [Val Acc 69%, Loss 0.674227]
    Iter 690. [Val Acc 72%, Loss 0.603755]
    Iter 700. [Val Acc 70%, Loss 0.643773]
    Iter 710. [Val Acc 70%, Loss 0.640813]
    Iter 720. [Val Acc 71%, Loss 0.632316]
    Iter 730. [Val Acc 70%, Loss 0.630254]
    Iter 740. [Val Acc 70%, Loss 0.649163]
    Iter 750. [Val Acc 71%, Loss 0.649936]
    Iter 760. [Val Acc 70%, Loss 0.640333]
    Iter 770. [Val Acc 70%, Loss 0.636234]
    Iter 780. [Val Acc 69%, Loss 0.672229]
    Iter 790. [Val Acc 70%, Loss 0.638844]
    Iter 800. [Val Acc 71%, Loss 0.634575]
    Iter 810. [Val Acc 71%, Loss 0.601655]
    Iter 820. [Val Acc 70%, Loss 0.632649]
    Iter 830. [Val Acc 72%, Loss 0.606974]
    Iter 840. [Val Acc 71%, Loss 0.628278]
    Iter 850. [Val Acc 67%, Loss 0.736416]
    Iter 860. [Val Acc 71%, Loss 0.610310]
    Iter 870. [Val Acc 70%, Loss 0.631057]
    Iter 880. [Val Acc 66%, Loss 0.721716]
    Iter 890. [Val Acc 70%, Loss 0.629276]
    Iter 900. [Val Acc 71%, Loss 0.611361]
    Iter 910. [Val Acc 71%, Loss 0.623801]
    Iter 920. [Val Acc 70%, Loss 0.612657]
    Iter 930. [Val Acc 71%, Loss 0.617298]
    Iter 940. [Val Acc 70%, Loss 0.633442]
    Iter 950. [Val Acc 68%, Loss 0.693318]
    Iter 960. [Val Acc 67%, Loss 0.728475]
    Iter 970. [Val Acc 71%, Loss 0.628805]
    Iter 980. [Val Acc 72%, Loss 0.613488]
    Iter 990. [Val Acc 70%, Loss 0.662641]
    Iter 1000. [Val Acc 70%, Loss 0.650241]
    -------------------------
    mu= 1 batch_size= 50000 :
    Iter 10. [Val Acc 71%, Loss 0.626459]
    Iter 20. [Val Acc 70%, Loss 0.663622]
    Iter 30. [Val Acc 70%, Loss 0.630805]
    Iter 40. [Val Acc 70%, Loss 0.624077]
    Iter 50. [Val Acc 70%, Loss 0.627647]
    Iter 60. [Val Acc 71%, Loss 0.636453]
    Iter 70. [Val Acc 71%, Loss 0.615262]
    Iter 80. [Val Acc 70%, Loss 0.622871]
    Iter 90. [Val Acc 69%, Loss 0.647021]
    Iter 100. [Val Acc 63%, Loss 0.914931]
    Iter 110. [Val Acc 67%, Loss 0.679086]
    Iter 120. [Val Acc 69%, Loss 0.648313]
    Iter 130. [Val Acc 69%, Loss 0.663981]
    Iter 140. [Val Acc 69%, Loss 0.696844]
    Iter 150. [Val Acc 71%, Loss 0.624825]
    Iter 160. [Val Acc 71%, Loss 0.617030]
    Iter 170. [Val Acc 70%, Loss 0.633178]
    Iter 180. [Val Acc 69%, Loss 0.628524]
    Iter 190. [Val Acc 70%, Loss 0.649084]
    Iter 200. [Val Acc 70%, Loss 0.623422]
    Iter 210. [Val Acc 71%, Loss 0.609200]
    Iter 220. [Val Acc 70%, Loss 0.627624]
    Iter 230. [Val Acc 70%, Loss 0.664023]
    Iter 240. [Val Acc 71%, Loss 0.620402]
    Iter 250. [Val Acc 71%, Loss 0.629401]
    Iter 260. [Val Acc 70%, Loss 0.624368]
    Iter 270. [Val Acc 71%, Loss 0.628890]
    Iter 280. [Val Acc 71%, Loss 0.610450]
    Iter 290. [Val Acc 70%, Loss 0.627236]
    Iter 300. [Val Acc 69%, Loss 0.651140]
    Iter 310. [Val Acc 68%, Loss 0.664734]
    Iter 320. [Val Acc 70%, Loss 0.618743]
    Iter 330. [Val Acc 69%, Loss 0.637422]
    Iter 340. [Val Acc 66%, Loss 0.753548]
    Iter 350. [Val Acc 69%, Loss 0.666854]
    Iter 360. [Val Acc 71%, Loss 0.634307]
    Iter 370. [Val Acc 71%, Loss 0.614273]
    Iter 380. [Val Acc 71%, Loss 0.610944]
    Iter 390. [Val Acc 71%, Loss 0.609151]
    Iter 400. [Val Acc 69%, Loss 0.647926]
    Iter 410. [Val Acc 70%, Loss 0.637192]
    Iter 420. [Val Acc 70%, Loss 0.625743]
    Iter 430. [Val Acc 69%, Loss 0.642357]
    Iter 440. [Val Acc 71%, Loss 0.619092]
    Iter 450. [Val Acc 71%, Loss 0.624259]
    Iter 460. [Val Acc 72%, Loss 0.599912]
    Iter 470. [Val Acc 69%, Loss 0.647424]
    Iter 480. [Val Acc 70%, Loss 0.638805]
    Iter 490. [Val Acc 72%, Loss 0.597873]
    Iter 500. [Val Acc 71%, Loss 0.633345]
    Iter 510. [Val Acc 72%, Loss 0.613755]
    Iter 520. [Val Acc 69%, Loss 0.635977]
    Iter 530. [Val Acc 69%, Loss 0.668317]
    Iter 540. [Val Acc 71%, Loss 0.619044]
    Iter 550. [Val Acc 71%, Loss 0.609404]
    Iter 560. [Val Acc 70%, Loss 0.631507]
    Iter 570. [Val Acc 71%, Loss 0.613583]
    Iter 580. [Val Acc 71%, Loss 0.611281]
    Iter 590. [Val Acc 70%, Loss 0.636085]
    Iter 600. [Val Acc 69%, Loss 0.644006]
    Iter 610. [Val Acc 70%, Loss 0.650189]
    Iter 620. [Val Acc 71%, Loss 0.647499]
    Iter 630. [Val Acc 70%, Loss 0.660596]
    Iter 640. [Val Acc 72%, Loss 0.622102]
    Iter 650. [Val Acc 71%, Loss 0.607361]
    Iter 660. [Val Acc 69%, Loss 0.653456]
    Iter 670. [Val Acc 71%, Loss 0.626598]
    Iter 680. [Val Acc 72%, Loss 0.608227]
    Iter 690. [Val Acc 71%, Loss 0.612732]
    Iter 700. [Val Acc 71%, Loss 0.621411]
    Iter 710. [Val Acc 70%, Loss 0.645016]
    Iter 720. [Val Acc 70%, Loss 0.636396]
    Iter 730. [Val Acc 71%, Loss 0.635381]
    Iter 740. [Val Acc 69%, Loss 0.656871]
    Iter 750. [Val Acc 71%, Loss 0.616633]
    Iter 760. [Val Acc 71%, Loss 0.627937]
    Iter 770. [Val Acc 70%, Loss 0.628213]
    Iter 780. [Val Acc 72%, Loss 0.610602]
    Iter 790. [Val Acc 69%, Loss 0.667862]
    Iter 800. [Val Acc 70%, Loss 0.633817]
    Iter 810. [Val Acc 70%, Loss 0.640995]
    Iter 820. [Val Acc 71%, Loss 0.607677]
    Iter 830. [Val Acc 71%, Loss 0.621537]
    Iter 840. [Val Acc 69%, Loss 0.637984]
    Iter 850. [Val Acc 71%, Loss 0.645876]
    Iter 860. [Val Acc 66%, Loss 0.776783]
    Iter 870. [Val Acc 70%, Loss 0.620939]
    Iter 880. [Val Acc 71%, Loss 0.621915]
    Iter 890. [Val Acc 71%, Loss 0.614946]
    Iter 900. [Val Acc 70%, Loss 0.652420]
    Iter 910. [Val Acc 69%, Loss 0.652943]
    Iter 920. [Val Acc 70%, Loss 0.617375]
    Iter 930. [Val Acc 69%, Loss 0.675866]
    Iter 940. [Val Acc 70%, Loss 0.624393]
    Iter 950. [Val Acc 68%, Loss 0.700144]
    Iter 960. [Val Acc 67%, Loss 0.682843]
    Iter 970. [Val Acc 70%, Loss 0.628097]
    Iter 980. [Val Acc 70%, Loss 0.639787]
    Iter 990. [Val Acc 72%, Loss 0.619058]
    Iter 1000. [Val Acc 71%, Loss 0.630849]
    -------------------------
    optimal mu= 0.1 optimal batch_size= 50000

We found the optimal weights and biases by scanning over a range of different learning rates and batch sizes for 1000 iterations each and then picking the weights and biases in which the loss over the validation set was minimal.
The best weights and biases were observed when $\mu=0.1$ and batch_size = 50000 (the size of the training data set). The best results are achieved when the batch size is maximal. The reason is that the number of iterations is the number of update steps. Because this number is constant (1000), we have the same number of updates independently of the batch size. Thus, when the batch size is large, as we learned in the lectures, we get a better approximation of the true gradient, and hence the results are better.
It is important to mention that we should not conclude that the best choice is always picking the largest batch size. The reason is that if we measure the validation loss per epoch (for a fixed number of epochs), the number of update steps with a large batch size will be smaller than the number of updates when the batch size is smaller which may lead to worse performance. Furthermore, a smaller batch size can assist the algorithm in avoiding it from stuck in a local minimum or a saddle point due to the noisier gradient.

```

# Write your code here
mu = 0.1
batch_sizes = [1,50000]
Iterations = 1000
plt.figure()
MC_SIZE = 3
losses_vec = np.zeros((MC_SIZE, len(batch_sizes), int(Iterations/10)))
for mc_indx in range(MC_SIZE):
    w0 = np.random.randn(90)
    b0 = np.random.randn(1)[0]
    print("monte_carlo:", mc_indx)
    for curr_idx, minibatch_size in enumerate(batch_sizes):
      print("mu=",mu, 'batch_size=', minibatch_size, ':')
      w,b,losses=run_gradient_descent(w0,b0,mu, max_iters=Iterations)
      losses_vec[mc_indx][curr_idx] = losses
      print("-------------------------")

mean = np.mean(losses_vec,axis=0)
std = np.std(losses_vec,axis=0)
for curr_idx, minibatch_size in enumerate(batch_sizes):
  plt.semilogy(range(0,Iterations,10),mean[curr_idx], label=f'{minibatch_size}')
  y_neg_error = mean[curr_idx]-std[curr_idx]
  y_pos_error = mean[curr_idx]+std[curr_idx]
  plt.fill_between(range(0,Iterations,10), y_neg_error, y_pos_error, alpha=0.3)
plt.legend(loc='best')
plt.xlabel("Iteration")
plt.ylabel("Loss")
plt.grid()
plt.title("Batch size convergance properties")
```

For $\mu=0.1$ and batch size 1 we can see from this figure that it takes more iterations for the validation loss to converge (compared with batch size 50000) because the estimation of the gradient is based on one sample, which makes this estimation very noisy. Through the training procedure we see that a minibatch with smaller size suffer from higher variance when applying Monte Carlo experiment with random parameters initialization.

### Part (h) -- 15%

Using the values of `w` and `b` from part (g), compute your training accuracy, validation accuracy,
and test accuracy. Are there any differences between those three values? If so, why?

```
train_ys=pred(opt_w,opt_b,train_norm_xs)
test_ys=pred(opt_w,opt_b,test_norm_xs)
val_ys=pred(opt_w,opt_b,val_norm_xs)

train_acc = get_accuracy(train_ys,train_ts)
val_acc = get_accuracy(val_ys,val_ts)
test_acc =  get_accuracy(test_ys,test_ts)

print('train_acc = ', train_acc, ' val_acc = ', val_acc, ' test_acc = ', test_acc)

```

    train_acc =  0.7315905877234328  val_acc =  0.73532  test_acc =  0.7247143133836916

Both train and validation accuracies are better than the test accuracy, we explain this phenomenoa by the fact that we didn't see the test data during the model building process, so we expect to encounter minor degradation in the model performance over the new unseen data.

Moreover, we can see that the gap between the train and test data is not significant and therefore we conclude that the model can extract meaningful information from the training dataset.
Meaning that the model is capable to perform similarly over unseen data, e.g., the model is well generalized.

Regarding the validation accuracy, we have tuned our model hyperparameters(Learning rate and Batch size) such that it achieves the best performance over the validation data, therefore, we shall expect to achieve better results in the validation data than the test data, and this is indeed the result that we've achieved.

### Part (i) -- 15%

Writing a classifier like this is instructive, and helps you understand what happens when
we train a model. However, in practice, we rarely write model building and training code
from scratch. Instead, we typically use one of the well-tested libraries available in a package.

Use `sklearn.linear_model.LogisticRegression` to build a linear classifier, and make predictions about the test set. Start by reading the
[API documentation here](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html).

Compute the training, validation and test accuracy of this model.

```
import sklearn.linear_model

model = sklearn.linear_model.LogisticRegression()
model.fit(train_norm_xs, train_ts.squeeze())
train_ys = model.predict(train_norm_xs)
test_ys = model.predict(test_norm_xs)
val_ys = model.predict(val_norm_xs)
train_acc = get_accuracy(train_ys,train_ts)
val_acc = get_accuracy(val_ys,val_ts)
test_acc =  get_accuracy(test_ys,test_ts)

print('train_acc = ', train_acc, ' val_acc = ', val_acc, ' test_acc = ', test_acc)
```

    train_acc =  0.7323979067715698  val_acc =  0.73642  test_acc =  0.7265736974627155

**This parts helps by checking if the code worked.**
**Check if you get similar results, if not repair your code**