Question 3. Determine the weights of a network with 4 input and 2 output units using a) Perceptron learning rule b) Delta learning rule with
\begin{align}
f(x) = \frac{1}{1+e^{-x}}
\end{align}
for the following input-output pairs

\begin{align}
Input: [1100]^T [1001]^T [0011]^T [0110]^T \\
Output: [11]^T [10]^T [01]^T [00]^T
\end{align}

Give your results for different choices of learning rate parameters.

Answer 3.

```python
import numpy as np
```

For Perceptron learning rule:

```python
class Perceptron(object):
    def __init__(self, input_size, lr=0.1, epochs=1000):
        self.W_a = np.zeros(input_size+1)
        self.W_b = np.zeros(input_size+1)
        self.epochs = epochs
        self.lr = lr

    def activation(self, x):
        return 1 if x>=0 else 0

    def predict(self, x):
        z1 = self.W_a.T.dot(x)
        z2 = self.W_b.T.dot(x)
        a1 = self.activation(z1)
        a2 = self.activation(z2)
        return a1, a2

    def fit(self, X, d):
        for _ in range(self.epochs):
            print("Coming in for epoch {}".format(_))
            for i in range(d.shape[0]):
                x = np.insert(X[i], 0, 1)
                y1, y2 = self.predict(x)
                e1 = d[i][0] - y1
                e2 = d[i][1] - y2
                self.W_a = self.W_a + self.lr * e1 * x
                self.W_b = self.W_b + self.lr * e2 * x
```

```python
X = np.array([
[1, 1, 0, 0],[1, 0, 0, 1],[0, 0, 1, 1],[0, 1, 1, 0]
])
d = np.array([[1, 1],[1, 0],[0, 1],[0, 0]])
perceptron = Perceptron(input_size=4)
perceptron.fit(X, d)
```

    Coming in for epoch 0
    Coming in for epoch 1
    Coming in for epoch 2
    Coming in for epoch 3
    Coming in for epoch 4
    Coming in for epoch 5
    Coming in for epoch 6
    Coming in for epoch 7
    Coming in for epoch 8
    Coming in for epoch 9
    Coming in for epoch 10
    Coming in for epoch 11
    Coming in for epoch 12
    Coming in for epoch 13
    Coming in for epoch 14
    Coming in for epoch 15
    Coming in for epoch 16
    Coming in for epoch 17
    Coming in for epoch 18
    Coming in for epoch 19
    Coming in for epoch 20
    Coming in for epoch 21
    Coming in for epoch 22
    Coming in for epoch 23
    Coming in for epoch 24
    Coming in for epoch 25
    Coming in for epoch 26
    Coming in for epoch 27
    Coming in for epoch 28
    Coming in for epoch 29
    Coming in for epoch 30
    Coming in for epoch 31
    Coming in for epoch 32
    Coming in for epoch 33
    Coming in for epoch 34
    Coming in for epoch 35
    Coming in for epoch 36
    Coming in for epoch 37
    Coming in for epoch 38
    Coming in for epoch 39
    Coming in for epoch 40
    Coming in for epoch 41
    Coming in for epoch 42
    Coming in for epoch 43
    Coming in for epoch 44
    Coming in for epoch 45
    Coming in for epoch 46
    Coming in for epoch 47
    Coming in for epoch 48
    Coming in for epoch 49
    Coming in for epoch 50
    Coming in for epoch 51
    Coming in for epoch 52
    Coming in for epoch 53
    Coming in for epoch 54
    Coming in for epoch 55
    Coming in for epoch 56
    Coming in for epoch 57
    Coming in for epoch 58
    Coming in for epoch 59
    Coming in for epoch 60
    Coming in for epoch 61
    Coming in for epoch 62
    Coming in for epoch 63
    Coming in for epoch 64
    Coming in for epoch 65
    Coming in for epoch 66
    Coming in for epoch 67
    Coming in for epoch 68
    Coming in for epoch 69
    Coming in for epoch 70
    Coming in for epoch 71
    Coming in for epoch 72
    Coming in for epoch 73
    Coming in for epoch 74
    Coming in for epoch 75
    Coming in for epoch 76
    Coming in for epoch 77
    Coming in for epoch 78
    Coming in for epoch 79
    Coming in for epoch 80
    Coming in for epoch 81
    Coming in for epoch 82
    Coming in for epoch 83
    Coming in for epoch 84
    Coming in for epoch 85
    Coming in for epoch 86
    Coming in for epoch 87
    Coming in for epoch 88
    Coming in for epoch 89
    Coming in for epoch 90
    Coming in for epoch 91
    Coming in for epoch 92
    Coming in for epoch 93
    Coming in for epoch 94
    Coming in for epoch 95
    Coming in for epoch 96
    Coming in for epoch 97
    Coming in for epoch 98
    Coming in for epoch 99
    Coming in for epoch 100
    Coming in for epoch 101
    Coming in for epoch 102
    Coming in for epoch 103
    Coming in for epoch 104
    Coming in for epoch 105
    Coming in for epoch 106
    Coming in for epoch 107
    Coming in for epoch 108
    Coming in for epoch 109
    Coming in for epoch 110
    Coming in for epoch 111
    Coming in for epoch 112
    Coming in for epoch 113
    Coming in for epoch 114
    Coming in for epoch 115
    Coming in for epoch 116
    Coming in for epoch 117
    Coming in for epoch 118
    Coming in for epoch 119
    Coming in for epoch 120
    Coming in for epoch 121
    Coming in for epoch 122
    Coming in for epoch 123
    Coming in for epoch 124
    Coming in for epoch 125
    Coming in for epoch 126
    Coming in for epoch 127
    Coming in for epoch 128
    Coming in for epoch 129
    Coming in for epoch 130
    Coming in for epoch 131
    Coming in for epoch 132
    Coming in for epoch 133
    Coming in for epoch 134
    Coming in for epoch 135
    Coming in for epoch 136
    Coming in for epoch 137
    Coming in for epoch 138
    Coming in for epoch 139
    Coming in for epoch 140
    Coming in for epoch 141
    Coming in for epoch 142
    Coming in for epoch 143
    Coming in for epoch 144
    Coming in for epoch 145
    Coming in for epoch 146
    Coming in for epoch 147
    Coming in for epoch 148
    Coming in for epoch 149
    Coming in for epoch 150
    Coming in for epoch 151
    Coming in for epoch 152
    Coming in for epoch 153
    Coming in for epoch 154
    Coming in for epoch 155
    Coming in for epoch 156
    Coming in for epoch 157
    Coming in for epoch 158
    Coming in for epoch 159
    Coming in for epoch 160
    Coming in for epoch 161
    Coming in for epoch 162
    Coming in for epoch 163
    Coming in for epoch 164
    Coming in for epoch 165
    Coming in for epoch 166
    Coming in for epoch 167
    Coming in for epoch 168
    Coming in for epoch 169
    Coming in for epoch 170
    Coming in for epoch 171
    Coming in for epoch 172
    Coming in for epoch 173
    Coming in for epoch 174
    Coming in for epoch 175
    Coming in for epoch 176
    Coming in for epoch 177
    Coming in for epoch 178
    Coming in for epoch 179
    Coming in for epoch 180
    Coming in for epoch 181
    Coming in for epoch 182
    Coming in for epoch 183
    Coming in for epoch 184
    Coming in for epoch 185
    Coming in for epoch 186
    Coming in for epoch 187
    Coming in for epoch 188
    Coming in for epoch 189
    Coming in for epoch 190
    Coming in for epoch 191
    Coming in for epoch 192
    Coming in for epoch 193
    Coming in for epoch 194
    Coming in for epoch 195
    Coming in for epoch 196
    Coming in for epoch 197
    Coming in for epoch 198
    Coming in for epoch 199
    Coming in for epoch 200
    Coming in for epoch 201
    Coming in for epoch 202
    Coming in for epoch 203
    Coming in for epoch 204
    Coming in for epoch 205
    Coming in for epoch 206
    Coming in for epoch 207
    Coming in for epoch 208
    Coming in for epoch 209
    Coming in for epoch 210
    Coming in for epoch 211
    Coming in for epoch 212
    Coming in for epoch 213
    Coming in for epoch 214
    Coming in for epoch 215
    Coming in for epoch 216
    Coming in for epoch 217
    Coming in for epoch 218
    Coming in for epoch 219
    Coming in for epoch 220
    Coming in for epoch 221
    Coming in for epoch 222
    Coming in for epoch 223
    Coming in for epoch 224
    Coming in for epoch 225
    Coming in for epoch 226
    Coming in for epoch 227
    Coming in for epoch 228
    Coming in for epoch 229
    Coming in for epoch 230
    Coming in for epoch 231
    Coming in for epoch 232
    Coming in for epoch 233
    Coming in for epoch 234
    Coming in for epoch 235
    Coming in for epoch 236
    Coming in for epoch 237
    Coming in for epoch 238
    Coming in for epoch 239
    Coming in for epoch 240
    Coming in for epoch 241
    Coming in for epoch 242
    Coming in for epoch 243
    Coming in for epoch 244
    Coming in for epoch 245
    Coming in for epoch 246
    Coming in for epoch 247
    Coming in for epoch 248
    Coming in for epoch 249
    Coming in for epoch 250
    Coming in for epoch 251
    Coming in for epoch 252
    Coming in for epoch 253
    Coming in for epoch 254
    Coming in for epoch 255
    Coming in for epoch 256
    Coming in for epoch 257
    Coming in for epoch 258
    Coming in for epoch 259
    Coming in for epoch 260
    Coming in for epoch 261
    Coming in for epoch 262
    Coming in for epoch 263
    Coming in for epoch 264
    Coming in for epoch 265
    Coming in for epoch 266
    Coming in for epoch 267
    Coming in for epoch 268
    Coming in for epoch 269
    Coming in for epoch 270
    Coming in for epoch 271
    Coming in for epoch 272
    Coming in for epoch 273
    Coming in for epoch 274
    Coming in for epoch 275
    Coming in for epoch 276
    Coming in for epoch 277
    Coming in for epoch 278
    Coming in for epoch 279
    Coming in for epoch 280
    Coming in for epoch 281
    Coming in for epoch 282
    Coming in for epoch 283
    Coming in for epoch 284
    Coming in for epoch 285
    Coming in for epoch 286
    Coming in for epoch 287
    Coming in for epoch 288
    Coming in for epoch 289
    Coming in for epoch 290
    Coming in for epoch 291
    Coming in for epoch 292
    Coming in for epoch 293
    Coming in for epoch 294
    Coming in for epoch 295
    Coming in for epoch 296
    Coming in for epoch 297
    Coming in for epoch 298
    Coming in for epoch 299
    Coming in for epoch 300
    Coming in for epoch 301
    Coming in for epoch 302
    Coming in for epoch 303
    Coming in for epoch 304
    Coming in for epoch 305
    Coming in for epoch 306
    Coming in for epoch 307
    Coming in for epoch 308
    Coming in for epoch 309
    Coming in for epoch 310
    Coming in for epoch 311
    Coming in for epoch 312
    Coming in for epoch 313
    Coming in for epoch 314
    Coming in for epoch 315
    Coming in for epoch 316
    Coming in for epoch 317
    Coming in for epoch 318
    Coming in for epoch 319
    Coming in for epoch 320
    Coming in for epoch 321
    Coming in for epoch 322
    Coming in for epoch 323
    Coming in for epoch 324
    Coming in for epoch 325
    Coming in for epoch 326
    Coming in for epoch 327
    Coming in for epoch 328
    Coming in for epoch 329
    Coming in for epoch 330
    Coming in for epoch 331
    Coming in for epoch 332
    Coming in for epoch 333
    Coming in for epoch 334
    Coming in for epoch 335
    Coming in for epoch 336
    Coming in for epoch 337
    Coming in for epoch 338
    Coming in for epoch 339
    Coming in for epoch 340
    Coming in for epoch 341
    Coming in for epoch 342
    Coming in for epoch 343
    Coming in for epoch 344
    Coming in for epoch 345
    Coming in for epoch 346
    Coming in for epoch 347
    Coming in for epoch 348
    Coming in for epoch 349
    Coming in for epoch 350
    Coming in for epoch 351
    Coming in for epoch 352
    Coming in for epoch 353
    Coming in for epoch 354
    Coming in for epoch 355
    Coming in for epoch 356
    Coming in for epoch 357
    Coming in for epoch 358
    Coming in for epoch 359
    Coming in for epoch 360
    Coming in for epoch 361
    Coming in for epoch 362
    Coming in for epoch 363
    Coming in for epoch 364
    Coming in for epoch 365
    Coming in for epoch 366
    Coming in for epoch 367
    Coming in for epoch 368
    Coming in for epoch 369
    Coming in for epoch 370
    Coming in for epoch 371
    Coming in for epoch 372
    Coming in for epoch 373
    Coming in for epoch 374
    Coming in for epoch 375
    Coming in for epoch 376
    Coming in for epoch 377
    Coming in for epoch 378
    Coming in for epoch 379
    Coming in for epoch 380
    Coming in for epoch 381
    Coming in for epoch 382
    Coming in for epoch 383
    Coming in for epoch 384
    Coming in for epoch 385
    Coming in for epoch 386
    Coming in for epoch 387
    Coming in for epoch 388
    Coming in for epoch 389
    Coming in for epoch 390
    Coming in for epoch 391
    Coming in for epoch 392
    Coming in for epoch 393
    Coming in for epoch 394
    Coming in for epoch 395
    Coming in for epoch 396
    Coming in for epoch 397
    Coming in for epoch 398
    Coming in for epoch 399
    Coming in for epoch 400
    Coming in for epoch 401
    Coming in for epoch 402
    Coming in for epoch 403
    Coming in for epoch 404
    Coming in for epoch 405
    Coming in for epoch 406
    Coming in for epoch 407
    Coming in for epoch 408
    Coming in for epoch 409
    Coming in for epoch 410
    Coming in for epoch 411
    Coming in for epoch 412
    Coming in for epoch 413
    Coming in for epoch 414
    Coming in for epoch 415
    Coming in for epoch 416
    Coming in for epoch 417
    Coming in for epoch 418
    Coming in for epoch 419
    Coming in for epoch 420
    Coming in for epoch 421
    Coming in for epoch 422
    Coming in for epoch 423
    Coming in for epoch 424
    Coming in for epoch 425
    Coming in for epoch 426
    Coming in for epoch 427
    Coming in for epoch 428
    Coming in for epoch 429
    Coming in for epoch 430
    Coming in for epoch 431
    Coming in for epoch 432
    Coming in for epoch 433
    Coming in for epoch 434
    Coming in for epoch 435
    Coming in for epoch 436
    Coming in for epoch 437
    Coming in for epoch 438
    Coming in for epoch 439
    Coming in for epoch 440
    Coming in for epoch 441
    Coming in for epoch 442
    Coming in for epoch 443
    Coming in for epoch 444
    Coming in for epoch 445
    Coming in for epoch 446
    Coming in for epoch 447
    Coming in for epoch 448
    Coming in for epoch 449
    Coming in for epoch 450
    Coming in for epoch 451
    Coming in for epoch 452
    Coming in for epoch 453
    Coming in for epoch 454
    Coming in for epoch 455
    Coming in for epoch 456
    Coming in for epoch 457
    Coming in for epoch 458
    Coming in for epoch 459
    Coming in for epoch 460
    Coming in for epoch 461
    Coming in for epoch 462
    Coming in for epoch 463
    Coming in for epoch 464
    Coming in for epoch 465
    Coming in for epoch 466
    Coming in for epoch 467
    Coming in for epoch 468
    Coming in for epoch 469
    Coming in for epoch 470
    Coming in for epoch 471
    Coming in for epoch 472
    Coming in for epoch 473
    Coming in for epoch 474
    Coming in for epoch 475
    Coming in for epoch 476
    Coming in for epoch 477
    Coming in for epoch 478
    Coming in for epoch 479
    Coming in for epoch 480
    Coming in for epoch 481
    Coming in for epoch 482
    Coming in for epoch 483
    Coming in for epoch 484
    Coming in for epoch 485
    Coming in for epoch 486
    Coming in for epoch 487
    Coming in for epoch 488
    Coming in for epoch 489
    Coming in for epoch 490
    Coming in for epoch 491
    Coming in for epoch 492
    Coming in for epoch 493
    Coming in for epoch 494
    Coming in for epoch 495
    Coming in for epoch 496
    Coming in for epoch 497
    Coming in for epoch 498
    Coming in for epoch 499
    Coming in for epoch 500
    Coming in for epoch 501
    Coming in for epoch 502
    Coming in for epoch 503
    Coming in for epoch 504
    Coming in for epoch 505
    Coming in for epoch 506
    Coming in for epoch 507
    Coming in for epoch 508
    Coming in for epoch 509
    Coming in for epoch 510
    Coming in for epoch 511
    Coming in for epoch 512
    Coming in for epoch 513
    Coming in for epoch 514
    Coming in for epoch 515
    Coming in for epoch 516
    Coming in for epoch 517
    Coming in for epoch 518
    Coming in for epoch 519
    Coming in for epoch 520
    Coming in for epoch 521
    Coming in for epoch 522
    Coming in for epoch 523
    Coming in for epoch 524
    Coming in for epoch 525
    Coming in for epoch 526
    Coming in for epoch 527
    Coming in for epoch 528
    Coming in for epoch 529
    Coming in for epoch 530
    Coming in for epoch 531
    Coming in for epoch 532
    Coming in for epoch 533
    Coming in for epoch 534
    Coming in for epoch 535
    Coming in for epoch 536
    Coming in for epoch 537
    Coming in for epoch 538
    Coming in for epoch 539
    Coming in for epoch 540
    Coming in for epoch 541
    Coming in for epoch 542
    Coming in for epoch 543
    Coming in for epoch 544
    Coming in for epoch 545
    Coming in for epoch 546
    Coming in for epoch 547
    Coming in for epoch 548
    Coming in for epoch 549
    Coming in for epoch 550
    Coming in for epoch 551
    Coming in for epoch 552
    Coming in for epoch 553
    Coming in for epoch 554
    Coming in for epoch 555
    Coming in for epoch 556
    Coming in for epoch 557
    Coming in for epoch 558
    Coming in for epoch 559
    Coming in for epoch 560
    Coming in for epoch 561
    Coming in for epoch 562
    Coming in for epoch 563
    Coming in for epoch 564
    Coming in for epoch 565
    Coming in for epoch 566
    Coming in for epoch 567
    Coming in for epoch 568
    Coming in for epoch 569
    Coming in for epoch 570
    Coming in for epoch 571
    Coming in for epoch 572
    Coming in for epoch 573
    Coming in for epoch 574
    Coming in for epoch 575
    Coming in for epoch 576
    Coming in for epoch 577
    Coming in for epoch 578
    Coming in for epoch 579
    Coming in for epoch 580
    Coming in for epoch 581
    Coming in for epoch 582
    Coming in for epoch 583
    Coming in for epoch 584
    Coming in for epoch 585
    Coming in for epoch 586
    Coming in for epoch 587
    Coming in for epoch 588
    Coming in for epoch 589
    Coming in for epoch 590
    Coming in for epoch 591
    Coming in for epoch 592
    Coming in for epoch 593
    Coming in for epoch 594
    Coming in for epoch 595
    Coming in for epoch 596
    Coming in for epoch 597
    Coming in for epoch 598
    Coming in for epoch 599
    Coming in for epoch 600
    Coming in for epoch 601
    Coming in for epoch 602
    Coming in for epoch 603
    Coming in for epoch 604
    Coming in for epoch 605
    Coming in for epoch 606
    Coming in for epoch 607
    Coming in for epoch 608
    Coming in for epoch 609
    Coming in for epoch 610
    Coming in for epoch 611
    Coming in for epoch 612
    Coming in for epoch 613
    Coming in for epoch 614
    Coming in for epoch 615
    Coming in for epoch 616
    Coming in for epoch 617
    Coming in for epoch 618
    Coming in for epoch 619
    Coming in for epoch 620
    Coming in for epoch 621
    Coming in for epoch 622
    Coming in for epoch 623
    Coming in for epoch 624
    Coming in for epoch 625
    Coming in for epoch 626
    Coming in for epoch 627
    Coming in for epoch 628
    Coming in for epoch 629
    Coming in for epoch 630
    Coming in for epoch 631
    Coming in for epoch 632
    Coming in for epoch 633
    Coming in for epoch 634
    Coming in for epoch 635
    Coming in for epoch 636
    Coming in for epoch 637
    Coming in for epoch 638
    Coming in for epoch 639
    Coming in for epoch 640
    Coming in for epoch 641
    Coming in for epoch 642
    Coming in for epoch 643
    Coming in for epoch 644
    Coming in for epoch 645
    Coming in for epoch 646
    Coming in for epoch 647
    Coming in for epoch 648
    Coming in for epoch 649
    Coming in for epoch 650
    Coming in for epoch 651
    Coming in for epoch 652
    Coming in for epoch 653
    Coming in for epoch 654
    Coming in for epoch 655
    Coming in for epoch 656
    Coming in for epoch 657
    Coming in for epoch 658
    Coming in for epoch 659
    Coming in for epoch 660
    Coming in for epoch 661
    Coming in for epoch 662
    Coming in for epoch 663
    Coming in for epoch 664
    Coming in for epoch 665
    Coming in for epoch 666
    Coming in for epoch 667
    Coming in for epoch 668
    Coming in for epoch 669
    Coming in for epoch 670
    Coming in for epoch 671
    Coming in for epoch 672
    Coming in for epoch 673
    Coming in for epoch 674
    Coming in for epoch 675
    Coming in for epoch 676
    Coming in for epoch 677
    Coming in for epoch 678
    Coming in for epoch 679
    Coming in for epoch 680
    Coming in for epoch 681
    Coming in for epoch 682
    Coming in for epoch 683
    Coming in for epoch 684
    Coming in for epoch 685
    Coming in for epoch 686
    Coming in for epoch 687
    Coming in for epoch 688
    Coming in for epoch 689
    Coming in for epoch 690
    Coming in for epoch 691
    Coming in for epoch 692
    Coming in for epoch 693
    Coming in for epoch 694
    Coming in for epoch 695
    Coming in for epoch 696
    Coming in for epoch 697
    Coming in for epoch 698
    Coming in for epoch 699
    Coming in for epoch 700
    Coming in for epoch 701
    Coming in for epoch 702
    Coming in for epoch 703
    Coming in for epoch 704
    Coming in for epoch 705
    Coming in for epoch 706
    Coming in for epoch 707
    Coming in for epoch 708
    Coming in for epoch 709
    Coming in for epoch 710
    Coming in for epoch 711
    Coming in for epoch 712
    Coming in for epoch 713
    Coming in for epoch 714
    Coming in for epoch 715
    Coming in for epoch 716
    Coming in for epoch 717
    Coming in for epoch 718
    Coming in for epoch 719
    Coming in for epoch 720
    Coming in for epoch 721
    Coming in for epoch 722
    Coming in for epoch 723
    Coming in for epoch 724
    Coming in for epoch 725
    Coming in for epoch 726
    Coming in for epoch 727
    Coming in for epoch 728
    Coming in for epoch 729
    Coming in for epoch 730
    Coming in for epoch 731
    Coming in for epoch 732
    Coming in for epoch 733
    Coming in for epoch 734
    Coming in for epoch 735
    Coming in for epoch 736
    Coming in for epoch 737
    Coming in for epoch 738
    Coming in for epoch 739
    Coming in for epoch 740
    Coming in for epoch 741
    Coming in for epoch 742
    Coming in for epoch 743
    Coming in for epoch 744
    Coming in for epoch 745
    Coming in for epoch 746
    Coming in for epoch 747
    Coming in for epoch 748
    Coming in for epoch 749
    Coming in for epoch 750
    Coming in for epoch 751
    Coming in for epoch 752
    Coming in for epoch 753
    Coming in for epoch 754
    Coming in for epoch 755
    Coming in for epoch 756
    Coming in for epoch 757
    Coming in for epoch 758
    Coming in for epoch 759
    Coming in for epoch 760
    Coming in for epoch 761
    Coming in for epoch 762
    Coming in for epoch 763
    Coming in for epoch 764
    Coming in for epoch 765
    Coming in for epoch 766
    Coming in for epoch 767
    Coming in for epoch 768
    Coming in for epoch 769
    Coming in for epoch 770
    Coming in for epoch 771
    Coming in for epoch 772
    Coming in for epoch 773
    Coming in for epoch 774
    Coming in for epoch 775
    Coming in for epoch 776
    Coming in for epoch 777
    Coming in for epoch 778
    Coming in for epoch 779
    Coming in for epoch 780
    Coming in for epoch 781
    Coming in for epoch 782
    Coming in for epoch 783
    Coming in for epoch 784
    Coming in for epoch 785
    Coming in for epoch 786
    Coming in for epoch 787
    Coming in for epoch 788
    Coming in for epoch 789
    Coming in for epoch 790
    Coming in for epoch 791
    Coming in for epoch 792
    Coming in for epoch 793
    Coming in for epoch 794
    Coming in for epoch 795
    Coming in for epoch 796
    Coming in for epoch 797
    Coming in for epoch 798
    Coming in for epoch 799
    Coming in for epoch 800
    Coming in for epoch 801
    Coming in for epoch 802
    Coming in for epoch 803
    Coming in for epoch 804
    Coming in for epoch 805
    Coming in for epoch 806
    Coming in for epoch 807
    Coming in for epoch 808
    Coming in for epoch 809
    Coming in for epoch 810
    Coming in for epoch 811
    Coming in for epoch 812
    Coming in for epoch 813
    Coming in for epoch 814
    Coming in for epoch 815
    Coming in for epoch 816
    Coming in for epoch 817
    Coming in for epoch 818
    Coming in for epoch 819
    Coming in for epoch 820
    Coming in for epoch 821
    Coming in for epoch 822
    Coming in for epoch 823
    Coming in for epoch 824
    Coming in for epoch 825
    Coming in for epoch 826
    Coming in for epoch 827
    Coming in for epoch 828
    Coming in for epoch 829
    Coming in for epoch 830
    Coming in for epoch 831
    Coming in for epoch 832
    Coming in for epoch 833
    Coming in for epoch 834
    Coming in for epoch 835
    Coming in for epoch 836
    Coming in for epoch 837
    Coming in for epoch 838
    Coming in for epoch 839
    Coming in for epoch 840
    Coming in for epoch 841
    Coming in for epoch 842
    Coming in for epoch 843
    Coming in for epoch 844
    Coming in for epoch 845
    Coming in for epoch 846
    Coming in for epoch 847
    Coming in for epoch 848
    Coming in for epoch 849
    Coming in for epoch 850
    Coming in for epoch 851
    Coming in for epoch 852
    Coming in for epoch 853
    Coming in for epoch 854
    Coming in for epoch 855
    Coming in for epoch 856
    Coming in for epoch 857
    Coming in for epoch 858
    Coming in for epoch 859
    Coming in for epoch 860
    Coming in for epoch 861
    Coming in for epoch 862
    Coming in for epoch 863
    Coming in for epoch 864
    Coming in for epoch 865
    Coming in for epoch 866
    Coming in for epoch 867
    Coming in for epoch 868
    Coming in for epoch 869
    Coming in for epoch 870
    Coming in for epoch 871
    Coming in for epoch 872
    Coming in for epoch 873
    Coming in for epoch 874
    Coming in for epoch 875
    Coming in for epoch 876
    Coming in for epoch 877
    Coming in for epoch 878
    Coming in for epoch 879
    Coming in for epoch 880
    Coming in for epoch 881
    Coming in for epoch 882
    Coming in for epoch 883
    Coming in for epoch 884
    Coming in for epoch 885
    Coming in for epoch 886
    Coming in for epoch 887
    Coming in for epoch 888
    Coming in for epoch 889
    Coming in for epoch 890
    Coming in for epoch 891
    Coming in for epoch 892
    Coming in for epoch 893
    Coming in for epoch 894
    Coming in for epoch 895
    Coming in for epoch 896
    Coming in for epoch 897
    Coming in for epoch 898
    Coming in for epoch 899
    Coming in for epoch 900
    Coming in for epoch 901
    Coming in for epoch 902
    Coming in for epoch 903
    Coming in for epoch 904
    Coming in for epoch 905
    Coming in for epoch 906
    Coming in for epoch 907
    Coming in for epoch 908
    Coming in for epoch 909
    Coming in for epoch 910
    Coming in for epoch 911
    Coming in for epoch 912
    Coming in for epoch 913
    Coming in for epoch 914
    Coming in for epoch 915
    Coming in for epoch 916
    Coming in for epoch 917
    Coming in for epoch 918
    Coming in for epoch 919
    Coming in for epoch 920
    Coming in for epoch 921
    Coming in for epoch 922
    Coming in for epoch 923
    Coming in for epoch 924
    Coming in for epoch 925
    Coming in for epoch 926
    Coming in for epoch 927
    Coming in for epoch 928
    Coming in for epoch 929
    Coming in for epoch 930
    Coming in for epoch 931
    Coming in for epoch 932
    Coming in for epoch 933
    Coming in for epoch 934
    Coming in for epoch 935
    Coming in for epoch 936
    Coming in for epoch 937
    Coming in for epoch 938
    Coming in for epoch 939
    Coming in for epoch 940
    Coming in for epoch 941
    Coming in for epoch 942
    Coming in for epoch 943
    Coming in for epoch 944
    Coming in for epoch 945
    Coming in for epoch 946
    Coming in for epoch 947
    Coming in for epoch 948
    Coming in for epoch 949
    Coming in for epoch 950
    Coming in for epoch 951
    Coming in for epoch 952
    Coming in for epoch 953
    Coming in for epoch 954
    Coming in for epoch 955
    Coming in for epoch 956
    Coming in for epoch 957
    Coming in for epoch 958
    Coming in for epoch 959
    Coming in for epoch 960
    Coming in for epoch 961
    Coming in for epoch 962
    Coming in for epoch 963
    Coming in for epoch 964
    Coming in for epoch 965
    Coming in for epoch 966
    Coming in for epoch 967
    Coming in for epoch 968
    Coming in for epoch 969
    Coming in for epoch 970
    Coming in for epoch 971
    Coming in for epoch 972
    Coming in for epoch 973
    Coming in for epoch 974
    Coming in for epoch 975
    Coming in for epoch 976
    Coming in for epoch 977
    Coming in for epoch 978
    Coming in for epoch 979
    Coming in for epoch 980
    Coming in for epoch 981
    Coming in for epoch 982
    Coming in for epoch 983
    Coming in for epoch 984
    Coming in for epoch 985
    Coming in for epoch 986
    Coming in for epoch 987
    Coming in for epoch 988
    Coming in for epoch 989
    Coming in for epoch 990
    Coming in for epoch 991
    Coming in for epoch 992
    Coming in for epoch 993
    Coming in for epoch 994
    Coming in for epoch 995
    Coming in for epoch 996
    Coming in for epoch 997
    Coming in for epoch 998
    Coming in for epoch 999

```python
print(perceptron.W_a)
print(perceptron.W_b)
```

    [ 0.   0.2  0.  -0.2  0. ]
    [-0.1 -0.1 -0.1  0.   0. ]

|Learning rate (lr)|Weights (A)|Bias (A)|Weights (B)|Bias (B)|
|------------------|-----------|--------|-----------|--------|
|1                 |[2.  0. -2.  0.]|[0.]|[-1. -1.  0.  0.]|[-1.0]|
|0.7               |[1.4  0.  -1.4  0. ]|[0.]|[-0.7 -0.7  0. 0.]|[-0.7]|
|0.5               |[1.  0. -1.  0.]|[0.]|[-0.5 -0.5  0. 0.]|[-0.5]|
|0.3               |[0.6  0.  -0.6  0. ]|[0.]|[-0.3 -0.3  0.   0. ]|[-0.3]|
|0.1               |[0.2  0.  -0.2  0. ]|[0.]|[-0.1 -0.1  0.   0. ]|[-0.1]|

For Delta learning rule:

```python
import math
```

```python
class Delta(object):
    def __init__(self, dinput_size, dlr=0.1, depochs=1000):
        self.dW_a = np.zeros(dinput_size+1)
        self.dW_b = np.zeros(dinput_size+1)
        self.depochs = depochs
        self.dlr = dlr

    def dactivation(self, dx):
        return 1 / (1 + math.exp(-dx))

    def dpredict(self, dx):
        dz1 = self.dW_a.T.dot(dx)
        dz2 = self.dW_b.T.dot(dx)
        da1 = self.dactivation(dz1)
        da2 = self.dactivation(dz2)
        return da1, da2, dz1, dz2

    def dfit(self, dX, dd):
        for _ in range(self.depochs):
            print("Coming in for epoch {}".format(_))
            for i in range(dd.shape[0]):
                dx = np.insert(dX[i], 0, 1)
                dy1, dy2, h1, h2 = self.dpredict(dx)
                de1 = dd[i][0] - dy1
                de2 = dd[i][1] - dy2
                self.dW_a = self.dW_a + self.dlr * de1 * dx * (math.exp(-h1))/((1+math.exp(-h1))*(1+math.exp(-h1)))
                self.dW_b = self.dW_b + self.dlr * de2 * dx * (math.exp(-h2))/((1+math.exp(-h2))*(1+math.exp(-h2)))
```

```python
dX = np.array([
[1, 1, 0, 0],[1, 0, 0, 1],[0, 0, 1, 1],[0, 1, 1, 0]
])
dd = np.array([[1, 1],[1, 0],[0, 1],[0, 0]])
delta = Delta(dinput_size=4)
delta.dfit(dX, dd)
```

    Coming in for epoch 0
    Coming in for epoch 1
    Coming in for epoch 2
    Coming in for epoch 3
    Coming in for epoch 4
    Coming in for epoch 5
    Coming in for epoch 6
    Coming in for epoch 7
    Coming in for epoch 8
    Coming in for epoch 9
    Coming in for epoch 10
    Coming in for epoch 11
    Coming in for epoch 12
    Coming in for epoch 13
    Coming in for epoch 14
    Coming in for epoch 15
    Coming in for epoch 16
    Coming in for epoch 17
    Coming in for epoch 18
    Coming in for epoch 19
    Coming in for epoch 20
    Coming in for epoch 21
    Coming in for epoch 22
    Coming in for epoch 23
    Coming in for epoch 24
    Coming in for epoch 25
    Coming in for epoch 26
    Coming in for epoch 27
    Coming in for epoch 28
    Coming in for epoch 29
    Coming in for epoch 30
    Coming in for epoch 31
    Coming in for epoch 32
    Coming in for epoch 33
    Coming in for epoch 34
    Coming in for epoch 35
    Coming in for epoch 36
    Coming in for epoch 37
    Coming in for epoch 38
    Coming in for epoch 39
    Coming in for epoch 40
    Coming in for epoch 41
    Coming in for epoch 42
    Coming in for epoch 43
    Coming in for epoch 44
    Coming in for epoch 45
    Coming in for epoch 46
    Coming in for epoch 47
    Coming in for epoch 48
    Coming in for epoch 49
    Coming in for epoch 50
    Coming in for epoch 51
    Coming in for epoch 52
    Coming in for epoch 53
    Coming in for epoch 54
    Coming in for epoch 55
    Coming in for epoch 56
    Coming in for epoch 57
    Coming in for epoch 58
    Coming in for epoch 59
    Coming in for epoch 60
    Coming in for epoch 61
    Coming in for epoch 62
    Coming in for epoch 63
    Coming in for epoch 64
    Coming in for epoch 65
    Coming in for epoch 66
    Coming in for epoch 67
    Coming in for epoch 68
    Coming in for epoch 69
    Coming in for epoch 70
    Coming in for epoch 71
    Coming in for epoch 72
    Coming in for epoch 73
    Coming in for epoch 74
    Coming in for epoch 75
    Coming in for epoch 76
    Coming in for epoch 77
    Coming in for epoch 78
    Coming in for epoch 79
    Coming in for epoch 80
    Coming in for epoch 81
    Coming in for epoch 82
    Coming in for epoch 83
    Coming in for epoch 84
    Coming in for epoch 85
    Coming in for epoch 86
    Coming in for epoch 87
    Coming in for epoch 88
    Coming in for epoch 89
    Coming in for epoch 90
    Coming in for epoch 91
    Coming in for epoch 92
    Coming in for epoch 93
    Coming in for epoch 94
    Coming in for epoch 95
    Coming in for epoch 96
    Coming in for epoch 97
    Coming in for epoch 98
    Coming in for epoch 99
    Coming in for epoch 100
    Coming in for epoch 101
    Coming in for epoch 102
    Coming in for epoch 103
    Coming in for epoch 104
    Coming in for epoch 105
    Coming in for epoch 106
    Coming in for epoch 107
    Coming in for epoch 108
    Coming in for epoch 109
    Coming in for epoch 110
    Coming in for epoch 111
    Coming in for epoch 112
    Coming in for epoch 113
    Coming in for epoch 114
    Coming in for epoch 115
    Coming in for epoch 116
    Coming in for epoch 117
    Coming in for epoch 118
    Coming in for epoch 119
    Coming in for epoch 120
    Coming in for epoch 121
    Coming in for epoch 122
    Coming in for epoch 123
    Coming in for epoch 124
    Coming in for epoch 125
    Coming in for epoch 126
    Coming in for epoch 127
    Coming in for epoch 128
    Coming in for epoch 129
    Coming in for epoch 130
    Coming in for epoch 131
    Coming in for epoch 132
    Coming in for epoch 133
    Coming in for epoch 134
    Coming in for epoch 135
    Coming in for epoch 136
    Coming in for epoch 137
    Coming in for epoch 138
    Coming in for epoch 139
    Coming in for epoch 140
    Coming in for epoch 141
    Coming in for epoch 142
    Coming in for epoch 143
    Coming in for epoch 144
    Coming in for epoch 145
    Coming in for epoch 146
    Coming in for epoch 147
    Coming in for epoch 148
    Coming in for epoch 149
    Coming in for epoch 150
    Coming in for epoch 151
    Coming in for epoch 152
    Coming in for epoch 153
    Coming in for epoch 154
    Coming in for epoch 155
    Coming in for epoch 156
    Coming in for epoch 157
    Coming in for epoch 158
    Coming in for epoch 159
    Coming in for epoch 160
    Coming in for epoch 161
    Coming in for epoch 162
    Coming in for epoch 163
    Coming in for epoch 164
    Coming in for epoch 165
    Coming in for epoch 166
    Coming in for epoch 167
    Coming in for epoch 168
    Coming in for epoch 169
    Coming in for epoch 170
    Coming in for epoch 171
    Coming in for epoch 172
    Coming in for epoch 173
    Coming in for epoch 174
    Coming in for epoch 175
    Coming in for epoch 176
    Coming in for epoch 177
    Coming in for epoch 178
    Coming in for epoch 179
    Coming in for epoch 180
    Coming in for epoch 181
    Coming in for epoch 182
    Coming in for epoch 183
    Coming in for epoch 184
    Coming in for epoch 185
    Coming in for epoch 186
    Coming in for epoch 187
    Coming in for epoch 188
    Coming in for epoch 189
    Coming in for epoch 190
    Coming in for epoch 191
    Coming in for epoch 192
    Coming in for epoch 193
    Coming in for epoch 194
    Coming in for epoch 195
    Coming in for epoch 196
    Coming in for epoch 197
    Coming in for epoch 198
    Coming in for epoch 199
    Coming in for epoch 200
    Coming in for epoch 201
    Coming in for epoch 202
    Coming in for epoch 203
    Coming in for epoch 204
    Coming in for epoch 205
    Coming in for epoch 206
    Coming in for epoch 207
    Coming in for epoch 208
    Coming in for epoch 209
    Coming in for epoch 210
    Coming in for epoch 211
    Coming in for epoch 212
    Coming in for epoch 213
    Coming in for epoch 214
    Coming in for epoch 215
    Coming in for epoch 216
    Coming in for epoch 217
    Coming in for epoch 218
    Coming in for epoch 219
    Coming in for epoch 220
    Coming in for epoch 221
    Coming in for epoch 222
    Coming in for epoch 223
    Coming in for epoch 224
    Coming in for epoch 225
    Coming in for epoch 226
    Coming in for epoch 227
    Coming in for epoch 228
    Coming in for epoch 229
    Coming in for epoch 230
    Coming in for epoch 231
    Coming in for epoch 232
    Coming in for epoch 233
    Coming in for epoch 234
    Coming in for epoch 235
    Coming in for epoch 236
    Coming in for epoch 237
    Coming in for epoch 238
    Coming in for epoch 239
    Coming in for epoch 240
    Coming in for epoch 241
    Coming in for epoch 242
    Coming in for epoch 243
    Coming in for epoch 244
    Coming in for epoch 245
    Coming in for epoch 246
    Coming in for epoch 247
    Coming in for epoch 248
    Coming in for epoch 249
    Coming in for epoch 250
    Coming in for epoch 251
    Coming in for epoch 252
    Coming in for epoch 253
    Coming in for epoch 254
    Coming in for epoch 255
    Coming in for epoch 256
    Coming in for epoch 257
    Coming in for epoch 258
    Coming in for epoch 259
    Coming in for epoch 260
    Coming in for epoch 261
    Coming in for epoch 262
    Coming in for epoch 263
    Coming in for epoch 264
    Coming in for epoch 265
    Coming in for epoch 266
    Coming in for epoch 267
    Coming in for epoch 268
    Coming in for epoch 269
    Coming in for epoch 270
    Coming in for epoch 271
    Coming in for epoch 272
    Coming in for epoch 273
    Coming in for epoch 274
    Coming in for epoch 275
    Coming in for epoch 276
    Coming in for epoch 277
    Coming in for epoch 278
    Coming in for epoch 279
    Coming in for epoch 280
    Coming in for epoch 281
    Coming in for epoch 282
    Coming in for epoch 283
    Coming in for epoch 284
    Coming in for epoch 285
    Coming in for epoch 286
    Coming in for epoch 287
    Coming in for epoch 288
    Coming in for epoch 289
    Coming in for epoch 290
    Coming in for epoch 291
    Coming in for epoch 292
    Coming in for epoch 293
    Coming in for epoch 294
    Coming in for epoch 295
    Coming in for epoch 296
    Coming in for epoch 297
    Coming in for epoch 298
    Coming in for epoch 299
    Coming in for epoch 300
    Coming in for epoch 301
    Coming in for epoch 302
    Coming in for epoch 303
    Coming in for epoch 304
    Coming in for epoch 305
    Coming in for epoch 306
    Coming in for epoch 307
    Coming in for epoch 308
    Coming in for epoch 309
    Coming in for epoch 310
    Coming in for epoch 311
    Coming in for epoch 312
    Coming in for epoch 313
    Coming in for epoch 314
    Coming in for epoch 315
    Coming in for epoch 316
    Coming in for epoch 317
    Coming in for epoch 318
    Coming in for epoch 319
    Coming in for epoch 320
    Coming in for epoch 321
    Coming in for epoch 322
    Coming in for epoch 323
    Coming in for epoch 324
    Coming in for epoch 325
    Coming in for epoch 326
    Coming in for epoch 327
    Coming in for epoch 328
    Coming in for epoch 329
    Coming in for epoch 330
    Coming in for epoch 331
    Coming in for epoch 332
    Coming in for epoch 333
    Coming in for epoch 334
    Coming in for epoch 335
    Coming in for epoch 336
    Coming in for epoch 337
    Coming in for epoch 338
    Coming in for epoch 339
    Coming in for epoch 340
    Coming in for epoch 341
    Coming in for epoch 342
    Coming in for epoch 343
    Coming in for epoch 344
    Coming in for epoch 345
    Coming in for epoch 346
    Coming in for epoch 347
    Coming in for epoch 348
    Coming in for epoch 349
    Coming in for epoch 350
    Coming in for epoch 351
    Coming in for epoch 352
    Coming in for epoch 353
    Coming in for epoch 354
    Coming in for epoch 355
    Coming in for epoch 356
    Coming in for epoch 357
    Coming in for epoch 358
    Coming in for epoch 359
    Coming in for epoch 360
    Coming in for epoch 361
    Coming in for epoch 362
    Coming in for epoch 363
    Coming in for epoch 364
    Coming in for epoch 365
    Coming in for epoch 366
    Coming in for epoch 367
    Coming in for epoch 368
    Coming in for epoch 369
    Coming in for epoch 370
    Coming in for epoch 371
    Coming in for epoch 372
    Coming in for epoch 373
    Coming in for epoch 374
    Coming in for epoch 375
    Coming in for epoch 376
    Coming in for epoch 377
    Coming in for epoch 378
    Coming in for epoch 379
    Coming in for epoch 380
    Coming in for epoch 381
    Coming in for epoch 382
    Coming in for epoch 383
    Coming in for epoch 384
    Coming in for epoch 385
    Coming in for epoch 386
    Coming in for epoch 387
    Coming in for epoch 388
    Coming in for epoch 389
    Coming in for epoch 390
    Coming in for epoch 391
    Coming in for epoch 392
    Coming in for epoch 393
    Coming in for epoch 394
    Coming in for epoch 395
    Coming in for epoch 396
    Coming in for epoch 397
    Coming in for epoch 398
    Coming in for epoch 399
    Coming in for epoch 400
    Coming in for epoch 401
    Coming in for epoch 402
    Coming in for epoch 403
    Coming in for epoch 404
    Coming in for epoch 405
    Coming in for epoch 406
    Coming in for epoch 407
    Coming in for epoch 408
    Coming in for epoch 409
    Coming in for epoch 410
    Coming in for epoch 411
    Coming in for epoch 412
    Coming in for epoch 413
    Coming in for epoch 414
    Coming in for epoch 415
    Coming in for epoch 416
    Coming in for epoch 417
    Coming in for epoch 418
    Coming in for epoch 419
    Coming in for epoch 420
    Coming in for epoch 421
    Coming in for epoch 422
    Coming in for epoch 423
    Coming in for epoch 424
    Coming in for epoch 425
    Coming in for epoch 426
    Coming in for epoch 427
    Coming in for epoch 428
    Coming in for epoch 429
    Coming in for epoch 430
    Coming in for epoch 431
    Coming in for epoch 432
    Coming in for epoch 433
    Coming in for epoch 434
    Coming in for epoch 435
    Coming in for epoch 436
    Coming in for epoch 437
    Coming in for epoch 438
    Coming in for epoch 439
    Coming in for epoch 440
    Coming in for epoch 441
    Coming in for epoch 442
    Coming in for epoch 443
    Coming in for epoch 444
    Coming in for epoch 445
    Coming in for epoch 446
    Coming in for epoch 447
    Coming in for epoch 448
    Coming in for epoch 449
    Coming in for epoch 450
    Coming in for epoch 451
    Coming in for epoch 452
    Coming in for epoch 453
    Coming in for epoch 454
    Coming in for epoch 455
    Coming in for epoch 456
    Coming in for epoch 457
    Coming in for epoch 458
    Coming in for epoch 459
    Coming in for epoch 460
    Coming in for epoch 461
    Coming in for epoch 462
    Coming in for epoch 463
    Coming in for epoch 464
    Coming in for epoch 465
    Coming in for epoch 466
    Coming in for epoch 467
    Coming in for epoch 468
    Coming in for epoch 469
    Coming in for epoch 470
    Coming in for epoch 471
    Coming in for epoch 472
    Coming in for epoch 473
    Coming in for epoch 474
    Coming in for epoch 475
    Coming in for epoch 476
    Coming in for epoch 477
    Coming in for epoch 478
    Coming in for epoch 479
    Coming in for epoch 480
    Coming in for epoch 481
    Coming in for epoch 482
    Coming in for epoch 483
    Coming in for epoch 484
    Coming in for epoch 485
    Coming in for epoch 486
    Coming in for epoch 487
    Coming in for epoch 488
    Coming in for epoch 489
    Coming in for epoch 490
    Coming in for epoch 491
    Coming in for epoch 492
    Coming in for epoch 493
    Coming in for epoch 494
    Coming in for epoch 495
    Coming in for epoch 496
    Coming in for epoch 497
    Coming in for epoch 498
    Coming in for epoch 499
    Coming in for epoch 500
    Coming in for epoch 501
    Coming in for epoch 502
    Coming in for epoch 503
    Coming in for epoch 504
    Coming in for epoch 505
    Coming in for epoch 506
    Coming in for epoch 507
    Coming in for epoch 508
    Coming in for epoch 509
    Coming in for epoch 510
    Coming in for epoch 511
    Coming in for epoch 512
    Coming in for epoch 513
    Coming in for epoch 514
    Coming in for epoch 515
    Coming in for epoch 516
    Coming in for epoch 517
    Coming in for epoch 518
    Coming in for epoch 519
    Coming in for epoch 520
    Coming in for epoch 521
    Coming in for epoch 522
    Coming in for epoch 523
    Coming in for epoch 524
    Coming in for epoch 525
    Coming in for epoch 526
    Coming in for epoch 527
    Coming in for epoch 528
    Coming in for epoch 529
    Coming in for epoch 530
    Coming in for epoch 531
    Coming in for epoch 532
    Coming in for epoch 533
    Coming in for epoch 534
    Coming in for epoch 535
    Coming in for epoch 536
    Coming in for epoch 537
    Coming in for epoch 538
    Coming in for epoch 539
    Coming in for epoch 540
    Coming in for epoch 541
    Coming in for epoch 542
    Coming in for epoch 543
    Coming in for epoch 544
    Coming in for epoch 545
    Coming in for epoch 546
    Coming in for epoch 547
    Coming in for epoch 548
    Coming in for epoch 549
    Coming in for epoch 550
    Coming in for epoch 551
    Coming in for epoch 552
    Coming in for epoch 553
    Coming in for epoch 554
    Coming in for epoch 555
    Coming in for epoch 556
    Coming in for epoch 557
    Coming in for epoch 558
    Coming in for epoch 559
    Coming in for epoch 560
    Coming in for epoch 561
    Coming in for epoch 562
    Coming in for epoch 563
    Coming in for epoch 564
    Coming in for epoch 565
    Coming in for epoch 566
    Coming in for epoch 567
    Coming in for epoch 568
    Coming in for epoch 569
    Coming in for epoch 570
    Coming in for epoch 571
    Coming in for epoch 572
    Coming in for epoch 573
    Coming in for epoch 574
    Coming in for epoch 575
    Coming in for epoch 576
    Coming in for epoch 577
    Coming in for epoch 578
    Coming in for epoch 579
    Coming in for epoch 580
    Coming in for epoch 581
    Coming in for epoch 582
    Coming in for epoch 583
    Coming in for epoch 584
    Coming in for epoch 585
    Coming in for epoch 586
    Coming in for epoch 587
    Coming in for epoch 588
    Coming in for epoch 589
    Coming in for epoch 590
    Coming in for epoch 591
    Coming in for epoch 592
    Coming in for epoch 593
    Coming in for epoch 594
    Coming in for epoch 595
    Coming in for epoch 596
    Coming in for epoch 597
    Coming in for epoch 598
    Coming in for epoch 599
    Coming in for epoch 600
    Coming in for epoch 601
    Coming in for epoch 602
    Coming in for epoch 603
    Coming in for epoch 604
    Coming in for epoch 605
    Coming in for epoch 606
    Coming in for epoch 607
    Coming in for epoch 608
    Coming in for epoch 609
    Coming in for epoch 610
    Coming in for epoch 611
    Coming in for epoch 612
    Coming in for epoch 613
    Coming in for epoch 614
    Coming in for epoch 615
    Coming in for epoch 616
    Coming in for epoch 617
    Coming in for epoch 618
    Coming in for epoch 619
    Coming in for epoch 620
    Coming in for epoch 621
    Coming in for epoch 622
    Coming in for epoch 623
    Coming in for epoch 624
    Coming in for epoch 625
    Coming in for epoch 626
    Coming in for epoch 627
    Coming in for epoch 628
    Coming in for epoch 629
    Coming in for epoch 630
    Coming in for epoch 631
    Coming in for epoch 632
    Coming in for epoch 633
    Coming in for epoch 634
    Coming in for epoch 635
    Coming in for epoch 636
    Coming in for epoch 637
    Coming in for epoch 638
    Coming in for epoch 639
    Coming in for epoch 640
    Coming in for epoch 641
    Coming in for epoch 642
    Coming in for epoch 643
    Coming in for epoch 644
    Coming in for epoch 645
    Coming in for epoch 646
    Coming in for epoch 647
    Coming in for epoch 648
    Coming in for epoch 649
    Coming in for epoch 650
    Coming in for epoch 651
    Coming in for epoch 652
    Coming in for epoch 653
    Coming in for epoch 654
    Coming in for epoch 655
    Coming in for epoch 656
    Coming in for epoch 657
    Coming in for epoch 658
    Coming in for epoch 659
    Coming in for epoch 660
    Coming in for epoch 661
    Coming in for epoch 662
    Coming in for epoch 663
    Coming in for epoch 664
    Coming in for epoch 665
    Coming in for epoch 666
    Coming in for epoch 667
    Coming in for epoch 668
    Coming in for epoch 669
    Coming in for epoch 670
    Coming in for epoch 671
    Coming in for epoch 672
    Coming in for epoch 673
    Coming in for epoch 674
    Coming in for epoch 675
    Coming in for epoch 676
    Coming in for epoch 677
    Coming in for epoch 678
    Coming in for epoch 679
    Coming in for epoch 680
    Coming in for epoch 681
    Coming in for epoch 682
    Coming in for epoch 683
    Coming in for epoch 684
    Coming in for epoch 685
    Coming in for epoch 686
    Coming in for epoch 687
    Coming in for epoch 688
    Coming in for epoch 689
    Coming in for epoch 690
    Coming in for epoch 691
    Coming in for epoch 692
    Coming in for epoch 693
    Coming in for epoch 694
    Coming in for epoch 695
    Coming in for epoch 696
    Coming in for epoch 697
    Coming in for epoch 698
    Coming in for epoch 699
    Coming in for epoch 700
    Coming in for epoch 701
    Coming in for epoch 702
    Coming in for epoch 703
    Coming in for epoch 704
    Coming in for epoch 705
    Coming in for epoch 706
    Coming in for epoch 707
    Coming in for epoch 708
    Coming in for epoch 709
    Coming in for epoch 710
    Coming in for epoch 711
    Coming in for epoch 712
    Coming in for epoch 713
    Coming in for epoch 714
    Coming in for epoch 715
    Coming in for epoch 716
    Coming in for epoch 717
    Coming in for epoch 718
    Coming in for epoch 719
    Coming in for epoch 720
    Coming in for epoch 721
    Coming in for epoch 722
    Coming in for epoch 723
    Coming in for epoch 724
    Coming in for epoch 725
    Coming in for epoch 726
    Coming in for epoch 727
    Coming in for epoch 728
    Coming in for epoch 729
    Coming in for epoch 730
    Coming in for epoch 731
    Coming in for epoch 732
    Coming in for epoch 733
    Coming in for epoch 734
    Coming in for epoch 735
    Coming in for epoch 736
    Coming in for epoch 737
    Coming in for epoch 738
    Coming in for epoch 739
    Coming in for epoch 740
    Coming in for epoch 741
    Coming in for epoch 742
    Coming in for epoch 743
    Coming in for epoch 744
    Coming in for epoch 745
    Coming in for epoch 746
    Coming in for epoch 747
    Coming in for epoch 748
    Coming in for epoch 749
    Coming in for epoch 750
    Coming in for epoch 751
    Coming in for epoch 752
    Coming in for epoch 753
    Coming in for epoch 754
    Coming in for epoch 755
    Coming in for epoch 756
    Coming in for epoch 757
    Coming in for epoch 758
    Coming in for epoch 759
    Coming in for epoch 760
    Coming in for epoch 761
    Coming in for epoch 762
    Coming in for epoch 763
    Coming in for epoch 764
    Coming in for epoch 765
    Coming in for epoch 766
    Coming in for epoch 767
    Coming in for epoch 768
    Coming in for epoch 769
    Coming in for epoch 770
    Coming in for epoch 771
    Coming in for epoch 772
    Coming in for epoch 773
    Coming in for epoch 774
    Coming in for epoch 775
    Coming in for epoch 776
    Coming in for epoch 777
    Coming in for epoch 778
    Coming in for epoch 779
    Coming in for epoch 780
    Coming in for epoch 781
    Coming in for epoch 782
    Coming in for epoch 783
    Coming in for epoch 784
    Coming in for epoch 785
    Coming in for epoch 786
    Coming in for epoch 787
    Coming in for epoch 788
    Coming in for epoch 789
    Coming in for epoch 790
    Coming in for epoch 791
    Coming in for epoch 792
    Coming in for epoch 793
    Coming in for epoch 794
    Coming in for epoch 795
    Coming in for epoch 796
    Coming in for epoch 797
    Coming in for epoch 798
    Coming in for epoch 799
    Coming in for epoch 800
    Coming in for epoch 801
    Coming in for epoch 802
    Coming in for epoch 803
    Coming in for epoch 804
    Coming in for epoch 805
    Coming in for epoch 806
    Coming in for epoch 807
    Coming in for epoch 808
    Coming in for epoch 809
    Coming in for epoch 810
    Coming in for epoch 811
    Coming in for epoch 812
    Coming in for epoch 813
    Coming in for epoch 814
    Coming in for epoch 815
    Coming in for epoch 816
    Coming in for epoch 817
    Coming in for epoch 818
    Coming in for epoch 819
    Coming in for epoch 820
    Coming in for epoch 821
    Coming in for epoch 822
    Coming in for epoch 823
    Coming in for epoch 824
    Coming in for epoch 825
    Coming in for epoch 826
    Coming in for epoch 827
    Coming in for epoch 828
    Coming in for epoch 829
    Coming in for epoch 830
    Coming in for epoch 831
    Coming in for epoch 832
    Coming in for epoch 833
    Coming in for epoch 834
    Coming in for epoch 835
    Coming in for epoch 836
    Coming in for epoch 837
    Coming in for epoch 838
    Coming in for epoch 839
    Coming in for epoch 840
    Coming in for epoch 841
    Coming in for epoch 842
    Coming in for epoch 843
    Coming in for epoch 844
    Coming in for epoch 845
    Coming in for epoch 846
    Coming in for epoch 847
    Coming in for epoch 848
    Coming in for epoch 849
    Coming in for epoch 850
    Coming in for epoch 851
    Coming in for epoch 852
    Coming in for epoch 853
    Coming in for epoch 854
    Coming in for epoch 855
    Coming in for epoch 856
    Coming in for epoch 857
    Coming in for epoch 858
    Coming in for epoch 859
    Coming in for epoch 860
    Coming in for epoch 861
    Coming in for epoch 862
    Coming in for epoch 863
    Coming in for epoch 864
    Coming in for epoch 865
    Coming in for epoch 866
    Coming in for epoch 867
    Coming in for epoch 868
    Coming in for epoch 869
    Coming in for epoch 870
    Coming in for epoch 871
    Coming in for epoch 872
    Coming in for epoch 873
    Coming in for epoch 874
    Coming in for epoch 875
    Coming in for epoch 876
    Coming in for epoch 877
    Coming in for epoch 878
    Coming in for epoch 879
    Coming in for epoch 880
    Coming in for epoch 881
    Coming in for epoch 882
    Coming in for epoch 883
    Coming in for epoch 884
    Coming in for epoch 885
    Coming in for epoch 886
    Coming in for epoch 887
    Coming in for epoch 888
    Coming in for epoch 889
    Coming in for epoch 890
    Coming in for epoch 891
    Coming in for epoch 892
    Coming in for epoch 893
    Coming in for epoch 894
    Coming in for epoch 895
    Coming in for epoch 896
    Coming in for epoch 897
    Coming in for epoch 898
    Coming in for epoch 899
    Coming in for epoch 900
    Coming in for epoch 901
    Coming in for epoch 902
    Coming in for epoch 903
    Coming in for epoch 904
    Coming in for epoch 905
    Coming in for epoch 906
    Coming in for epoch 907
    Coming in for epoch 908
    Coming in for epoch 909
    Coming in for epoch 910
    Coming in for epoch 911
    Coming in for epoch 912
    Coming in for epoch 913
    Coming in for epoch 914
    Coming in for epoch 915
    Coming in for epoch 916
    Coming in for epoch 917
    Coming in for epoch 918
    Coming in for epoch 919
    Coming in for epoch 920
    Coming in for epoch 921
    Coming in for epoch 922
    Coming in for epoch 923
    Coming in for epoch 924
    Coming in for epoch 925
    Coming in for epoch 926
    Coming in for epoch 927
    Coming in for epoch 928
    Coming in for epoch 929
    Coming in for epoch 930
    Coming in for epoch 931
    Coming in for epoch 932
    Coming in for epoch 933
    Coming in for epoch 934
    Coming in for epoch 935
    Coming in for epoch 936
    Coming in for epoch 937
    Coming in for epoch 938
    Coming in for epoch 939
    Coming in for epoch 940
    Coming in for epoch 941
    Coming in for epoch 942
    Coming in for epoch 943
    Coming in for epoch 944
    Coming in for epoch 945
    Coming in for epoch 946
    Coming in for epoch 947
    Coming in for epoch 948
    Coming in for epoch 949
    Coming in for epoch 950
    Coming in for epoch 951
    Coming in for epoch 952
    Coming in for epoch 953
    Coming in for epoch 954
    Coming in for epoch 955
    Coming in for epoch 956
    Coming in for epoch 957
    Coming in for epoch 958
    Coming in for epoch 959
    Coming in for epoch 960
    Coming in for epoch 961
    Coming in for epoch 962
    Coming in for epoch 963
    Coming in for epoch 964
    Coming in for epoch 965
    Coming in for epoch 966
    Coming in for epoch 967
    Coming in for epoch 968
    Coming in for epoch 969
    Coming in for epoch 970
    Coming in for epoch 971
    Coming in for epoch 972
    Coming in for epoch 973
    Coming in for epoch 974
    Coming in for epoch 975
    Coming in for epoch 976
    Coming in for epoch 977
    Coming in for epoch 978
    Coming in for epoch 979
    Coming in for epoch 980
    Coming in for epoch 981
    Coming in for epoch 982
    Coming in for epoch 983
    Coming in for epoch 984
    Coming in for epoch 985
    Coming in for epoch 986
    Coming in for epoch 987
    Coming in for epoch 988
    Coming in for epoch 989
    Coming in for epoch 990
    Coming in for epoch 991
    Coming in for epoch 992
    Coming in for epoch 993
    Coming in for epoch 994
    Coming in for epoch 995
    Coming in for epoch 996
    Coming in for epoch 997
    Coming in for epoch 998
    Coming in for epoch 999

```python
print(delta.dW_a)
print(delta.dW_b)
```

    [-2.93099017e-04  2.83050749e+00  9.56203098e-04 -2.83080059e+00
     -1.24930211e-03]
    [-0.00630858 -0.00315429 -0.00946284 -0.00315429  0.00315426]

|Learning rate (lr)|Weights (A)|Bias (A)|Weights (B)|Bias (B)|
|------------------|-----------|--------|-----------|--------|
|1                 |[4.09940513e+00,  1.50935779e-03, -4.09966701e+00, -1.77123527e-03]|[-2.61877479e-04]|[-0.0340762, -0.1022286, -0.0340762,  0.0340762]|[-0.0681524]|
|0.7|[3.91075692e+00,  1.42039822e-03, -3.91102156e+00, -1.68503484e-03]|[-2.64636618e-04]|[-0.02328688, -0.06986065, -0.02328688, 0.02328688]|[-0.04657377]|
|0.5|[3.73086876e+00, 1.33879709e-03, -3.73113651e+00, -1.60655388e-03]|[-2.67756784e-04]|[-0.01635148, -0.04905445, -0.01635148,  0.01635148]|[-0.03270297]|
|0.3|[3.45331815e+00,  1.21744666e-03, -3.45359183e+00, -1.49113116e-03]|[-2.73684504e-04]|[-0.00963793, -0.0289138,  -0.00963793,  0.00963793]|[-0.01927586]|
|0.1|[2.83050749e+00,  9.56203098e-04, -2.83080059e+00, -1.24930211e-03]]|[-2.93099017e-04]|[-0.00315429, -0.00946284, -0.00315429,  0.00315426]|[-0.00630858]|

```python

```


# Lambdify in symbolic module
### Importing required modules

```python
import sympy
from sympy.abc import x, y
from sympy import symbols
from einsteinpy.symbolic import BaseRelativityTensor

sympy.init_printing()
```

### Calculating a Base Relativity Tensor

```python
syms = symbols("x y")
x, y = syms
T = BaseRelativityTensor([[x, 1],[0, x+y]], syms, config="ll")
```

### Calling the lambdify function

```python
args, func = T.tensor_lambdify()
args
```

`args` indicates the order in which arguments should be passed to the returned function `func`

### Executing the returned function for some value

```python
func(2, 1)
```


# Let's generate some data from a state space model

```julia
using Distributions
using PyPlot
plt.style[:use]("ggplot")
srand(12345)  # set random number seed
```

    WARNING: could not import Base.help into PyCall

    Base.Random.MersenneTwister(Base.dSFMT.DSFMT_state(Int32[-870096391,1072918504,-1812426662,1073255081,-733866021,1073404543,807620846,1073368448,1919433844,1072852359  …  -362113007,1073100625,-166402106,1073460158,-1907020342,721295190,-750225566,-1300227565,382,32650]),[1.85709,1.17005,1.48908,1.2646,1.52351,1.22143,1.65142,1.75928,1.47494,1.78851  …  1.08892,1.25852,1.41644,1.45456,1.91416,1.84281,1.96421,1.23656,1.88541,1.83922],382,UInt32[0x00003039])

We'll generate data according to a simple state space model and then recover the underlying model via the Kalman filter/smoother. We follow the conventions of [Durbin and Koopman (2012)](http://www.ssfpack.com/DKbook.html).

## Observation model

$$\begin{align}
y_t &= Z_t \alpha_t + \varepsilon_t \\
\varepsilon_t &\sim \mathcal{N}(0, H_t)
\end{align}$$

```julia
Z = [1 -1 ; 0 1 ; 1 0]
H = 0.1 * [1 0 0 ; 0 1 0.2 ; 0 0.2 1]

Np, Nm = size(Z);
```

```julia
assert(size(H) == (Np, Np))
```

## State evolution model

$$
\begin{align}
\alpha_{t+1} &= T_t \alpha_t + R_t \eta_t \\
\eta_t &\sim \mathcal{N}(0, Q_t)
\end{align}$$

```julia
th = 10 * pi / 180
T = [cos(th) sin(th) ; -sin(th) cos(th)]
R = [1.1 1]'
Q = [0.05]

_, Nr = size(R);
```

```julia
assert(size(T) == (Nm, Nm))
```

## Prior
$$
\alpha_1 \sim \mathcal{N}(a_1, P_1)
$$

```julia
a_init = zeros(Nm)
P_init = 5 * eye(Nm);
```

```julia
assert(size(P_init) == (Nm, Nm))
```

# Now generate some data

```julia
Nt = 100  # number of samples
```

    100

```julia
# preallocate
α = Array(Float64, Nm, Nt);
y = Array(Float64, Np, Nt);
ϵ = Array(Float64, Np, Nt);
η = Array(Float64, Nr, Nt);
```

Initialize:

```julia
α[:, 1] = rand(MvNormal(a_init, P_init));
```

```julia
ϵrng = MvNormal(H)
ηrng = MvNormal(Q)
for t in 1:Nt
    ϵ[:, t] = rand(ϵrng)
    η[:, t] = rand(ηrng)

    y[:, t] = Z * α[:, t] + ϵ[:, t]

    if t + 1 <= Nt
        α[:, t + 1] = T * α[:, t] + R * η[:, t]
    end
end
```

```julia
plot(y');
```

```julia
plot(α')
```

# Kalman filter (forward)

Now filter! In DK conventions, this is
$$\begin{align}
v_t &= y_t - Z_t a_t  \\
F_t &= Z_t P_t Z_t^\top + H_t  \\
aa \equiv \mathbb{E}(\alpha_t\vert y_{1:t}) &= a_t + P_t Z_t^\top F_t^{-1} v_t  \\
PP \equiv \mathrm{var}(\alpha_t \vert y_{1:t}) &= P_t - P_t Z_t^\top F_t^{-1} Z_t P_t  \\
K_t &= T_t P_t Z_t^\top F_t^{-1}  \\
L_t &= T_t - K_t Z_t  \\
a_{t + 1} &= T_t a_t + K_t v_t  \\
P_{t + 1} &= T_t P_t(T_t - K_t Z_t)^\top + R_t Q_t R_t^\top
\end{align}$$

```julia
#preallocate

a = Array(Float64, Nm, Nt);
P = Array(Float64, Nm, Nm, Nt);
aa = Array(Float64, Nm, Nt);
PP = Array(Float64, Nm, Nm, Nt);
v = Array(Float64, Np, Nt);
F = Array(Float64, Np, Np, Nt);
K = Array(Float64, Nm, Np, Nt);
L = Array(Float64, Nm, Nm, Nt);
```

```julia
# set priors
a[:, 1] = a_init;
P[:, :, 1] = P_init;
```

```julia
# iterate

for t in 1:Nt
    v[:, t] = y[:, t] - Z * a[:, t]
    Ft = Z * P[:, :, t] * Z' + H
    F[:, :, t] = Ft
    Finv = inv(Ft)
    aa[:, t] = a[:, t] + P[:, :, t] * Z' * Finv * v[:, t]
    PP[:, :, t] = P[:, :, t] - P[:, :, t] * Z' * Finv * Z * P[:, :, t]
    K[:, :, t] = T * P[:, :, t] * Z' * Finv
    L[:, :, t] = T - K[:, :, t] * Z

    if t < Nt
        a[:, t + 1] = T * a[:, t] + K[:, :, t] * v[:, t]
        P[:, :, t + 1] = T * P[:, :, t] * L[:, : , t]' + R * Q * R'
    end

end
```

```julia
plot(aa')
plot(α')
title("Actual and recovered states");
```

# Kalman smoother (backward)

First define the posterior of the state vector given all the data as
$$\begin{align}
p(\alpha_t \vert y_{1:N}) = \mathcal{N}(\hat{\alpha}_t, V_t)
\end{align}$$

The recursions for the smoothed state $\hat{\alpha}$ are then

$$\begin{align}
r_n &= 0  \\
N_n &= 0  \\
r_{t - 1} &= Z_t^\top F_t^{-1} v_t + L_t^\top r_t  \\
N_{t - 1} &= Z_t^\top F_t^{-1} Z_t + L_t^\top N_t L_t  \\
\hat{\alpha}_t &= a_t + P_t r_{t - 1}  \\
V_t &= P_t - P_t N_{t - 1} P_t
\end{align}$$

```julia
r = Array(Float64, Nm, Nt)
ahat = Array(Float64, Nm, Nt)
N = Array(Float64, Nm, Nm, Nt)
V = Array(Float64, Nm, Nm, Nt)

ahat[:, 1] = a_init
V[:, :, 1] = P_init
r[:, end] = 0
N[:, :, end] = 0;
```

```julia
for t in Nt:-1:2

    local F_t = F[:, :, t]
    local Finv = inv(F_t)
    local r_t = r[:, t]
    local v_t = v[:, t]
    local L_t = L[:, :, t]
    local N_t = N[:, :, t]
    local a_t = a[:, t]
    local P_t = P[:, :, t]

    r[:, t - 1] = Z' * Finv * v_t + L_t' * r_t
    N[:, :, t - 1] = Z' * Finv * Z + L_t' * N_t * L_t

    ahat[:, t] = a_t + P_t * r[:, t - 1]
    V[:, :, t] = P_t - P_t * N[:, :, t - 1] * P_t
end
```

```julia
plot(ahat')
plot(α')
title("Actual and recovered states");
```

# Disturbance smoother/fast state smoother

As an alternative to the Kalman smoother, Durbin and Koopman recommend a disturbance smoother for the noise terms $\varepsilon$ and $\eta$. This recursion has lower memory requirements (using only $v_t$, $F_t$, and $K_t$) and does not require multiplication by the often dense $a_t$ and $P_t$.

This system of recursions (for the posterior means) is given by

$$\begin{align}
u_t &= F_t^{-1} v_t - K_t^\top r_t  \\
\hat{\varepsilon}_t &= H_t u_t  \\
\hat{\eta}_t &= Q_t R_t^\top r_t  \\
r_{t - 1} &= Z_t^\top u_t + T_t^\top r_t
\end{align}$$

with additional recursions for the variances (cf. Durbin and Koopman 4.5.3).

These recursions provide a particularly efficient means of calculating the smoothed state estimates $\hat{\alpha}_t$:

$$\begin{align}
\hat{\alpha}_1 &= a_1 + P_1 r_0  \\
\hat{\alpha}_{t + 1} &= T_t \hat{\alpha}_t + R_t Q_t R_t^\top r_t
\end{align}$$

This takes advantage of the typical sparsity of $T$ and $RQR$. However, it does not lend itself to calculating the posterior smoothed variances $V$.

Thus, if we only care about the posterior smoothed state means, we can do so very efficiently by saving only $F$, $K$,  and $v$ from the filtering step, calculate $r$ from the recursion using $u$ above, and run the model forward in time to get $\hat{\alpha}$.

## Implementation of the fast state smoother

First, we initialize $r$ and $\hat{\alpha}$:

```julia
# initialize r
r = Array(Float64, Nm, Nt)
r[:, end] = 0
alphat = Array(Float64, Nm, Nt);
```

Then we recursively calculate $r$, working backward in time:

```julia
# calculate r
r_init = 1.

for t in Nt:-1:1
    local Finv = inv(F[:, :, t])
    local v_t = v[:, t]
    local K_t = K[:, :, t]
    local r_t = r[:, t]

    local u = Finv * v_t - K_t' * r_t
    local thisr = Z' * u + T' * r_t

    if t - 1 > 0
        r[:, t - 1] = thisr
    else
        r_init = thisr
    end
end
```

Finally, we run the state space model forward in time

```julia
# run model forward
alphat[:, 1] = a_init + P_init * r_init
RQR = R * Q * R'

for t in 1:(Nt - 1)
    alphat[:, t + 1] = T * alphat[:, t] + RQR * r[:, t]
end
```

```julia
plot(alphat')
plot(α')
title("Actual and recovered states");
```

# Drawing posterior samples (simulation smoothing)

We want to be able to sample from the smoothed posterior for the states, $p(\alpha_t|y_{1:N})$. We can do this using the technique of [Durbin and Koopman (2002)](http://biomet.oxfordjournals.org/content/89/3/603.short), which involves following steps:

1. Draw disturbance terms ($\varepsilon$ and $\eta$) from the *prior*, denoted $w^+$.
1. Simulate the model using these disturbances, generating a state sequence, $\alpha^+$ and observations $y^+$.
1. Using the Kalman filter and smoother, calculate smoothed estimates of the state sequence conditioned
    on the actual and the simulated data, $\hat{\alpha} = \mathbb{E}(\alpha|y_{1:n})$ and
    $\hat{\alpha}^+ = \mathbb{E}(\alpha|y^+_{1:n})$.
1. Combine these estimates with the simulated state sequence. The result,
    $\alpha^+ - \hat{\alpha}^+ + \hat{\alpha}$, is a draw from the posterior.

## Implementing the posterior sampler

```julia
# preallocate

alpha_plus = Array(Float64, Nm, Nt)
y_plus = Array(Float64, Np, Nt);
```

## Draw Disturbances

```julia
ϵ_sim = rand(MvNormal(H), Nt)
η_sim = rand(MvNormal(Q), Nt);
```

## Simulate model

```julia
alpha_plus[:, 1] = rand(MvNormal(a_init, P_init))
for t in 1:Nt
    y_plus[:, t] = Z * alpha_plus[:, t] + ϵ_sim[:, t]
    if t < Nt
        alpha_plus[:, t + 1] = T * alpha_plus[:, t] + R * η_sim[:, t]
    end
end
```

## Get Smoothed Means

Now run the data through...

```julia
import SStools
```

```julia
vv, KK, FFinv, a, P = SStools.kalman_filter(y, a_init, P_init, Z, H, T, R, Q)
alpha_hat = SStools.fast_state_smoother(vv, KK, FFinv, a_init, P_init, Z, T, R, Q)
```

    2x100 Array{Float64,2}:
     -0.162108  -0.507299  -0.836324  -1.10313  …  1.78589   1.92274   2.04495
     -2.071     -2.0005    -1.87233   -1.65719     0.645101  0.372424  0.111742

```julia
vv, KK, FFinv, a, P = SStools.kalman_filter(y_plus, a_init, P_init, Z, H, T, R, Q)
alpha_hat_plus = SStools.fast_state_smoother(vv, KK, FFinv, a_init, P_init, Z, T, R, Q)
```

    2x100 Array{Float64,2}:
     2.18963   2.1674     2.17749     2.28646   …  -1.00361  -0.927515  -0.684532
     0.873268  0.351952  -0.0462079  -0.287197      1.3547    1.34985    1.4854

## And Combine

```julia
alpha_draw = alpha_plus - alpha_hat_plus + alpha_hat
```

    2x100 Array{Float64,2}:
     -0.352918  -0.539733  -0.768694  -1.08096  …  1.855     2.0492    2.14891
     -2.33182   -2.04171   -1.81026   -1.658       0.729629  0.483414  0.162868

```julia
plot(alpha_draw');
```

And we can do this with a single function:

```julia
adraw = SStools.simulate(y, a_init, P_init, Z, H, T, R, Q; interleaved=false)
plt.plot(adraw');
```

# Let's make a posterior

```julia
Ns = 1000  # number of samples

alpha_samples = Array(Float64, Nm, Nt, Ns);
```

```julia
@time begin
    for idx in 1:Ns
        alpha_samples[:, :, idx] = SStools.simulate(y, a_init, P_init, Z, H, T, R, Q; interleaved=false)
    end
end
```

```julia
trans = 10 / Ns
figure(figsize=(10, 8))

for idx in 1:Nm
    subplot(Nm, 1, idx)
    plot(squeeze(alpha_samples[idx, :, :], 1), color="black", alpha=trans)
    plot(α[idx, :]', linestyle="--", label="Actual")
    plot(alphat[idx, :]', label="Posterior")
    legend()
end
```

```julia

```


# trigonometric functions

## definitions

六个三角函数. 设直角三角形中, $\theta$ 的对边、邻边和斜边分别是
$a$, $b$, $c$, 得到
-   sine. $$\sin\theta = \frac{a}{c}$$
-   cosine. $$\cos\theta = \frac{b}{c}$$
-   tangent. $$\tan\theta = \frac{a}{b}$$
-   cotangent. $$\cot\theta = \frac{b}{a}$$
-   secant. $$\sec\theta = \frac{c}{b}$$
-   cosecant. $$\csc\theta = \frac{c}{a}$$

三个不同适用范围的定义.
-   直角三角形定义. 根据此定义, $\theta \in [0, \frac{\pi}{2}]$.
-   单位圆定义. 圆心在原点的单位圆上点的座标为 $(\cos \theta, \sin \theta)$.
    根据此定义, $\theta \in \mathbb{R}$. 该定义与双曲函数的双曲线定义对应.
-   Taylor 级数定义. 根据此定义, $\theta \in \mathbb{C}$.
\begin{align}
\sin x &= \sum_{n=0}^\infty \frac{(-1)^nx^{2n+1}}{(2n+1)!}\\
\cos x &= \sum_{n=0}^\infty \frac{(-1)^nx^{2n}}{(2n)!}
\end{align}

## geometric relations
两种不同的单位圆表示方法.

曲线和值的关系.

## properties

-   关系.
    \begin{align}
    \tan\theta &= \frac{\sin\theta}{\frac\cos\theta}\\
    \cot\theta &= \frac{1}{\tan\theta}\\
    \sec\theta &= \frac{1}{\cos\theta}\\
    \csc\theta &= \frac{1}{\sin\theta}
    \end{align}
-   周期性. (三角函数的单位圆定义的必然推论.)
    $\tan$ and $\cot$ 的周期是 $k\pi$, 其他 4 个周期是 $2k\pi$.
-   复数域上的性质. 由 $\sin x$, $\cos x$, $e^x$ 的 Taylor expansion, 可以得到 Euler's
    formula 成立.
    $$e^{ix} = \cos x + i \sin x$$
    $$\cos x = \operatorname{Re}(e^{ix}), \sin x = \operatorname{Im}(e^{ix})$$
    $$\sin x = \frac{e^{ix} - e^{-ix}}{2i}, \cos x = \frac{e^{ix} + e^{-ix}}{2}$$
    complex sine, cosine 和 real sine, cosine 以及 real hyperbolic sine, cosine
    的关系
    \begin{align}
    \sin(x+iy) &= \sin x \cosh y + i \cos x \sinh y\\
    \cos(x+iy) &= \cos x \cosh y - i \sin x \sinh y
    \end{align}

## etymology
-   sine. Latin sinus for gulf or bay, means "open".
-   cosine. The sine of the complementary or co-angle.
-   tangent. So called because it can be represented as a line segment tangent
    to the circle, that is the line that touches the circle but not cuts it.
    从图中看得很清楚.
-   cotangent. Tangent of co-angle.
-   secant. So called because it represents the line that cuts the circle (from
    Latin: secare, to cut). 从图中看得很清楚.
-   cosecant. The secant of co-angle.

```python

```


# Approximate solvers for the Euler equations of gas dynamics

In this chapter we discuss approximate solvers for the one-dimensional Euler equations:

\begin{align}
    \rho_t + (\rho u)_x & = 0 \\
    (\rho u)_t + (\rho u^2 + p)_x & = 0 \\
    E_t + ((E+p)u)_x & = 0.
\end{align}

As in [Euler_equations.ipynb](Euler_equations.ipynb), we focus on the case of an ideal gas, for which the total energy is given by

\begin{align} \label{EA:EOS}
    E = \frac{p}{\gamma-1} + \frac{1}{2}\rho u^2.
\end{align}

## Roe solver

We first derive a Roe solver for the Euler equations, following the same approach as in [shallow_water_approximate_solvers.ipynb](shallow_water_approximate_solvers.ipynb).  Namely, we assume that $\hat{A} = f'(\hat{q})$ for some average state $\hat{q}$, and impose the condition of conservation:

\begin{align} \label{EA:cons}
    f'(\hat{q}) (q_r - q_l) & = f(q_r) - f(q_l).
\end{align}

We will need the following quantities:

\begin{align}
q & = \begin{pmatrix} \rho \\ \rho u \\ E \end{pmatrix}, \ \ \ \ \ \  f(q) = \begin{pmatrix} \rho u \\ \rho u^2 + p \\ H u \rho \end{pmatrix}, \\
f'(\hat{q}) & = \begin{pmatrix}
                0 & 1 & 0 \\
                \frac{\gamma-3}{2}\hat{u}^2 & (3-\gamma)\hat{u} & \gamma-1 \\
                \frac{\gamma-1}{2}\hat{u}^3 - \hat{u}\hat{H} & \hat{H} - (\gamma-1)\hat{u}^2 & \gamma \hat{u} \end{pmatrix}.
\end{align}

Here $H = \frac{E+p}{\rho}$ is the enthalpy.  We have rewritten most expressions involving $E$ in terms of $H$ because it simplifies the derivation that follows.  We now solve (\ref{EA:cons}) to find $\hat{u}$ and $\hat{H}$.  It turns out that, for the case of a polytropic ideal gas, the average density $\hat{\rho}$ plays no role in the Roe solver.

The first equation of (\ref{EA:cons}) is an identity, satisfied independently of our choice of $\hat{q}$.  The second equation is (using (\ref{EA:EOS}))

\begin{align}
    \frac{\gamma-3}{2}\hat{u}^2 (\rho_r - \rho_l) + (3-\gamma)\hat{u}(\rho_r u_r - \rho_l u_l) + (\gamma-1)\left( \frac{p_r-p_l}{\gamma-1} + \frac{1}{2}(\rho_r u_r^2 - \rho_l u_l^2) \right) & = \rho_r u_r^2 - \rho_l u_l^2 + p_r - p_l,
\end{align}

which simplifies to a quadratic equation for $\hat{u}$:

\begin{align} \label{EA:u_quadratic}
    (\rho_r - \rho_l)\hat{u}^2 - 2(\rho_r u_r - \rho_l u_l) \hat{u} + (\rho_r u_r^2 - \rho_l u_l^2) & = 0,
\end{align}

with roots

\begin{align}
    \hat{u}_\pm & = \frac{\rho_r u_r - \rho_l u_l \mp \sqrt{\rho_r \rho_l} (u_l - u_r)}{\rho_r - \rho_l} = \frac{\sqrt{\rho_r} u_r \pm \sqrt{\rho_l} u_l}{\sqrt{\rho_r}\pm\sqrt{\rho_l}}
\end{align}

Notice that this is identical to the Roe average of the velocity for the shallow water equations, if we replace the density $\rho$ with depth $h$.  As before, we choose the root $u_+$ since it is well defined for all values of $\rho_r, \rho_l$.

Next we find $\hat{H}$ by solving the last equation of (\ref{EA:cons}), which reads
\begin{align}
    \left( \frac{\gamma-1}{2}\hat{u}^3 - \hat{u}\hat{H} \right)(\rho_r - \rho_l) + \left( \hat{H} - (\gamma-1)\hat{u}^2 \right)(\rho_r u_r - \rho_l u_l) + \gamma \hat{u}(E_r - E_l) & = H_r u_r \rho_r - H_l u_l \rho_l.
\end{align}
We can simplify this using the equality $\gamma E = \rho H + \frac{\gamma-1}{2}\rho u^2$ and solve for $\hat{H}$ to find
\begin{align}
    \hat{H} & = \frac{\rho_r H_r (u_r - \hat{u}_+) - \rho_l H_l (u_l - \hat{u}_+)}{\rho_r u_r - \rho_l u_l - \hat{u}_\pm(\rho_r -\rho_l)} \\
    & = \frac{\rho_r H_r (u_r - \hat{u}_+) - \rho_l H_l (u_l - \hat{u}_+)}{\pm\sqrt{\rho_r \rho_l}(u_r-u_l)} \\
    & = \frac{\rho_r H_r - \rho_l H_l \mp\sqrt{\rho_r \rho_l}(H_r - H_l)}{\rho_r - \rho_l} \\
    & = \frac{\sqrt{\rho_r}H_r \pm \sqrt{\rho_l} H_l}{\sqrt{\rho_r}\pm\sqrt{\rho_l}}.
\end{align}
Once more, we take the plus sign in the final expression for $\hat{H}$.

To implement the Roe solver, we also need the eigenvalues and eigenvectors of the averaged flux Jacobian $f'(\hat{q})$.  These are just the eigenvalues of the true Jacobian, evaluated at the averaged state:
\begin{align}
    \lambda_1 & = \hat{u} - \hat{c}, & \lambda_2 & = \hat{u} & \lambda_3 & = \hat{u} + \hat{c},
\end{align}
\begin{align}
r_1 & = \begin{bmatrix} 1 \\ \hat{u}-\hat{c} \\ \hat{H}-\hat{u}\hat{c}\end{bmatrix} &
r_2 & = \begin{bmatrix} 1 \\ \hat{u} \\ \frac{1}{2}\hat{u}^2 \end{bmatrix} &
r_3 & = \begin{bmatrix} 1 \\ \hat{u}+\hat{c} \\ \hat{H}+\hat{u}\hat{c}\end{bmatrix}.
\end{align}
Here $\hat{c} = \sqrt{(\gamma-1)(\hat{H}-\hat{u}^2/2)}$.

Solving the system of equations
\begin{align}
q_r - q_l & = \sum_{p=1}^3 {\mathcal W}_p = \sum_{p=1}^3 \alpha_p r_p
\end{align}
for the wave strengths gives
\begin{align}
    \alpha_2 & = \delta_1 + (\gamma-1)\frac{\hat{u}\delta_2 - \delta_3}{\hat{c}^2} \\
    \alpha_3 & = \frac{\delta_2 + (\hat{c}-\hat{u})\delta_1 - \hat{c}\alpha_2}{2\hat{c}} \\
    \alpha_1 & = \delta_1 - \alpha_2 - \alpha_3,
\end{align}
where $\delta = q_r - q_l$.  We now have everything we need to implement the Roe solver.

```python
%matplotlib inline
```

```python
%config InlineBackend.figure_format = 'svg'
import numpy as np
from exact_solvers import Euler
from utils import riemann_tools
from ipywidgets import interact
from ipywidgets import widgets
State = Euler.Primitive_State
```

```python
def roe_averages(q_l, q_r, gamma=1.4):
    rho_sqrt_l = np.sqrt(q_l[0])
    rho_sqrt_r = np.sqrt(q_r[0])
    p_l = (gamma-1.)*(q_l[2]-0.5*(q_l[1]**2)/q_l[0])
    p_r = (gamma-1.)*(q_r[2]-0.5*(q_r[1]**2)/q_r[0])
    denom = rho_sqrt_l + rho_sqrt_r
    u_hat = (q_l[1]/rho_sqrt_l + q_r[1]/rho_sqrt_r)/denom
    H_hat = ((q_l[2]+p_l)/rho_sqrt_l + (q_r[2]+p_r)/rho_sqrt_r)/denom
    c_hat = np.sqrt((gamma-1)*(H_hat-0.5*u_hat**2))

    return u_hat, c_hat, H_hat

def Euler_roe(q_l, q_r, gamma=1.4):
    """
    Approximate Roe solver for the Euler equations.
    """

    rho_l = q_l[0]
    rhou_l = q_l[1]
    u_l = rhou_l/rho_l
    rho_r = q_r[0]
    rhou_r = q_r[1]
    u_r = rhou_r/rho_r

    u_hat, c_hat, H_hat = roe_averages(q_l, q_r, gamma)

    delta = q_r - q_l

    s1 = u_hat - c_hat
    s2 = u_hat
    s3 = u_hat + c_hat

    alpha2 = (gamma-1.)/c_hat**2 *((H_hat-u_hat**2)*delta[0]+u_hat*delta[1]-delta[2])
    alpha3 = (delta[1] + (c_hat - u_hat)*delta[0] - c_hat*alpha2) / (2.*c_hat)
    alpha1 = delta[0] - alpha2 - alpha3

    r1 = np.array([1., u_hat-c_hat, H_hat - u_hat*c_hat])
    r2 = np.array([1., u_hat, 0.5*u_hat**2])
    q_l_star = q_l + alpha1*r1
    q_r_star = q_l_star + alpha2*r2

    states = np.column_stack([q_l,q_l_star,q_r_star,q_r])
    speeds = [s1, s2, s3]
    wave_types = ['contact','contact', 'contact']

    def reval(xi):
        rho = (xi<s1)*states[0,0] + (s1<=xi)*(xi<s2)*states[0,1] + (s2<=xi)*(xi<s3)*states[0,2] + (s3<=xi)*states[0,3]
        mom = (xi<s1)*states[1,0] + (s1<=xi)*(xi<s2)*states[1,1] + (s2<=xi)*(xi<s3)*states[1,2] + (s3<=xi)*states[1,3]
        E = (xi<s1)*states[2,0] + (s1<=xi)*(xi<s2)*states[2,1] + (s2<=xi)*(xi<s3)*states[2,2] + (s3<=xi)*states[2,3]
        return rho, mom, E

    return states, speeds, reval, wave_types
```

### Examples

Let's compare the Roe approximation to the exact solution.  As a first example, we use the Sod shock tube.

```python
def compare_solutions(left_state, right_state, solvers=['Exact','HLL']):
    q_l = np.array(Euler.primitive_to_conservative(*left_state))
    q_r = np.array(Euler.primitive_to_conservative(*right_state))

    outputs = []
    states = {}

    for solver in solvers:
        if solver.lower() == 'exact':
            outputs.append(Euler.exact_riemann_solution(q_l,q_r))
        if solver.lower() == 'hll':
            outputs.append(Euler_hll(q_l, q_r))
            states['hll'] = outputs[-1][0]
        if solver.lower() == 'roe':
            outputs.append(Euler_roe(q_l, q_r))
            states['roe'] = outputs[-1][0]

    plot_function = riemann_tools.make_plot_function([val[0] for val in outputs],
                                                     [val[1] for val in outputs],
                                                     [val[2] for val in outputs],
                                                     [val[3] for val in outputs],
                                                     solvers,
                                                     layout='vertical',
                                                     variable_names=Euler.primitive_variables,
                                                     derived_variables=Euler.cons_to_prim)

    interact(plot_function, t=widgets.FloatSlider(min=0,max=0.9,step=0.1));

    return states
```

```python
left_state  = State(Density = 3.,
                    Velocity = 0.,
                    Pressure = 3.)
right_state = State(Density = 1.,
                    Velocity = 0.,
                    Pressure = 1.)

states = compare_solutions(left_state,right_state,solvers=['Exact','Roe'])
```

```python
Euler.phase_plane_plot(left_state,right_state,approx_states=states['roe'])
```

Recall that in the true solution the middle wave is a contact discontinuity and carries only a jump in the density. For that reason the three-dimensional phase space plot is generally shown projected onto the pressure-velocity plane as shown above: The two intermediate states in the true solution have the same pressure and velocity, and so are denoted by a single Middle state in the phase plane plot.

The Roe solver, on the other hand, generates a middle wave that carries a jump in all 3 variables  and there are two green dots appearing in the plot above for the two middle states (though the pressure jump is quite small in this example).  For a Riemann problem like this one with zero initial velocity on both sides, the Roe average velocity must also be zero, so the middle wave is stationary; this is of course not typically true in the exact solution, even when $u_l=u_r=0$.

Here is a second example.  Experiment with the initial states to explore how the Roe solution compares to the exact solution.

```python
left_state  = State(Density = 0.1,
                    Velocity = 0.,
                    Pressure = 0.1)
right_state = State(Density = 1.,
                    Velocity = 1.,
                    Pressure = 1.)

states = compare_solutions(left_state,right_state,solvers=['Exact','Roe'])
```

```python
Euler.phase_plane_plot(left_state,right_state,approx_states=states['roe'])
```

### Single-shock solution
Next we demonstrate the exactness property of the Roe solver by applying it to a case where the left and right states are connected by a single shock wave.

```python
M = 2.  # Mach number of the shock wave
gamma = 1.4
mu = 2*(M**2-1)/(M*(gamma+1.))
right_state = State(Density = 1.,
              Velocity = 0.,
              Pressure = 1.)
c_r = np.sqrt(gamma*right_state.Pressure/right_state.Density)

rho_l = right_state.Density * M/(M-mu)
p_l = right_state.Pressure * ((2*M**2-1)*gamma+1)/(gamma+1)
u_l = mu*c_r

left_state = State(Density = rho_l,
             Velocity = u_l,
            Pressure = p_l)

states = compare_solutions(left_state,right_state,solvers=['Exact','Roe'])
```

```python
Euler.phase_plane_plot(left_state,right_state,approx_states=states['roe'])
```

It is evident that the solution consists of a single right-going shock.  The exact solution cannot be seen because it coincides exactly with the Roe solution.  The path of the shock in the first plot also cannot be seen since it is plotted under the path of the rightmost Roe solution wave.  The two solutions differ only in the wave speeds predicted for the other two waves, but since these waves have zero strength this makes no difference.

### Transonic rarefactions and an entropy fix
Here is an example of a Riemann problem whose solution includes a transonic 2-rarefaction:

```python
left_state  = State(Density = 0.1,
                    Velocity = -2.,
                    Pressure = 0.1)
right_state = State(Density = 1.,
                    Velocity = -1.,
                    Pressure = 1.)

states = compare_solutions(left_state,right_state,solvers=['Exact','Roe'])
```

Notice that in the exact solution, the right edge of the rarefaction travels to the right.  In the Roe solution, all waves travel to the left.  As in the case of the shallow water equations, here too this behavior can lead to unphysical solutions when this approximate solver is used in a numerical discretization.  In order to correct this, we can split the single wave into two when a transonic rarefaction is present, in a way similar to what is done in the shallow water equations.  We do not go into details here.

## HLLE Solver

Recall that an HLL solver uses only two waves with a constant state between them.  The Euler equations are our first example for which the number of waves in the true solution is larger than the number of waves in the approximate solution.  As one might expect, this leads to noticeable inaccuracy in solutions produced by the solver.

The left-going wave speed is chosen to be the minimum of the Roe speed for the 1-wave and the characterstic speed $\lambda^1$ in the left state $q_\ell$.  The right-going wave speed is chosen to be the maximum of the Roe speed for the 3-wave and the characteristic speed $\lambda^3$ in the right state $q_r$.  Effectively, this means that
\begin{align}
    s_1 & = \min(u_l - c_l, \hat{u}-\hat{c}) \\
    s_2 & = \max(u_r + c_r, \hat{u}+\hat{c})
\end{align}

Recall that once we have chosen these two wave speeds, conservation dictates the value of the intermediate state:
\begin{align}  \label{SWA:hll_middle_state}
q_m = \frac{f(q_r) - f(q_l) - s_2 q_r + s_1 q_l}{s_1 - s_2}.
\end{align}

```python
def Euler_hll(q_l, q_r, gamma=1.4):
    """HLL approximate solver for the Euler equations."""

    rho_l = q_l[0]
    rhou_l = q_l[1]
    u_l = rhou_l/rho_l
    rho_r = q_r[0]
    rhou_r = q_r[1]
    u_r = rhou_r/rho_r
    E_r = q_r[2]
    E_l = q_l[2]

    u_hat, c_hat, H_hat = roe_averages(q_l, q_r, gamma)
    p_r = (gamma-1.) * (E_r - rho_r*u_r**2/2.)
    p_l = (gamma-1.) * (E_l - rho_l*u_l**2/2.)
    H_r = (E_r+p_r) / rho_r
    H_l = (E_l+p_l) / rho_l
    c_r = np.sqrt((gamma-1.)*(H_r-u_r**2/2.))
    c_l = np.sqrt((gamma-1.)*(H_l-u_l**2/2.))

    s1 = min(u_l-c_l,u_hat-c_hat)
    s2 = max(u_r+c_r,u_hat+c_hat)

    rho_m = (rhou_r - rhou_l - s2*rho_r + s1*rho_l)/(s1-s2)
    rhou_m = (rho_r*u_r**2 - rho_l*u_l**2 + p_r - p_l - s2*rhou_r + s1*rhou_l)/(s1-s2)
    E_m = ( u_r*(E_r+p_r) - u_l*(E_l+p_l) -s2*E_r + s1*E_l)/(s1-s2)
    q_m = np.array([rho_m, rhou_m, E_m])

    states = np.column_stack([q_l,q_m,q_r])
    speeds = [s1, s2]
    wave_types = ['contact','contact']

    def reval(xi):
        rho  = (xi<s1)*rho_l + (s1<=xi)*(xi<=s2)*rho_m + (s2<xi)*rho_r
        mom  = (xi<s1)*rhou_l + (s1<=xi)*(xi<=s2)*rhou_m + (s2<xi)*rhou_r
        E  = (xi<s1)*E_l + (s1<=xi)*(xi<=s2)*E_m + (s2<xi)*E_r
        return rho, mom, E

    return states, speeds, reval, wave_types
```

### Examples

```python
left_state  = State(Density = 3.,
                    Velocity = 0.,
                    Pressure = 3.)
right_state = State(Density = 1.,
                    Velocity = 0.,
                    Pressure = 1.)

states = compare_solutions(left_state, right_state, solvers=['Exact','HLL'])
```

```python
Euler.phase_plane_plot(left_state,right_state,approx_states=states['hll'])
```

### Preservation of positivity
Just as we saw in the case of the shallow water equations, the Roe solver (or any linearized solver) for the Euler equations fails to preserve positivity of the pressure and/or density in some situations.  Here is one example.

```python
left_state  = State(Density = 1.,
                    Velocity = -5.,
                    Pressure = 1.)
right_state = State(Density = 1.,
                    Velocity = 1.,
                    Pressure = 1.)

states = compare_solutions(left_state, right_state, solvers=['Exact', 'Roe'])
```

As we can see, in this example each Roe solver wave moves much more slowly than the leading edge of the corresponding true rarefaction.  In order to maintain conservation, this implies that the middle Roe state must have lower density than the true middle state.  This leads to a negative density.  Note that the velocity and pressure take huge values in the intermediate state.

The HLL solver, on the other hand, guarantees positivity of the density and pressure.  Since the HLL wave speed in the case of a rarefaction is always the speed of the leading edge of the true rarefaction, and since the HLL solution is conservative, the density in a rarefaction will always be at least as great as that of the true solution.  This can be seen clearly in the example below.

```python
left_state  = State(Density = 1.,
                    Velocity = -10.,
                    Pressure = 1.)
right_state = State(Density = 1.,
                    Velocity = 1.,
                    Pressure = 1.)

states = compare_solutions(left_state, right_state, solvers=['Exact', 'HLL']);
```

```python
Euler.phase_plane_plot(left_state,right_state,approx_states=states['hll'])
```